A method and system for multimodal relation extraction in multi-image scenes
By filtering key objects using visual semantic similarity and hierarchical clustering, and combining multi-head attention and residual mechanisms for multimodal fusion, the problem of irrelevant information interference in multimodal relationship extraction in multi-image scenarios is solved, thereby improving the accuracy and stability of relationship extraction.
Patent Information
- Application Number
- CN202611233430.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-14
- Publication Date
- 2026-11-10
AI Technical Summary
In multi-image scenarios, existing multimodal relationship extraction methods struggle to effectively filter and fuse key visual objects from multiple images, leading to interference from irrelevant information and affecting the accuracy and stability of relationship judgment. This is especially true when head and tail entities are scattered across different images, making it difficult for the model to capture local spatial interaction semantics.
By constructing a set of key object visual regions through visual semantic similarity calculation and hierarchical clustering, key visual objects related to the head and tail entities are selected. Multimodal fusion is then performed through multi-head attention and residual mechanisms, combining object-level and joint region visual features to enhance the expression of semantic relationships and local interaction information between entities.
It reduces the interference of redundant images and irrelevant visual regions on relationship judgment, improves the accuracy and stability of multimodal relationship extraction in multi-image scenarios, and can adaptively adjust the multimodal feature fusion weights to enhance the model's ability to express semantic relationships and local interaction information between entities.
Smart Images

Figure CN123069262A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence and multimodal information extraction technology, specifically relating to a method and system for multimodal relationship extraction for multi-image scenes. Background Technology
[0002] Multimodal relation extraction aims to combine textual semantics with visual cues from images to determine the relationship category between specified entity pairs in a given text. Compared to pure text relation extraction, images can provide supplementary visual information such as object category, scene environment, action state, and spatial layout, which is of great value in application scenarios such as social media content understanding, news report analysis, and knowledge graph construction. Existing research on multimodal relation extraction mainly revolves around cross-modal fusion. Early methods typically used the entire image as input, extracted global visual features, and then concatenated them with the text representation or interacted with cross-modal attention to assist in relation prediction. However, the features of the entire image simultaneously contain target objects, background regions, and other irrelevant visual content. The model may focus on regions in the image that are significant but unrelated to the target relation. When there is a semantic discrepancy between the image and the text, this direct fusion method may also introduce erroneous visual information into the relation representation. With the development of research, some methods have begun to shift from coarse-grained whole-image fusion to fine-grained entity-object alignment. By extracting local candidate regions in the image and using methods such as cross-modal retrieval, region reweighting, or contrastive learning, the correspondence between text entities and visual objects is established. In addition, to reduce interference from irrelevant visual content, some methods introduce mechanisms such as feature denoising, information bottlenecking, or mutual information maximization to screen visual evidence related to relationship judgment and suppress the influence of background noise.
[0003] However, the aforementioned mainstream methods and corresponding datasets primarily employ an input format of one image per piece of text, typically assuming that visual cues related to relationships can be fully provided by a single image. In real-world applications, however, the same text in social media posts, news reports, and online content is often associated with multiple images. Different images may showcase entity appearances, event backgrounds, and interactive behaviors from different angles, collectively providing the visual evidence needed for relationship judgment. While multiple image inputs can provide richer visual information, they also introduce new technical challenges: the increased amount of visual content leads to the dispersion of effective evidence, while also introducing more irrelevant information and visual redundancy. If the model only performs average fusion, feature stitching, or coarse-grained attention aggregation on multiple images, it is easily affected by irrelevant images and salient background objects. On the other hand, the target relationship depends not only on the category or attributes of a single visual object but also on the local spatial layout and interactive semantics between the head and tail entities, such as the distance between two people, their postures, and the shared scene context, which plays a crucial role in distinguishing fine-grained relationships. Existing methods mainly focus on region feature selection and image-level correlation modeling, lacking explicit modeling of the joint local regions formed by entity-related objects and their spatial interaction relationships. When the head and tail entities in multiple images are scattered across different images, or when the target entity lacks a clear and independent visual subject, the model may also capture significant visual patterns unrelated to the relationship.
[0004] Therefore, multimodal relation extraction in multi-image scenarios cannot be simply regarded as an input extension of single-image tasks. It requires a more refined object-level visual screening mechanism, a more explicit alignment method between text entities and visual objects, and a visual fusion structure that can characterize the semantic interaction of entities in local space, so as to reduce the interference of irrelevant visual information on relation judgment and improve the accuracy of relation extraction in multi-image scenarios. Summary of the Invention
[0005] To address the shortcomings of existing technologies, the present invention provides the following technical solution: A method for multimodal relation extraction in multi-image scenes includes: Obtain the input text, multiple images associated with the input text, and the head and tail entities in the input text; Candidate visual objects are detected in the image, and visual semantic similarity calculation and hierarchical clustering are used to construct a set of key object visual regions. Based on the text representations of the head entity and the tail entity, a first key visual object and a second key visual object that meet the preset relevance requirements are selected from the set of key object visual regions, respectively. Based on the location information of the first key visual object and the location information of the second key visual object, a joint region covering the first key visual object and the second key visual object is determined, and the visual features of the joint region are extracted from the corresponding image to obtain the joint visual region features. The text representations of the head entity, the text representations of the tail entity, the visual features of the key object visual region set, and the joint visual region features are fused in a multimodal manner to obtain multimodal fusion features. Relationship classification is performed based on the multimodal fusion features to obtain the relationship category between the head entity and the tail entity.
[0006] Preferred methods for obtaining the set of key visual object regions include: The image associated with the input text is detected using an object detector to obtain candidate visual objects; The visual encoder is used to extract features from candidate visual objects in each image to obtain the features of candidate visual objects; Calculate the similarity between the features of each candidate visual object, and divide the candidate visual objects into several object clusters based on the similarity using a hierarchical clustering method; Calculate the central feature of each object cluster, and select the candidate visual object that is closest to the central feature as the representative object of the object cluster; The representative objects corresponding to each object cluster are combined to form the key object visual region set.
[0007] Preferably, the method for obtaining the first key visual object and the second key visual object includes: Calculate a first correlation score between the text representation of the head entity and the features of each candidate visual object in the set of key object visual regions; based on the first correlation score, select the candidate visual object with the highest correlation to the head entity from the set of key object visual regions as the first key visual object; Calculate a second correlation score between the text representation of the tail entity and the features of each candidate visual object in the set of key object visual regions; based on the second correlation score, select the candidate visual object with the highest correlation to the tail entity from the set of key object-level visual regions as the second key visual object.
[0008] Preferably, methods for obtaining multimodal fusion features include: Using the text representation of the head entity as a query, a multi-head attention operation is performed on the visual features of the key object visual region set to obtain a visually enhanced representation of the head entity. Using the text representation of the tail entity as a query, a multi-head attention operation is performed on the visual features of the key object visual region set to obtain a visually enhanced representation of the tail entity. The joint visual region features are mapped to the same feature space as the entity-to-text representation, and then added to the entity-to-text representation in a residual manner to obtain the joint region enhancement representation; The visual enhancement representations of the head entity, the tail entity, and the joint region enhancement representation are concatenated and mapped to obtain a unified visual enhancement feature. A gating vector is calculated based on the entity-to-text representation and the unified visual enhancement feature. The gating vector is then used to control the injection ratio of the unified visual enhancement feature into the entity-to-text representation, thereby obtaining the multimodal fusion feature.
[0009] Preferably, the method for obtaining the entity-to-text representation includes: Entity tagging is performed on the head and tail entities of the input text to obtain the entity-tagged text sequence; The text sequence is encoded using a text encoder to obtain the contextual semantic representation of each word in the text sequence; Based on the positions of the head entity and the tail entity in the text sequence, the context semantic representation is averaged to obtain the head entity representation and the tail entity representation; The head entity representation and the tail entity representation are concatenated and mapped to obtain the entity-to-text representation.
[0010] Preferably, the visual features of the key object visual region set include A person region features, B person region features, building features, road features, and representative street scene object features; The joint visual region features are used to represent spatial and scene information when a person and the surrounding environment coexist.
[0011] The present invention also provides a multimodal relation extraction system for multi-image scenes, used to implement the method, comprising: A multimodal data acquisition module is used to acquire input text, multiple images associated with the input text, and head and tail entities in the input text; The key object visual region acquisition module is used to detect candidate visual objects in the image and construct a set of key object visual regions by using visual semantic similarity calculation and hierarchical clustering. The key visual object selection module is used to select a first key visual object and a second key visual object that meet preset relevance requirements from the key object visual region set, based on the text representation of the head entity and the text representation of the tail entity, respectively. The joint visual region feature acquisition module is used to determine the joint region covering the first key visual object and the second key visual object based on the position information of the first key visual object and the position information of the second key visual object, and extract the visual features of the joint region from the corresponding image to obtain the joint visual region features. The multimodal fusion module is used to perform multimodal fusion of the text representation of the head entity, the text representation of the tail entity, the visual features of the key object visual region set, and the joint visual region features to obtain multimodal fusion features; The relationship extraction module is used to classify relationships based on the multimodal fusion features to obtain the relationship category between the head entity and the tail entity.
[0012] Preferably, the key object visual region acquisition module includes: The candidate object acquisition unit is used to perform target detection on the image associated with the input text using a target detector to obtain candidate visual objects; The feature extraction unit is used to extract features of candidate visual objects in each image using a visual encoder to obtain the features of the candidate visual objects. Clustering units are used to calculate the similarity between the features of each candidate visual object, and to divide the candidate visual objects into several object clusters based on the similarity using hierarchical clustering. The representative object selection unit is used to calculate the central feature of each object cluster and select the candidate visual object that is closest to the central feature as the representative object of the object cluster. The set construction unit is used to assemble the key object visual region set by combining the representative objects corresponding to each object cluster.
[0013] Compared with existing technologies, the beneficial effects of this invention are as follows: This application provides a multimodal relation extraction (MRE-MI) method for multi-image scenes, which can filter candidate visual objects in multiple images, reducing the interference of redundant images and irrelevant visual regions on relation judgment; it can align key visual objects in images based on text entity information to determine visual evidence related to subject and object entities; it can enhance the model's ability to express semantic relationships and local interaction information between entities through the fusion of object-level visual regions and joint visual regions; and it can adaptively adjust the fusion weights of multimodal features according to the importance of text and visual information in different samples. This application achieves the goal of improving the accuracy and stability of multimodal relation extraction in multi-image scenes by filtering, aligning, and fusing key visual information in multiple images. Attached Figure Description
[0014] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0015] Figure 1 This is a flowchart of a method according to an embodiment of the present invention; Figure 2 This is an example diagram of multimodal relationship extraction from multiple images according to an embodiment of the present invention. Detailed Implementation
[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0018] Example 1 like Figure 1 As shown, a multimodal relation extraction method for multi-image scenes includes: S1: Obtain the input text, multiple images associated with the input text, and the head and tail entities in the input text; such as... Figure 2 As shown, the input text is "Person A and Person B in Location Ba thread", and the input images are four images associated with the text. The four images showcase the relevant persons and their environment from different perspectives, and include persons, buildings, roads, street scenes, and other background objects. In this embodiment, person entity B is defined as the head entity. Location B is identified as the tail entity. The person A in the text and other people and scene objects appearing in the images serve as auxiliary contextual information in relation determination, but do not belong to the current entity pair to be identified. Through joint analysis of the semantics of the text and the visual information in the four associated images, the model outputs the relationship category between the head entity person B and the tail entity location B: This relationship indicates that a person has visited or appeared at location B.
[0019] S2: Perform candidate visual object detection on the image, and construct a set of key object visual regions using visual semantic similarity calculation and hierarchical clustering. A further implementation method for obtaining the set of key visual object regions includes: using an object detector to perform object detection on the image associated with the input text to obtain candidate visual objects; using a visual encoder to extract features from the candidate visual objects in each image to obtain candidate visual object features; calculating the similarity between the features of each candidate visual object, and using hierarchical clustering to divide the candidate visual objects into several object clusters based on the similarity; calculating the central feature of each object cluster, and selecting the candidate visual object closest to the central feature as the representative object of the object cluster; and forming a set of key object visual regions from the representative objects corresponding to each object cluster.
[0020] Specifically, suppose the input text is associated with multiple images that constitute an image set: in, Indicates the first A number of related images, Indicates the number of associated images. Figure 2 In the illustrated embodiment, An object detector is used to detect each associated image separately. For the first... The detection results for the images are as follows: in, Indicates the first The number of candidate objects detected in the image. Indicates the first The bounding boxes of the candidate objects This represents the detection confidence level of the corresponding candidate object.
[0021] bounding box The confidence threshold is determined by the coordinates of the top-left and bottom-right corners of the candidate object. , retain satisfaction Candidate objects are identified, and candidate object regions are cropped from the original image based on their corresponding bounding boxes. exist Figure 2 In the illustrated embodiment, candidate objects include the person regions corresponding to person A and person B, as well as buildings, roads, street scene facilities, and other background objects. The same person, similar buildings, and similar street scenes may appear repeatedly in multiple images; therefore, directly encoding all candidate regions can easily lead to visual information redundancy.
[0022] To reduce the interference of repeated objects and irrelevant background information in multiple related images on relationship judgment, this application first performs unified image preprocessing on each candidate object region. Specifically, the candidate object regions are scaled to a preset size and pixel normalization is performed. In this embodiment, a pre-trained CLIP ViT-B / 32 visual model is used as a visual encoder to perform visual semantic encoding on the candidate object regions. Candidate object regions Its original visual features are represented as follows: in, The CLIP ViT-B / 32 visual encoding network is used, and the output vectors corresponding to the classification labels in the visual Transformer are used as the original visual features of the candidate object regions. Represents the candidate object region Visual features.
[0023] Since the dimensionality of the output features of the visual encoder may differ from the dimensionality of the features used by the subsequent relation extraction model, this application projects the original candidate object features onto a unified visual semantic space through an object feature mapping layer: in, and These represent the weight parameters and bias parameters of the object feature mapping layer, respectively. This represents the mapped candidate object features. To ensure the comparability of candidate object features from different images, the mapped candidate object features are... Normalization: in, This represents the normalized features of the candidate objects. Representing vectors Norm.
[0024] Subsequently, the features of any two candidate objects are calculated. and Cosine similarity between them: Based on the similarity between the features of candidate objects, hierarchical clustering is used to group candidate objects with similar visual semantics into the same cluster. For example, areas of people that appear repeatedly in multiple images can be grouped into the same or similar clusters, and similar building and road areas can also form corresponding clusters.
[0025] For any cluster First, calculate the central features of the cluster, and then select the candidate object closest to the cluster center as the representative object: in, This indicates the number of candidate objects contained in cluster c. Represents clusters The corresponding representative object number. Represents clusters The central feature.
[0026] The key object visual region set is composed of representative objects from each cluster: in, This indicates the number of representative objects that are ultimately retained.
[0027] S3: Based on the text representations of the head entity and the tail entity, select a first key visual object corresponding to the head entity and a second key visual object corresponding to the tail entity from the key object visual region set, respectively; a further implementation method includes: Calculate the first relevance score between the text representation of the head entity and the features of each candidate visual object in the key object visual region set; based on the first relevance score, select the candidate visual object with the highest relevance to the head entity from the key object visual region set and determine it as the first key visual object; calculate the second relevance score between the text representation of the tail entity and the features of each candidate visual object in the key object visual region set; based on the second relevance score, select the candidate visual object with the highest relevance to the tail entity from the key object visual region set and determine it as the second key visual object. Specifically, through the above processing, it is possible to... Figure 2 The four associated images shown retain representative figures, buildings, and street scenes, reducing information redundancy caused by the recurrence of the same figure or similar background in multiple images.
[0028] Based on the set of key object visual regions, and according to the textual semantics of the head and tail entities, key visual objects related to the current entity pair are determined. For the head entity, the relevant visual object is mainly the person region corresponding to person B; for the tail entity, the relevant visual information mainly comes from buildings, roads, and the overall street view area. Two key objects with a high degree of correlation with the current entity pair are selected from the same image.
[0029] S4: Based on the position information of the first key visual object and the second key visual object, determine the joint region (UnionCrop joint region) covering the first and second key visual objects. Extract the visual features of the joint region from the corresponding image to obtain the joint visual region features. Specifically, let the bounding boxes of the two key objects be: ; The joint bounding box of the two key objects is represented as follows: Based on the joint bounding box, the UnionCrop joint region is cropped from the corresponding original image: Among them, the object-level visual region is used to represent the local semantic information of key objects such as people, buildings and roads, while the UnionCrop region is used to preserve the relative position, spatial layout and local scene information between two key objects.
[0030] Through the above technical solutions, this proposal can filter key visual objects in multi-image scenarios, align text entities with image regions, and fuse object-level visual information and joint region visual information, thereby reducing the interference of redundant image content on relationship judgment and improving the accuracy and stability of multimodal relationship extraction in multi-image scenarios.
[0031] exist Figure 2 In the illustrated embodiment, the UnionCrop region can simultaneously include the person object and its surrounding buildings, roads, or street scene environment, thereby supplementing the visual co-occurrence information between the person entity and the location entity.
[0032] Prior to feature fusion, this embodiment also includes multimodal feature encoding. A further implementation method for obtaining entity-to-text representations includes: performing entity tagging on the head and tail entities of the input text to obtain a text sequence with entity tags; encoding the text sequence using a text encoder to obtain the contextual semantic representation corresponding to each word in the text sequence; performing average pooling on the contextual semantic representation based on the positions of the head and tail entities in the text sequence to obtain head entity representations and tail entity representations; and concatenating and mapping the head entity representations and tail entity representations to obtain entity-to-text representations.
[0033] Specifically, firstly, entity tagging is performed on the head and tail entities in the input text. Figure 2 The text shown is represented by entity tags as follows: in, and These represent the start and end markers of the header entity, respectively. and These represent the start and end markers of the tail entity, respectively.
[0034] Text sequence after entity tagging Input the text encoder to obtain the text hiding state: ,in, This includes the contextual semantic representation of each word in the input text. Based on the positions of the head and tail entities in the text, the hidden states of the corresponding words are average-pooled to obtain the head entity representations. Tail-end entity representation By concatenating and mapping the head and tail entity representations, we obtain the entity-to-text representation: The semicolon indicates a vector concatenation operation. and These represent the weight matrix and bias vector of the entity to the feature mapping layer, respectively.
[0035] In this embodiment, The contextual semantic features of entity A are represented. The contextual semantic features representing location entity B, This represents the semantic features of the entity pair composed of the two entities.
[0036] Simultaneously, the obtained set of key object visual regions is input into the visual encoder to obtain an object-level visual feature sequence; the UnionCrop joint region is input into the visual encoder to obtain joint region visual features: in, Represents an object-level visual feature sequence. Indicates the number of key objects to be retained. This represents the visual features of the UnionCrop region.
[0037] exist Figure 2 In the illustrated embodiment, the object-level visual feature sequence may include A person region features, B person region features, building features, road features, and other representative street scene object features; UnionCrop joint region features are used to represent the spatial and scene information when the person object and the surrounding location environment coexist.
[0038] S5: Multimodal fusion is performed on the text representations of the head entity, the text representations of the tail entity, the visual features of the key object visual region set, and the joint visual region features to obtain multimodal fusion features; a further implementation method is that the method for obtaining multimodal fusion features includes: Using the text representation of the head entity as the query, a multi-head attention operation is performed on the visual features of the key object visual region set to obtain the visually enhanced representation of the head entity; using the text representation of the tail entity as the query, a multi-head attention operation is performed on the visual features of the key object visual region set to obtain the visually enhanced representation of the tail entity; specifically, based on the head entity representation and the tail entity representation, visual information related to the corresponding entity is aggregated from the object-level visual feature sequence. For both head and tail entities, the entity-level visually enhanced representation is uniformly represented as: in, This represents a multi-head attention operation, where entity representations are used as queries and object-level visual features are used as keys and values.
[0039] for Figure 2 In the illustrated embodiment, the head entity, person B, primarily establishes a high attentional association with the person region, while the tail entity, location B, primarily establishes a high attentional association with buildings, roads, and street view areas. Other people and background objects with lower association with the current entity receive lower weights. The above operations are used to obtain the visual enhancement representation of the head entity. Augmented visual representation of tail entity .
[0040] The joint visual region features are mapped to the same feature space as the entity pair text representation, and then added to the entity pair text representation in a residual manner to obtain the joint region enhancement representation; specifically, further, the UnionCrop joint region features are mapped to the same feature space as the entity pair representation, and then injected into the entity pair representation in a residual manner: in, and These represent the weight matrix and bias vector of the UnionCrop feature map layer, respectively. This represents an enhanced representation of entity pairs after incorporating spatial layout and local scene information. The representation represents the visually enhanced representation of entity pairs obtained by integrating spatial layout and local scene information within the UnionCrop region.
[0041] Object-level visual augmentation primarily reflects the local semantics of objects such as people, buildings, and roads, while UnionCrop augmentation primarily reflects the spatial co-occurrence and local context between person objects and location / scene objects. Both supplement the visual information needed to determine entity relationships from different perspectives.
[0042] The visual enhancement representations of head entities, tail entities, and joint region enhancements are concatenated and mapped to obtain unified visual enhancement features. Specifically, the visual enhancement representations of head entities, tail entities, and UnionCrop enhancements are concatenated and mapped to obtain unified visual enhancement features. in, and These represent the weight matrix and bias vector of the unified visual feature mapping layer, respectively.
[0043] Calculate the gate vector based on the text entity representation and the visual augmentation representation: in, Indicates the Sigmoid activation function Representation of text by entities and unified visual enhancement features The gated input features formed by concatenation, and the gated vector Used to control the proportion of visual information injected into entity pairs. and Let represent the weight matrices of the first and second fully connected layers of the gated network, respectively. and These represent the bias vectors of the first and second fully connected layers of the gated network, respectively.
[0044] The gating mechanism is used to adaptively adjust the weights of various features in relation judgment based on the importance of textual information, object-level visual information, and joint visual information in different samples, thereby reducing the impact of irrelevant images, redundant visual objects, or erroneous regions on the relation extraction results.
[0045] A gating vector is calculated based on the entity-to-text representation and the unified visual augmentation features. This gating vector controls the injection ratio of the unified visual augmentation features into the entity-to-text representation, resulting in a multimodal fusion feature. Specifically, the unified visual augmentation features are injected into the entity-to-text representation using a gated residual method to obtain the final multimodal fusion representation. in, This indicates element-wise multiplication.
[0046] When the input text already contains a relatively clear expression of the relationship between people and places, the gating module mainly retains the text entity pair information and uses image information as supplementary evidence; when the text expression is incomplete or ambiguous, the gating module increases the proportion of visual information injected, so that the model can make more use of the information about people and places provided by multiple images.
[0047] S6: Classify relationships based on multimodal fusion features to obtain the relationship categories between head and tail entities. Specifically, the final multimodal fusion representation... Input the relation classifier to obtain the predicted probability of each candidate relation category: in, This represents the predicted probability of candidate relation category r given the input text, multiple associated images, and head and tail entities. This indicates the input text. This represents a set of images composed of multiple related images. and These represent the head entity and the tail entity, respectively. This represents the final multimodal representation obtained after fusing textual information and dual-view visual information; and These represent the weight matrices of the first and second fully connected layers of the relation classifier, respectively. and These represent the bias vectors of the first and second fully connected layers of the relation classifier, respectively. Represents a nonlinear activation function; This represents a normalized function that converts the classification scores of each candidate relation category into predicted probabilities. The relation category with the highest predicted probability is selected as the final relation extraction result. in, Represents the set of candidate relation categories. This indicates the final predicted relationship category. For Figure 2 The illustrated embodiment ultimately predicts the visit location relationship between person entities and location entities, and generates corresponding structured relationship triples: The relation triples can be written into a multimodal entity relation database or knowledge graph for social media content structuring, people's activity location identification, event retrieval, and cross-modal information analysis.
[0048] During the model training phase, relation classification loss is used to optimize the model's prediction results, and combined with entity object alignment loss, the model learns the correspondence between text entities and visual objects. The relation classification loss is calculated using cross-entropy loss, expressed as: in, Represents the set of candidate relation categories. Indicates candidate relation categories The corresponding real label; when the candidate relationship category When the training samples are the true relation categories, The value is 1 if it is set to 1, otherwise the value is 0. Represents the candidate relation categories output by the relation classifier. The predicted probability.
[0049] The entity alignment loss is used to reduce the feature distance between text entities and their orthogonally aligned visual objects, and to increase the feature differences between text entities and irrelevant visual objects. For head and tail entities, the relevance scores between them and each candidate visual object in the key object visual region set are calculated. The alignment losses for head and tail entities are expressed as follows: The entity object alignment loss is expressed as: in, Indicates the number of candidate visual objects in the set of key object visual regions; The text representation of the head entity and the first Correlation score between features of candidate visual objects; The text representation of the tail entity and the first Correlation score among the features of candidate visual objects and These represent the orthogonally aligned visual object numbers corresponding to the head and tail entities, respectively. This represents the temperature coefficient used to adjust the distribution of relevance scores; the remaining candidate visual objects are used as negative alignment objects in the loss calculation. The total training loss of the model is expressed as: in, Represents the loss of relation classification. This represents the alignment loss of entity objects. These represent the weight coefficients of the entity alignment loss. The entity alignment loss is used to enhance the correspondence between text entities and relevant visual objects, and to suppress irrelevant visual information. The encoding, fusion, and relation classification modules are jointly trained by minimizing the total loss.
[0050] exist Figure 2 In the illustrated embodiment, the person region corresponding to person A can be used as the positive alignment visual object for the head entity, while buildings, roads, and the overall street view region can be used as the visual information corresponding to the location tail entity. Other objects unrelated to the current entity relationship are used as negative alignment objects. By minimizing the total training loss, the text encoder, visual encoder, object-level visual alignment module, UnionCrop spatial enhancement module, gating fusion module, and relation classifier are jointly optimized.
[0051] Through the above implementation methods, this application can filter representative key objects from multiple related images, reducing visual redundancy caused by duplicate objects and irrelevant backgrounds; simultaneously, by combining the object-level visual region UnionCrop, it preserves the local semantics, spatial layout, and scene information of entity-related objects. After multimodal feature fusion, the output relation triples are: This solution can reduce the interference of redundant visual information on relationship judgment and improve the accuracy and stability of multimodal relationship extraction in multi-image scenarios. Specifically, This indicates the visitation relationship between people and locations. The above method fully utilizes complementary visual information from multiple associated images while avoiding noise introduced by invalid visual content, thus improving the accuracy and stability of multimodal relationship extraction results in multi-image environments.
[0052] To verify the effectiveness of the method of the present invention, it was compared with HVFormer-MI, which is extended from the single-image multimodal relation extraction model, the general multimodal model GPT-4o, and the multi-image relation extraction model GLRA proposed from the MRE-MI dataset. The experimental results are shown in Table 1.
[0053] Table 1 As shown in Table 1, the method proposed in this application achieves the highest results in both accuracy and Micro-F1 score. Compared to HVFormer-MI, which is an extension of the single-image method, the accuracy of the method proposed in this application is improved by 7.16 percentage points, and the Micro-F1 score is improved by 6.05 percentage points. Compared to the existing multi-image relation extraction method GLRA, the accuracy is improved by 5.04 percentage points, and the Micro-F1 score is improved by 3.18 percentage points. These results demonstrate that the key object screening, UnionCrop joint region construction, and dual-view visual fusion mechanism proposed in this application can reduce the interference of duplicate objects and irrelevant backgrounds in multiple images, while preserving the local semantic and spatial information of entity-related objects, thereby improving the performance of multi-image multimodal relation extraction.
[0054] To further verify the role of the key technology modules in this application, while maintaining consistency in dataset partitioning, model parameters, and training settings, the key object filtering and UnionCrop joint region module, as well as the entity object alignment loss, were removed, and compared with the complete method. The experimental results are shown in Table 2. Specific ablation settings are as follows: 1. Remove key object screening and joint region module: Instead of clustering candidate objects and selecting representative objects from multiple images, use all candidate object features directly; 2. Remove entity object alignment loss: The model is trained using only relation classification loss; 3. Remove the dual-view fusion module: The model is trained using only standard cross-modal attention. 4. Complete approach: Simultaneously employs key object filtering, UnionCrop joint regions, gating fusion, and entity object alignment loss.
[0055] Table 2 As shown in Table 2, the complete method achieves the best results in both accuracy and Micro-F1 score. Removing the key object screening and UnionCrop joint region reduces accuracy by 1.31 percentage points and Micro-F1 by 1.08 percentage points, indicating that key object screening can reduce duplicate objects and irrelevant background interference in multiple images, and UnionCrop joint region can supplement the spatial layout and local contextual information between entity-related objects. Removing the entity object alignment loss reduces accuracy by 0.13 percentage points and Micro-F1 by 0.71 percentage points, indicating that this loss can enhance the correspondence between text entities and related visual objects. Removing the dual-view fusion module reduces accuracy by 0.94 percentage points and Micro-F1 by 1.69 percentage points, indicating that the synergistic fusion of object-level visual information and joint region visual information plays an important role in relation judgment. These results demonstrate that the key object screening, UnionCrop joint region, entity object alignment loss, and dual-view fusion module in this application can all improve the performance of multimodal relation extraction in multi-image scenes.
[0056] To further illustrate the processing effect of the proposed method on actual samples, three multi-image relationship extraction examples were selected for analysis, and the results are shown in Table 3. In the table, "√" indicates that the model prediction is correct, and "×" indicates that the model prediction is incorrect.
[0057] Table 3 In the first example, the input text describes person A appearing on the cover of magazine D, and two related images provide information about person A's appearance and magazine D's cover, respectively. Existing methods produce incorrect predictions because they fail to accurately associate the person's region with the magazine logo. The method of this invention obtains the correct relationship category by filtering key visual objects such as the person's region and the magazine logo, and fusing complementary visual information from multiple images.
[0058] In the second example, the input text describes the co-occurrence of character A and character B. Three related images are presented, showcasing different combinations of characters, close-ups, and interactive scenes. The effective information contained in a single image is incomplete, and there are also distracting elements such as captions and background figures. Comparative methods failed to yield correct results. The method of this invention reduces irrelevant visual information by filtering key objects and retains co-occurrence and local interaction information between the two entities using joint regions, ultimately obtaining the correct relationship category.
[0059] In the third example, the input text describes person A and person B as appearing together at location C. One associated image mainly shows the distant person and their surrounding environment, while another associated image provides clearer features of the person and the scene features of location C. Since a single image cannot simultaneously provide complete information about both the person and the location, the comparison methods failed to correctly identify the corresponding relationship. The method of this invention comprehensively utilizes the features of both the person and the scene from multiple images to align the text entity with the visual evidence, thereby obtaining a correct prediction result.
[0060] The above examples demonstrate that the method of the present invention can complementarily fuse scattered entity information, scene information, and interaction information in different images, and reduce interference caused by background, captions, and repetitive objects. Especially when the evidence in a single image is incomplete, the method of the present invention can still obtain correct relationship extraction results through key object screening, UnionCrop joint region construction, and dual-view visual fusion.
[0061] Example 2 The present invention also provides a multimodal relationship extraction system for multi-image scenes, used to implement the method of Embodiment 1, including: The multimodal data acquisition module is used to acquire input text, multiple images associated with the input text, and head and tail entities in the input text; The key object visual region acquisition module is used to detect candidate visual objects in images. It uses visual semantic similarity calculation and hierarchical clustering to construct a set of key object visual regions. The key visual object selection module is used to select the first key visual object and the second key visual object that meet the preset relevance requirements from the key object visual region set, based on the text representation of the head entity and the text representation of the tail entity, respectively. The joint visual region feature acquisition module is used to determine the joint region covering the first key visual object and the second key visual object based on the position information of the first key visual object and the second key visual object, and extract the visual features of the joint region from the corresponding image to obtain the joint visual region features. The multimodal fusion module is used to perform multimodal fusion of the text representation of the head entity, the text representation of the tail entity, the visual features of the key object visual region set, and the joint visual region features to obtain multimodal fusion features. The relation extraction module is used to classify relations based on multimodal fusion features and obtain the relationship categories between head entities and tail entities.
[0062] A further implementation method includes a key object visual region acquisition module comprising: The candidate object acquisition unit is used to perform object detection on the image associated with the input text using an object detector to obtain candidate visual objects; The feature extraction unit is used to extract features of candidate visual objects in each image using a visual encoder to obtain the features of the candidate visual objects. Clustering units are used to calculate the similarity between the features of each candidate visual object, and to divide the candidate visual objects into several object clusters based on the similarity using hierarchical clustering. The representative object selection unit is used to calculate the central feature of each object cluster and select the candidate visual object that is closest to the central feature as the representative object of the object cluster. The set construction unit is used to assemble the representative objects corresponding to each object cluster into a set of key object visual regions.
[0063] The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.
Claims
1. A method for extracting multimodal relationships in multi-image scenes, characterized in that, include: Obtain the input text, multiple images associated with the input text, and the head and tail entities in the input text; Candidate visual objects are detected in the image, and visual semantic similarity calculation and hierarchical clustering are used to construct a set of key object visual regions. Based on the text representation of the head entity and the text representation of the tail entity, the candidate visual object with the highest correlation to the head entity is selected from the set of key object visual regions as the first key visual object and the second key visual object corresponding to the tail entity. Based on the location information of the first key visual object and the location information of the second key visual object, a joint region covering the first key visual object and the second key visual object is determined, and the visual features of the joint region are extracted from the corresponding image to obtain the joint visual region features. The text representations of the head entity, the text representations of the tail entity, the visual features of the key object visual region set, and the joint visual region features are fused in a multimodal manner to obtain multimodal fusion features. Relationship classification is performed based on the multimodal fusion features to obtain the relationship category between the head entity and the tail entity.
2. The method according to claim 1, characterized in that, Methods for obtaining a set of key visual object regions include: The image associated with the input text is detected using an object detector to obtain candidate visual objects; The visual encoder is used to extract features from candidate visual objects in each image to obtain the features of candidate visual objects; Calculate the similarity between the features of each candidate visual object, and divide the candidate visual objects into several object clusters based on the similarity using a hierarchical clustering method; Calculate the central feature of each object cluster, and select the candidate visual object that is closest to the central feature as the representative object of the object cluster; The representative objects corresponding to each object cluster are combined to form the key object visual region set.
3. The method according to claim 2, characterized in that, The method for obtaining the first key visual object and the second key visual object includes: Calculate a first correlation score between the text representation of the head entity and the features of each candidate visual object in the set of key object visual regions; based on the first correlation score, select the candidate visual object with the highest correlation to the head entity from the set of key object visual regions as the first key visual object; Calculate a second correlation score between the text representation of the tail entity and the features of each candidate visual object in the set of key object visual regions; based on the second correlation score, select the candidate visual object with the highest correlation to the tail entity from the set of key object-level visual regions as the second key visual object.
4. The method according to claim 2, characterized in that, Methods for obtaining multimodal fusion features include: Using the text representation of the head entity as a query, a multi-head attention operation is performed on the visual features of the key object visual region set to obtain a visually enhanced representation of the head entity. Using the text representation of the tail entity as a query, a multi-head attention operation is performed on the visual features of the key object visual region set to obtain a visually enhanced representation of the tail entity. The joint visual region features are mapped to the same feature space as the entity-to-text representation, and then added to the entity-to-text representation in a residual manner to obtain the joint region enhancement representation; The visual enhancement representations of the head entity, the tail entity, and the joint region enhancement representation are concatenated and mapped to obtain a unified visual enhancement feature. A gating vector is calculated based on the entity-to-text representation and the unified visual enhancement feature. The gating vector is then used to control the injection ratio of the unified visual enhancement feature into the entity-to-text representation, thereby obtaining the multimodal fusion feature.
5. The method according to claim 4, characterized in that, The methods for obtaining the text representation of the entity include: Entity tagging is performed on the head and tail entities of the input text to obtain the entity-tagged text sequence; The text sequence is encoded using a text encoder to obtain the contextual semantic representation of each word in the text sequence; Based on the positions of the head entity and the tail entity in the text sequence, the context semantic representation is averaged to obtain the head entity representation and the tail entity representation; The head entity representation and the tail entity representation are concatenated and mapped to obtain the entity-to-text representation.
6. The method according to claim 4, characterized in that, The visual features of the key object visual region set include features of A person region, B person region, building features, road features, and features of representative street scene objects; The joint visual region features are used to represent spatial and scene information when a person and the surrounding environment coexist.
7. A multimodal relation extraction system for multi-image scenes, used to implement the method described in any one of claims 1-6, characterized in that, include: A multimodal data acquisition module is used to acquire input text, multiple images associated with the input text, and head and tail entities in the input text; The key object visual region acquisition module is used to detect candidate visual objects in the image and construct a set of key object visual regions by using visual semantic similarity calculation and hierarchical clustering. The key visual object selection module is used to select a first key visual object and a second key visual object that meet preset relevance requirements from the key object visual region set, based on the text representation of the head entity and the text representation of the tail entity, respectively. The joint visual region feature acquisition module is used to determine the joint region covering the first key visual object and the second key visual object based on the position information of the first key visual object and the position information of the second key visual object, and extract the visual features of the joint region from the corresponding image to obtain the joint visual region features. The multimodal fusion module is used to perform multimodal fusion of the text representation of the head entity, the text representation of the tail entity, the visual features of the key object visual region set, and the joint visual region features to obtain multimodal fusion features; The relationship extraction module is used to classify relationships based on the multimodal fusion features to obtain the relationship category between the head entity and the tail entity.
8. The system according to claim 7, characterized in that, The key object visual region acquisition module includes: The candidate object acquisition unit is used to perform target detection on the image associated with the input text using a target detector to obtain candidate visual objects; The feature extraction unit is used to extract features of candidate visual objects in each image using a visual encoder to obtain the features of the candidate visual objects. Clustering units are used to calculate the similarity between the features of each candidate visual object, and to divide the candidate visual objects into several object clusters based on the similarity using hierarchical clustering. The representative object selection unit is used to calculate the central feature of each object cluster and select the candidate visual object that is closest to the central feature as the representative object of the object cluster. The set construction unit is used to assemble the key object visual region set by combining the representative objects corresponding to each object cluster.
