A multi-modal named entity recognition method for multi-document scenarios

Abstract

The application discloses a kind of multi-modal named entity recognition methods for multi-graph scene, belong to computer vision and multi-modal information extraction technical field, including: constructing entity cross-graph candidate visual region set and determining initial visual support weight;Based on entity type label, entity condition multi-instance supervision is carried out, and false matching is inhibited by non-entity negative sample constraint;Based on the entity type prediction result of set, visual region disturbance is carried out, and contribution degree and discriminativeness are determined according to target type prediction score change;Based on contribution degree and discriminativeness, visual reconstruction is guided by contribution, and support maintenance ability is determined according to target type prediction score retention degree;Further, visual contribution calibration score is generated to calibrate aggregation, and calibrated cross-graph entity level visual representation is directed to text representation, to obtain multi-modal named entity recognition result.The method can improve the reliability and accuracy of entity recognition in multi-graph scene.

Description

Technical Field

[0001] This invention relates to a multimodal named entity recognition method for multi-image scenarios, belonging to the field of computer vision and multimodal information extraction technology. Background Technology

[0002] Named entity recognition (NER) is a fundamental task in the field of information extraction, aiming to identify the boundaries and types of entities with clear semantic categories, such as names of people, places, organizations, and others, from unstructured text. With the development of mobile internet and social platforms, more and more text content is associated with multiple images simultaneously, creating multi-image, multimodal NER scenarios. In these scenarios, multiple images can provide external visual cues for ambiguous or context-deficient entities in the text, thereby assisting in entity boundary recognition and entity type determination.

[0003] Existing multimodal named entity recognition methods mostly suffer from the following shortcomings: First, some methods are mainly geared towards single-image scenarios, making it difficult to handle the cross-image visual region selection problem when a text sequence is associated with multiple images simultaneously. Second, some methods adopt a unified visual fusion approach at the sample level or sequence level in multi-image scenarios, which easily introduces background co-occurrence, image redundancy, or visual information unrelated to the current entity from multiple images into the text representation. Third, existing data often lack explicit correspondence annotations between entities and local image regions, relying solely on similarity, attention weights, or relevance scores to select visual regions, which easily leads to misidentification of pseudo-related local image regions caused by topic similarity or cross-entity mismatches as valid visual regions. Fourth, some methods lack a visual region rejection mechanism for non-entity fragments, which may lead to non-entity fragments being forcibly matched with visual information, affecting the stability of entity boundary recognition and entity type prediction. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal named entity recognition method for multi-image scenarios, which can reduce interference from background co-occurrence, image redundancy, and pseudo-correlated local image regions, thereby improving the reliability and accuracy of entity recognition in multi-image scenarios.

[0005] To achieve the above objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides a multimodal named entity recognition method for multi-graph scenarios, comprising: A set of cross-image candidate visual regions for entities is constructed from the local region visual representations of multiple valid images of the same multi-image sample, conditioned on the entity-level text representation of candidate entity fragments, and the initial visual support weight of each candidate visual region in the set of cross-image candidate visual regions for entities is determined. Based on the entity type labels of candidate entity fragments, entity conditional multi-instance supervision is performed on the entity cross-graph candidate visual region set, and the mismatch between non-entity fragments and irrelevant visual regions is suppressed by non-entity negative sample constraints. The entity type prediction results corresponding to the complete entity cross-image candidate visual region set are used as the benchmark prediction results to perturb the entity cross-image candidate visual region set, and the contribution and discriminativeness of each candidate visual region are determined according to the change in the target type prediction score of the candidate entity fragments before and after the perturbation. Visual reconstruction guided by contribution of each candidate visual region set across the graph is performed based on the contribution and discriminativeness of each candidate visual region, and the support maintenance capability of each candidate visual region is determined according to the degree of retention of the target type prediction score of the candidate entity fragment after reconstruction. A visual contribution calibration score is generated based on the initial visual support weight, contribution, discriminativeness, and support maintenance capability of each candidate visual region. The entity cross-graph candidate visual region set is then calibrated and aggregated based on the visual contribution calibration score to obtain the calibrated cross-graph entity-level visual representation. The calibrated cross-graph entity-level visual representation is applied to the text representation of candidate entity fragments to obtain multimodal named entity recognition results.

[0006] In conjunction with the first aspect, furthermore, based on the entity-level text representation of candidate entity fragments, a set of cross-image candidate visual regions for entities is constructed from the local region visual representations of multiple valid images from the same multi-image sample, and the initial visual support weights of each candidate visual region in the set of cross-image candidate visual regions for entities are determined, including: Based on the valid image labels used to distinguish between valid images and completed images, the visual representations of the local regions corresponding to the completed images are excluded, and the visual representations of the local regions of multiple valid images in the same multi-image sample are aggregated into a cross-image visual instance set. The matching calculation is performed on each visual instance in the cross-graph visual instance set based on the entity-level text representation of the candidate entity fragment, so as to obtain the initial visual support score of each visual instance relative to the candidate entity fragment. A unified weight allocation is performed across the graph for the initial visual support scores of each visual instance relative to the candidate entity fragments to obtain the initial visual support weight of each visual instance. Visual instances with non-zero initial visual support weights are used as candidate visual regions to form a set of cross-graph candidate visual regions for the entity fragments. The initial visual support weight of the visual instance is used as the initial visual support weight of the candidate visual region. In this context, different candidate entity fragments within the same multi-graph sample share a cross-graph visual instance set, and each has its own corresponding cross-graph candidate visual region set.

[0007] In conjunction with the first aspect, further, entity-conditional multi-instance supervision of entity cross-graph candidate visual region sets based on entity type labels of candidate entity fragments includes: The set of cross-graph candidate visual regions corresponding to the candidate entity fragment is taken as a multi-instance package, and each candidate visual region in the set of cross-graph candidate visual regions is taken as an instance in the multi-instance package. Based on the initial visual support weights of each candidate visual region in the entity cross-graph candidate visual region set, the instances in the multi-instance package are weighted and aggregated to generate the visual region aggregated representation of the candidate entity fragment, and then fused with the entity-level text representation of the candidate entity fragment to obtain the entity visual joint representation of the candidate entity fragment. The entity visual joint representation of candidate entity fragments is used to predict the entity type of the candidate entity fragments. The entity type label of the candidate entity fragments is used as a bag-level supervision signal to apply a classification loss to the entity type prediction result of the candidate entity fragments, so that the overall representation of the multi-instance bag learns the correspondence with the entity type label of the candidate entity fragments.

[0008] In conjunction with the first aspect, further, suppressing mismatches between non-entity fragments and irrelevant visual regions through non-entity negative sample constraints includes: Construct O-class negative samples from non-entity locations in a text sequence and generate text representations of the O-class negative samples; Using the text representation of the negative O samples as a condition, a set of candidate visual regions for the O-class negative samples is constructed from the local region visual representations of multiple valid images of the same multi-image sample, and the initial visual support weights of each candidate visual region in the set of candidate visual regions for the O-class negative samples are determined. Based on the initial visual support weights of each candidate visual region in the candidate visual region set of O-class negative samples, the candidate visual regions in the O-class negative sample candidate visual region set are weighted and aggregated to generate the visual region aggregated representation of O-class negative samples. This is then fused with the text representation of O-class negative samples to obtain the entity visual joint representation of O-class negative samples. We use O-class labels to supervise the joint entity visual representation of O-class negative samples, thereby suppressing mismatches between non-entity segments and irrelevant visual regions.

[0009] In conjunction with the first aspect, furthermore, using the entity type prediction results corresponding to the complete set of cross-image candidate visual regions as the benchmark prediction results, visual region perturbation is applied to the set of cross-image candidate visual regions. The contribution and discriminative power of each candidate visual region are determined based on the changes in the target type prediction scores of the candidate entity fragments before and after the perturbation, including: Within the scope of the entity cross-graph candidate visual region set, the initial visual support weights of each candidate visual region are reallocated and normalized to obtain a complete visual region representation. The baseline prediction result is obtained by combining the entity-level text representation of the candidate entity fragments. Determine the target type of the candidate entity fragment. If the candidate entity fragment has a true entity type label, then the true entity type label is taken as the target type; otherwise, the entity type with the highest score in the baseline prediction results is taken as the target type. Using the baseline prediction results as a reference, the visual regions of the candidate visual regions across the graph are perturbed, including single-region removal verification and cross-entity negative region replacement verification. The contribution and discriminativeness of each candidate visual region are determined based on the change in the target type prediction score of the candidate entity fragments before and after the perturbation.

[0010] In conjunction with the first aspect, further, single-region removal verification includes: One candidate visual region is removed sequentially from the entity cross-graph candidate visual region set to obtain multiple candidate visual region sets after single region removal; For each single region removal, the remaining candidate visual regions in the candidate visual region set are reassigned weights, and entity type prediction is performed based on the reassigned remaining candidate visual regions to obtain the single region removal prediction result. The difference between the baseline prediction result and the single-region removal prediction result in the prediction score of the target type of the candidate entity fragment is calculated to obtain the contribution of the removed candidate visual region. This contribution is used to characterize the actual support of the removed candidate visual region for the entity type prediction of the candidate entity fragment, and the contribution of the removed candidate visual region is used as the basis for contribution-guided visual reconstruction.

[0011] In conjunction with the first aspect, further cross-entity negative region replacement verification includes: Negative visual regions are selected from the set of cross-graph candidate visual regions corresponding to other candidate entity segments that are different from the current candidate entity segment. When there are no negative visual regions that meet the conditions in the same multi-graph sample, negative visual regions are selected from the candidate entity segments, negative region cache pool, or random valid visual regions of the cross-graph candidate visual regions set that do not belong to the current candidate entity segment in other multi-graph samples in the same training batch. Replace at least one candidate visual region in the set of cross-image candidate visual regions corresponding to the current candidate entity fragment with a negative visual region to obtain a negative region replacement set, and obtain the negative region replacement prediction result based on the negative region replacement set; The difference between the baseline prediction result and the negative region substitution prediction result in the target type of the candidate entity fragment is calculated to obtain the negative region substitution advantage, which is used to characterize the prediction advantage of the candidate entity fragment's own visual region relative to the negative visual region. The negative region substitution advantage is used as a measure of the discriminativeness of the candidate visual region as the basis for contribution-guided visual reconstruction.

[0012] In conjunction with the first aspect, furthermore, based on the contribution and discriminativeness of each candidate visual region, a contribution-guided visual reconstruction of the entity cross-graph candidate visual region set is performed, and the support maintenance capability of each candidate visual region is determined according to the degree of retention of the target type prediction score of the reconstructed candidate entity fragments, including: The reconstruction support weights are determined based on the contribution and discriminative power of each candidate visual region; Based on the reconstruction support weights, a contribution-guided visual reconstruction is performed on the entity cross-graph candidate visual region set to obtain a contribution-guided visual representation. The support maintenance capability of each candidate visual region is determined by the degree to which the target type prediction score of the candidate entity fragment is preserved based on the reconstruction prediction results corresponding to the contribution-guided visual representation.

[0013] In conjunction with the first aspect, furthermore, based on the visual contribution calibration scores of each candidate visual region, the entity cross-graph candidate visual region set is calibrated and aggregated to obtain the calibrated cross-graph entity-level visual representation, including: The calibration weight is obtained by fusing the visual contribution calibration score of each candidate visual region with the initial visual support weight of each candidate visual region. The entity cross-graph candidate visual region set is weighted and aggregated according to the calibration aggregation weight to obtain the calibrated cross-graph entity-level visual representation.

[0014] In conjunction with the first aspect, the calibrated cross-graph entity-level visual representation is further applied to the text representation of candidate entity fragments to obtain multimodal named entity recognition results, including: The calibrated cross-graph entity-level visual representation is mapped to the text feature space to obtain the entity-level visual update representation; Based on the position of the candidate entity fragment in the text sequence, the entity-level visual update representation is injected into the text representation of the corresponding region of the candidate entity fragment, and the updated text representation is used for named entity label prediction to obtain the multimodal named entity recognition result.

[0015] Secondly, the present invention provides a multimodal named entity recognition system for multi-graph scenarios, comprising: The candidate visual region construction module is used to construct a set of cross-image candidate visual regions for entities from the local region visual representations of multiple valid images of the same multi-image sample, based on the entity-level text representation of the candidate entity fragments, and to determine the initial visual support weight of each candidate visual region in the set of cross-image candidate visual regions for entities. The entity conditional multi-instance supervision module is used to perform entity conditional multi-instance supervision on the set of candidate visual regions across the graph based on the entity type label of the candidate entity fragment, and to suppress mismatches between non-entity fragments and irrelevant visual regions through non-entity negative sample constraints. The visual contribution calibration module is used to perturb the entity type prediction results corresponding to the complete entity cross-graph candidate visual region set as the benchmark prediction results, and determine the contribution and discriminativeness of each candidate visual region based on the change in the target type prediction score of the candidate entity fragments before and after the perturbation; based on the contribution and discriminativeness of each candidate visual region, the module performs contribution-guided visual reconstruction of the entity cross-graph candidate visual region set, and determines the support maintenance capability of each candidate visual region based on the degree of preservation of the target type prediction score of the candidate entity fragments after reconstruction. The calibration fusion module is used to generate a visual contribution calibration score based on the initial visual support weight, contribution, discriminativeness and support maintenance capability of each candidate visual region, and to perform calibration aggregation on the entity cross-graph candidate visual region set based on the visual contribution calibration score to obtain the calibrated cross-graph entity-level visual representation. The entity fragment update and recognition module is used to apply the calibrated cross-graph entity-level visual representation to the text representation of candidate entity fragments to obtain multimodal named entity recognition results.

[0016] Thirdly, the present invention provides a computer device, comprising: Storage medium: used to store computer programs; Processor: Used to execute the computer program to implement the multimodal named entity recognition method for multi-graph scenarios as described in the first aspect.

[0017] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal named entity recognition method for multi-graph scenarios described in the first aspect.

[0018] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the multimodal named entity recognition method for multi-graph scenarios described in the first aspect.

[0019] Compared with the prior art, the beneficial effects of the present invention are: The multimodal named entity recognition method for multi-image scenarios provided by this invention is designed for multi-image multimodal named entity recognition scenarios in which a text sequence is associated with multiple images. Instead of simply averaging or globally fusing multiple images, it constructs a set of cross-image candidate visual regions for each candidate entity fragment, thereby improving the entity targeting of visual region discovery.

[0020] In the absence of corresponding annotations between entities and local image regions, this invention models the candidate visual region set as an entity-conditional multi-instance package and uses the entity type labels of candidate entity fragments to perform package-level supervision on the aggregated representation of candidate visual regions. This enables the model to learn potential visual regions related to the current candidate entity fragment type judgment from multiple images under weak supervision.

[0021] This invention introduces non-entity negative sample constraints, enabling the model to not only learn the effective visual regions corresponding to entities, but also to learn to reject erroneous matches between non-entity text fragments and irrelevant visual regions, thereby reducing the interference of visual information on entity boundary recognition.

[0022] This invention calibrates the visual contribution of candidate visual regions by using benchmark prediction results, single-region removal verification, cross-entity negative region replacement verification, and contribution-guided visual reconstruction. It not only selects visual regions based on initial relevance scores, but also further determines whether the visual region truly supports the type judgment of the current candidate entity fragment.

[0023] This invention fuses the visual contribution calibration score with the initial visual support weight and performs secondary weighted aggregation on the candidate visual regions. This reduces the impact of pseudo-related local regions caused by background co-occurrence, topic similarity, image redundancy, and cross-entity mismatch, thereby improving the accuracy and reliability of multi-image multimodal named entity recognition. Attached Figure Description

[0024] Figure 1 This is a flowchart of a multimodal named entity recognition method for multi-graph scenarios provided in an embodiment of the present invention. Detailed Implementation

[0025] The technical solution of the present invention will be further described in detail below with reference to specific embodiments.

[0026] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. Unless otherwise specified, embodiments of the present invention and the technical features thereof can be combined with each other.

[0027] This invention provides a multimodal named entity recognition method for multi-graph scenarios, including: A set of cross-image candidate visual regions for entities is constructed from the local region visual representations of multiple valid images of the same multi-image sample, conditioned on the entity-level text representation of candidate entity fragments, and the initial visual support weight of each candidate visual region in the set of cross-image candidate visual regions for entities is determined. Based on the entity type labels of candidate entity fragments, entity conditional multi-instance supervision is performed on the entity cross-graph candidate visual region set, and the mismatch between non-entity fragments and irrelevant visual regions is suppressed by non-entity negative sample constraints. The entity type prediction results corresponding to the complete entity cross-image candidate visual region set are used as the benchmark prediction results to perturb the entity cross-image candidate visual region set, and the contribution and discriminativeness of each candidate visual region are determined according to the change in the target type prediction score of the candidate entity fragments before and after the perturbation. Visual reconstruction guided by contribution of each candidate visual region set across the graph is performed based on the contribution and discriminativeness of each candidate visual region, and the support maintenance capability of each candidate visual region is determined according to the degree of retention of the target type prediction score of the candidate entity fragment after reconstruction. A visual contribution calibration score is generated based on the initial visual support weight, contribution, discriminativeness, and support maintenance capability of each candidate visual region. The entity cross-graph candidate visual region set is then calibrated and aggregated based on the visual contribution calibration score to obtain the calibrated cross-graph entity-level visual representation. The calibrated cross-graph entity-level visual representation is applied to the text representation of candidate entity fragments to obtain multimodal named entity recognition results.

[0028] The multimodal named entity recognition method for multi-image scenarios provided by this invention addresses the scenario of multimodal named entity recognition in a multi-image context where a text sequence is associated with multiple images. It constructs a set of cross-image candidate visual regions based on entity conditions around candidate entity fragments in the text. In the absence of region-level annotations, it learns the correspondence between candidate visual regions and candidate entity fragment types through multi-instance supervised learning based on entity conditions. Furthermore, it uses visual contribution calibration to determine whether local visual regions in multiple images truly support the type judgment of the current candidate entity fragment. This reduces interference from background co-occurrence, image redundancy, and pseudo-correlated local image regions, thereby improving the reliability and accuracy of entity recognition in multi-image scenarios.

[0029] Figure 1 This is a flowchart of a multimodal named entity recognition method for multi-graph scenarios provided in this embodiment of the invention. This flowchart only shows the logical order of the method in this embodiment of the invention. Provided there are no conflicts, different flowcharts may be used. Figure 1 Complete the steps shown or described in the order indicated.

[0030] The multimodal named entity recognition method for multi-graph scenarios provided in this invention can be applied to a terminal and can be executed by a multimodal named entity recognition system for multi-graph scenarios. This system can be implemented by software and / or hardware and can be integrated into the terminal, such as any tablet computer or computer device with communication functions.

[0031] In one possible embodiment, such as Figure 1 As shown, the multimodal named entity recognition method for multi-graph scenarios specifically includes the following steps: Step 1: Construct a set of cross-image candidate visual regions for entities from the local region visual representations of multiple valid images of the same multi-image sample, based on the entity-level text representation of the candidate entity fragments, and determine the initial visual support weight of each candidate visual region in the set of cross-image candidate visual regions for entities. In this embodiment, the construction of a cross-image candidate visual region set for entities based on the entity-level text representation of candidate entity fragments from the local region visual representations of multiple valid images of the same multi-image sample, and the determination of the initial visual support weight of each candidate visual region in the cross-image candidate visual region set, specifically includes the following steps: Step 1: Based on the valid image labels used to distinguish between valid images and completed images, exclude the local region visual representations corresponding to the completed images, and aggregate the local region visual representations of multiple valid images in the same multi-image sample into a cross-image visual instance set; Step 2: Match and calculate the visual instances in the cross-graph visual instance set based on the entity-level text representation of the candidate entity fragments to obtain the initial visual support score of each visual instance in the cross-graph visual instance set relative to the candidate entity fragments. Step 3: Perform a unified weight allocation across the graph for the initial visual support scores of each visual instance in the cross-graph visual instance set relative to the candidate entity fragments, and obtain the initial visual support weights of each visual instance in the cross-graph visual instance set. Use the visual instances with non-zero initial visual support weights as candidate visual regions to form a set of cross-graph candidate visual regions corresponding to the candidate entity fragments, and use the initial visual support weights of the visual instances as the initial visual support weights of the candidate visual regions.

[0032] In this embodiment, different candidate entity fragments in the same multi-image sample share a cross-image visual instance set, and each has its own corresponding cross-image candidate visual region set.

[0033] Step 2: Perform multi-instance supervision of entity conditions on the candidate visual region set across the graph based on the entity type label of the candidate entity fragment, and suppress mismatch between non-entity fragments and irrelevant visual regions by non-entity negative sample constraints; In this embodiment, the entity conditional multi-instance supervision of the entity cross-graph candidate visual region set based on the entity type label of the candidate entity fragment specifically includes the following steps: Step 1: Take the set of candidate visual regions across the entity graph corresponding to the candidate entity fragment as a multi-instance package, and take each candidate visual region in the set of candidate visual regions across the entity graph as an instance in the multi-instance package; Step 2: Based on the initial visual support weights of each candidate visual region in the entity cross-graph candidate visual region set, perform weighted aggregation on each instance in the multi-instance package to generate the visual region aggregation representation of the candidate entity fragment, and fuse it with the entity-level text representation of the candidate entity fragment to obtain the entity-visual joint representation of the candidate entity fragment. Step 3: Based on the joint visual representation of candidate entity fragments, predict the entity type of the candidate entity fragments. Apply classification loss to the entity type prediction results of the candidate entity fragments using the entity type label of the candidate entity fragments as a bag-level supervision signal, so that the overall representation of the multi-instance bag learns the correspondence with the entity type label of the candidate entity fragments.

[0034] In this embodiment, suppressing mismatches between non-entity segments and irrelevant visual regions through non-entity negative sample constraints specifically includes the following steps: Step 1: Construct Class O negative samples from non-entity locations in the text sequence and generate text representations of Class O negative samples; Step 2: Construct a candidate visual region set for class O negative samples from the local region visual representations of multiple valid images of the same multi-image sample, based on the text representation of class O negative samples, and determine the initial visual support weight of each candidate visual region in the candidate visual region set for class O negative samples. Step 3: Based on the initial visual support weights of each candidate visual region in the candidate visual region set of the O-class negative samples, perform weighted aggregation on each candidate visual region in the candidate visual region set of the O-class negative samples to generate the visual region aggregation representation of the O-class negative samples, and fuse it with the text representation of the O-class negative samples to obtain the entity visual joint representation of the O-class negative samples. Step 4: Use O-class labels to supervise the joint entity visual representation of O-class negative samples, and suppress mismatches between non-entity segments and irrelevant visual regions.

[0035] Step 3: Using the entity type prediction results corresponding to the complete entity cross-image candidate visual region set as the benchmark prediction results, perform visual region perturbation on the entity cross-image candidate visual region set, and determine the contribution and discriminativeness of each candidate visual region in the entity cross-image candidate visual region set based on the change in the target type prediction score of the candidate entity fragments before and after perturbation. In this embodiment, the entity type prediction result corresponding to the complete entity cross-image candidate visual region set is used as the benchmark prediction result to perturb the entity cross-image candidate visual region set, and the contribution and discriminativeness of each candidate visual region in the entity cross-image candidate visual region set are determined based on the change in the target type prediction score of the candidate entity fragments before and after the perturbation. The specific steps include the following: Step 1: Within the scope of the entity cross-graph candidate visual region set, the initial visual support weights of each candidate visual region are reallocated and normalized to obtain a complete visual region representation. The baseline prediction result is obtained by combining the entity-level text representation of the candidate entity fragments. Step 2: Determine the target type of the candidate entity fragment. If the candidate entity fragment has a true entity type label, then the true entity type label is taken as the target type; otherwise, the entity type with the highest score in the baseline prediction results is taken as the target type. Step 3: Using the baseline prediction results as a reference, perform visual region perturbation on the entity cross-graph candidate visual region set, which includes single-region removal verification and cross-entity negative region replacement verification. Based on the change in the target type prediction score of the candidate entity fragments before and after the perturbation, determine the contribution and discriminativeness of each candidate visual region in the entity cross-graph candidate visual region set.

[0036] Specifically, the single-area removal verification includes the following steps: Step ①: Remove one candidate visual region from the entity cross-graph candidate visual region set one by one, to obtain multiple candidate visual region sets after removing a single region; Step 2: Reassign weights to the remaining candidate visual regions in the candidate visual region set after each single region removal, and perform entity type prediction based on the remaining candidate visual regions after weight reassignment to obtain the single region removal prediction result. Step 3: Calculate the difference between the baseline prediction result and the single-region removal prediction result in the prediction score of the target type of the candidate entity fragment, and obtain the contribution of the removed candidate visual region. This contribution is used to characterize the actual support of the removed candidate visual region for the entity type prediction of the candidate entity fragment, and the contribution of the removed candidate visual region is used as the basis for contribution-guided visual reconstruction.

[0037] Specifically, cross-entity negative region replacement verification includes the following steps: Step ①: Select a negative visual region from the set of cross-graph candidate visual regions corresponding to other candidate entity segments that are different from the current candidate entity segment. When there is no negative visual region that meets the conditions in the same multi-graph sample, select a negative visual region from the candidate entity segments, negative region cache pool, or random valid visual regions of the set of cross-graph candidate visual regions that do not belong to the current candidate entity segment in other multi-graph samples in the same training batch. Step 2: Replace at least one candidate visual region in the set of cross-image candidate visual regions corresponding to the current candidate entity fragment with a negative visual region to obtain a negative region replacement set, and obtain the negative region replacement prediction result based on the negative region replacement set; Step 3: Calculate the difference in prediction scores between the baseline prediction result and the negative region replacement prediction result for the target type of the candidate entity fragment, and obtain the negative region replacement advantage. This advantage is used to characterize the prediction advantage of the candidate entity fragment's own visual region relative to the negative visual region. The negative region replacement advantage is used as a measure of the discriminativeness of the candidate visual region as the basis for contribution-guided visual reconstruction.

[0038] Step 4: Based on the contribution and discriminativeness of each candidate visual region in the entity cross-graph candidate visual region set, perform contribution-guided visual reconstruction of the entity cross-graph candidate visual region set, and determine the support and maintenance capability of each candidate visual region in the entity cross-graph candidate visual region set for entity type prediction of candidate entity fragments based on the degree of retention of the target type prediction score of the candidate entity fragments after reconstruction. In this embodiment, the visual reconstruction of the entity cross-graph candidate visual region set is guided by contribution based on the contribution and discriminativeness of each candidate visual region in the entity cross-graph candidate visual region set. The support and maintenance capability of each candidate visual region in the entity cross-graph candidate visual region set for entity type prediction of candidate entity fragments is determined according to the degree of retention of the target type prediction score of the candidate entity fragments after reconstruction. The specific steps include the following: Step 1: Determine the reconstruction support weights based on the contribution and discriminative power of each candidate visual region in the entity cross-graph candidate visual region set; Specifically, the reconstruction support weight is determined based on the contribution and distinctiveness of each candidate visual region in the entity cross-graph candidate visual region set, so that candidate visual regions with high contribution and distinctiveness receive higher support during the reconstruction process, while candidate visual regions with low contribution or weak distinctiveness are suppressed.

[0039] Step 2: Perform contribution-guided visual reconstruction on the entity cross-graph candidate visual region set according to the reconstruction support weights to obtain the contribution-guided visual representation; Step 3: Determine the support and maintenance capability of each candidate visual region in the entity cross-graph candidate visual region set for entity type prediction of candidate entity fragments based on the degree of preservation of the target type prediction score of the candidate entity fragment according to the reconstruction prediction results corresponding to the contribution-guided visual representation.

[0040] Step 5: Generate a visual contribution calibration score for each candidate visual region based on the initial visual support weight, contribution, discriminativeness, and support maintenance capability of each candidate visual region in the entity cross-graph candidate visual region set. Then, perform calibration and aggregation on the entity cross-graph candidate visual region set based on the visual contribution calibration scores of each candidate visual region to obtain the calibrated cross-graph entity-level visual representation. In this embodiment, the calibration and aggregation of the entity cross-graph candidate visual region set based on the visual contribution calibration score of each candidate visual region to obtain the calibrated cross-graph entity-level visual representation specifically includes the following steps: Step 1: Fuse the visual contribution calibration score of each candidate visual region with the initial visual support weight of each candidate visual region to obtain the calibration aggregate weight; Step 2: Perform weighted aggregation on the candidate visual regions of entities across the graph according to the calibration aggregation weights to obtain the calibrated cross-graph entity-level visual representation.

[0041] Step 6: Apply the calibrated cross-graph entity-level visual representation to the text representation of the candidate entity fragment to obtain the multimodal named entity recognition result.

[0042] In this embodiment, the calibration of the cross-graph entity-level visual representation is applied to the text representation of the candidate entity fragment to obtain the multimodal named entity recognition result, specifically including the following steps: Step 1: Map the calibrated cross-graph entity-level visual representation to the text feature space to obtain the entity-level visual update representation; Step 2: Based on the position of the candidate entity fragment in the text sequence, inject the entity-level visual update representation into the text representation of the corresponding region of the candidate entity fragment, and use the updated text representation for named entity label prediction to obtain the multimodal named entity recognition result.

[0043] This invention provides a visual contribution calibration method for multimodal named entity recognition in multi-image scenarios, applicable to multimodal named entity recognition tasks where a text sequence is simultaneously associated with multiple images. Given an input sample, the input sample includes a text sequence, multiple images associated with the text sequence, and named entity labels corresponding to the text sequence. Since the number of real images associated with different samples may vary, in actual processing, each sample can be padded to a uniform number of images, and the real valid images and padded images can be distinguished by valid image labels.

[0044] The text encoding model employs a robustly optimized bidirectional encoder representation from transformers approach (RoBERTa) pre-trained language model, while the visual encoding model uses a vision transformer (ViT) model. RoBERTa is used to extract the contextual semantic representation of each token in the text sequence, and ViT is used to extract the global visual representation and local region visual representation of each image. Alternatively, other text encoding models capable of generating contextual text representations, and other visual encoding models capable of generating local region visual representations of images, can also be used.

[0045] In one possible embodiment, the visual contribution calibration method for multimodal named entity recognition in multi-graph scenarios specifically includes the following steps: Step S110: Representation of samples and their candidate entity fragments, visual representation, and effective image labeling.

[0046] In step S110, an input sample consisting of a text sequence and its associated multiple images is obtained, and candidate entity fragments, entity-level text representations, visual representations of local image regions of multiple images, and effective image labels corresponding to the input sample are obtained.

[0047] Specifically, the process begins by reading the text token sequence, named entity label sequence, and associated image list from the input sample. For the text sequence, the Begin-Inside-Outside (BIO) annotation system can be used to represent entity boundaries and entity types. Entity types include person names, organization names, location names, and other types of entities. The label set includes non-entity position O, other type entity start B-MIS, other type entity inside I-MIS, person name start B-PER, person name inside I-PER, organization name start B-ORG, organization name inside I-ORG, location name start B-LOC, location name inside I-LOC, and non-first sub-word positions or special positions X generated by sub-word segmentation.

[0048] For the text sequence, the RoBERTa tokenizer is used for sub-word segmentation. If a word is segmented into multiple sub-words, the first sub-word inherits the entity label of the original word, and the remaining sub-words are marked with X. Subsequently, the start and end tags required by RoBERTa are added to the beginning and end of the text sequence, and the text is truncated or padded according to the preset maximum text length to generate the input token number, attention mask, and label sequence.

[0049] Input the token number and attention mask into the RoBERTa encoder to obtain the context text representation for each token in the text sequence. Let the length of the text sequence be... The token-level text representation output by RoBERTa is denoted as ,in, , … … They represent the 1st, 2nd, ..., … The context text representation of each token.

[0050] Candidate entity fragments can be obtained from the real named entity label sequence during the training phase; during the prediction phase, the predicted label sequence can be obtained first based on the first round of sequence labeling results of the model, and then candidate entity fragments can be constructed based on the predicted label sequence. Each candidate entity fragment includes its sample number, start position, end position, and entity type.

[0051] For a candidate entity fragment, let its starting position be... The end position is Then the entity-level text representation of the current candidate entity fragment can be represented by the fragment's starting position token. End position token indicates And the token representation within the fragment is generated together with the mean: ; in, The entity-level text representation of the current candidate entity. This represents vector concatenation. This represents a learnable mapping matrix. This entity-level text representation simultaneously contains entity boundary information and intra-fragment semantic information, used for subsequent entity conditional matching of local image regions.

[0052] For image input, each sample is pre-defined to contain a maximum of M images. If the number of truly associated images is less than M, padding is used to fill in the image positions; if the number of truly associated images is greater than M, the first M images are used. For each truly valid image, the ViT image processor is used to convert it into a pixel tensor of uniform size.

[0053] Simultaneously, valid image markers are constructed, with the real image location marked as 1 and the padded image location marked as 0. After inputting multiple images into the ViT encoder, a global representation and a local region marker representation for each image are obtained. The global representation can be used to obtain an image-level or sample-level global visual representation, while the local region marker representation serves as the visual representation of the local image region. A region mask is generated based on the valid image markers, ensuring that the local image region corresponding to the padded image is not considered a valid visual region in subsequent calculations.

[0054] Through step S110, we can obtain the candidate entity fragments corresponding to the current input sample, the entity-level text representation of the candidate entity fragments, the visual representation of local regions of multiple images, and the effective image labels.

[0055] Step S120: Construct a set of candidate visual regions for entity conditions across the graph and determine the initial visual support weights.

[0056] In step S120, for each candidate entity fragment, the visual representations of the local regions of all valid images in the same sample are constructed into a set of cross-image candidate visual regions for entity conditions. Entity condition matching is performed based on the entity-level text representation of the candidate entity fragment to obtain the initial visual support score and initial visual support weight of each visual region relative to the current candidate entity fragment.

[0057] For a candidate entity fragment, the visual representations of local image regions from all valid images in the same sample are aggregated into a cross-image visual instance set. Since the visual instances in this set come from multiple valid images of the same sample, it can cover local visual cues scattered across different images in multi-image scenes.

[0058] Specifically, the local region representation of the image output by ViT is first projected onto the same latent space as the RoBERTa text representation through a linear mapping layer. Then, entity condition matching is performed on each visual instance in the cross-graph visual instance set, using the entity-level text representation of the current candidate entity fragment as a condition.

[0059] For the entity-level text representation of the current candidate entity fragment and the Each visual region represents Entity conditional matching features can be constructed and initial visual support scores can be calculated: ; in, This represents entity condition matching features. Represents element-wise product. This indicates a learnable scoring network. Indicates the first The initial visual support score of each visual region relative to the current candidate entity fragment.

[0060] When calculating the initial visual support score, a region mask generated using valid image labels is used to mask and fill in the corresponding invalid visual regions of the image, preventing them from participating in subsequent weight allocation and the formation of the candidate visual region set. Let... Indicates the first Whether a visual region is valid, if the visual region comes from a real and valid image, then ,otherwise The initial visual support weights can then be expressed as: ; in, Indicates the first The initial visual support weights of each visual region relative to the current candidate entity fragment. Indicates the first The initial visual support score of each visual region relative to the current candidate entity fragment. Indicates the first Is each visual region effective?

[0061] Within the cross-graph visual instance set corresponding to all valid images of the same sample, the initial visual support score is uniformly weighted across the graph to obtain the initial visual support weight, forming the entity condition cross-graph candidate visual region set corresponding to the current candidate entity fragment. The candidate visual region set retains the corresponding weight, visual region representation, validity mask, and its index in the cross-graph visual instance set.

[0062] As can be seen from step S120, this invention does not simply average multiple images, nor does it uniformly fuse the global image representation. Instead, it forms a set of cross-image candidate visual regions for entity conditions from the local region sets of multiple valid images, focusing on different candidate entity fragments. Therefore, the same visual region can generate different initial visual support weights for different candidate entity fragments, thereby enhancing the entity-specificity of visual region selection.

[0063] Step S130: Perform entity condition multi-instance supervision and non-entity negative sample constraints.

[0064] In step S130, during the training phase, under the condition of lacking corresponding annotations between entities and local regions of the image, entity conditional multi-instance supervision is performed based on the aggregated representation of the candidate visual region set and the entity type label of the candidate entity fragment. The correspondence between the candidate visual region and the current candidate entity fragment type is learned. At the same time, non-entity negative sample constraints are introduced to reduce mismatches between non-entity text fragments and irrelevant visual regions.

[0065] Since multi-image, multimodal named entity recognition data typically only provides entity boundaries and entity type labels in the text, without providing corresponding annotations between entities and local image regions, the model cannot directly determine which local image region a given entity corresponds to. Therefore, this invention models the candidate visual region set as an entity-conditional multi-instance package, where candidate visual regions are instances within the package, and the entity type labels of candidate entity fragments serve as package-level supervision signals to constrain the overall representation of the candidate visual region set.

[0066] Specifically, based on the initial visual support weights obtained in step S120, the set of candidate visual regions corresponding to the current candidate entity fragment is weighted and aggregated to obtain an entity-level visual representation. Then, this entity-level visual representation is fused with the entity-level text representation of the current candidate entity fragment to obtain a joint entity-visual representation, and the entity type of the current candidate entity fragment is predicted based on this joint entity-visual representation.

[0067] During the training phase, the entity type labels of the current candidate entity fragments are used as bag-level supervision signals to apply a classification loss to the entity type prediction results. Entity-conditional multi-instance supervision here means that the supervision signal does not require labeling each local region of the image to determine whether it is a real and valid region, but rather constrains the overall representation of the aggregated set of candidate visual regions to support the type determination of the current candidate entity fragment. In this way, the model can learn the correspondence between latent visual regions and candidate entity fragment types even in the absence of region-level annotations.

[0068] This embodiment introduces a non-entity negative sample constraint. During the training phase, in addition to constructing positive entity fragments based on entity labels, O-class negative samples are also constructed from non-entity locations labeled O in the text sequence. O-class negative samples can be a single non-entity token or short fragments formed by non-entity locations. For O-class negative samples, their text representations are also generated, and visual regions are aggregated in the effective image local region set in the same way as candidate entity fragments to obtain the corresponding entity-visual joint representation. Then, the O-class labels are used to supervise this joint representation, enabling the model to learn that non-entity text fragments should not be forcibly matched as effective visual regions.

[0069] Through step S130, the model can not only learn the potential visual regions corresponding to entities, but also learn to reject erroneous associations between non-entity text fragments and irrelevant visual regions, thereby reducing the interference of background regions, redundant images, and irrelevant local regions of images on entity boundary recognition and entity type judgment.

[0070] Step S140: Perform visual contribution calibration and generate a visual contribution calibration score.

[0071] In step S140, visual contribution calibration is performed on the candidate visual region set. Visual contribution calibration includes four sub-steps: obtaining the baseline prediction result, performing single-region removal verification, performing cross-entity negative region replacement verification, and performing contribution-guided visual reconstruction. It is used to evaluate the degree of support of the candidate visual regions for the current candidate entity segment type judgment from three aspects: the actual contribution of the candidate visual regions, cross-entity distinguishability, and prediction preservation ability after reconstruction, and to generate a visual contribution calibration score.

[0072] The candidate visual region set obtained in step S120 is mainly formed based on the initial visual support weights. The initial visual support weights reflect the initial matching degree between the candidate visual region and the current candidate entity fragment; however, a high initial matching degree does not necessarily mean that the candidate visual region truly supports the type judgment of the current candidate entity fragment. For example, some background regions may be related to the text topic but do not directly support the entity type of the current candidate entity fragment; some visual regions may be more relevant to other candidate entity fragments but are incorrectly assigned to the current candidate entity fragment. Therefore, step S140 further verifies and corrects the actual role of the candidate visual regions through visual contribution calibration.

[0073] Obtaining the baseline prediction result: For the current candidate entity fragment, the initial visual support weights are first redistributed within its corresponding candidate visual region set, ensuring that visual regions within the candidate set participate in the comparison within the same range. Then, the candidate visual regions are aggregated based on the redistributed weights to obtain a complete visual region representation. This complete visual region representation is fused with the entity-level text representation of the current candidate entity fragment and input into the entity type prediction layer to obtain the entity type prediction result corresponding to the complete candidate visual region set. This prediction result is used as the baseline prediction result.

[0074] During the training phase, the target type of the current candidate entity fragment is taken from the true entity type label of that candidate entity fragment; during the prediction phase, the target type of the current candidate entity fragment is taken from the entity type with the highest prediction score in the benchmark prediction results. In this way, the training phase can use the true label to constrain the visual contribution calibration process, and the prediction phase can use the prediction results formed by the complete candidate visual region set as a reference for subsequent perturbation verification when the true label is missing.

[0075] Perform single-region removal verification: Single-region removal verification is used to measure the actual contribution of each candidate visual region to the prediction of the target type of the current candidate entity fragment. For each candidate visual region in the candidate visual region set, it is removed from the candidate visual region set in turn, and the weights are reassigned and the visual regions are aggregated based on the remaining candidate visual regions to obtain the corresponding single-region removal prediction result.

[0076] The single-region removal prediction result is compared with the baseline prediction result, and the difference in prediction scores between the two for the current candidate entity fragment target type is calculated to obtain the contribution of the removed candidate visual region. The contribution can be expressed as: ; in, Indicates the first The contribution of each candidate visual region This represents the prediction score for the target type in the baseline prediction results. Indicates the removal of the first The predicted score for the target type after each candidate visual region.

[0077] when When the value is large, it indicates that removing the candidate visual region significantly reduces the current target type score, and the candidate visual region provides strong support for the current candidate entity fragment type judgment; when When the value is small or negative, it indicates that the candidate visual region contributes weakly to the current candidate entity fragment type judgment, and may even introduce interference information.

[0078] Perform cross-entity negative region replacement verification: Cross-entity negative region replacement verification is used to measure the discriminative power of candidate visual regions relative to negative visual regions. Single region removal verification can determine whether a candidate visual region contributes to the current prediction result, but it cannot fully determine whether the candidate visual region specifically supports the current candidate entity fragment, or is only weakly correlated with the sample subject, background, or other entities. Therefore, this embodiment further introduces cross-entity negative region replacement verification.

[0079] Specifically, for the candidate visual regions of the current candidate entity fragment, negative visual regions are selected from the candidate visual regions corresponding to other candidate entity fragments that are different from the current candidate entity fragment. During the training phase, negative visual regions are preferentially selected from the candidate visual regions corresponding to other candidate entity fragments with different entity type labels; during the prediction phase, negative visual regions are preferentially selected from the candidate visual regions corresponding to other candidate entity fragments with different predicted entity types. When there are no negative visual regions that meet the conditions in the same sample, negative visual regions can be selected from other sample candidate entity fragments in the same training batch, the negative region cache pool, or random valid visual regions that do not belong to the candidate set of the current candidate entity fragment.

[0080] At least one candidate visual region in the candidate visual region set corresponding to the current candidate entity fragment is replaced with a negative visual region to obtain a negative region replacement set. A negative region replacement prediction result is then obtained based on this set. The negative region replacement prediction result is compared with the baseline prediction result, and the difference in prediction scores between the two for the target type of the current candidate entity fragment is calculated to obtain a discriminative measure of the candidate visual region. The discriminative measure can be expressed as: ; in, A measure of the discriminative power of candidate visual regions. This represents the predicted score for the target type after the negative region is replaced.

[0081] when When the value is large, it indicates that after replacing the current candidate visual region with the negative visual region, the target type prediction score of the current candidate entity segment drops significantly, indicating that the current candidate visual region has a stronger discriminative advantage compared to the negative visual region; when When the value is small, it indicates that the discrimination difference between the current candidate visual region and the negative visual region is weak, which may indicate cross-entity mismatch or spurious correlation problems.

[0082] Contribution-guided visual reconstruction is performed to verify whether the visual representation formed by high-contribution and discriminative candidate visual regions can maintain the type judgment of the current candidate entity fragment. Specifically, based on the contribution obtained from single-region removal verification and the discriminability obtained from cross-entity negative region replacement verification, the reconstruction support weight of candidate visual regions is determined, so that high-contribution and discriminative candidate visual regions receive higher support during the reconstruction process, while low-contribution or weakly discriminative candidate visual regions are suppressed.

[0083] Based on the reconstruction support weights, a contribution-guided visual reconstruction is performed on the candidate visual region set to obtain a contribution-guided visual representation. This contribution-guided visual representation is then used to predict the entity type of the current candidate entity fragment, yielding the reconstructed prediction result. The support maintenance capability is determined by comparing the degree to which the reconstructed prediction result and the baseline prediction result preserve the prediction score for the target type of the current candidate entity fragment.

[0084] If the reconstructed prediction result can maintain the target type prediction score well, it indicates that the reconstructed visual representation formed by the candidate visual regions with high contribution and discriminativeness can maintain the type judgment of the current candidate entity fragment, and the support maintenance ability is strong; if the target type prediction score drops significantly after reconstruction, it indicates that the candidate visual region set may still rely on low contribution regions, weak discriminative regions or other interference regions, and the support maintenance ability is weak.

[0085] By combining the initial visual support weights, contribution, discriminative power, and support maintenance ability, a visual contribution calibration score is generated for each candidate visual region. For the , The visual contribution calibration score of each candidate visual region can be expressed as: ; in, Indicates the first Visual contribution calibration score of each candidate visual region This indicates a learnable calibration scoring network. This represents the Sigmoid function. This indicates support for maintaining the capability.

[0086] Through step S140, the present invention no longer utilizes visual regions solely based on initial visual support weights, but further combines the actual contribution of candidate visual regions to the current candidate entity fragment type judgment, their discriminability relative to negative visual regions, and the prediction preservation ability after contribution-guided reconstruction to generate a visual contribution calibration score, thereby improving the reliability of visual region selection and utilization in multi-image scenarios.

[0087] Step S150: Fuse the visual contribution calibration score with the initial visual support weights and generate a calibrated cross-graph entity-level visual representation.

[0088] In step S150, the visual contribution calibration score is fused with the initial visual support weights corresponding to the candidate visual region set, and the candidate visual region set is subjected to secondary weighted aggregation to obtain the calibrated cross-graph entity-level visual representation.

[0089] Specifically, the visual contribution calibration score obtained in step S140 can be fused with the initial visual support weight obtained in step S120 to obtain the calibration aggregate weight. The two can be multiplied and then normalized to obtain the calibration aggregate weight. Based on the calibration aggregate weight, a second weighted aggregation is performed on the candidate visual region set to obtain the calibrated cross-graph entity-level visual representation. ; in, Indicates the first Calibration aggregation weights for candidate visual regions Indicates the first The initial visual support weights of each visual region relative to the current candidate entity fragment. Indicates the first Visual contribution calibration score of each candidate visual region This represents the calibrated cross-graph entity-level visual representation.

[0090] Compared to entity-level visual representations obtained using only initial visual support weights, the calibrated cross-graph entity-level visual representation considers both the initial correlation between candidate visual regions and current candidate entity segments, as well as the contribution calibration results obtained after visual contribution calibration of candidate visual regions. Therefore, this representation can reduce the interference of spurious local image regions caused by background co-occurrence, image redundancy, subject similarity, and cross-entity mismatches on entity recognition.

[0091] Step S160: Apply the calibrated cross-graph entity-level visual representation orientation to the entity fragment and output the recognition result.

[0092] In step S160, the calibrated cross-graph entity-level visual representation is applied to the text representation of the corresponding candidate entity fragment, and a multi-graph multimodal named entity recognition result is generated based on the updated text representation.

[0093] Specifically, the calibrated cross-graph entity-level visual representation obtained in step S150 is mapped to the text feature space to obtain the entity-level visual update representation. A gating vector is calculated based on the entity-level text representation of the current candidate entity fragment and the calibrated cross-graph entity-level visual representation. This gating vector is used to control the injection strength of the calibrated visual representation onto the text representation.

[0094] Based on the start and end positions of the current candidate entity fragments in the text sequence, the fragment regions that need to be updated are determined, and the entity-level visual update representation is injected into the token representation within that fragment region. If multiple candidate entity fragments update the same token position, the multiple update results can be averaged or processed using other fusion methods.

[0095] In one possible implementation, the model first performs a first round of named entity recognition prediction based on the preliminary fused text representation to obtain candidate entity fragments. Then, it performs entity conditional candidate visual region discovery and visual contribution calibration on these predicted fragments, and writes the calibrated visual representation back to the corresponding fragment region. After fragment writing back, the updated text representation is input into a linear classification layer to obtain the label emission score for each token, and then decoded using a conditional random field to output the final named entity recognition label sequence. Based on the label sequence, the entity boundaries and entity types in the text can be determined.

[0096] During the training phase, the model's total loss can include the named entity recognition main loss, entity-conditional multi-instance supervision loss, and visual contribution calibration loss. The named entity recognition main loss can be obtained from the conditional random field negative log-likelihood loss; the entity-conditional multi-instance supervision loss is used to constrain the entity type prediction based on the aggregated representation of candidate visual regions; and the visual contribution calibration loss is used to constrain the baseline prediction results, negative region replacement, and contribution-guided visual reconstruction process. By jointly optimizing these losses, the model can simultaneously learn text sequence annotation capabilities, entity-conditional visual region learning capabilities, and candidate visual region contribution calibration capabilities.

[0097] Through the above steps, this invention can construct entity-conditional cross-graph candidate visual region sets for different candidate entity fragments in multi-graph scenarios. Without region-level annotation, it uses entity-conditional multi-instance supervised learning to learn the correspondence between candidate visual regions and candidate entity fragment types. Then, visual contribution calibration is used to determine whether the candidate visual regions truly support the type judgment of the current candidate entity fragment. Compared to methods that rely solely on attention weights or similarity scores for visual fusion, this invention can further reduce interference from pseudo-related visual regions caused by background co-occurrence, image redundancy, topic similarity, and cross-entity mismatches, thereby improving the accuracy and reliability of multi-graph multimodal named entity recognition results.

[0098] This invention provides a multimodal named entity recognition system for multi-graph scenarios, comprising: The candidate visual region construction module is used to construct a set of cross-image candidate visual regions for entities from the local region visual representations of multiple valid images of the same multi-image sample, based on the entity-level text representation of the candidate entity fragments, and to determine the initial visual support weight of each candidate visual region in the set of cross-image candidate visual regions for entities. The entity conditional multi-instance supervision module is used to perform entity conditional multi-instance supervision on the set of candidate visual regions across the graph based on the entity type label of the candidate entity fragment, and to suppress mismatches between non-entity fragments and irrelevant visual regions through non-entity negative sample constraints. The visual contribution calibration module is used to perturb the entity type prediction results corresponding to the complete entity cross-graph candidate visual region set as the benchmark prediction results, and determine the contribution and discriminativeness of each candidate visual region based on the change in the target type prediction score of the candidate entity fragments before and after the perturbation; based on the contribution and discriminativeness of each candidate visual region, the module performs contribution-guided visual reconstruction of the entity cross-graph candidate visual region set, and determines the support maintenance capability of each candidate visual region based on the degree of preservation of the target type prediction score of the candidate entity fragments after reconstruction. The calibration fusion module is used to generate a visual contribution calibration score based on the initial visual support weight, contribution, discriminativeness and support maintenance capability of each candidate visual region, and to perform calibration aggregation on the entity cross-graph candidate visual region set based on the visual contribution calibration score to obtain the calibrated cross-graph entity-level visual representation. The entity fragment update and recognition module is used to apply the calibrated cross-graph entity-level visual representation to the text representation of candidate entity fragments to obtain multimodal named entity recognition results.

[0099] The multimodal named entity recognition system for multi-graph scenarios provided in this embodiment of the invention can execute the multimodal named entity recognition method for multi-graph scenarios provided in this embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0100] In this embodiment, the candidate visual region construction module uses the entity-level text representation of candidate entity fragments as a condition to perform cross-image comparison of visual regions in multiple valid images, enabling different candidate entity fragments to form their own corresponding visual region distributions. The entity condition multi-instance supervision module suppresses mismatches between irrelevant visual regions and non-entity text fragments through non-entity negative sample constraints. The visual contribution calibration module determines whether the candidate visual region truly supports the type judgment of the current candidate entity fragment by comparing the changes in type judgment before and after visual region processing. For the calibration fusion module, the initial visual support weight is used to represent the initial matching degree between the candidate visual region and the current candidate entity fragment, and the visual contribution calibration score is used to represent the actual support degree of the candidate visual region for the type judgment of the current candidate entity fragment. By fusing the two, the influence of pseudo-related visual regions caused by background co-occurrence, image redundancy, and cross-entity mismatches can be weakened, making the aggregated visual representation more focused on the candidate visual regions that truly support the type judgment of the current candidate entity fragment. Unlike the unified visual fusion of the entire text, the entity fragment update and recognition module only performs visual enhancement on the candidate entity fragment regions, thereby avoiding irrelevant visual information from affecting non-entity locations or other entity fragments.

[0101] The multimodal named entity recognition system for multi-image scenarios provided in this invention can reduce the interference of background co-occurrence, image redundancy, and pseudo-correlated local regions of images on entity recognition in multi-image scenarios, thereby improving the accuracy and reliability of multimodal named entity recognition results.

[0102] This invention provides a computer device, comprising: Storage medium: used to store computer programs; Processor: Used to execute computer programs to implement the multimodal named entity recognition method for multi-graph scenarios provided in the embodiments of the present invention.

[0103] This invention provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the multimodal named entity recognition method for multi-graph scenarios provided in this invention.

[0104] This invention provides a computer program product, including a computer program that, when executed by a processor, implements the multimodal named entity recognition method for multi-graph scenarios provided in this invention.

[0105] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0107] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0108] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0109] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal named entity recognition method for multi-graph scenarios, characterized in that, include: A set of cross-image candidate visual regions for entities is constructed from the local region visual representations of multiple valid images of the same multi-image sample, conditioned on the entity-level text representation of candidate entity fragments, and the initial visual support weight of each candidate visual region in the set of cross-image candidate visual regions for entities is determined. Based on the entity type labels of candidate entity fragments, entity conditional multi-instance supervision is performed on the entity cross-graph candidate visual region set, and the mismatch between non-entity fragments and irrelevant visual regions is suppressed by non-entity negative sample constraints. The entity type prediction results corresponding to the complete entity cross-image candidate visual region set are used as the benchmark prediction results to perturb the entity cross-image candidate visual region set, and the contribution and discriminativeness of each candidate visual region are determined according to the change in the target type prediction score of the candidate entity fragments before and after the perturbation. Visual reconstruction guided by contribution of each candidate visual region set across the graph is performed based on the contribution and discriminativeness of each candidate visual region, and the support maintenance capability of each candidate visual region is determined according to the degree of retention of the target type prediction score of the candidate entity fragment after reconstruction. A visual contribution calibration score is generated based on the initial visual support weight, contribution, discriminativeness, and support maintenance capability of each candidate visual region. The entity cross-graph candidate visual region set is then calibrated and aggregated based on the visual contribution calibration score to obtain the calibrated cross-graph entity-level visual representation. The calibrated cross-graph entity-level visual representation is applied to the text representation of candidate entity fragments to obtain multimodal named entity recognition results.

2. The multimodal named entity recognition method for multi-graph scenarios according to claim 1, characterized in that, A set of cross-image candidate visual regions for entities is constructed from the local region visual representations of multiple valid images from the same multi-image sample, conditioned on the entity-level text representation of candidate entity fragments. The initial visual support weights for each candidate visual region in the set of cross-image candidate visual regions are determined, including: Based on the valid image labels used to distinguish between valid images and completed images, the visual representations of local regions corresponding to completed images are excluded, and the visual representations of local regions of multiple valid images in the same multi-image sample are aggregated into a cross-image visual instance set. The matching calculation is performed on each visual instance in the cross-graph visual instance set based on the entity-level text representation of the candidate entity fragment, so as to obtain the initial visual support score of each visual instance relative to the candidate entity fragment. A unified weight allocation is performed across the graph for the initial visual support scores of each visual instance relative to the candidate entity fragments to obtain the initial visual support weight of each visual instance. Visual instances with non-zero initial visual support weights are used as candidate visual regions to form a set of cross-graph candidate visual regions for the entity fragments. The initial visual support weight of the visual instance is used as the initial visual support weight of the candidate visual region. In this context, different candidate entity fragments within the same multi-graph sample share a cross-graph visual instance set, and each has its own corresponding cross-graph candidate visual region set.

3. The multimodal named entity recognition method for multi-graph scenarios according to claim 1, characterized in that, Entity-conditional multi-instance supervision of entity cross-graph candidate visual region sets based on entity type labels of candidate entity fragments includes: The set of cross-graph candidate visual regions corresponding to the candidate entity fragment is taken as a multi-instance package, and each candidate visual region in the set of cross-graph candidate visual regions is taken as an instance in the multi-instance package. Based on the initial visual support weights of each candidate visual region in the entity cross-graph candidate visual region set, the instances in the multi-instance package are weighted and aggregated to generate the visual region aggregated representation of the candidate entity fragment, and then fused with the entity-level text representation of the candidate entity fragment to obtain the entity visual joint representation of the candidate entity fragment. The entity visual joint representation of candidate entity fragments is used to predict the entity type of the candidate entity fragments. The entity type label of the candidate entity fragments is used as a bag-level supervision signal to apply a classification loss to the entity type prediction result of the candidate entity fragments, so that the overall representation of the multi-instance bag learns the correspondence with the entity type label of the candidate entity fragments.

4. The multimodal named entity recognition method for multi-graph scenarios according to claim 1, characterized in that, Suppressing mismatches between non-entity fragments and irrelevant visual regions through non-entity negative sample constraints includes: Construct O-class negative samples from non-entity locations in a text sequence and generate text representations of the O-class negative samples; Using the text representation of the negative O samples as a condition, a set of candidate visual regions for the O-class negative samples is constructed from the local region visual representations of multiple valid images of the same multi-image sample, and the initial visual support weights of each candidate visual region in the set of candidate visual regions for the O-class negative samples are determined. Based on the initial visual support weights of each candidate visual region in the candidate visual region set of O-class negative samples, the candidate visual regions in the O-class negative sample candidate visual region set are weighted and aggregated to generate the visual region aggregated representation of O-class negative samples. This is then fused with the text representation of O-class negative samples to obtain the entity visual joint representation of O-class negative samples. We use O-class labels to supervise the joint entity visual representation of O-class negative samples, thereby suppressing mismatches between non-entity segments and irrelevant visual regions.

5. The multimodal named entity recognition method for multi-graph scenarios according to claim 1, characterized in that, Using the entity type prediction results corresponding to the complete set of cross-image candidate visual regions as the baseline prediction results, visual region perturbation is applied to the set of cross-image candidate visual regions. The contribution and discriminative power of each candidate visual region are determined based on the changes in the target type prediction scores of candidate entity fragments before and after the perturbation, including: Within the scope of the entity cross-graph candidate visual region set, the initial visual support weights of each candidate visual region are reallocated and normalized to obtain a complete visual region representation. The baseline prediction result is obtained by combining the entity-level text representation of the candidate entity fragments. Determine the target type of the candidate entity fragment. If the candidate entity fragment has a true entity type label, then the true entity type label is taken as the target type; otherwise, the entity type with the highest score in the baseline prediction results is taken as the target type. Using the baseline prediction results as a reference, the visual regions of the candidate visual regions across the graph are perturbed, including single-region removal verification and cross-entity negative region replacement verification. The contribution and discriminativeness of each candidate visual region are determined based on the change in the target type prediction score of the candidate entity fragments before and after the perturbation.

6. The multimodal named entity recognition method for multi-graph scenarios according to claim 5, characterized in that, Single-area removal verification includes: One candidate visual region is removed sequentially from the entity cross-graph candidate visual region set to obtain multiple candidate visual region sets after single region removal; For each single region removal, the remaining candidate visual regions in the candidate visual region set are reassigned weights, and entity type prediction is performed based on the reassigned remaining candidate visual regions to obtain the single region removal prediction result. The difference between the baseline prediction result and the single-region removal prediction result in the prediction score of the target type of the candidate entity fragment is calculated to obtain the contribution of the removed candidate visual region. This contribution is used to characterize the actual support of the removed candidate visual region for the entity type prediction of the candidate entity fragment, and the contribution of the removed candidate visual region is used as the basis for contribution-guided visual reconstruction.

7. The multimodal named entity recognition method for multi-graph scenarios according to claim 5, characterized in that, Cross-entity negative region replacement verification includes: Negative visual regions are selected from the set of cross-graph candidate visual regions corresponding to other candidate entity segments that are different from the current candidate entity segment. When there are no negative visual regions that meet the conditions in the same multi-graph sample, negative visual regions are selected from the candidate entity segments, negative region cache pool, or random valid visual regions of the set of cross-graph candidate visual regions that do not belong to the current candidate entity segment in other multi-graph samples in the same training batch. Replace at least one candidate visual region in the set of cross-image candidate visual regions corresponding to the current candidate entity fragment with a negative visual region to obtain a negative region replacement set, and obtain the negative region replacement prediction result based on the negative region replacement set; The difference between the baseline prediction result and the negative region substitution prediction result in the target type of the candidate entity fragment is calculated to obtain the negative region substitution advantage, which is used to characterize the prediction advantage of the candidate entity fragment's own visual region relative to the negative visual region. The negative region substitution advantage is used as a measure of the discriminativeness of the candidate visual region as the basis for contribution-guided visual reconstruction.

8. The multimodal named entity recognition method for multi-graph scenarios according to claim 1, characterized in that, Visual reconstruction guided by the contribution and discriminative power of each candidate visual region is performed on the entity cross-graph candidate visual region set. The support maintenance capability of each candidate visual region is determined based on the degree of preservation of the target type prediction score of the reconstructed candidate entity fragments, including: The reconstruction support weights are determined based on the contribution and discriminative power of each candidate visual region. Based on the reconstruction support weights, a contribution-guided visual reconstruction is performed on the entity cross-graph candidate visual region set to obtain a contribution-guided visual representation. The support maintenance capability of each candidate visual region is determined by the degree to which the target type prediction score of the candidate entity fragment is preserved based on the reconstruction prediction results corresponding to the contribution-guided visual representation.

9. The multimodal named entity recognition method for multi-graph scenarios according to claim 1, characterized in that, Based on the visual contribution calibration scores of each candidate visual region, the entity cross-graph candidate visual region set is calibrated and aggregated to obtain the calibrated cross-graph entity-level visual representation, including: The calibration weight is obtained by fusing the visual contribution calibration score of each candidate visual region with the initial visual support weight of each candidate visual region. The entity cross-graph candidate visual region set is weighted and aggregated according to the calibration aggregation weight to obtain the calibrated cross-graph entity-level visual representation.

10. The multimodal named entity recognition method for multi-graph scenarios according to claim 1, characterized in that, The calibrated cross-graph entity-level visual representation is applied to the text representation of candidate entity fragments to obtain multimodal named entity recognition results, including: The calibrated cross-graph entity-level visual representation is mapped to the text feature space to obtain the entity-level visual update representation; Based on the position of the candidate entity fragment in the text sequence, the entity-level visual update representation is injected into the text representation of the corresponding region of the candidate entity fragment, and the updated text representation is used for named entity label prediction to obtain the multimodal named entity recognition result.

Images