A multi-modal knowledge enhanced cross-document event coreference resolution method, system, computer and storage medium

Abstract

The application discloses a kind of multi-modal knowledge enhanced cross-document event co-reference resolution method, system, computer and storage medium, through big language model based on abstract-type-entity three-layer prompt template accurate extraction event information in document, structured multi-modal event graph including text and visual node is constructed, for original image, semantic alignment is carried out using multi-modal big model, and no image event is generated using stable diffusion model Synthesis image and easy-to-confuse negative sample;Text encoder and visual encoder are used to vectorize the representation of event text content and image respectively, and multidimensional co-reference relationship between cross-document events is established;Based on semantic distance calculation and contrast learning optimization, cross-document event co-reference resolution is realized, and the same event dispersed in different documents is uniformly associated.The application breaks through the limitations of traditional pure text event co-reference resolution in the aspects of insufficient multi-modal knowledge utilization and rough semantic association modeling.

Description

Technical Field

[0001] This invention relates to computer multimodal processing, specifically to a method, system, computer, and storage medium for cross-document event coreference resolution with multimodal knowledge enhancement. Background Technology

[0002] Cross-document event coreference resolution is an important and challenging task in the field of information extraction. Its core objective is to address the problem of heterogeneous representations of the same event in different documents, achieving semantic unity of events across documents. This task aims to identify different representations of the same real-world event in multiple documents and associate these scattered events into a unified event. Unlike intra-document event coreference resolution, cross-document event coreference resolution requires models to have stronger contextual understanding and knowledge generalization capabilities.

[0003] This task faces two core challenges: First, descriptions of similar events often have high similarity, and the model needs to identify subtle differences to distinguish different events; second, the same event may have different emphases in different sources, and deep semantic evidence is needed to determine the core reference relationship.

[0004] Current cross-document event coreference resolution techniques primarily focus on in-document event coreference resolution and are mostly limited to pure text processing. While some research has attempted to incorporate multimodal information such as images to supplement textual evidence—for example, proposing multimodal neural networks that fuse text, images, and event graphs, or employing visual-linguistic linear mapping combined with hybrid modality integration—these existing methods generally rely on post-fusion mechanisms and lack effective modeling of structured cross-document cross-modal relationships. Furthermore, although contrastive learning has been used to improve entity and event representations, existing frameworks still struggle to achieve joint optimization of cross-modal and cross-document alignment mechanisms within a unified graph-based modeling framework. Therefore, in real-world multi-source datasets where text and visual semantic mismatches or complete image loss are common, existing methods still have significant limitations in effectively resolving event coreference. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal knowledge-enhanced cross-document event coreference resolution method, system, computer, and storage medium that overcomes the limitations of traditional text event coreference methods, such as difficulty in effectively utilizing visual information, inability to align text and image events, and difficulty in handling documents without images. This enables comprehensive, accurate, and efficient multimodal event recognition and unified cross-document association.

[0006] The technical solution to achieve the purpose of this invention is: a cross-document event coreference resolution method with multimodal knowledge enhancement, comprising the following steps:

[0007] Step 1: Extract structured event information from the document. For each document, using a large language model and based on a three-layer prompt template of summary-type-entity, extract the structured event information within the document. The structured event information includes event type, event trigger words, event description, and at least one event element and its corresponding event entity.

[0008] Step 2: Constructing a Multimodal Event Graph within the Document: For each event extracted in Step 1, its associated visual information is used to construct a multimodal event graph. If the document contains original images that semantically match the event description, a multimodal large model is used to semantically align the event description with the original image, and the matched original image is used as the positive sample image for that event. For events without matching original images, a stable diffusion model is used to generate synthetic images based on the event description, which are then used as positive sample images. Furthermore, the event description is modified using a cyclic shift strategy consistent with entity type, and a stable diffusion model is used to generate easily confused negative sample images based on the modified event description. The structured information of the event and its associated positive and negative sample images together constitute the nodes of the multimodal event graph for that event, and all nodes constitute the multimodal event graph of the document.

[0009] Step 3: Implement cross-document event coreference resolution: A text encoder is used to vectorize event trigger words, event elements, corresponding event entities, and event descriptions in the multimodal event graph to obtain corresponding text representations. A visual encoder is used to vectorize positive sample images and easily confused negative sample images in the multimodal event graph to obtain corresponding visual representations. For any two multimodal event graphs from different documents, semantic distances are calculated based on the obtained representations in the dimensions of event trigger words, event elements, and event image-text, respectively, to model the multidimensional coreference relationship between the two events. An optimal transfer method is used to calculate the global semantic distance between the two multimodal event graphs based on the multidimensional semantic distance. Simultaneously, a contrastive learning loss function is used to optimize the similarity between the event description representation and its positive sample image representation. By jointly optimizing the global semantic distance and the contrastive learning loss, the cross-document event with the closest global semantic distance is determined as the coreference, achieving unified association of cross-document events.

[0010] Furthermore, in step one, the three-tiered prompt template of summary-type-entity is as follows:

[0011] The first level of prompts is used to extract event trigger words and their context from the document and generate corresponding event summaries;

[0012] The second layer of prompts is used to cluster all event summaries generated in the first layer, summarize the event types of each event cluster, and select representative event trigger words;

[0013] The third layer of hints is used to extract event elements and corresponding event entities from the context of each event cluster, and generate a complete event description for that event cluster.

[0014] Furthermore, in step two, the cyclic shift strategy with consistent entity types is as follows: when generating easily confused negative samples, the event elements remain unchanged, and only the event entities with the same entity type are cyclically shifted to adjust their order, thereby forming an adjusted event description, and generating negative sample images accordingly.

[0015] Furthermore, in step three, the multi-dimensional semantic distance between any two events in the two multimodal event graphs includes:

[0016] Semantic distance of event trigger words The calculation method is as follows:

[0017] ;

[0018] in, Represents the cosine similarity function. and These represent the trigger words for two events. and These represent the trigger words for two events, with subscripts indicating their respective triggers. , An index of events representing different documents;

[0019] Semantic distance of event elements The calculation method is as follows:

[0020] ;

[0021] in, and These represent the event elements and corresponding event entities of two events, respectively. and These represent the event element representation and the event entity representation of the two events, respectively.

[0022] Event image-text semantic distance The calculation method is as follows:

[0023] ;

[0024] in, and These represent the event descriptions and corresponding event images for the two events, respectively. and These represent the event description representation and the event image representation of the two events, respectively.

[0025] Furthermore, in step three, the optimal transmission method is used to calculate the two multimodal event graphs. and Global semantic distance between Specifically:

[0026] ;

[0027] Wherein, the symbol ⊙ represents the Hadamard product. Represents the transmission matrix. and These are multimodal event graphs. and The number of nodes, The cost matrix consists of the cost matrix transmitted between event trigger words. Cost matrix of transmission between event elements Cost matrix for transmission between event images and text The cost submatrix is ​​composed of elements representing the semantic distances between the corresponding nodes.

[0028] Furthermore, in step three, joint optimization is achieved by minimizing the overall optimization function. Implementation, represented as:

[0029] ;

[0030] ;

[0031] ;

[0032] in, This represents the global semantic distance optimization function. This represents the contrastive learning optimization function. and They represent the first Event description representation and corresponding event image representation for each event. This represents the total number of event images and negative samples of the event images. This represents a temperature hyperparameter; It is an indicator function, used to determine... Yes or no The purpose is to identify negative samples of event images and other event images.

[0033] Furthermore, the large language model includes ChatGPT, LLaMA, Qwen, or DeepSeek; the multimodal large model includes LLaVA, Qwen-Omni, or GPT-4o; and the stable diffusion model is the Stable Diffusion model.

[0034] A multimodal knowledge-enhanced cross-document event coreference resolution system implements the multimodal knowledge-enhanced cross-document event coreference resolution method, comprising three modules that respectively execute steps one through three.

[0035] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the multimodal knowledge-enhanced cross-document event coreference resolution method.

[0036] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal knowledge-enhanced cross-document event coreference resolution method.

[0037] Compared with existing technologies, the significant advantages of this invention are as follows: It accurately identifies event types, trigger words, descriptions, and elements through a large language model; intelligently aligns events with the original image using a multimodal large model; and generates synthetic images and easily confused negative samples for events without images using a diffusion model, constructing a structured multimodal event graph containing text and visual nodes. It extracts multimodal representations through text and visual encoders, establishes trigger words, elements, and image-text coreference relationships between cross-document events, and optimizes coreference resolution based on semantic distance calculation and contrastive learning. This method deeply integrates large language models and multimodal technologies, solving problems such as insufficient utilization of multimodal knowledge, coarse semantic association modeling, and biased coreference resolution decisions in traditional methods. It significantly improves the accuracy and robustness of cross-document event coreference resolution, providing an innovative technical path for multimodal document semantic understanding. Attached Figure Description

[0038] Figure 1 A flowchart for cross-document event coreference resolution to enhance multimodal knowledge. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0040] like Figure 1 As shown, the present invention provides a cross-document event coreference resolution method with multimodal knowledge enhancement, comprising the following steps:

[0041] Step 1: Extract structured event information from the document. Using a large language model (such as chat-GPT, LLaMA, Qwen, DeepSeek, etc.), extract structured event information from the document using a three-layered prompt template of summary-type-entity, including event type, event trigger words, event description, and event elements. The specific process is as follows:

[0042] (1) By using the first layer of the three-layer prompt template of summary-type-entity, the event in each document is detected, the event trigger word and event context are extracted, and the corresponding event summary is generated.

[0043] (2) Use hierarchical clustering algorithm to cluster all event summaries in the document to form event clusters representing different events in the document.

[0044] (3) Through the second layer of the three-layer prompt template of summary-type-entity, summarize the event type (Type) of each event cluster and the representative event trigger word (Trigger).

[0045] (4) Extract event elements (Arguments) and corresponding event entities from the event context of the event cluster through the third layer of the three-layer prompt template of summary-type-entity, and generate the event description of the event cluster.

[0046] The three-tiered suggestion template for Summary-Type-Entity is as follows:

[0047] 1) First-level hint

[0048] Input: Document

[0049] Role: Event Recognition and Summary Generation

[0050] Task:

[0051] - Event detection: Identify all event-triggered words in the document.

[0052] -Summary generation: Generates an event summary based on the event context.

[0053] rule:

[0054] - Output only the event trigger words, event context, and event summary.

[0055] - The event summary should include all the key elements of the event as much as possible.

[0056] Output format:

[0057] - event[trigger]:{ trigger, context, summary},

[0058] 2) Second layer prompt

[0059] Input: Event clusters "cluster1:{ event[trigger], ……}, cluster2:{ event[trigger], ……}, ……"

[0060] Characters: Event Cluster Summary

[0061] Task:

[0062] - Event Type Summarization: Summarize the event types (Type) in each event cluster.

[0063] - Representative event trigger word selection: Select one event trigger word from each event cluster as the representative event trigger word (Trigger) for that event cluster.

[0064] rule:

[0065] - Each event cluster categorizes only one event type.

[0066] - The representative event trigger word must be most semantically similar to the event type.

[0067] - The event summary should include all the key elements of the event as much as possible.

[0068] Output format:

[0069] - Cluster[Type, Trigger]:{ event[trigger], …},

[0070] 3) Third-level prompt

[0071] Input: Cluster[Type, Trigger]:{ event[trigger], …}, …

[0072] Role: Event Element Extraction and Event Description Generation

[0073] Task:

[0074] - Event Element Extraction: Extract event elements (Arguments) from each event cluster.

[0075] - Event description generation: Summarize the event information for each event cluster and generate an event description for the event cluster.

[0076] rule:

[0077] - Event elements must be closely related to the event type of the event cluster.

[0078] - Event elements include key elements such as the perpetrator, the victim, time, people, institutions, location, cause, and result.

[0079] - The event description should include all event elements as much as possible.

[0080] Output format:

[0081] - Event[Type]:{ Type, Trigger, Description, Arguments: [arg1,arg2...]},

[0082] ...

[0083] Step 2: Constructing a multimodal event graph within the document: The multimodal event graph includes nodes such as event type, event trigger words, event description, event elements, and event image. The specific construction process is as follows:

[0084] (1) If the document contains the original image, the event is aligned with the original image and used as the event image for that event. Specifically, for a document containing a unique event, the event is naturally aligned with the original image; for a document containing multiple events, a multimodal large model (such as LLaVA, Qwen-Omni, GPT4o, etc.) is used to detect the event in the document that best matches the original image through the event image alignment prompt template, and the original image is used as the event image for that event.

[0085] (2) For events that are not aligned with the original image, a stable diffusion model is used to generate image pseudo-labels that are consistent with the semantics of the event description, thereby constructing the event image corresponding to the event.

[0086] (3) In order to further improve the feature representation ability of the text encoder and the visual encoder so that they can effectively distinguish similar events with different event elements, this invention modifies the event elements in the event description to form different event descriptions, and uses the Stable Diffusion Model to generate a set of negative samples that are easily confused with the event images. The contrastive learning framework guides the model to distinguish the semantic differences between the event images (positive samples) and negative samples, thereby realizing the joint optimization training of the text encoder and the visual encoder.

[0087] The strategy for constructing negative samples of event images is as follows: For event elements and their corresponding event entities, the event elements are fixed, and the event entities are cyclically shifted to adjust their order. To avoid the adjusted event elements and entities deviating from reality, cyclic shifting is only performed between event entities of the same entity type. Then, the adjusted event elements and corresponding event entities are updated in the event description to form the adjusted event description. Finally, the Stable DiffusionModel is used to generate the adjusted negative samples of the event images.

[0088] (4) Combine the event information in the document (including event type, event trigger words, event description and event elements) with the event graph to form a multimodal event graph in the document.

[0089] Step 3, Cross-Document Event Core Referencing Resolution: First, the multimodal event graph within a document is represented. Then, core referencing relationships are established between multimodal event graphs from different documents, specifically including: core referencing relationships of event trigger words, core referencing relationships of event elements, and core referencing relationships between event text and image (including core referencing relationships of event descriptions and event images). Finally, by jointly optimizing the global semantic distance and contrastive learning loss, the cross-document events with the closest semantic distance are identified as core referencing events, thereby achieving unified association of cross-document events.

[0090] (1) Multimodal event graph representation within the document

[0091] Using a text encoder (Text Transformer) to analyze event-triggered words within a document , event elements Event Entity Event Description After vectorization, the event trigger word representation is obtained. Event element representation Event entity representation Event description and representation Event text representation; event images are represented using a visual encoder, Vision Transformer. Vectorization is performed to obtain an event image representation. .

[0092] (2) Cross-document multimodal event common index graph modeling

[0093] For those from the and the Any two multimodal event graphs of a document and The process of modeling the common index graph of multimodal events is as follows:

[0094] 1) Modeling coreference relationships of event-triggered words

[0095] Coreference relationships between event trigger words can be found through event trigger words. and We model the semantic distance between events. The smaller the semantic distance, the more similar the trigger words of the two events are, and the greater the probability of co-reference between the event trigger words.

[0096] Semantic distance of event trigger words The calculation method is as follows:

[0097]

[0098] in, Represents the cosine similarity function. and They represent and Trigger words, and They represent and Trigger word representation, subscript , An index of events representing different documents.

[0099] 2) Coreference modeling of event elements

[0100] Core reference relationships between event elements can be established through event elements and their corresponding event entities. and We model the semantic distance between events. The smaller the semantic distance, the more similar the elements of the two events are, and the greater the probability of core reference between the event elements.

[0101] Semantic distance of event elements The calculation method is as follows:

[0102]

[0103] in, and They represent and Event elements and corresponding event entities, and These represent the event element representation and the event entity representation, respectively.

[0104] 3) Event-text coreference modeling

[0105] The coreference relationship between event descriptions and corresponding event images can be established through the event description and the corresponding event image. and We model the semantic distance between events. The smaller the semantic distance, the more similar the text and images of the two events are, and the greater the probability of co-reference between the text and images.

[0106] Event image-text semantic distance The calculation method is as follows:

[0107]

[0108] in, and They represent and Event description and corresponding event image, and These represent event description and event image representation, respectively.

[0109] (3) Joint optimization of global semantic distance and contrastive learning

[0110] First, the semantic distance between multimodal event graphs across documents is calculated, and a global semantic distance optimization function is constructed. Then, a contrastive learning optimization function is constructed, which optimizes the similarity between event descriptions and event images, such that the cosine similarity between event descriptions and event images (positive samples) is close to 1, while the cosine similarity between event descriptions and negative samples of event images is close to 0. Finally, the global semantic distance and contrastive learning loss are jointly optimized to associate the same event scattered in different documents into a unified event, thereby achieving cross-document event coreference resolution.

[0111] 1) Global semantic distance optimization

[0112] To measure two multimodal event graphs ( and Semantic differences at the global structural level are calculated using the optimal transmission method to determine their semantic distance. The optimal transmission method, based on probability distribution matching theory, constructs a cost matrix to quantify the semantic differences between nodes and edges, and obtains the globally optimal transmission scheme through constraint optimization, thereby achieving accurate measurement of semantic distance between complex cross-modal graph structures. (Semantic distance across document multimodal event graphs) and global semantic distance optimization function It is expressed as follows:

[0113]

[0114]

[0115] The symbol ⊙ represents the Hadamard product (i.e., element-wise multiplication). Represents the transfer matrix, and its learning objective is to optimize the coreference relationships between nodes in two graphs. and These are multimodal event graphs. and The number of nodes, which means, Any node in the process can be transferred to different weights. Multiple nodes in the process. The cost matrix is ​​divided into the cost matrix for transmission between event trigger words. The cost matrix of transmission between event elements The cost matrix for transmission between event images and text Since the greater the semantic distance between nodes in two graphs, the higher the transmission cost, semantic distance is used to measure the cost matrix.

[0116]

[0117]

[0118]

[0119] Solving for the optimal transfer matrix The problem can be efficiently approximated using the differentiable Sinkhorn-Knopp algorithm. This algorithm, based on an entropy regularization strategy, transforms the discrete optimal transport problem into a smooth convex optimization problem, which can then be stably solved using an iterative scaling method (row and column normalization iteration).

[0120] 2) Comparison of learning optimization

[0121] Normalized temperature-scaled crossentropy loss (NT-Xent) is used as the optimization function for contrastive learning. Calculate the cosine similarity between the event description and the event image, such that the event description... With event images The cosine similarity between (positive samples) is close to 1, while the cosine similarity between the event image negative samples and other event images is close to 0.

[0122]

[0123] in, This represents the total number of event images and negative samples of event images across all documents. This represents a temperature hyperparameter; It is an indicator function, used to determine... Yes or no The purpose is to identify negative samples of event images and other event images.

[0124] 3) Joint optimization

[0125] Finally, by minimizing the overall optimization function We jointly optimize the semantic distance and contrastive learning of multimodal event graphs.

[0126]

[0127] This invention also proposes a multimodal knowledge-enhanced cross-document event coreference resolution system. The implementation of the multimodal knowledge-enhanced cross-document event coreference resolution method includes three modules that respectively execute steps one through three.

[0128] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the multimodal knowledge-enhanced cross-document event coreference resolution method.

[0129] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal knowledge-enhanced cross-document event coreference resolution method.

[0130] In summary, the core innovation of this invention lies in:

[0131] (1) A cross-document event coreference resolution framework based on graph structure is proposed. By integrating multimodal information in a unified graph structure, the complex relationships between documents and between modalities are explicitly modeled, overcoming the shortcomings of existing methods in capturing intermodal dependencies.

[0132] (2) Design a structured alignment mechanism based on optimal transmission to match the overall structure of cross-document multimodal graphs from a global perspective, breaking through the limitations of traditional pairwise similarity measurement and effectively bridging the cross-modal semantic gap;

[0133] (3) Contrastive learning is used to optimize the multimodal joint representation space. By bringing core-referenced text-image pairs closer together and pushing away non-core-referenced pairs, the discriminative power of the representation is enhanced, thereby improving the discriminative ability in semantically heterogeneous scenarios.

[0134] (4) Innovatively apply the event visual representation generated by the diffusion model for knowledge enhancement. This technique can achieve robust multimodal representation learning in scenes with scarce or noisy images, reducing the dependence on perfect alignment of the original images.

[0135] Example

[0136] To verify the effectiveness of the present invention, the following simulation experiment was conducted.

[0137] Input: Multiple news documents containing images, such as the following 3 news documents;

[0138] Document 1

[0139] Text: "...At 8:00 AM on May 20th, the 2025 City Marathon kicked off at the city center square. 15,000 runners participated in the full marathon, half marathon, and mini marathon events. After fierce competition, runner A won the men's full marathon with a time of 2 hours, 7 minutes, and 36 seconds, breaking the race record. The women's full marathon was won by runner B, with a time of 2 hours, 22 minutes, and 15 seconds. The organizing committee stated that this year, a digital real-time track monitoring system was introduced for the first time, effectively ensuring the safety of runners and the fairness of the race. Tens of thousands of citizens along the route enthusiastically watched the race, cheering on the runners. After the race, many runners took photos at the finish line area, sharing their joy of finishing..."

[0140] Image: [Original Image 1] (showing the grand starting scene);

[0141] Document 2

[0142] The text reads: "...From May 20th to 22nd, the food festival was held at the convention center next to the city center square, bringing together nearly 100 food exhibitors. The festival featured themed zones, local specialty zones, creative cuisine zones, and food culture experience zones. Chefs demonstrated molecular gastronomy techniques, attracting many visitors. Traditional pastries from local established brands were highly praised. The organizers also arranged interactive activities such as 'Parent-Child Baking Classes' and 'Food Photography Contests.' Preliminary statistics indicate that the festival received over 80,000 visitors over the three days..."

[0143] Image: [Original Image 2] (A corner of the food festival);

[0144] Document 3

[0145] Text: "...On May 20th, the city marathon course passed through the convention center, coinciding with the peak of the food festival crowds. The organizing committee made advance arrangements, setting up a temporary crowd control area at the intersection of the course and the food festival entrance, and assigning volunteers to guide pedestrians in an orderly manner. Many marathon participants said after the race that they found the aroma of food very interesting as they ran past the convention center area. One parent who participated in the mini-marathon with their child said, 'My child kept asking to go to the food festival after finishing the race, so it's a win-win situation.' The food festival organizers also provided free tasting vouchers to finishers, creating a positive interaction between the two events..."

[0146] Image: [Original Image 3] (Volunteers guide pedestrians across the side of the track);

[0147] Regarding the aforementioned document, the following are some implementation details of cross-document event coreference resolution for multimodal knowledge enhancement:

[0148] Step 1: For each document, use the DeepSeek large language model to extract structured event information from the document through a three-layer prompt template of summary-type-entity, including event type, event trigger words, event description and event elements.

[0149] 1) First-level prompts (event identification and summary generation)

[0150] Input: Document 1

[0151] Output (Example):

[0152] event[trigger1]: { trigger: "Starting gun", context: "At 8:00 AM on May 20th, the 2025 City Marathon started with a starting gun at the city center square. 15,000 runners participated in the full marathon, half marathon, and mini marathon events.", summary: "On the morning of May 20th, the 2025 City Marathon started with a starting gun at the city center square, with 15,000 runners participating."}

[0153] event[trigger2]: { trigger: "Winning the Championship", context: "After fierce competition, athlete A won the men's full marathon with a time of 2 hours, 07 minutes, and 36 seconds, breaking the course record. The women's full marathon champion was athlete B, with a time of 2 hours, 22 minutes, and 15 seconds.", summary: "Athletes A and B won the men's and women's full marathon championships respectively, and athlete A broke the course record."}

[0154] event[trigger3]: { trigger: "System Introduction", context: "The organizing committee stated that this year marks the first time a digital track real-time monitoring system has been introduced, effectively ensuring athlete safety and fair competition." summary: "The event has introduced a digital track real-time monitoring system for the first time, ensuring safety and fairness."}

[0155] ...

[0156] Process documents 2 and 3 similarly.

[0157] 2) Second-level hints (event cluster summarization)

[0158] Cluster all event summaries within the document to form event clusters, and then summarize the event types and representative trigger words.

[0159] Input: Event cluster of document 1

[0160] cluster1: { event[trigger1], event[trigger2], event[trigger3], …}

[0161] cluster2: { ……}

[0162] Output (Example):

[0163] Cluster1[Type: "Marathon Event", Trigger: "Start Gun"]: { event[trigger1], event[trigger2], event[trigger3], ……}

[0164] Cluster2[Type: "Award Winner", Trigger: "Winner of the Championship"]: { event[trigger2],……}

[0165] (3) Third-level prompts (event element extraction and event description generation)

[0166] Input: Cluster1[Type: "Marathon Event", Trigger: "Start Gun"]: { event[trigger1], event[trigger2], event[trigger3], ……}

[0167] Output:

[0168] Event [Marathon Race]:

[0169] {

[0170] Type: "Marathon Event"

[0171] Trigger: "Start the race with a gunshot"

[0172] Description: "At 8:00 AM on May 20th, the 2025 City Marathon kicked off at the city center square, with 15,000 runners participating. Runner A and Runner B ultimately won the men's and women's full marathon titles respectively, with Runner A breaking the race record. The event introduced a digital real-time track monitoring system for the first time, ensuring safety and fairness."

[0173] Arguments: [

[0174] Time: 8:00 AM, May 20th

[0175] Location: "City Center Square"

[0176] Number of participants: 15,000

[0177] Men's Champion: "Player A"

[0178] Women's Champion: "Contestant B"

[0179] Innovative measure: "Digital track real-time monitoring system" ]

[0181] }

[0182] Similarly, the "athlete winning" event can be extracted from document 1, but for simplicity, let's assume that document 1 mainly contains a "marathon event" event. Document 2 extracts the "food festival" event, and document 3 extracts the "marathon and food festival interaction" event.

[0183] Step two: For each document, construct a multimodal event graph. Each document's multimodal event graph includes nodes such as event type, event trigger words, event description, event elements, and event image. The specific steps are as follows:

[0184] Step 2.1, Document 1 (containing the original image and unique event)

[0185] Document 1 contains [Original Image 1] (starting point), which is naturally aligned with the "Marathon Event" event and serves as a positive sample image for the event.

[0186] Step 2.2, Document 2 (containing the original image and multiple events)

[0187] Document 2 contains only one image, [Original Image 2] (a corner of the food festival). Using the LLaVA multimodal large model, the image is found to be the best match for the "food festival" event in Document 2 by aligning the event image with the cue template. Therefore, [Original Image 2] is used as the positive sample image for this event.

[0188] Step 2.3, Document 3 (No event aligned with the original image)

[0189] Document 3 contains [Original Image 3] (volunteers guiding pedestrians), but this image does not completely correspond to the "Marathon and Food Festival Interaction" event in Document 3 (the image only shows the guiding scene and does not reflect the interaction). Therefore, a stable diffusion model is used to generate synthetic images based on the event description as positive sample images.

[0190] Step 2.4: Generate easily confused negative sample images

[0191] For each event, a cyclic shift strategy with consistent entity types is used to modify the event description, and a stable diffusion model is used to generate negative sample images. Taking the "Marathon Event" event in Document 1 as an example, the adjustment only involves cyclic shifting between entities of the same entity type; "Men's Champion" and "Women's Champion" are both human entities, so they are swapped.

[0192] Before adjustment:

[0193] 1) Event elements and corresponding event entities

[0194] Time: 8:00 AM, May 20th

[0195] Location: "City Center Square"

[0196] Number of participants: 15,000

[0197] Men's Champion: "Player A"

[0198] Women's Champion: "Contestant B"

[0199] Innovative measure: "Digital Track Real-time Monitoring System"

[0200] 2) Event Description

[0201] At 8:00 AM on May 20th, the 2025 City Marathon kicked off at the city center square, with 15,000 runners participating. Runner A and Runner B ultimately won the men's and women's full marathon titles respectively, with Runner A breaking the race record. The event also marked the first time a digital real-time course monitoring system was introduced.

[0202] 3) Event Images

[0203] [Original Image 1]

[0204] 4) Sample type

[0205] Positive samples

[0206] After adjustment:

[0207] 1) Event elements and corresponding event entities

[0208] Time: 8:00 AM, May 20th

[0209] Location: "City Center Square"

[0210] Number of participants: 15,000

[0211] Men's Champion: "Contestant B"

[0212] Women's Champion: "Player A"

[0213] Innovative measure: "Digital Track Real-time Monitoring System"

[0214] 2) Event Description

[0215] At 8:00 AM on May 20th, the 2025 City Marathon kicked off at the city center square, with 15,000 runners participating. Runner B and Runner A ultimately won the men's and women's full marathon titles respectively, with Runner A breaking the race record. The event also marked the first time a digital real-time course monitoring system was introduced.

[0216] 3) Event Images

[0217] [Synthetic Image Negative Samples]

[0218] 4) Sample type

[0219] Negative samples;

[0220] Step 2.5: Create a multimodal event graph within the document.

[0221] The multimodal event graph nodes for the three documents are as follows:

[0222] 1) Document 1

[0223] Event [Marathon Race]:

[0224] {

[0225] Type: "Marathon Event"

[0226] Trigger: "Start the race with a gunshot"

[0227] Description: "At 8:00 AM on May 20th, the 2025 City Marathon kicked off at the city center square, with 15,000 runners participating. Runner A and Runner B ultimately won the men's and women's full marathon titles respectively, with Runner A breaking the race record. This event marked the first time a digital real-time course monitoring system was introduced."

[0228] Arguments: [Time: "8:00 AM, May 20th", Location: "City Center Square", Number of Participants: "15,000", Men's Champion: "Participant A", Women's Champion: "Participant B", Innovative Measures: "Digital Track Real-time Monitoring System"], PositiveImages: [Original Image 1]

[0229] NegativeImages: [Synthetic Image Negative Samples]

[0230] }

[0231] Document 2

[0232] Event [Food Festival]:

[0233] {

[0234] Type: "Food Festival"

[0235] Trigger: "Hold",

[0236] Description: "From May 20th to 22nd, the food festival was held at the convention center, bringing together nearly 100 exhibitors. Chefs demonstrated molecular gastronomy on-site, and local time-honored pastries received high praise. Activities included parent-child baking classes and a food photography competition, attracting a total of over 80,000 visitors."

[0237] Arguments: [Dates: May 20-22, Location: Convention Center, Number of Exhibitors: Nearly 100, Featured Activities: Molecular Gastronomy Demonstration, Parent-Child Baking Class, Food Photography Contest, Total Attendance: 80,000+]

[0238] PositiveImages: [Original Image 2],

[0239] NegativeImages: [Synthetic Image Negative Samples]

[0240] }

[0241] Document 3

[0242] Event [Marathon and Food Festival Interaction]:

[0243] {

[0244] Type: "Interactive Activities"

[0245] Trigger: "intersection",

[0246] Description: "On May 20th, the city marathon course intersected with the peak flow of the food festival as it passed the convention center. The organizing committee set up a temporary guidance area and arranged volunteers to guide the participants. After the race, the runners expressed that it was fun. The food festival provided free tasting vouchers to finishers, creating a positive interaction."

[0247] Arguments: [Time: "May 20th", Location: "Near the Convention Center", Interactive Measures: "Temporary crowd control area, volunteer guidance, free tasting vouchers", Participants: "Marathon runners, food festival visitors"]

[0248] PositiveImages: [Synthetic Images],

[0249] NegativeImages: [Synthetic Image Negative Samples]

[0250] }

[0251] Step 3: Cross-document event coreference resolution

[0252] Step 3.1, Example of Multimodal Event Graph Representation Vector

[0253] Taking the "Marathon Event" event in Document 1 as an example, we used a text encoder (Text Transformer, outputting a 768-dimensional vector) and a vision encoder (Vision Transformer, outputting a 512-dimensional vector) to vectorize the content of each modality of the event. The following lists the first 10 dimensions of each representation vector to visually demonstrate the vector form (the actual vectors are floating-point numbers, but three decimal places are retained here).

[0254] 1. Text representation vector of the event trigger word "fire the starting gun" (768 dimensions, truncated to the first 10 dimensions)

[0255] [0.213, -0.547, 0.882, 0.124, -0.336, 0.759, -0.468, 0.991, -0.205, 0.643]

[0256] 2. Textual representation vector of the event description (768 dimensions, truncated to the first 10 dimensions)

[0257] Original description of the event:

[0258] "At 8:00 a.m. on May 20, the 2025 City Marathon started at the city center square, with 15,000 runners participating. Runner A and runner B won the men's and women's full marathon championships respectively, and runner A broke the race record. The event introduced a digital real-time track monitoring system for the first time, ensuring safety and fairness."

[0259] Vector example:

[0260] [0.425, 0.178, -0.892, 0.361, -0.073, 0.514, 0.667, -0.239, 0.805, -0.154]

[0261] 3. The entity representation vector (768 dimensions, truncated to the first 10 dimensions) of the event entity text "May 20th, 8:00 AM" corresponding to the event element "Time".

[0262] [-0.112, 0.953, -0.478, 0.306, 0.841, -0.629, 0.274, -0.790, 0.518, 0.387]

[0263] 4. Visual representation vector of positive sample image (original image 1: marathon starting scene) (512 dimensions, truncated to the first 10 dimensions)

[0264] [0.736, 0.259, -0.408, 0.892, -0.145, 0.621, 0.377, -0.553, 0.964, -0.211]

[0265] 5. Visual representation vector of negative sample images (synthetic images: easily confused images generated after entity swapping) (512 dimensions, truncated to the first 10 dimensions)

[0266] [0.514, -0.367, 0.805, 0.438, 0.192, -0.760, 0.683, 0.249, -0.591, 0.847]

[0267] Step 3.2, calculate the multi-dimensional semantic distance

[0268] Taking the "Marathon Event" event in Document 1 and the "Marathon and Food Festival Interaction" event in Document 3 as examples, we calculate the semantic distance between the two events in multiple dimensions, including the semantic distance of event trigger words, the semantic distance of event elements, and the semantic distance of event images and text.

[0269] Step 3.3, Calculate the global semantic distance

[0270] The global semantic distance between two multimodal event graphs is calculated using an optimal transmission method. The cost matrix is ​​composed of the semantic distances between each node, and the global distance is obtained by solving the transmission matrix using the Sinkhorn algorithm.

[0271] Step 3.4, Comparative Learning Optimization

[0272] A contrastive learning loss function is constructed to bring the event description closer to its representation in positive sample images, and to push it further away from its representation in negative sample images and other event images. The global semantic distance and the contrastive learning loss are jointly optimized.

[0273] Step 3.5, Common Pointer Determination

[0274] For any two cross-document events, calculate their joint optimized total distance, and determine the cross-document event with the smallest distance as the core reference. For example, "marathon event" in document 1 and "marathon and food festival interaction" in document 3 are highly semantically related (sharing time, location, and some elements), and are therefore identified as core reference events, achieving unified association.

[0275] Through the above steps, the present invention successfully associated the same event (marathon event and its derivative interactions) scattered in different documents and effectively distinguished different events (marathon and food festival), verifying the accuracy and robustness of the method under multimodal knowledge enhancement.

[0276] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0277] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multi-modal knowledge enhanced cross-document event coreference resolution method, characterized in that, Includes the following steps: Step 1: Extract Structured Event Information from Documents: For each document, using a large language model and a three-layer prompt template (summary-type-entity), extract the structured event information within the document. This structured event information includes event type, event trigger words, event description, and at least one event element and its corresponding event entity. The three-layer prompt template (summary-type-entity) is specifically as follows: The first level of prompts is used to extract event trigger words and their context from the document and generate corresponding event summaries; The second layer of prompts is used to cluster all event summaries generated in the first layer, summarize the event types of each event cluster, and select representative event trigger words; The third layer of prompts is used to extract event elements and corresponding event entities from the context of each event cluster, and generate a complete event description for that event cluster; Step 2: Construct a multimodal event graph within the document: For each event extracted in Step 1, associate it with visual information to construct a multimodal event graph. If the document contains original images and there are original images that semantically match the event description, use a multimodal large model to semantically align the event description with the original images, and use the matched original images as positive sample images for the event. For events that do not match the original image, a stable diffusion model is used to generate a synthetic image as a positive sample image based on the event description. Furthermore, the event description is modified using a cyclic shift strategy consistent with entity type, and the stable diffusion model is used to generate easily confused negative sample images based on the modified event description. The structured information of the event and its associated positive and negative sample images together constitute the multimodal event graph nodes of the event, and all nodes constitute the multimodal event graph of the document. The cyclic shift strategy consistent with entity type specifically means that when generating easily confused negative samples, the event elements remain unchanged, and only cyclic shifts are performed between event entities of the same entity type to adjust their order, thereby forming an adjusted event description, and negative sample images are generated accordingly. Step 3: Implement cross-document event coreference resolution: Use a text encoder to vectorize the event trigger words, event elements, event entities corresponding to the event elements, and event descriptions in the multimodal event graph to obtain the corresponding text representations. Use a visual encoder to vectorize the positive sample images and easily confused negative sample images in the multimodal event graph to obtain the corresponding visual representations. For any two multimodal event graphs from different documents, based on the obtained representations, calculate the semantic distance in the event trigger word dimension, event element dimension, and event graph-text dimension to model the multidimensional coreference relationship between the two events. The optimal transmission method is adopted to calculate the global semantic distance between two multimodal event graphs based on multidimensional semantic distance. At the same time, the contrastive learning loss function is used to optimize the similarity between the event description representation and its positive sample image representation. By jointly optimizing the global semantic distance and the contrastive learning loss, the cross-document event with the closest global semantic distance is determined as the coreference, thereby realizing the unified association of cross-document events.

2. The multi-modal knowledge augmented cross-document event coreference resolution method of claim 1, wherein, In step three, the multi-dimensional semantic distance between any two events in the two multimodal event graphs includes: Event trigger word semantic distance The calculation is as follows: ； wherein, denotes a cosine similarity function, and denote trigger words of two events, respectively, and denote trigger word representations of two events, respectively, with subscript , represent event indices of different documents; Event element semantic distance The calculation is as follows: ； in, and These represent the event elements and corresponding event entities of two events, respectively. and These represent the event element representation and the event entity representation of the two events, respectively. Event graph-text semantic distance The calculation is as follows: ； wherein, and respectively represent an event description of two events and a corresponding event image, and respectively represent an event description representation of two events and an event image representation.

3. The multi-modal knowledge augmented cross-document event coreference resolution method of claim 2, wherein, In step three, the optimal transport method is used to compute the global semantic distance between the two multi-modal event graphs and , in particular:​ ； Wherein, the symbol ⊙ represents the Hadamard product. Represents the transmission matrix. and These are multimodal event graphs. and The number of nodes, The cost matrix consists of the cost matrix transmitted between event trigger words. Cost matrix of transmission between event elements Cost matrix for transmission between event images and text The cost submatrix is ​​composed of elements representing the semantic distances between the corresponding nodes.

4. The method for cross-document event coreference resolution with multimodal knowledge enhancement according to claim 3, characterized in that, In step three, the joint optimization is performed by minimizing the total optimization function Implementation, denoted as: ； ； ； in, This represents the global semantic distance optimization function. This represents the contrastive learning optimization function. and They represent the first Event description representation and corresponding event image representation for each event. This represents the total number of event images and negative samples of the event images. This represents a temperature hyperparameter; It is an indicator function, used to determine... Yes or no The purpose is to identify negative samples of event images and other event images.

5. The multi-modal knowledge augmented cross-document event coreference resolution method of claim 1, wherein, The large language model includes ChatGPT, LLaMA, Qwen, or DeepSeek; the multimodal large model includes LLaVA, Qwen-Omni, or GPT-4o; and the stable diffusion model is the Stable Diffusion model.

6. A multi-modal knowledge augmented cross-document event coreference resolution system, characterized in that, The method for resolving cross-document event coreference with multimodal knowledge enhancement as described in any one of claims 1-5 includes three modules that respectively execute steps one through three.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the multimodal knowledge-enhanced cross-document event coreference resolution method according to any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, wherein when executed by a processor, the computer program implements the multimodal knowledge-enhanced cross-document event coreference resolution method according to any one of claims 1-5.

Images