Multi-modal event extraction method and system based on iterative visual verification

Abstract

This invention discloses a multimodal event extraction method and system based on iterative visual verification, belonging to the interdisciplinary field of natural language processing and computer vision. The method generates an initial event list using a large multimodal model, and uses an optimized dynamic resolution visual verifier for inspection and feedback, driving the iterative process and enhancing generator performance, forming a closed-loop optimization mechanism of "generation-verification-correction." It provides two verifier training strategies: adjusting the number of trainable parameters to only 0.1%-2% of the original model, resulting in extremely low computational resource consumption; and a reinforcement learning strategy to further improve performance, suitable for scenarios with extremely high accuracy requirements. By training on a weakly aligned random perturbation dataset, the model can learn to detect incompleteness, type errors, and hallucination elements in the event list, and output structured feedback containing four types of error information, effectively solving the incompleteness and hallucination problems of large language models in multimodal event extraction.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of natural language processing and computer vision, and involves technologies such as artificial intelligence, multimodal learning, event extraction and model verification. Specifically, it relates to a multimodal event extraction method and system based on iterative visual verification. Background Technology

[0002] Cross-modal event extraction aims to identify and structure event information (event type, trigger words, participants, and semantic roles, etc.) from text-image pairs, and has wide application value in scenarios such as news analysis, social media monitoring, intelligent question answering, and public opinion analysis. With the explosive growth of social media and multimedia news, pure text event extraction can no longer cover the rich visual information, prompting the rapid development of multimodal methods into a research hotspot.

[0003] Existing solutions mainly include: direct generation based on general multimodal large models, prompt-based extraction based on closed-source APIs, and single-round text post-validation. Although the generation capabilities of large models are constantly improving, existing technologies still have significant shortcomings in terms of generation quality control, cross-modal consistency verification, and resource efficiency, severely restricting practical application deployment. 1) Large model generation quality instability and hallucination problem Multimodal hallucinations are severe: the model tends to fill in the gaps in the image based on textual cues, with a hallucination rate as high as 15%-20%. For example, the text mentions "police arresting a suspect," but the image only shows "police cars," yet the model creates fictitious entities of "police" and "suspect"; or the image shows "peaceful assembly," but the model misidentifies it as a "conflict event," leading to misjudgment of the event type.

[0004] The event structure is frequently incomplete: the generated event list often lacks key arguments (such as missing Agent, Location, etc.), with a missing rate of 20%-35%, or the arguments do not match the event type, resulting in incomplete event information extraction and difficulty in supporting downstream applications.

[0005] Cross-modal inconsistency: The model may correctly understand the text content but fail to make full use of the image information, or vice versa, resulting in inconsistencies between the extracted results and the multimodal input, with visual-text alignment accuracy of only 60%-75%.

[0006] 2) Lack of effective quality verification and feedback mechanisms The inherent flaw of one-way generative architecture: Most existing generative frameworks are one-way. Once the model makes a mistake in the initial stage, such as identifying the wrong event type or missing key arguments, subsequent processing can only continue on the basis of the error and cannot self-correct, leading to the accumulation and propagation of errors.

[0007] Lack of verification loop: There is a lack of a closed-loop control mechanism that can perform "verification-feedback-correction". The quality of the generated results depends entirely on the randomness of a single inference, without the opportunity for iterative optimization, making it difficult to guarantee the reliability and stability of the output.

[0008] Feedback information is coarse-grained and not actionable: Even with the introduction of a verification step, existing methods often only provide a simple binary judgment of "correct / incorrect", lacking specific error location (which arguments are missing), error type classification (whether it is deletion, replacement or illusion) and correction suggestions, and cannot effectively guide the generative model to make targeted improvements.

[0009] 3) The high cost of training validators limits their practical application. The memory bottleneck in training multimodal validators: Building high-quality cross-modal validators requires fine-tuning based on large-scale visual language models to ensure sufficient multimodal understanding capabilities. However, fine-tuning such models with full parameters (e.g., 7B-8B parameter scale) requires 80-100GB of video memory, far exceeding the capacity of mainstream single-card GPUs, resulting in high training costs that are difficult for small and medium-sized research institutions and enterprises to afford.

[0010] There is a lack of fine-tuning solutions for multimodal validation tasks: While existing efficient parameter fine-tuning strategies have been widely applied in unimodal and general dialogue tasks, their application in the specific task of multimodal validation is insufficient. Multimodal validation requires the model to simultaneously understand the correspondence between image content, text semantics, and event structure, which places higher demands on the design of fine-tuning strategies (such as target module selection, positive and negative sample construction, and training hyperparameters), and there is a lack of mature training schemes and configuration experience.

[0011] Labeling validation data is challenging: training a high-quality validator requires a large number of positive and negative sample pairs, but manually labeling whether the event extraction was correct and the specific type and location of errors is extremely costly. How to construct high-quality training data automatically while ensuring the diversity and balance of positive and negative samples is a key challenge in training multimodal validators.

[0012] Based on the above problems, a lightweight and efficient multimodal event extraction method is urgently needed. This method should have: (1) effective cross-modal consistency verification capability, which can detect illusions and errors in the generated results; (2) an iterative feedback closed-loop mechanism, which guides the generator to improve the output through specific feedback from the verifier; (3) a parameter-efficient training strategy, which reduces the resource consumption of the verifier and enables it to be trained and deployed on a single GPU; and (4) fast iterative convergence, which ensures the practicality of multi-round verification. Summary of the Invention

[0013] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multimodal event extraction method and system based on iterative visual verification. This method combines the ability to generate large multimodal models with the quality control capabilities of lightweight visual verifiers, improves the accuracy and completeness of cross-modal event extraction through iterative feedback mechanisms, and reduces computational resource consumption and inference costs.

[0014] To achieve the above objectives, in a first aspect, the present invention provides a multimodal event extraction method based on iterative visual verification, comprising the following steps: Step 1, Enter text description and corresponding images Load pre-trained large-scale multimodal models M gen As a generator, it loads a pre-trained visual validator and performs parameter-efficient fine-tuning. Initialize the iteration counter Set the maximum number of iterations. ; Step 2, using the generator M gen text and images Generate an initial event list E 0. The event list contains zero or more events, each event e i It can be represented as:

[0015] in, For event category (event_class). For event subtypes (event_subtype). It is a trigger word. For the list of arguments (each argument contains a semantic role) and entity ), Optional visual grounding information; Step 3, iterate through the visual verification loop when If the termination condition is not met, execute the following sub-steps: Step 3.1, Multimodal Verification List of current events Text description and corresponding images Visual verification device Visual validator Check the cross-modal consistency of the event list and output structured feedback. :

[0016] feedback It includes the validation result (correct field, boolean value) and specific error information, which is divided into four categories: missing events, wrong event types, missing arguments, and wrong argument roles. Step 3.2, dynamically update the prompt template like If `.correct` is true, terminate the iteration and output the result. As a final result; if If error messages are included, a prompt template will be built. :

[0017] Inject specific feedback from the visual validator into the prompt template to guide the generator. Make targeted corrections; Step 3.3, Feedback-driven regeneration Use enhanced prompts Call generator Generate an improved event list. :

[0018] Update iterative counter ; Step 3.4, repeat steps 3.1 through 3.3. Continue executing the iterative verification loop until verification is successful, the maximum number of iterations is reached, or the convergence condition is met; specifically, the termination condition includes the following three-level decision mechanism: (1) Verifier decision: If The verification result is passed, and the iteration is terminated; (2) Convergence criterion: If the similarity of the event lists generated in two consecutive iterations exceeds the threshold ,Right now The iteration is terminated after the condition is determined to have converged. (3) Maximum number of iterations limit: .

[0019] Step 4, return to the final event list Output the number of iterations and the verification history.

[0020] Furthermore, the verifier The training includes the following steps: 1) Construct positive samples: ,in ; 2) Negative samples are generated using the following perturbation strategy: Let the original event list be The list of events after the disturbance is as follows : Perturbation Method 1: Deletion Perturbation: Randomly delete some events or arguments. E'=E\ { e i}or This generates a missing class error; Perturbation Method 2, Replacement Perturbation: Randomly replace the event type or argument role. or The generated type is incorrect; Perturbation Method 3, Illusion Perturbation: Add illusionary event elements that are inconsistent with the image content. An error occurred while generating an illusion. The labels for negative samples include correct: false and a specific error description; the number of entries in the training dataset obtained after the above three perturbations is approximately four times that of the original dataset, including one positive sample and three negative samples; 3) Train a validator based on a visual language model using the parameter tuning method. Let the original model parameters be... The adaptation parameters are: The fine-tuned model parameters are:

[0021] The number of trainable parameters accounts for 0.1%-2% of the original model; 4) The training objective is to minimize the negative log-likelihood loss from sequence to sequence:

[0022] in, The output sequence of the validator. Indicates the preceding A token.

[0023] Furthermore, the validator can also be trained using a reinforcement learning strategy to further optimize its decision-making ability based on parameter tuning training. Reinforcement learning training includes: designing a reward function that provides positive rewards for correct identification errors, heavier negative rewards for missed detection errors, and lighter negative rewards for false positives; using a policy optimization algorithm to update the validator parameters, with the optimization objective being to maximize the expected cumulative reward; and using KL divergence constraints to control the policy update magnitude. Compared to parameter fine-tuning, reinforcement learning can further improve the validator's performance by 2 to 3 percentage points, but the training cost increases by approximately 10 to 15 times, making it suitable for critical application scenarios with extremely high accuracy requirements.

[0024] Furthermore, the policy optimization algorithm can employ either GRPO (Group Relative Policy Optimization) or PPO (Proximal Policy Optimization).

[0025] Furthermore, the input encoding method of the visual verifier is as follows: Image features, text descriptions, and a list of events to be verified are fused and encoded using a multimodal method. Let the image features be... Text features are The event list is characterized by The input after fusion is:

[0026] Visual features, text features, and features of the event to be verified are used as inputs to the cross-attention module. The three are aligned and fused through cross-attention to obtain a joint representation, which is used to carry multimodal consistency information and support subsequent verification.

[0027] Secondly, the present invention also provides a multimodal event extraction system based on iterative visual verification, including an input preprocessing module, a generator module, a verifier module, an iteration control module, and a post-processing module. The input preprocessing module is used to encode text, preprocess images, and load the initial prompt template; The generator module, based on a large-scale multimodal model, receives text, images, and prompt templates, and outputs structured event information. It is compatible with mainstream general-purpose multimodal model interfaces. The validator module, based on a visual language model with efficient parameter fine-tuning, receives text, images, and event lists, and outputs structured feedback containing verification results and specific error information. The iteration control module includes a prompt template manager, a termination condition determiner, and a history manager, which manage prompt template updates, termination condition determination, and history records. The post-processing module formats the final event list, calculates quality scores, and provides result visualization.

[0028] Thirdly, the present invention also provides a computer device, including a processor and a memory, wherein the memory is used to store a computer executable program, the processor reads part or all of the computer executable program from the memory and executes it, and the processor can implement the above-mentioned multimodal event extraction method based on iterative visual verification when executing part or all of the computer executable program.

[0029] A computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, it can implement the above-described multimodal event extraction method based on iterative visual verification.

[0030] Compared with the prior art, the present invention has at least the following beneficial effects: Significantly improves the quality of event extraction Through an iterative verification mechanism, this invention, compared to a single-generation baseline method, can effectively detect and correct missing, erroneous, and hallucination problems in event extraction, thereby improving the completeness and accuracy of event extraction. Significantly reduces the training cost of visual validators and provides flexible training strategy options. This invention provides two validator training strategies: a parameter-efficient fine-tuning strategy that reduces training memory requirements by 70%-85%, enabling validator training to be completed on a single mainstream GPU; and a reinforcement learning fine-tuning strategy that further improves performance by 2 to 3 percentage points, suitable for scenarios with extremely high accuracy requirements. The two strategies can be flexibly selected based on actual resource conditions and performance needs. Provide fine-grained error feedback The validator outputs structured error information, including four categories: missing events, error types, missing arguments, and incorrect roles. This allows for precise identification of problems and guidance for the generator to make targeted corrections, rather than a simple binary judgment of correct / incorrect.

[0031] Achieving cross-modal consistency guarantee The multimodal feature fusion mechanism of the visual verifier can effectively detect hallucinatory event elements that are not present in the image, ensuring that the generated event list is consistent with the visual content, and significantly improving the reliability of multimodal understanding. Good scalability and generalization ability The validator can be migrated to other event extraction datasets, supports different general multimodal large models as generators (compatible with open source and closed source models), can adapt to different event extraction mode definitions, and has strong versatility and practical value. Attached Figure Description

[0032] To more clearly illustrate the technical solutions of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without any creative effort.

[0033] Figure 1 The schematic diagram of the overall system architecture provided for the embodiments of the present invention illustrates the interaction relationship between the generator module, the verifier module, and the iteration control module, as well as the data flow of the iteration verification closed loop. Figure 2 The iterative verification process provided in this embodiment of the invention is illustrated in detail, showing the complete process from initial generation to iterative optimization, including termination condition determination and enhancement prompt construction. Figure 3 The diagram illustrates the validator training data construction process provided in this embodiment of the invention, showing the generation of positive samples and three negative sample perturbation strategies, including deletion perturbation, replacement perturbation and illusion perturbation. Figure 4 The diagram of the LoRA fine-tuning architecture provided in the embodiment of the present invention shows the insertion position and parameter configuration of the LoRA adapter in all linear layers of the Qwen3-VL-8B model, including rank r=16, scaling factor α=32, and target module target=all-linear; Figure 5 The schematic diagram of multimodal input encoding provided in the embodiments of the present invention illustrates the feature extraction and fusion encoding methods of image features, text descriptions, and candidate event lists; Figure 6 This is a schematic diagram of the automatic construction process of weakly aligned datasets provided in an embodiment of the present invention, illustrating a five-stage pipeline from raw data acquisition to training data generation. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0035] This invention provides a multimodal event extraction method based on iterative visual verification, comprising the following steps: Step 1, Enter text description and corresponding images Load pre-trained large-scale multimodal models M gen As a generator, it loads a pre-trained visual validator and performs parameter-efficient fine-tuning. Initialize the iteration counter Set the maximum number of iterations. ; Step 2, using the generator M gen text and images Generate an initial event list E 0. The event list contains zero or more events, each event e i It can be represented as:

[0036] in, For event category (event_class). For event subtypes (event_subtype). It is a trigger word. For the list of arguments (each argument contains a semantic role) and entity ), Optional visual grounding information; Step 3, iterate through the visual verification loop when If the termination condition is not met, execute the following sub-steps: Step 3.1, Multimodal Verification List of current events Text description and corresponding images Visual verification device Visual validator Check the cross-modal consistency of the event list and output structured feedback. :

[0037] feedback It includes the validation result (correct field, boolean value) and specific error information, which is divided into four categories: missing events, wrong event types, missing arguments, and wrong argument roles. Step 3.2, dynamically update the prompt template like If `.correct` is true, terminate the iteration and output the result. As a final result; if If error messages are included, a prompt template will be built. :

[0038] Inject specific feedback from the visual validator into the prompt template to guide the generator. Make targeted corrections; Step 3.3, Feedback-driven regeneration Use enhanced prompts Call generator Generate an improved event list. :

[0039] Update iterative counter ; Step 3.4, repeat steps 3.1 through 3.3. Continue executing the iterative verification loop until verification is successful, the maximum number of iterations is reached, or the convergence condition is met; specifically, the termination condition includes the following three-level decision mechanism: (1) Verifier decision: If The verification result is passed, and the iteration is terminated; (2) Convergence criterion: If the similarity of the event lists generated in two consecutive iterations exceeds the threshold ,Right now The iteration is terminated after the condition is determined to have converged. (3) Maximum number of iterations limit: .

[0040] Step 4, return to the final event list Output the number of iterations and the verification history.

[0041] Furthermore, validator The training includes the following steps: 1) Construct positive samples: ,in ; 2) Negative samples are generated using the following perturbation strategy: Let the original event list be The list of events after the disturbance is as follows : Perturbation Method 1: Deletion Perturbation: Randomly delete some events or arguments. E'=E\ { e i}or This generates a missing class error; Perturbation Method 2, Replacement Perturbation: Randomly replace the event type or argument role. or The generated type is incorrect; Perturbation Method 3, Illusion Perturbation: Add illusionary event elements that are inconsistent with the image content. An error occurred while generating an illusion. The labels for negative samples include correct: false and a specific error description; the number of entries in the training dataset obtained after the above three perturbations is approximately four times that of the original dataset, including one positive sample and three negative samples; 3) Train a validator based on a visual language model using the parameter tuning method. Let the original model parameters be... The adaptation parameters are: The fine-tuned model parameters are:

[0042] The number of trainable parameters accounts for 0.1%-2% of the original model; 4) The training objective is to minimize the negative log-likelihood loss from sequence to sequence:

[0043] in, The output sequence of the validator. Indicates the preceding A token.

[0044] The visual validator can also be trained using a reinforcement learning strategy to further optimize its decision-making ability based on parameter tuning. Reinforcement learning training includes: designing a reward function that provides positive rewards for correct identification errors, heavier negative rewards for missed detection errors, and lighter negative rewards for false positives; using a policy optimization algorithm to update the validator parameters, with the optimization objective being to maximize the expected cumulative reward; and using KL divergence constraints to control the policy update magnitude. Compared to parameter fine-tuning, reinforcement learning can further improve the validator's performance by 2 to 3 percentage points, but the training cost increases by approximately 10 to 15 times, making it suitable for critical application scenarios with extremely high accuracy requirements.

[0045] Policy optimization algorithms can be either GRPO (Group Relative Policy Optimization) or PPO (Proximal Policy Optimization).

[0046] The input encoding method of the visual verifier is as follows: Image features, text descriptions, and a list of events to be verified are fused and encoded using a multimodal method. Let the image features be... Text features are The event list is characterized by The input after fusion is:

[0047] Visual features, text features, and features of the event to be verified are used as inputs to the cross-attention module. The three are aligned and fused through cross-attention to obtain a joint representation, which is used to carry multimodal consistency information and support subsequent verification.

[0048] Example 1: Complete System Implementation This embodiment provides a complete implementation scheme based on Python, detailing the various modules and implementation details of the system. The overall system architecture is as follows: Figure 1 As shown, it includes three core components: a generator module, a validator module, and an iteration control module.

[0049] like Figure 1 As shown, the system adopts a closed-loop structure of "generation-verification-control": the left side is the multimodal input, including text description. T and corresponding images I The right side shows the final event list. E final The area within the dashed box represents the iterative verification closed loop, which includes the generator module. M gen Verifier module The system consists of three parts: an iterative control module, a generator (a large multimodal model), and a template generator. P k Input multimodal information, output a list of candidate events E k The validator is a small multimodal model with a built-in LoRA adapter, receiving images, text, and a list of candidate events. E k Output structured verification feedback F k The iteration control module is responsible for managing prompt templates, determining termination conditions, and recording iteration counts: when verification passes, it outputs "Correct" and terminates; otherwise, it operates based on feedback. F k Generate enhanced prompts P k+1 This drives the next round of generation.

[0050] The process includes the following steps: 1) Initial input enters the generator; 2) The generator outputs candidate events. E k3) The validator performs consistency verification on candidate events and generates feedback. F k 4) The iterative control module injects feedback into the prompt template to form P k+1 5) When the correctness condition is met, output the final event list. E final The example output in the diagram includes the event type, trigger word, and role arguments (such as Event:Justice:Arrest, Trigger:“arrested”, Agent:Police, Person:suspect, Place:near the blue car), demonstrating the gradual improvement of the event structure by the closed-loop mechanism.

[0051] Environment configuration The specific hardware requirements are: GPU is NVIDIA A100 80GB or NVIDIA RTX 4090 24GB×4, CPU is Intel Xeon or AMD EPYC with at least 16 cores, and memory is 64GB or more.

[0052] Software environment: Operating system is Ubuntu 22.04 LTS or CentOS 7.9, Python version is 3.10 or 3.11, PyTorch version is 2.1.2 or higher, Transformers version is 4.37.0 or higher, PEFT version is 0.7.0 or higher for LoRA implementation, MS-SWIFT version is 2.0 or higher or LLaMA-Factory is optional to simplify the training process.

[0053] Detailed steps for validator training Step 1: Data Preparation The visual validator training of this invention supports two types of data sources.

[0054] The first category consists of publicly available benchmark datasets. Training samples are extracted from the M2E2 cross-modal event extraction dataset. Each sample contains: a text description T representing a sentence from a news article, a corresponding image I representing the news image, and an event list E containing manually labeled event types, participants, and role relationships. The dataset statistics are as follows: approximately 700 to 800 samples in the training set, approximately 100 to 150 samples in the validation set, and approximately 100 to 150 samples in the test set.

[0055] The second category involves self-built large-scale weakly aligned datasets, the source of which is optional. To improve the generalization ability of the validator, this invention provides an automatic method for constructing large-scale weakly aligned datasets, such as... Figure 6 As shown. This method comprises the following five stages: Phase one involves raw data collection. This involves collecting text-image pairing data from publicly available news websites. Each data entry includes: a news headline and summary text, one or more accompanying images, and a lead paragraph from the news article. The lead paragraph, typically written by professional journalists, contains the 5W1H elements of the event (Who, What, When, Where, Why, How), and is a high-quality source of event descriptions.

[0056] Phase two involves image-text consistency filtering. A visual semantic similarity model is used to calculate the semantic similarity between the image and the text. The calculation formula is:

[0057] in, For the visual feature vector of the image, This is the semantic feature vector of the text. A similarity score threshold is set. (For SigLIP2, it is approximately 0.005), filtering out samples with inconsistent images and text. This stage can filter out about 80% of low-quality samples; for example, visual semantic similarity models can use SigLIP (Sigmoid Loss for Language Image Pre-Training) and CLIP (Constrastive Language-Image Pre-training).

[0058] Phase three involves the automatic generation of event annotations. A large multimodal model, such as GPT-4o or Gemini, is used to annotate the filtered image-text pairs. The input is an image I plus text T (including a summary and introduction), and the output is a list of events E. To improve annotation quality, a structured cue template is used, requiring the model to output a JSON-formatted list of events conforming to a predefined schema.

[0059] Phase four involves quality review and deduplication. The automatically generated annotations undergo quality review, including: format validation to check if the output conforms to the JSON Schema; integrity checks to verify the existence of required fields; deduplication based on image hashing and text similarity to remove duplicate samples; and anomaly filtering to remove samples with abnormal event counts.

[0060] Phase five involves training data transformation. The approved data is converted into a validator training format, including: positive samples using the original annotations as a list of correct events, and negative samples automatically generated using a perturbation strategy.

[0061] The dataset statistics are as follows: approximately 600,000 image-text pairs were originally collected, approximately 50,000 to 60,000 were filtered for consistency, approximately 30,000 to 40,000 were reviewed for quality, and approximately 20,000 high-quality samples were finally available.

[0062] The advantages of this method include: no need for manual annotation and low construction cost, large data scale covering diverse scenarios, high-quality introductory paragraphs with complete event elements, sustainable expansion and support for incremental updates.

[0063] Step 2: Event Schema Definition This invention employs a structured event representation schema to standardize the input and output formats for event extraction.

[0064] The event extraction output schema is defined as follows: Visual grounding information includes two fields: bounding box (bbox), which is a quadruple representing the position coordinates of an entity in the image, and confidence, which is a floating-point number between 0 and 1 representing the degree of confidence in the grounding.

[0065] An argument contains four fields: semantic role, which represents the role of the argument in the event (such as Agent, Patient, Location, Time, etc.); entity, which represents the name of the entity corresponding to the argument; text offset, which is an optional field that represents the start and end positions of the entity in the original text; and visual_grounding, which is an optional field that represents the position of the entity in the image.

[0066] The trigger word contains two fields: the trigger word text, which represents the keyword that triggers the event, and the text offset, which represents the start and end positions of the trigger word in the original text.

[0067] An event contains six fields: event_id (used to uniquely identify the event), event_class (indicating the general category of the event, such as Justice, Life, Conflict, etc.), event_subtype (indicating the specific type of the event, such as Arrest-Jail, Die, Attack, etc.), trigger (a structure similar to the Trigger structure described above), arguments (an array of Argument structures), and event_description (an optional field used to briefly describe the event content). The event list is an array of Event structures, representing all events extracted from a single text-image pair.

[0068] The validator feedback schema is defined as follows: MissingEvent contains three fields: event_class, event_subtype, and trigger word to be identified.

[0069] The WrongEventType contains three fields: the trigger word that classifies the error, the current event type (in the format Class:Subtype), and the correct event type.

[0070] Missing Argument contains three fields: the event trigger word for the missing argument, the role of the missing argument, and the entity text to be extracted.

[0071] The WrongArgumentRole contains three fields: the entity that was wrongly assigned the role, the current role (current_role), and the correct role (correct_role).

[0072] The VerifierFeedback contains five fields: a boolean value indicating whether the extraction result is correct ("correct"), a list of missing events ("missing_events") (an array of MissingEvents), a list of error types ("wrong_types") (an array of WrongEventTypes), a list of missing arguments ("missing_arguments") (an array of MissingArguments), and a list of incorrect roles ("wrong_roles") (an array of WrongArgumentRoles). All list fields use empty arrays to indicate that there are no errors in that category.

[0073] Step 3: Constructing positive and negative samples The process of constructing validator training data is as follows: Figure 3 As shown. When constructing positive samples, the validator should output "Correct". An example of a positive sample is as follows: the text is "Two Chicago police officers take a man into custody", the image is a photo of an arrest scene, the event list contains an event type "Justice:Arrest-Jail" and participant roles (Agent is officers, Person is man), and the label is "Correct".

[0074] When constructing negative samples, the validator should output specific errors. There are three types of negative samples: Negative sample type 1 involves removing perturbations, with a perturbation ratio ranging from 10% to 30%. Example: Based on the above positive samples, remove the officers of the Agent role and only keep the man of the Person role. In this case, the label is "Missing argument:(Agent,officers)".

[0075] Negative sample type 2 is a replacement perturbation, with a perturbation rate of 5% to 15%. Example: If the role of officers is incorrectly replaced from Agent to Patient, the label will be "Wrong role: officers should be Agent, not Patient".

[0076] Negative sample type 3 is hallucination perturbation, with a perturbation rate of 10% to 20%. Example: When an instrument role of "police car" is incorrectly added even though the police car does not exist in the image, the label will be "Hallucination: 'police car' not visible in image".

[0077] Step 4: Validator Training Strategy This invention provides two validator training strategies, which are suitable for different resource conditions and performance requirements.

[0078] Strategy 1: LoRA (Lo Efficient Parameter Tuning) This strategy is suitable for scenarios with limited computing resources or those requiring rapid iteration, and its architecture is as follows: Figure 4 As shown in the figure, the input-model-output flow and LoRA fine-tuning structure of the validator are illustrated: the input layer contains images, text, and a list of candidate events, which are then processed by a visual encoder and a projection layer before entering Qwen3. VL The model is an 8B language model. The model layers apply LoRA adaptation to the attention and feedforward sub-modules in multi-layer Transformers. The output layer generates structured validation feedback (such as Correct and Missing) from the decoder. Parameter statistics and LoRA configuration are shown in the lower right corner, and the bottom illustrates the insertion position of the LoRA adapter and the lightweight incremental update mechanism. Load the base model Qwen3-VL-8B (or other models of similar scale) and apply the LoRA adapter. The LoRA configuration parameters are as follows: rank r is set to 16 (8 to 16 can be used for smaller models); scaling factor lora_alpha is set to 32, typically twice r; target_modules is set to all-linear, applying LoRA to all linear layers; dropout probability lora_dropout is set to 0.05; bias is set to none, no bias training; task_type is set to CAUSAL_LM, i.e., a causal language model.

[0079] The training parameters are set as follows: batch size is 4 to 8, and the learning rate is 1 to 10. -4 Up to 1×10 -5 The training run consists of 5 to 10 rounds and stops early based on validation set performance. The optimizer used is AdamW, and the learning rate is scheduled using either Cosine Annealing or Linear Warmup.

[0080] The training objective is to minimize the negative log-likelihood loss from sequence to sequence:

[0081] in The output sequence of the validator. Indicates the preceding A token.

[0082] The advantages of the LoRA strategy include: low training memory requirements (approximately 24GB), short training time (approximately 2 to 4 hours for a single A100 card), rapid iterative experimentation, and suitability for resource-constrained scenarios.

[0083] Strategy 2: Fine-tuning RLHF (Reinforcement Learning from Human Feedback) / GRPO (Group Relative Policy Optimization) using reinforcement learning This strategy is suitable for scenarios where optimal performance is sought and computational resources are sufficient. Building upon LoRA fine-tuning, reinforcement learning methods are further employed to optimize the validator's decision-making capabilities.

[0084] The reinforcement learning framework employs either GRPO or PPO (Proximal Policy Optimization) algorithms. The reward function is designed as follows: a positive reward is given for correctly identifying errors. Negative rewards will be given for cases of missed or incorrect reporting. A negative reward will be given for cases where a correct sample is falsely reported as an error. The specific reward value is set as follows: =1.0, =0 (the penalty for missing a report is more severe because missing a report can lead to the propagation of errors). =0.

[0085] The reinforcement learning training parameters are set as follows: batch size is 2 to 4, and learning rate is 1×10⁻⁶. -6 Up to 5×10 -6 It is an order of magnitude lower than supervised learning, with 3 to 5 training rounds, and the KL divergence coefficient is... Set to 0.01 to 0.1 to control the policy update magnitude, and set the sampling temperature to 0.7 to 1.0.

[0086] The advantages of reinforcement learning strategies include more accurate validator decisions, the ability to learn finer-grained error judgment criteria, and better performance in complex scenarios. The trade-off is a significant increase in training costs: training memory requirements are approximately 2 to 3 times that of LoRA (approximately 48 to 72 GB), training time is approximately 10 to 15 times that of LoRA (approximately 24 to 48 hours for a single A100 card), and more hyperparameter tuning is required.

[0087] As shown in Tables 1 and 2, on the experimental small-scale dataset, the event extraction based on the reinforcement learning strategy outperforms the event extraction based on the LoRA strategy in all metrics. However, the LoRA strategy has a lower training cost, requiring approximately 24GB of GPU memory and 3 hours of training time, while the reinforcement learning strategy has a higher cost, requiring approximately 36 hours of training time and approximately 80GB of GPU memory.

[0088] Strategy selection recommendations: For rapid prototyping, resource-constrained scenarios, or applications with high real-time requirements, the LoRA strategy is recommended; for critical applications that pursue optimal performance, have sufficient computing resources, or have extremely high accuracy requirements, the reinforcement learning strategy is recommended; incremental training can also be adopted, first using LoRA to quickly obtain a baseline model, and then using reinforcement learning for further optimization.

[0089] Step 5: Monitoring the training process Key monitoring metrics include: training loss, validation accuracy, F1 score (for error detection), recall (how many real errors the validator can identify), and precision (how many of the errors reported by the validator are real).

[0090] Detailed process of iterative reasoning The complete process of iterative verification is as follows: Figure 2 As shown. The multimodal input encoding method of the validator is as follows. Figure 5 As shown, the image features, text descriptions, and list of events to be verified are fused and encoded before being input into the model.

[0091] refer to Figure 2 and Figure 5 The system also provides the input encoding specifications and iterative closed-loop process for the validator. The input includes text descriptions, corresponding images, and a candidate event list: the corresponding images are processed by a visual encoder to obtain visual feature sequences; the text descriptions and candidate event lists are tokenized to obtain text sequences and event sequences; the visual feature sequences, text sequences, and event sequences are concatenated according to a preset format to form a complete input sequence, which is then fed into the multimodal model; the multimodal model outputs structured verification feedback, which includes at least a correctness flag and fields for missing events, error types, missing arguments, and error roles. This feedback serves as the basis for iterative control: when the feedback is deemed correct, the final event list is directly output; when the feedback is deemed incorrect, the feedback is injected into a prompt template to form the input for the next round, driving the generator to generate an improved event list and enter the next round of verification, until the termination condition is met or the maximum number of iterations is reached. This design achieves a closed-loop collaboration of "encoding specifications—verification feedback—iterative correction".

[0092] Input example: Text T is "Police arrested the suspect near the blue car.", and image I is a street photo showing a police arrest scene.

[0093] Round 0 (Initial Generation): Using the generator (e.g., GPT-4o) Generate an initial event list This includes the unique event type "Justice:Arrest-Jail" and the participant roles: Agent is Police, Person is suspect. The problem is: Location information is missing (near the blue car).

[0094] Round 1 Validation: Input text, images, and a list of events into the validator. Verifier output feedback for "Missing argument: (Location, near the blue car)".

[0095] Round 2 Generation: Build Enhancement Hints This includes the results of previous attempts and the problems found, requesting the generator to add missing location information. The generator outputs an improved list of events. The Location role has been added as "near the blue car" to the existing one.

[0096] Third round of validation: Input the validator again, output... The value is "Correct". Validation passed, iteration terminated, and the function returned. As the final result.

[0097] Iteration statistics: The total number of iterations is 2, the generator is called 2 times, the validator is called 2 times, and the total time is about 3 to 5 seconds, depending on the speed of the generator and the validator.

[0098] Preferred technical parameters Based on experimental verification, the recommended technical parameters are as follows: LoRA rank Setting it to 16 balances performance and parameter count, suitable for 8B-level models; setting the LoRA target module to all-linear yields better results when applied to all linear layers; setting the learning rate to 2×10 -4 Experiments show that 1×10 -4 Up to 5×10 -4 The optimal ratio is between 8 and 5; the batch size is set to 8, which is limited by the video memory and can be supported by a single A100 card; the maximum number of iterations (epochs) is set to 5, and experiments show that the average convergence time is 2.3, so 5 is sufficient; the number of training epochs is set to 10, as too many will lead to overfitting.

[0099] Example 2: Offline Batch Verification Mode In some application scenarios, users may need to perform offline verification of large batches of event extraction results, rather than iterating in real time. This embodiment describes a batch verification mode.

[0100] Application scenarios include: quality auditing of news event databases, post-processing of existing event extraction results, and automatic annotation of large-scale datasets.

[0101] The implementation involves defining a batch validation function. Inputs include a list of text, images, events, and a validator model. Outputs include a feedback list, indices of correct and incorrect samples, and error messages. The validator is called for each sample, and correct and incorrect samples are categorized and statistically analyzed based on the feedback results.

[0102] Example 3: Multi-verifier integration To further improve the reliability of verification, this embodiment adopts an integration strategy of multiple verifiers.

[0103] The implementation involves training multiple validators (e.g., 3 to 5), each using a different random seed, slightly different training data splits, and different LoRA configurations (r=8, 16, 32).

[0104] The ensemble strategy is as follows: collect feedback from all validators, count the number of "Correct" outputs, and use a majority voting mechanism. If more than half of the validators consider it correct, output "Correct" and calculate the confidence score; otherwise, merge all error feedback and return the error that occurred most frequently and its confidence score.

[0105] The advantages of this method include: reducing the false positive rate of a single validator, improving the robustness of the validation, and providing a confidence score.

[0106] Example 4: Real-world application scenario - News event analysis system The application context is that a news organization needs to automatically extract event information from hundreds of news articles and accompanying images published every day to build an event knowledge base.

[0107] The system deployment scheme is as follows: the generator uses the GPT-4o API for cloud calls, and the verifier is deployed on a local server using a Qwen3-VL-4B with LoRA, with the hardware being a single server equipped with an NVIDIA RTX 4090.

[0108] The workflow includes: scraping newly published news articles and images every hour, using a generator to initially extract an event list, using a validator to perform batch verification, iteratively optimizing samples that fail verification, and storing verified events into a knowledge base.

[0109] The actual results are as follows: On another self-built test dataset based on Chinese news, the average processing time for each news article is 5 to 8 seconds, and the event extraction recall rate reaches 83.9%, which is 11.2 percentage points higher than the 72.7% without verification.

[0110] To further evaluate the practical effectiveness of the validator in more detail, regarding the test set partitioning on the aforementioned perturbation dataset, this invention implemented and analyzed the results of two evaluation schemes: Offline validation: The images, texts, and positive / negative samples in the dataset are fed into the validator to evaluate the validator's judgment on sample completeness and error detection performance. The final results are shown in Table 1.

[0111] End-to-end online verification: The images and text in the dataset are fed into an end-to-end iterative verification system composed of specific validators. The final output of the system as a whole is evaluated and compared with the dataset annotation. The final results are shown in Table 2.

[0112] Table 1

[0113] Table 2

[0114] In Table 1, Cohen's Kappa > 0 indicates better performance than random guessing, and Cohen's Kappa > 0.6 indicates strong performance in decision-making tasks. In Table 2, “—” indicates no iteration.

[0115] The offline event extraction evaluation results are as follows: The accuracy of the unadjusted baseline was 71.7%, Cohen's Kappa was -0.04 (almost entirely random guessing), and the F1 score for error detail matching was 45.4%; the accuracy of the LoRA fine-tuned validator of this invention was 91.2%, Cohen's Kappa was 0.688, and the F1 score for error detail matching was 84.5%; the accuracy of the reinforcement learning fine-tuned validator of this invention was 92.2%, Cohen's Kappa was 0.701, and the F1 score for error detail matching was 86.4%. The Cohen's Kappa metric, which excludes the influence of random guessing, improved from negative values ​​to 0.688 and 0.701, indicating that the validator truly learned effective validation and discrimination capabilities. The improvement in the F1 score also indicates that the validator began to notice more features of the input and incorporate them into the problem-solving process.

[0116] The end-to-end event extraction evaluation results are as follows: the pure GPT-4o event recall rate is 71.1% and the argument recall rate is 62.9%; the GPT-4o plus fixed resolution validator event recall rate is 68.3%, the argument recall rate is 59.8%, and the average number of iterations is 2.9. The poor validator performance resulted in the overall system performance being weaker than that without a validator framework; the cross-modal validation method (LoRA training) of this invention has an event recall rate of 81.9%, an argument recall rate of 74.3%, and an average number of iterations of 2.4. The cross-modal verification method (reinforcement learning training) of this invention has an event recall rate of 84.1%, an argument recall rate of 77.3%, and an average of 2.2 iterations.

[0117] Key findings based on the method described in this invention include: First, the advantages of cross-modal validation. Compared to plain text validation, cross-modal validation offers performance improvements thanks to enhanced information, which helps the iterative validation system make more accurate judgments.

[0118] Second, further improvements in reinforcement learning. Compared to LoRA training (event recall rate of 81.9%), reinforcement learning training (event recall rate of 84.1%) improved the main event recall performance by approximately 2.2%. Third, iteration efficiency. The LoRA validator converges on average in 2.4 iterations, while the reinforcement learning validator converges on average in 2.2 iterations; 87% of the samples are completed within 3 iterations, and a very small number of samples (less than 5%) require 4 to 5 iterations.

[0119] Fourth, the trade-off between resource efficiency and performance. LoRA training requires approximately 24GB of VRAM and takes about 3 hours, making it suitable for resource-constrained scenarios; reinforcement learning training requires approximately 60GB of VRAM and takes about 36 hours, making it suitable for scenarios seeking maximum performance. The two strategies can be flexibly chosen based on actual needs.

[0120] Based on the above-mentioned technical concept, the present invention also provides a multimodal event extraction system based on iterative visual verification, including an input preprocessing module, a generator module, a verifier module, an iteration control module, and a post-processing module. The input preprocessing module is used to encode the text description, preprocess the image corresponding to the text description, and load the initial prompt template; The generator module is based on a large multimodal model, receives text, images, and cue templates, and outputs a list of structured events; The validator module is based on a visual language model with efficient parameter fine-tuning. It receives text descriptions, corresponding images, and an initial event list, performs cross-modal consistency checks, and outputs structured feedback containing verification results and specific error information. The iteration control module is used to manage prompt template updates, termination condition determination, and historical records; The post-processing module is used to format and score the quality of the final event list.

[0121] The above embodiments demonstrate the complete implementation scheme and practical application effects of the present invention. In summary, the technical features of the present invention include: First, iterative verification closed loop. Through a cycle of generation, verification, feedback, and improvement, the quality of event extraction is continuously improved.

[0122] Second, flexible training strategies. It provides two training strategies: LoRA (Locally Aligned Parameters) and reinforcement learning, which are suitable for resource-constrained scenarios and scenarios that pursue ultimate performance, respectively.

[0123] Third, cross-modal consistency checks. Detecting illusions and errors by combining textual and image information.

[0124] Fourth, provide specific error feedback. Offer detailed corrective suggestions, rather than simply judging whether it's correct or incorrect.

[0125] This invention provides a flexible resource-performance trade-off while maintaining high accuracy, and has good practical value and prospects for promotion.

[0126] In addition, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the multimodal event extraction method based on iterative visual verification described in the present invention.

[0127] The present invention can also provide a computer device, including a processor and a memory, wherein the memory is used to store a computer executable program, the processor reads the computer executable program from the memory and executes it, and the processor can implement the multimodal event extraction method based on iterative visual verification described in the present invention when executing the computer executable program.

[0128] The computer device may be a laptop, tablet, desktop computer, or workstation.

[0129] The processor can be a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or an off-the-shelf programmable gate array (FPGA).

[0130] The memory described in this invention can be an internal storage unit of a laptop, tablet, desktop computer, or workstation, such as memory or hard disk; or it can be an external storage unit, such as a portable hard disk or flash memory card.

[0131] Computer-readable storage media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media can include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. Random access memory can include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

[0132] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A multimodal event extraction method based on iterative visual verification, characterized in that, Includes the following steps: S1, Input text description T and corresponding images I Using a large multimodal model as the generator M gen Generate an initial event list E 0, the event list contains zero or more events, each event containing an event type, trigger word, argument list and optional visual positioning information; S2, list the current events E k Text description T and corresponding images I Input parameters adjust the visual validator pre-trained or reinforcement learning trained The visual verifier checks the cross-modal consistency of the event list and outputs structured feedback. F k The structured feedback F k It includes a validation result field and specific error information, which is divided into four categories: missing event, incorrect event type, missing argument, and incorrect argument role; S3, if structured feedback F k If the verification result is successful, then output the current event list. E k This serves as the final result and terminates the iteration. If structured feedback F k If it contains error messages, then a prompt will be built. P k+1 Inject the specific feedback from the visual validator into the prompt template; S4, use full prompts P k+1 Call generator M gen Generate an improved event list E k+1 Update the iteration counter k = k +1; S5, repeat S2 to S4 until verification is passed and the maximum number of iterations is reached. K max Or, if the convergence condition is met, the final list of events can be obtained. E final .

2. The multimodal event extraction method based on iterative visual verification according to claim 1, characterized in that, The visual verification device The training includes the following steps: Constructing positive samples ( T , I , E , y ),in y This indicates that the verification has passed; Negative samples are generated using the following perturbation strategy, assuming the original event list is... E The list of events after the disturbance is as follows E’ : Perturbation removal: Randomly remove some events or arguments. E'=E\ { e i }, generates a missing class error; Replacement perturbation: Randomly replaces event type or argument role, generating type errors; Hallucination Disturbance: Adds hallucination event elements that are inconsistent with the image content. An error occurred while generating an illusion. The training dataset obtained after the above three perturbations has approximately four times the number of entries as the original dataset, including one positive sample and three negative samples; Adjusting parameters to train the validator, assuming the original model parameters are... The adaptation parameters are: ,but: The training objective is to minimize the negative log-likelihood loss from sequence to sequence: in The output sequence of the validator. Indicates the preceding A token.

3. The multimodal event extraction method based on iterative visual verification according to claim 2, characterized in that, The parameters for training the validator are adjusted as follows: The number of trainable parameters accounts for 0.1%-2% of the original model's parameters; The memory requirements for validator training have been reduced to 15%-30% of the original full parameter fine-tuning.

4. The multimodal event extraction method based on iterative visual verification according to claim 2, characterized in that, The visual validator is trained using reinforcement learning, including the following steps: Based on parameter adjustment, reinforcement learning is used to further optimize the decision-making ability of the visual validator; Design reward function R Positive rewards will be given for correctly identifying errors. r correct Negative rewards will be given for cases of missed or incorrect reporting. r miss A negative reward will be given for cases where a correct sample is falsely reported as an error. r false ,in ; The visual validator parameters are updated using a policy optimization algorithm, with the optimization objective being to maximize the expected cumulative reward. The amplitude is updated using a KL divergence constraint control strategy.

5. The multimodal event extraction method based on iterative visual verification according to claim 1, characterized in that, Prompt Template P k+1 Build as: in, P base Basic prompt template, E k This is the current event list. F k This is the feedback from the verifier; Instruction is the correction instruction.

6. The multimodal event extraction method based on iterative visual verification according to claim 1, characterized in that, The input encoding method of the visual verifier is as follows: The image features V, text description T, and list of events to be verified are fused and encoded using a multimodal method. Let the image features be... Text features are The characteristics of the list of events to be verified are: The input after fusion is: Visual features, text features, and features of the event to be verified are used as inputs to the cross-attention module. By aligning and fusing the three features through cross-attention, a joint representation is obtained.

7. The multimodal event extraction method based on iterative visual verification according to claim 1, characterized in that, The structured feedback output by the visual verifier F k Includes the following fields: The validation result field indicates whether the event list is correct; A list of missing events records events that should exist but are missing. Error type list, recording errors categorized by event type; A list of missing arguments, recording arguments that should exist but are missing; Error Role List: Records incorrect assignment of argument roles.

8. A multimodal event extraction system based on iterative visual verification, characterized in that, It includes an input preprocessing module, a generator module, a validator module, an iteration control module, and a post-processing module; The input preprocessing module is used to encode the text description, preprocess the image corresponding to the text description, and load the initial prompt template; The generator module is based on a large multimodal model, receives text, images, and cue templates, and outputs a list of structured events; The validator module is based on a visual language model with efficient parameter fine-tuning. It receives text descriptions, corresponding images, and an initial event list, performs cross-modal consistency checks, and outputs structured feedback containing verification results and specific error information. The iteration control module is used to manage prompt template updates, termination condition determination, and historical records; The post-processing module is used to format and score the quality of the final event list.

9. A computer device, characterized in that, It includes a processor and a memory, the memory being used to store a computer-executable program, the processor reading part or all of the computer-executable program from the memory and executing it, and the processor executing part or all of the computer-executable program is able to implement the multimodal event extraction method based on iterative visual verification as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the multimodal event extraction method based on iterative visual verification as described in any one of claims 1-7.

Images