Multi-modal fine-grained instruction fine-tuning data construction method based on reverse verification

Abstract

The application discloses a kind of based on reverse verification's multimodal fine-grained instruction fine-tuning data construction method, comprising: obtaining multimodal original data, the structured analysis of multimodal original data is obtained Verifiable visual evidence;Based on visual evidence generation fine-grained instruction, and then generate candidate answer set;For the candidate answer set of each fine-grained instruction, combined with verifiable visual evidence executes reverse verification, to obtain reverse verification score and failure reason label;To the candidate answer that passes through reverse verification, execute multi-criteria scoring and obtain multi-criteria score, and determine quality score based on reverse verification score and multi-criteria score;Filter based on quality score, output multimodal training data for instruction fine-tuning.The present application can construct traceable, verifiable, controllable difficulty and noise multimodal fine-grained instruction fine-tuning data without significantly increasing labor cost, improve the compliance ability and robustness of multimodal large model to fine-grained instruction.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and multimodal learning technology, and in particular to a method for constructing multimodal fine-grained instruction-based fine-tuning data based on reverse verification. Background Technology

[0002] Multimodal large-scale models excel in tasks such as image understanding, video understanding, and visual question answering, but they still have shortcomings in stable adherence to fine-grained instructions, evidence alignment, and hallucination suppression. Traditional multimodal instruction fine-tuning data typically relies on manual annotation or weak rule generation, which suffers from three typical problems: First, visual evidence is untraceable, and the model easily learns descriptions that are "seemingly reasonable but unverifiable"; second, sample noise and hallucinations are difficult to detect systematically, leading to training signal contamination; and third, the difficulty distribution is uncontrollable, with too many "too easy / too difficult" samples weakening training efficiency. Therefore, a scalable, verifiable, and controllable method for constructing multimodal fine-grained instruction fine-tuning data with manageable noise and difficulty is needed. Summary of the Invention

[0003] The purpose of this invention is to provide a method for constructing multimodal fine-grained instruction fine-tuning data based on reverse verification, so as to solve the problem that existing multimodal instruction data is difficult to balance in terms of fine-grained alignment, visual illusion, weak reference, noise filtering and distribution balance.

[0004] According to a first aspect of the embodiments of this application, a method for constructing multimodal fine-grained instruction-based fine-tuning data based on reverse verification is provided, characterized in that it includes:

[0005] Acquire multimodal raw data, which includes at least images or videos, as well as related text information and metadata;

[0006] The multimodal raw data is structured and parsed by calling at least one domain model to obtain verifiable visual evidence;

[0007] Based on the visual evidence, fine-grained instructions are generated, and a set of candidate answers is generated according to the fine-grained instructions;

[0008] For each set of candidate answers to a fine-grained instruction, reverse verification is performed in conjunction with the verifiable visual evidence to obtain a reverse verification score and a failure reason label. Candidate answers that pass the reverse verification are marked as positive examples and enter the subsequent scoring process, while candidate answers that fail are marked as negative examples and do not enter the subsequent scoring process. The negative examples are used as control answers for preference pair derivation.

[0009] A multi-criteria scoring method is applied to the candidate answers that pass the reverse validation to obtain a multi-criteria score, and a quality score is determined based on the reverse validation score and the multi-criteria score.

[0010] Based on the quality score, the data is filtered and output as multimodal training data for instruction fine-tuning.

[0011] According to a second aspect of the embodiments of this application, a multimodal fine-grained instruction fine-tuning data construction apparatus based on reverse verification is provided, comprising:

[0012] The acquisition module is used to acquire multimodal raw data, which includes at least images or videos and related text information and metadata.

[0013] The parsing module is used to call at least one domain model to perform structured parsing of the multimodal raw data to obtain verifiable visual evidence;

[0014] The candidate generation module is used to generate fine-grained instructions based on the visual evidence, and to generate a set of candidate answers based on the fine-grained instructions.

[0015] The reverse verification module is used to perform reverse verification on the candidate answer set for each fine-grained instruction, combined with the verifiable visual evidence, to obtain a reverse verification score and a failure reason label; candidate answers that pass the reverse verification are marked as positive examples and enter the subsequent scoring, candidate answers that fail are marked as negative examples and do not enter the subsequent scoring, and the negative examples are used as control answers for preference pair derivation;

[0016] The quality scoring module is used to perform multi-criteria scoring on candidate answers that have passed reverse validation to obtain multi-criteria scores, and to determine a quality score based on the reverse validation scores and the multi-criteria scores.

[0017] The filtering module is used to filter data based on the quality score and output multimodal training data for instruction fine-tuning.

[0018] According to a third aspect of the embodiments of this application, an electronic device is provided, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to perform the method as described in the first aspect.

[0019] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0020] Compared with the prior art, the embodiments of the present invention have at least the following beneficial effects:

[0021] (1) Verifiable and traceable supervision signals: Verifiable visual evidence is obtained by performing structured parsing on multimodal raw data, and objects / attributes / relationships / textual facts are bound to masks, bounding boxes, text boxes or relation edges with evidence indexes, so as to achieve the traceability of "answer-evidence" and solve the problem of unclear and uninterpretable supervision basis in multimodal instruction data.

[0022] (2) Noise in candidate answers can be automatically eliminated and attributed: Reverse verification is performed on the candidate answer set for each fine-grained instruction, and the reverse verification score and failure reason label are output. It can identify semantic deviation, missing / mismatched evidence, templated or pseudo-relevant answers, and solve the problem that training noise caused by illusion, weak reference and label mismatch is difficult to detect.

[0023] (3) Simultaneously consider "evidence" and "expression quality": Perform multi-criteria scoring on candidate answers that pass reverse verification and combine them with the reverse verification score to determine the quality score. This ensures that sample selection is subject to both visual evidence and quality constraints such as factuality, completeness, logical consistency, and clarity, thus solving the problem of "logically reasonable but visually untenable" selection based solely on linguistic fluency or a single indicator.

[0024] (4) Positive and negative comparisons can be directly derived to support preference learning / contrast learning: answers that pass the reverse verification are marked as positive examples and enter the subsequent scoring, while those that fail are marked as negative examples and retained as comparisons for preference pair derivation. High and low quality comparison samples can be stably formed under the same instruction, solving the problems of high cost of constructing positive and negative samples and unstable comparisons in preference learning training.

[0025] (5) More effective filtering, reduced redundancy and improved data utilization: Filtering based on quality scores and outputting training data for fine-tuning instructions can reduce the number of homogeneous low-information samples entering the training set and prioritize the retention of high-value sample groups, thereby improving training efficiency and model robustness. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0027] Figure 1 This is a schematic diagram illustrating the overall flow of a multimodal fine-grained instruction-based data construction method based on reverse verification, according to an exemplary embodiment.

[0028] Figure 2 This is a structural block diagram illustrating a multimodal fine-grained instruction fine-tuning data construction device based on reverse verification, according to an exemplary embodiment.

[0029] Figure 3This is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0030] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0031] Figure 1 This is a flowchart illustrating a multimodal fine-grained instruction-based data construction method using reverse verification, according to an exemplary embodiment. (Reference) Figure 1 The multimodal fine-grained instruction fine-tuning data construction method based on reverse verification provided in this embodiment of the invention may include the following steps:

[0032] S1: Obtain multimodal raw data, which includes at least images or videos and related text information and metadata;

[0033] Specifically, the original multimodal dataset can be represented as ,in Represents the original dataset (subscript) (meaning "original") Represents an image. Indicates a video (can be empty). Text information related to an image or video (such as title, description, etc.). This represents metadata (shooting device, timestamp, source domain, question type tags, etc.). For example, taking a "short video showcasing e-commerce products" as... (For example, product unboxing / rotation display videos filmed with a mobile phone), its For product titles / descriptions or video caption text (such as "Black genuine leather jacket with metal zipper" or "Brand: XX, Model: YY"), its This includes data collection time, device model, source platform / target category, etc. In this embodiment, the final output dataset may include a supervised fine-tuning dataset. Preferences for datasets and optional benchmark sets ,in This indicates a fine-tuning of oversight. Indicates preference, This indicates the evaluation benchmark.

[0034] For ease of subsequent verification, this embodiment records the structured analysis result of each sample as follows: ,in This represents the structured parsing result. For the candidate region set, This is a scene image. This is a collection of OCR text evidence. Meta-information for recording scores, reasons for failure, perturbation methods, etc.

[0035] To achieve traceable verification, this embodiment assigns an evidence index to each element in the structured evidence. The evidence index is used to indicate the evidence element corresponding to a certain object, attribute, relationship, or text fact, such as mask number, bounding box number, text box number, or relationship edge number. At the same time, a failure reason label is recorded for each candidate answer. The failure reason label is used to indicate the failure of semantic consistency verification, evidence alignment verification, or counterfactual sensitivity test, so as to perform positive and negative example division, quality statistics, and preference pair derivation for candidate answers in subsequent steps.

[0036] The reasons and advantages of this design are as follows: First, incorporating textual information and metadata into the S1 stage provides the necessary context for subsequent task type sampling, instruction generation, and verification, reducing the proportion of weakly correlated and unverifiable samples; second, it utilizes a unified structured format. Furthermore, the predefined evidence index / failure reason labels allow subsequent analysis, verification, and scoring results to be traceably written back to the sample, achieving closed-loop management from "visual evidence—instruction—response—verification—screening—export"; thirdly, it can simultaneously support and The structure allows validated answers to be used as positive examples in supervised fine-tuning, while unvalidated answers are used as negative examples for preference pairs, thereby improving data utilization efficiency and training robustness.

[0037] S2: Invoke at least one domain model to perform structured parsing on the multimodal raw data to obtain verifiable visual evidence;

[0038] Specifically, in this embodiment, taking the "e-commerce product image / short video" scenario as an example: the multimodal raw data includes a main product image or a short video showcasing the product's appearance, and the image contains packaging text (which can be recognized by OCR), product color / material attributes, and the spatial relationships of multiple components. For image samples, the image is directly used as input; for video samples, frame sampling is first performed to obtain a video frame sequence. (For example, sampling at a fixed frame rate or by shot boundary / keyframe strategy), and performing structured parsing on each frame separately; the domain models can be called individually or in combination, and the outputs of each model are uniformly backfilled into the structured parsing results. This is used for subsequent instruction generation and reverse verification calls. The domain model is selected from: segmentation models, open vocabulary detection / localization models, object detection models, referential localization models, character recognition models, depth / normal estimation models, human keypoint / pose estimation models, etc.

[0039] Specifically, the multimodal raw data is subjected to structured parsing to obtain verifiable visual evidence, including:

[0040] S21: Generate instance-level masks and bounding boxes for the image or video frame to form a candidate instance set;

[0041] Specifically, taking e-commerce product images / short videos as an example, an image or video frame contains multiple separable instances such as the product itself, packaging box, hang tag, and brand logo. This embodiment calls a segmentation model to perform instance-level segmentation on the image or video frame, obtaining a set of instance masks. The bounding box is obtained from the bounding rectangle of each mask. Assign a unique instance number to each instance. This forms a set of candidate instances. For video frame sequences, cross-frame instances can be further associated based on mask IoU or appearance features to form a trajectory number for the same instance (which can be written into meta) to support subsequent timing tasks (e.g., "at which seconds does the tag appear" or "when is the packaging box opened").

[0042] S22: Extract the category information and attribute information of each instance based on the candidate instance set;

[0043] Specifically, taking e-commerce product images / short videos as an example, candidate instances may include "the product itself, packaging box, hang tag, bottle cap / zipper, brand logo area," etc. This embodiment defines each candidate instance area (by...) or (After cropping) The open vocabulary detection / localization model or object detection model is called to output the category label and confidence score (e.g., category is "shoes / clothes / bottled beverage / packaging box / tag / logo"), and the attribute recognition branch or vision-language model is called to output the attribute words and their confidence scores (e.g., color "black / white", material "leather / metal / plastic", status "open / unopened", action "pick up / open", etc.). The category information and attribute information of each instance are recorded as follows: and with instance number Binding storage.

[0044] S23: Determine the relationships between instances based on the instance pairs in the candidate instance set, obtain a set of relationship information, and construct a scene graph using candidate instances as nodes and relationship information as edges;

[0045] Specifically, taking e-commerce product images / short videos as an example, common relationships include "logo on the packaging box," "hang tag attached to clothing," "bottle cap covering the bottle opening," and "product being held in hand," etc. This embodiment provides examples... It calculates geometric relationship features (relative position, containment, overlap, distance, etc.) and can combine them with relationship classification models or vision-language models to output relationship types (e.g., left / right / up / down / containment / contact / holding / pointing, etc.) and their confidence scores, forming a set of relationship triples. Construct a scenario graph using candidate instances as nodes and relation triples as edges. ,in , .

[0046] S24: Generate a traceable evidence index for the category information, attribute information and relationship information, which is used to indicate the mask number, bounding box number, text box number or relationship edge number of the category information, attribute information or relationship information and the candidate instance set.

[0047] Specifically, taking e-commerce product images / short videos as an example, product packaging may contain text areas such as brand names and specifications (OCR evidence), and relationships such as "logo on the packaging box" and "bottle cap covering the bottle opening" need to be traceable and locating. This embodiment assigns a mask number to each instance of evidence element. With bounding box number And assign relationship edge number to each relationship edge. For text blocks output by the character recognition model, assign text box numbers. and form Based on this, evidence index mappings are established for category information and attribute information respectively, for example... , Establish evidence index mappings for relational information, for example... ; Establish evidence index mappings for textual facts, for example This allows any category / attribute / relationship / textual fact to be traced back to the corresponding visual evidence element.

[0048] S25: Based on candidate instances, category information, attribute information, relationship information, scene graph, and evidence index, form verifiable visual evidence;

[0049] Specifically, taking e-commerce product images / short videos as an example, candidate instances such as "product body / packaging / tag / logo" and their categories and attributes, as well as relationships such as "contains / covers / connects / is located" are uniformly organized with OCR text evidence to facilitate the subsequent generation of evidence-based instructions and responses and their execution for verification. This embodiment sets the candidate instance set... Category information set Attribute information set Relationship information set Scene diagram OCR text evidence set Furthermore, the evidence index set is uniformly encapsulated into verifiable visual evidence and written into the structured parsing results. In this context, verifiability is reflected in the fact that any subsequently generated atomic fact can be located to a specific mask / box / text box / relational edge through the evidence index, and its reliability is further supported by the confidence level output by the corresponding domain model.

[0050] S3: Generate fine-grained instructions based on the visual evidence, and generate a set of candidate answers based on the fine-grained instructions; this step includes the following sub-steps:

[0051] S31: Select or sample task types from a predefined task set, and generate fine-grained instructions based on the scene graph and the evidence corresponding to the evidence index;

[0052] Specifically, taking an e-commerce product image and its title description as input, the image contains brand text and product attributes, used to generate fine-grained instructions such as "read partial text / identify color and material / compare quantity". This is derived from a predefined task set. Select or sample task type Based on the structured analysis results Scene diagram And the fine-grained instructions for generating evidence elements indicated by the evidence index. The task types include at least one or more of the following: attribute recognition, relationship understanding, counting and comparison, referential resolution, local text reading, temporal event localization, and cross-perspective consistency checking. When generating fine-grained instructions, the task type is considered. Select target instance nodes and relation edges from the scene graph, or locate target mask numbers / boundary box numbers / text box numbers / relation edge numbers from the evidence index, and organize them into instruction constraints to form the fine-grained instructions.

[0053] The task types include at least one or more of the following: attribute recognition, relationship understanding, counting and comparison, referential resolution, local text reading, time-series event localization, and cross-perspective consistency checking.

[0054] S32: Generate at least one candidate answer for each fine-grained instruction to form a set of candidate answers corresponding to the fine-grained instruction;

[0055] Specifically, taking e-commerce product images / short videos as an example, fine-grained instructions This can be configured to retrieve details such as "read the brand name on the packaging," "what color / material the product is," and "whether the logo is on the packaging box." It can be tailored to each fine-grained instruction. Generate a set of candidate answers ,in Represents the set of candidate answers. Indicates the first One candidate answer, Candidate index, This indicates the number of candidate answers. The set of candidate answers is generated from at least one of the following sources: first, a symbolic solver based on the visual evidence. First, candidate answers that can be traced through an evidence index are directly generated. Second, one or more multimodal generation models are invoked to sample the fine-grained instructions and generate diverse candidate answers to form the candidate answer set required for subsequent reverse verification and positive / negative comparison. To improve coverage, the same fine-grained instructions can be sampled repeatedly, and the candidate answers can be deduplicated and normalized.

[0056] S4: For each set of candidate answers to a fine-grained instruction, perform reverse verification in conjunction with the verifiable visual evidence to obtain a reverse verification score and a failure reason label; mark candidate answers that pass the reverse verification as positive examples and proceed to subsequent scoring, and mark candidate answers that fail as negative examples and do not proceed to subsequent scoring, and use the negative examples as control answers for preference pair derivation; this step includes the following sub-steps:

[0057] S41: Based on the candidate answers, reverse the verification question and perform a consistency check with the fine-grained instructions to obtain the consistency check result;

[0058] Specifically, taking e-commerce product images / short videos as an example, fine-grained instructions Possibly "reading the brand name on the packaging," one candidate answer. The answer is "Brand X". In this embodiment, candidate answers are not directly accepted; instead, they are required to be able to reverse-interpret the semantics of their corresponding instructions. This is achieved through a reverse-engineering generator. In a given candidate answer (and optional image / video frames and structured evidence) Generating a reverse verification problem under the condition of ) Subsequently, the fine-grained instructions were... Related to the reverse verification problem Perform a consistency check and obtain the consistency check result. In one implementation, let... For a text embedding model, the consistency score can be calculated using the following formula:

[0059]

[0060] When the consistency score is below the threshold If the candidate answer deviates from the semantics of the fine-grained instruction, it is recorded as a consistency check failure.

[0061] S42: If the consistency check passes, the candidate answer is parsed into a set of atomic facts, and the evidence index and visual evidence are aligned and checked one by one to obtain the evidence alignment check result and the target instance or evidence area.

[0062] Specifically, taking e-commerce product images / short videos as an example, when the fine-grained instruction is "Does the packaging have 'XX' written on it?" or "Is the logo on the packaging box?", the candidate answers are... This can correspond to several verifiable fact items. This embodiment will provide candidate answers. Analysis into a set of atomic facts ,in Indicates the first A set of atomic facts for each candidate answer. Indicates the first Atomic facts, Indicates the number of atomic facts. Candidate index, This indexes atomic facts. Atomic facts include at least one or more of the following: object existence assertions, attribute assertions, spatial / temporal relation assertions, and textual evidence assertions. For each atomic fact... Based on the evidence index, locate the corresponding mask number, bounding box number, text box number, or relation edge number, and then call the structured evidence. Candidate region set Scene diagram and OCR text evidence set Perform line-by-line alignment verification. In one implementation, define verifiable functions for atomic facts and structured evidence. The evidence alignment score can be calculated using the following formula:

[0063] When the evidence alignment score is below the threshold If the candidate answer lacks visual evidence or is a hallucination, it is recorded as an evidence alignment verification failure. At the same time, the area corresponding to the instance number or mask / text box involved in the evidence alignment verification is determined as the target instance or evidence area for subsequent counterfactual perturbation.

[0064] S43: Perform counterfactual perturbation on the target instance or evidence region determined by the evidence alignment check, and test the sensitivity of the candidate response before and after the perturbation to obtain the sensitivity test results;

[0065] Specifically, taking e-commerce product images / short videos as an example, when the fine-grained instruction is "read the brand name on the packaging" or "whether the logo is on the packaging box," the evidence alignment verification can determine the target evidence area as the brand text box or the logo instance area. This embodiment verifies the target instance or evidence area on the original input image / video frame. Applying counterfactual perturbations generates the perturbed input. or And on the perturbed input, for the same fine-grained instruction Regenerating or re-evaluating candidate answers Then compare and Changes at the atomic level yielded sensitivity test results.

[0066] The counterfactual perturbation includes one or more of the following: occlusion, replacement, deletion, or style perturbation of the target instance or evidence region; and requires that the candidate response's change in the atomic fact dimension that should be affected by the perturbation meets a preset difference threshold. Alternatively, it can maintain consistency across atomic fact dimensions that should not be affected by disturbances, satisfying a preset consistency threshold. In one implementation, the disturbance sensitivity score can be defined as:

[0067]

[0068] in For semantic similarity or task-specific consistency functions; when If the sensitivity test fails, the reason for the failure is recorded and labeled as "Fake fact sensitivity test failed".

[0069] S44: Generate a reverse verification score based on the consistency verification result, evidence alignment verification result, and sensitivity test result, and generate a failure reason label based on the verification steps that failed.

[0070] Specifically, taking e-commerce product images / short videos as an example, for the same fine-grained instruction... The following are candidate answers If the reverse verification problem is inconsistent with the semantics of the instruction, or if its key facts cannot be aligned with evidence such as packaging text / logo areas, or if it is not sensitive to disturbances that obscure the evidence area, its reverse verification score should be reduced and the corresponding reason for failure should be recorded. This embodiment is based on consistency scores. Evidence alignment score and sensitivity score Reverse validation score for generating candidate answers In one implementation, the reverse verification score can be obtained by a weighted combination of the scores:

[0071] in The system sets preset weighting coefficients and generates failure reason labels based on failed consistency checks, evidence alignment checks, or sensitivity tests, which are used for subsequent statistical analysis and preference pair export.

[0072] S45: Mark the candidate answers that pass the reverse verification as positive examples and proceed to subsequent scoring; mark the candidate answers that fail as negative examples and do not proceed to subsequent scoring; and use the negative examples as control answers for preference pair derivation.

[0073] Specifically, taking e-commerce product images / short videos as an example, for the same fine-grained instruction... (e.g., "Read the brand name on the packaging") A set of candidate answers If an answer passes all three tests—consistency, evidence alignment, and perturbation sensitivity—it should be selected as the preferred answer and included in the subsequent quality scoring; otherwise, it should be used as a control answer for preference pair construction. This embodiment marks candidate answers that meet a preset reverse validation threshold as positive examples, for example, when... If no failure reason label exists, add it to the subsequent multi-criteria scoring set; mark candidate answers that do not meet the preset threshold conditions or have failure reason labels as negative examples and remove them from the subsequent multi-criteria scoring; at the same time, pair or group positive and negative examples under the same fine-grained instruction to form preference pair data. In this context, positive examples are selected as preferred answers, while negative examples serve as control answers.

[0074] S5: Perform multi-criteria scoring on the candidate answers that pass the reverse verification to obtain a multi-criteria score, and determine a quality score based on the reverse verification score and the multi-criteria score;

[0075] Specifically, taking e-commerce product images / short videos as an example, fine-grained instructions could be "read the brand name on the packaging / determine the color and material / compare the quantity," and candidate answers would be verified through reverse validation. The merits and demerits should be further differentiated based on dimensions such as "visual alignment, evidentiary value, factual basis, completeness, and clarity of expression." This embodiment examines each candidate answer that passes reverse verification. A multi-criteria scoring model is invoked to perform scoring, resulting in corresponding multi-criteria scores. The multi-criteria scoring model takes the image or video frame, fine-grained instructions, candidate answers, and visual evidence or evidence indexes associated with the candidate answers as input, outputs the scoring results of each criterion, and aggregates them to obtain the multi-criteria score.

[0076] The multi-criteria score is output by a multi-criteria scoring model, and the criteria include one or more of the following: visual alignment, evidentiaryness, factuality, no illusion, completeness, clarity of expression, logical consistency, and conciseness.

[0077] Furthermore, this embodiment determines a quality score based on the reverse verification score and the multi-criteria score for subsequent data filtering and export. In one implementation, the scores of each criterion output by the multi-criteria scoring model are aggregated to obtain a multi-criteria score, which is then fused with the reverse verification score to obtain a quality score. In another implementation, a joint threshold determination method is used: the quality score of the candidate answer is determined to be passed only when both the reverse verification score and the multi-criteria score meet a preset threshold.

[0078] S6: Filter based on the quality score and output multimodal training data for instruction fine-tuning.

[0079] Specifically, taking e-commerce product images / short videos as an example, the same fine-grained instruction (e.g., "Read the brand name on the packaging") corresponds to a set of candidate answers. This includes both definitively correct answers and unclear or inconsistent control answers. This embodiment organizes candidate answer groups by fine-grained instruction and performs filtering based on the quality score to determine the final exported samples. For candidate answer groups under the same fine-grained instruction, candidate answers that meet preset quality score conditions are first retained as reference answers that pass reverse validation; candidate answers that fail reverse validation or whose quality scores do not meet preset conditions are retained as control answers but are not used as target answers for supervised fine-tuning in the training loss calculation.

[0080] The filtering includes static filtering and dynamic filtering;

[0081] The static filtering is used to remove samples whose quality score variance is less than a threshold or whose difference between the maximum and minimum values ​​is less than a threshold within the candidate answer group under the same instruction.

[0082] The dynamic filtering is used to retain samples that contain both high-quality and low-quality responses under the same instruction, in order to construct the positive and negative samples required for contrastive learning or preference learning.

[0083] The output multimodal training data for instruction fine-tuning should include at least: multimodal input, fine-grained instructions, target evidence index, reference answers that pass back-validation, control answers that fail back-validation, and labels indicating the reasons for their failure.

[0084] Corresponding to the aforementioned embodiment of a multimodal fine-grained instruction fine-tuning data construction method based on reverse verification, this application also provides an embodiment of a multimodal fine-grained instruction fine-tuning data construction apparatus based on reverse verification.

[0085] Figure 2 This is a structural block diagram illustrating a multimodal fine-grained instruction-based data construction apparatus based on reverse verification, according to an exemplary embodiment. (Reference) Figure 2 The device may include:

[0086] Acquisition module 1 is used to acquire multimodal raw data, which includes at least images or videos and related text information and metadata;

[0087] Parsing module 2 is used to call at least one domain model to perform structured parsing of the multimodal raw data to obtain verifiable visual evidence;

[0088] Candidate generation module 3 is used to generate fine-grained instructions based on the visual evidence, and generate a set of candidate answers according to the fine-grained instructions;

[0089] The reverse verification module 4 is used to perform reverse verification on the candidate answer set for each fine-grained instruction, combined with the verifiable visual evidence, to obtain a reverse verification score and a failure reason label; the candidate answers that pass the reverse verification are marked as positive examples and enter the subsequent scoring, the candidate answers that fail are marked as negative examples and do not enter the subsequent scoring, and the negative examples are used as control answers for preference pair derivation;

[0090] The quality scoring module 5 is used to perform multi-criteria scoring on the candidate answers that have passed the reverse verification to obtain a multi-criteria score, and to determine a quality score based on the reverse verification score and the multi-criteria score.

[0091] The filtering module 6 is used to filter based on the quality score and output multimodal training data for instruction fine-tuning.

[0092] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0093] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0094] Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the above-described method for constructing multimodal fine-grained instruction tuning data based on reverse verification. Figure 3 The diagram shown is a hardware structure diagram of any data processing-capable device, including a multimodal fine-grained instruction fine-tuning data construction device based on reverse verification provided in an embodiment of the present invention. (Except for...) Figure 3 In addition to the processor, memory, DMA controller, disk, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0095] Accordingly, this application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the aforementioned multimodal fine-grained instruction fine-tuning data construction method based on reverse verification. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0096] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only.

[0097] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for constructing multimodal fine-grained instruction-based fine-tuning data based on reverse verification, characterized in that, include: Acquire multimodal raw data, which includes at least images or videos, as well as related text information and metadata; The multimodal raw data is structured and parsed by calling at least one domain model to obtain verifiable visual evidence; Based on the visual evidence, fine-grained instructions are generated, and a set of candidate answers is generated according to the fine-grained instructions; For each set of candidate answers to a fine-grained instruction, reverse verification is performed in conjunction with the verifiable visual evidence to obtain a reverse verification score and a failure reason label. Candidate answers that pass the reverse validation are marked as positive examples and proceed to subsequent scoring; candidate answers that fail are marked as negative examples and do not proceed to subsequent scoring; and negative examples are used as control answers for preference pair derivation. A multi-criteria scoring method is applied to the candidate answers that pass the reverse validation to obtain a multi-criteria score, and a quality score is determined based on the reverse validation score and the multi-criteria score. Based on the quality score, filter the data and output multimodal training data for instruction fine-tuning. Specifically, for each fine-grained instruction's candidate response set, reverse verification is performed in conjunction with the verifiable visual evidence to obtain a reverse verification score and a failure reason label, including: The verification question is deduced from the candidate answer and then checked for consistency with the fine-grained instruction to obtain the consistency check result. If the consistency check passes, the candidate answer is parsed into a set of atomic facts, and then aligned and checked one by one according to the evidence index and visual evidence to obtain the evidence alignment check result and the target instance or evidence area. Perform counterfactual perturbation on the target instance or evidence region determined by the evidence alignment check, and test the sensitivity of candidate responses before and after the perturbation to obtain the sensitivity test results; A reverse verification score is generated based on the consistency verification result, the evidence alignment verification result, and the sensitivity test result, and a failure reason label is generated based on the verification steps that fail.

2. The method according to claim 1, characterized in that, The multimodal raw data is subjected to structured analysis to obtain verifiable visual evidence, including: Instance-level masks and bounding boxes are generated for the image or video frame to form a candidate instance set; Based on the candidate instance set, extract the category information and attribute information of each instance; Based on the instance pairs in the candidate instance set, the relationships between instances are determined to obtain a set of relationship information, and a scene graph is constructed using candidate instances as nodes and relationship information as edges. A traceable evidence index is generated for the category information, attribute information, and relationship information to indicate the mask number, bounding box number, text box number, or relationship edge number corresponding to the category information, attribute information, or relationship information in the candidate instance set. Verifiable visual evidence is formed based on candidate instances, category information, attribute information, relationship information, scene graph, and evidence index.

3. The method according to claim 1, characterized in that, Based on the visual evidence, fine-grained instructions are generated, and a set of candidate answers is generated according to the fine-grained instructions, including: Select or sample task types from a predefined set of tasks, and generate fine-grained instructions based on the scene graph and evidence index corresponding to the visual evidence; For each fine-grained instruction, at least one candidate answer is generated to form a set of candidate answers corresponding to that fine-grained instruction; The task types include at least one or more of the following: attribute recognition, relationship understanding, counting and comparison, referential resolution, local text reading, temporal event localization, and cross-perspective consistency checking.

4. The method according to claim 1, characterized in that, The counterfactual perturbations include: The target instance or evidence area is subject to one or more of the following: occlusion, replacement, deletion, or style perturbation. Candidate responses are required to meet a preset difference threshold in terms of the magnitude of change on the atomic fact dimension that should be affected by the perturbation, or to meet a preset consistency threshold in terms of consistency on the atomic fact dimension that should not be affected by the perturbation.

5. The method according to claim 1, characterized in that, Filtering includes static filtering and dynamic filtering; The static filtering is used to remove samples whose quality score variance is less than a threshold or whose difference between the maximum and minimum values ​​is less than a threshold within the candidate answer group under the same instruction. The dynamic filtering is used to retain samples that contain both high-quality and low-quality responses under the same instruction, in order to construct the positive and negative samples required for contrastive learning or preference learning.

6. The method according to claim 1, characterized in that, The output multimodal training data for instruction fine-tuning should include at least: multimodal input, fine-grained instructions, target evidence index, reference answers that pass back-validation, control answers that fail back-validation, and labels indicating the reasons for their failure.

7. A multimodal fine-grained instruction fine-tuning data construction device based on reverse verification, characterized in that, include: The acquisition module is used to acquire multimodal raw data, which includes at least images or videos and related text information and metadata. The parsing module is used to call at least one domain model to perform structured parsing of the multimodal raw data to obtain verifiable visual evidence; The candidate generation module is used to generate fine-grained instructions based on the visual evidence, and to generate a set of candidate answers based on the fine-grained instructions. The reverse verification module is used to perform reverse verification on the candidate answer set for each fine-grained instruction, combined with the verifiable visual evidence, to obtain a reverse verification score and a failure reason label; candidate answers that pass the reverse verification are marked as positive examples and enter the subsequent scoring, candidate answers that fail are marked as negative examples and do not enter the subsequent scoring, and negative examples are used as control answers for preference pair derivation; The quality scoring module is used to perform multi-criteria scoring on candidate answers that have passed reverse validation to obtain multi-criteria scores, and to determine a quality score based on the reverse validation scores and the multi-criteria scores. The filtering module is used to filter data based on the quality score and output multimodal training data for fine-tuning instructions. Specifically, for each fine-grained instruction's candidate response set, reverse verification is performed in conjunction with the verifiable visual evidence to obtain a reverse verification score and a failure reason label, including: The verification question is deduced from the candidate answer and then checked for consistency with the fine-grained instruction to obtain the consistency check result. If the consistency check passes, the candidate answer is parsed into a set of atomic facts, and then aligned and checked one by one according to the evidence index and visual evidence to obtain the evidence alignment check result and the target instance or evidence area. Perform counterfactual perturbation on the target instance or evidence region determined by the evidence alignment check, and test the sensitivity of candidate responses before and after the perturbation to obtain the sensitivity test results; A reverse verification score is generated based on the consistency verification result, the evidence alignment verification result, and the sensitivity test result, and a failure reason label is generated based on the verification steps that fail.

8. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the steps of any one of claims 1-6.

Images