A multi-source question-answer sample self-calibration construction method and system for complex visual scenes

Abstract

The application provides a multi-source question and answer sample self-calibration construction method and system for complex visual scenes, comprising: acquiring multi-source visual images and screening to generate an image pool; generating candidate question and answer samples containing question texts and reference answers based on the image pool; performing no-image solvability screening on the candidate question and answer samples, eliminating candidate samples that can be answered without visual information, to obtain retained samples; performing scene adaptation review and visual revision on the retained samples, checking scene types, target visibility, and question determinability, to obtain samples that pass the review; generating structured annotation fields containing target unit lists, relationship sets, and segmented inference explanations for the samples that pass the review; performing consistency closed-loop verification on the structured annotation fields, and if inconsistent with the reference answers, feeding back for regeneration until passing the verification or reaching an iteration upper limit; and encapsulating and outputting the samples that pass the verification. The application realizes high-quality construction of complex visual scene question and answer data.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence, computer vision, and multimodal data engineering technology, and specifically to a method and system for constructing self-calibrated multi-source question-answering samples for complex visual scenarios. Background Technology

[0002] Existing multimodal question-answering datasets are typically constructed using questions and answers as the smallest labeled units. These samples often lack verifiable intermediate evidence, such as target units, spatial or functional connections, or inference paths. Therefore, when training models using this type of data, while the models can learn the form of the answer, they struggle to learn the complete inference process from visual evidence to the answer.

[0003] Furthermore, conventional data production methods often rely on manual one-time annotation or single-round output from a model, lacking a systematic interception mechanism for issues such as language priors, scene incompatibility, and self-contradictory annotations. This problem is particularly prominent for complex images containing a large number of objects, occlusions, local relationships, and potential abnormal states, easily leading to questions that can be answered without looking at the image, answers that cannot be supported by the image, and samples where intermediate descriptions are inconsistent with the final conclusion.

[0004] A search revealed a Chinese patent application (application number 202110729964.X) that discloses a question-answer pair expansion method based on keyword replacement. However, this method is primarily limited to the plain text domain and suffers from the following drawbacks: First, it does not address visual scenarios, making it impossible to ensure the samples' reliance on image evidence and resulting in linguistic prior bias. Second, it lacks structured reasoning process annotation, leading to poor data interpretability. Third, it lacks a closed-loop verification mechanism for image-text consistency, making it difficult to guarantee the logical consistency and quality stability of the samples, and thus failing to meet the requirements for constructing high-quality question-answer samples in complex visual scenarios. Summary of the Invention

[0005] In view of the deficiencies in the prior art, the purpose of this application is to provide a method and system for constructing self-calibrated multi-source question-answering samples for complex visual scenes.

[0006] The first aspect of this application provides a method for constructing a self-calibrated multi-source question-answering sample for complex visual scenes, including: Acquire multi-source visual images and filter them to generate an image pool; Based on the image pool, candidate question-and-answer samples containing question text and reference answers are generated; At least two independent language models are invoked to perform graph unsolvability screening on the candidate question-answer samples, and candidate samples that can be answered without visual information are eliminated to obtain the retained samples; The retained samples are subjected to scene adaptation review and visual revision to verify scene type, target visibility and problem identifiability, and samples that pass the review are obtained. Generate structured annotation fields for the reviewed samples, including a list of target units, a set of relations, and segmented inference instructions; A consistency closed-loop check is performed on the structured annotation fields. If they are inconsistent with the reference answer, feedback is provided and the data is regenerated until the check is passed or the iteration limit is reached. The validated samples will be packaged and output.

[0007] Optionally, the step of acquiring multi-source visual images and filtering to generate an image pool includes: Acquire multi-source visual images; Identify the number of targets, spatial relationship types, and key region occlusion rates for each image; The image pool is generated by retaining images in which the number of targets is greater than a preset threshold, the spatial relationship type belongs to a preset relationship set, and the key area occlusion rate is lower than a preset occlusion threshold.

[0008] Optionally, generating candidate question-answer samples containing question text and reference answers based on the image pool includes: For each image in the image pool, generate multiple candidate question-answer samples and corresponding reference answers; The task types of the multiple candidate question-answer samples cover attribute recognition, spatial relationship judgment, functional purpose inference, abnormal state detection, and premise invalidation recognition.

[0009] Optionally, generating candidate question-answer samples containing question text and reference answers based on the image pool includes: Generate question text according to the preset task type distribution ratio; If the question text is a multiple-choice question, construct distracting options that do not match the content of the image; If the question text is an open-ended question, generate an answer that is supported by image evidence and meets the preset length requirement.

[0010] Optionally, the step of calling at least two independent language models to perform graph-free solvability screening on the candidate question-answer samples, eliminating candidate samples that can be answered without visual information, and obtaining the retained samples, includes: Without inputting an image, the at least two independent language models are invoked to answer the question text of the candidate question-answering samples respectively; Compare the responses from at least two independent language models with the reference answer; If the answers from at least two independent language models are consistent with the reference answer, the sample is determined to be a sample that can answer without visual information and is removed, and the remaining sample is taken as the retained sample.

[0011] Optionally, the step of performing scene adaptation verification and visual revision on the retained samples, verifying scene type, target visibility, and problem determineability, to obtain samples that pass the verification includes: Perform scenario type and common sense compatibility screening on the retained samples; Parallel target visibility and scale checks, as well as problem determineability checks, are performed on the samples that pass the screening. For samples that were not eliminated in the above checks, a visual review and minor revision were performed, and the status of approval, revision approval, or deletion was given based on the image evidence. Samples whose processing status is "passed" or "passed after revision" are identified as the samples that have passed the review.

[0012] Optionally, the consistency closed-loop verification includes: The structured annotation fields are logically verified to detect whether there are relationship contradictions, missing target units, or inference conclusions that are inconsistent with the reference answer.

[0013] Optionally, the feedback further generates the structured annotation fields, including: Based on the inconsistency type detected by the consistency closed-loop verification, a corresponding correction prompt is generated; The correction prompt is fed back to the step of generating structured annotation fields, triggering the regeneration of the structured annotation fields.

[0014] Optionally, the step of packaging and outputting the validated samples includes: sampling, archiving, and formatting the samples that have passed closed-loop validation to form a data file that can be used for training, validation, or testing.

[0015] A second aspect of this application provides a multi-source question-answering sample self-calibration construction system for complex visual scenes, comprising: The multi-source image access module acquires multi-source visual images and filters them to generate an image pool. The candidate question-answer generation module generates candidate question-answer samples containing question text and reference answers based on the image pool; The no-image solvability screening module calls at least two independent language models to perform no-image solvability screening on the candidate question-answer samples, eliminating candidate samples that can be answered without visual information, and obtaining the retained samples; The scene adaptation verification module performs scene adaptation verification and visual revision on the retained samples, verifies the scene type, target visibility and problem determination, and obtains samples that pass the verification. The structure field generation module generates structured annotation fields for the reviewed samples, including a list of target units, a set of relations, and segmentation inference instructions. The cross-consistency verification module performs a consistency closed-loop verification on the structured annotation fields. If it is inconsistent with the reference answer, it will be fed back and regenerated until the verification is passed or the iteration limit is reached. The sample encapsulation module encapsulates and outputs the validated samples.

[0016] This application provides a self-calibrated construction method for multi-source question-answering samples in complex visual scenes. By employing cross-model graph-free solvability screening technology, it effectively eliminates samples that rely primarily on linguistic priors for answers, significantly increasing the data's dependence on visual evidence. Through scene-adaptive hierarchical verification technology, it reduces the incidence of scene premise errors, unclear target references, and unverifiable answers, thereby improving the stability of sample quality. By using structured field generation and cross-consistency closed-loop verification technology, it ensures that target units, relational descriptions, inference explanations, and answers mutually support each other, enhancing the interpretability and traceability of the data. Through the synergistic effect of multi-source image screening, linguistic leakage elimination, visual verification, and structured self-calibration, high-quality construction of question-answering data for complex visual scenes is achieved, effectively overcoming the shortcomings of existing technologies such as single final answers, insufficient quality control links, and missing intermediate evidence.

[0017] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0018] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating a method for constructing self-calibrated multi-source question-answering samples for complex visual scenes, according to an exemplary embodiment. Figure 2 This is a schematic diagram illustrating the process framework of a multi-source question-answering sample self-calibration construction method for complex visual scenes, according to an exemplary embodiment. Figure 3 This is a structural diagram of a multi-source question-answering sample self-calibration construction system for complex visual scenes, according to an exemplary embodiment. Detailed Implementation

[0019] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0020] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations.

[0021] Existing samples only contain the final answers and lack procedural evidence. To address these issues, this application provides a method for self-calibrating and constructing multi-source question-answering samples for complex visual scenarios.

[0022] Reference Figure 1 and Figure 2 As shown, the multi-source question-answering sample self-calibration construction method for complex visual scenes in this embodiment includes: S100: Acquire multi-source visual images and filter them to generate an image pool; S200, generates candidate question-answer samples containing question text and reference answers based on image pooling; S300: Call at least two independent language models to perform graph unsolvability screening on candidate question-answer samples, remove candidate samples that can be answered without visual information, and obtain the retained samples; S400 performs scenario adaptation review and visual revision on the retained samples, verifies the scenario type, target visibility and problem identifiability, and obtains samples that pass the review; S500 generates structured annotation fields for samples that have passed the review, including a list of target units, a set of relations, and a segmented inference description. S600 performs a consistency closed-loop check on the structured annotation fields. If it is inconsistent with the reference answer, it will be fed back and regenerated until the check is passed or the iteration limit is reached. The S700 will encapsulate and output the verified sample.

[0023] The above embodiments connect the quality control links into a reversible link through the unsolvable screening of no image in S300, the scene adaptation review and visual revision in S400, and the consistency closed-loop verification in S600, thereby significantly reducing the proportion of unverifiable samples and contradictory samples.

[0024] In the above embodiment, S500 generates a structured annotation field containing a list of target units, a set of relationships, and segmented inference instructions, so that each sample has both the final answer and traceable intermediate evidence. This allows the samples to be used for supervised training as well as for locating errors in the model in the target recognition, relationship understanding, or logical inference stages.

[0025] To avoid samples with sparse targets, simple relationships, or severe occlusion, in one specific embodiment of this application, in step S100, obtaining a multi-source visual image pool through screening can be achieved by the following steps: S101, acquires multi-source visual images; Specifically, "multi-source" refers to an image collection accessed by the system from different sources and covering various scene types. These images can come from public datasets, proprietary acquisition devices, or legally authorized libraries; their scenes cover various environmental types such as indoor, street scenes, and everyday life scenes, and include high-density images.

[0026] S102, identify the number of targets, spatial relationship type and key area occlusion rate of each image; Specifically, the number of targets refers to the total number of detectable individual object instances in an image. Spatial relationship type refers to the relative positional relationship between objects, including top-bottom, left-right, inside-outside, and adjacent relationships. Key region occlusion rate refers to the proportion of pixel area of ​​a target object's key recognition region that is occluded by other objects to the total area of ​​that key region. Identifying the number of targets, spatial relationship type, and key region occlusion rate for each image can be achieved using existing technologies, such as the YOLO object detection model, spatial relationship parsers, or occlusion analysis networks.

[0027] S103, retain images whose number of targets is greater than a preset threshold, whose spatial relationship type belongs to a preset relationship set, and whose key area occlusion rate is lower than a preset occlusion threshold, and generate an image pool to be labeled.

[0028] Specifically, the preset quantity threshold refers to the minimum number of targets required to ensure scene complexity, such as 3; the preset relationship set refers to a predefined list of spatial relationships available for question-answering reasoning; and the preset occlusion threshold refers to the maximum allowable occlusion ratio to ensure target identifiability, such as 50%.

[0029] The above embodiments ensure that the image has sufficient scene complexity and visual determinability from the source by quantifying the number of targets, spatial relationships and occlusion rate, effectively avoiding the problem of unusable samples caused by sparse targets, simple relationships or severe occlusion.

[0030] In order to achieve diversity and comprehensiveness in the distribution of sample data, in a specific embodiment of this application, S200, candidate question-answer samples containing question text and reference answers are generated based on the image pool. Specifically, this is done by filling the sample with the image target detection results based on a predefined task template library, or by calling a multimodal generation model to input images and task instructions.

[0031] Among them, the task types of multiple candidate question-answering samples cover attribute recognition, spatial relationship judgment, function and purpose inference, abnormal state detection, and premise invalidation recognition.

[0032] Candidate question-answer samples refer to question texts generated based on image visual content and their corresponding reference answers; reference answers refer to standard answers used for subsequent verification and training; attribute recognition refers to the task of recognizing features such as target color, shape, and material; spatial relationship judgment refers to the task of judging the positional relationships between targets, such as up / down, left / right, and inside / outside; function and purpose inference refers to the reasoning task of inferring the purpose of a target based on its type; abnormal state detection refers to the task of identifying whether a target is damaged, missing, or abnormally placed; premise invalidity recognition refers to the task of generating questions based on premises that do not exist in the image and expecting the answer to negate the premise.

[0033] For example, for an attribute recognition task, if an image contains a red apple, the question "What color is the apple?" and the reference answer "red" are generated. For spatial relationship judgment tasks, if the cat in the image is located above the mat, the question "Where is the cat on the mat?" is generated, along with the suggested answer "Above". For the function and purpose inference task, if the image contains a hammer, generate the question "What is the purpose of this tool?" and the reference answer "to hammer nails"; For anomaly detection tasks, if the image contains a broken cup, generate the question "Is the cup intact?" and the reference answer "No". For recognition tasks where the premise is not true, if there are no dogs in the image, the question "How many dogs are in the picture?" is generated and the reference answer "zero".

[0034] The above embodiments, by generating candidate samples covering multiple task types, ensure the diversity and comprehensiveness of data distribution and avoid learning bias caused by a single task.

[0035] Some questions contain strong linguistic cues, allowing the model to answer without visual information, which is detrimental to model training. In one specific embodiment of this application, in step S300, at least two independent language models are invoked to perform a visual unsolvability screening on candidate question-answer samples, eliminating candidate samples that can be answered without visual information, thus obtaining the retained samples. This can be achieved through the following steps: S301, without inputting an image, call at least two independent language reasoning models to answer the candidate question-answering samples respectively; Specifically, independent language reasoning models refer to pure text-based large language models with different architectures or training data. Their input consists only of the question and answer texts from candidate question-answer samples, without any image features. Their output is the predicted answer generated by the model based on prior language knowledge. Examples include different series of models such as LLaMA, Qwen, or ChatGLM. S302, if the answers from the language reasoning model are all consistent with the reference answer, then the sample is determined to be a sample that can answer without visual information and is removed, resulting in the retained sample.

[0036] The above embodiments, by employing cross-model graph-free solvability screening technology, can effectively eliminate samples that rely primarily on linguistic priors to answer questions, thereby increasing the data's reliance on visual evidence.

[0037] Images from different sources vary significantly in terms of scene type, target scale, occlusion status, and denotation sharpness. In one specific embodiment of this application, S400, scene adaptation verification and visual revision are performed on the retained samples to check scene type, target visibility, and problem determineability, obtaining samples that pass the verification, such as... Figure 2 As shown, the following steps can be taken: S401, Scene type and common sense compatibility screening.

[0038] First, scene type labels are obtained. For each retained sample, a multimodal model generates a scene or room type summary based on the image. For a single outdoor scene, a preset fixed scene label can be directly used.

[0039] Specifically, the input to the multimodal model is the image data of the retained samples; its output is a text label or summary describing the image environment category, such as "indoor kitchen", "outdoor street view" or "natural park".

[0040] For example, the multimodal model can be an LLaVA or BLIP series model.

[0041] Secondly, assess suitability. Without inputting image pixels, and based solely on scene type labels and text content, determine whether the question, options, and reference answer are clearly unreasonable, using common sense rules or text review models.

[0042] Specifically, scene type refers to the category of environment depicted in the image, including indoor rooms, outdoor street scenes, or natural scenes; common sense compatibility refers to whether the premise of the question conforms to the physical laws or common sense of life of that scene type.

[0043] For example, if the scenario type label is "bedroom" and the question text is "Where is the boat moored?", based on common sense, it is impossible for a boat to exist in a bedroom, so it is judged as obviously unreasonable and removed.

[0044] Finally, clearly identifiable bad samples are directly removed, and the remaining samples proceed to the next step.

[0045] S402, Target visibility and scale inspection.

[0046] For samples that have passed the S401 screening, the target referent in the text question of the sample is first analyzed to determine the visual entity to be checked.

[0047] Secondly, the image is examined to determine whether the target object is identifiable, whether its relative scale or area is too small, whether key parts are occluded too much, and whether the spatial reference is ambiguous (the ambiguity of the spatial reference refers to whether the spatial orientation description in the question can uniquely lock the target object in the image; if there are multiple similar targets in the image and the question does not contain enough spatial limiting attributes to distinguish the targets, it is determined that there is ambiguity in the spatial reference). Subsequently, classification and processing were carried out based on the inspection results: If the target is unidentifiable, too small, or too heavily occluded, the sample is marked as a candidate for deletion. If the target is visible but the spatial reference is ambiguous, the sample is marked as a candidate for revision, and S404 will eliminate the ambiguity by tightening the reference. If the target visibility, scale, and reference all meet the requirements, this check will pass and you will proceed to the next step.

[0048] Specifically, the target referent refers to the specific object entity that is explicitly required to be identified or described in the question text; scale check refers to determining whether the target pixel area is lower than a preset area threshold; whether the occlusion of key parts is too high refers to determining whether the proportion of pixels occluded in key parts exceeds a preset occlusion threshold; whether the target is identifiable refers to whether the target pixel area is higher than the area threshold and the occlusion proportion is lower than the occlusion threshold; tightening referential elimination refers to the operation of locking the referent from multiple targets of the same type into a unique target by adding limiting attributes such as color, position or state.

[0049] S403, Problem Determinability Check.

[0050] For the samples that enter this step (samples after S401 screening), without changing the images, based on visible visual evidence and optional semantic or instance-level truth labels, it is determined whether the questions have verifiable basis for answering and whether the reference answers are clearly supported by the images. Secondly, distinguish between task types for judgment: For samples whose task type is to identify the premise that is not true, the premise of the problem is not true in the image, which is part of the expected design. The reference answer must refute the premise based on the image evidence, and it is not considered a defect. For other types of samples, if the premise is not valid, the evidence is missing, or the answer cannot support it, they are marked as candidates to be deleted. Finally, samples that meet the expected design criteria will proceed to the next steps.

[0051] Specifically, problem decidability refers to whether the correct answer to the problem can be uniquely determined based on the current image content; premise invalidity recognition refers to a task type specifically designed to test whether a model can identify objects that do not exist in an image, and its correct answer is the existence of a negative premise.

[0052] S404, Visual Review and Lightweight Revision.

[0053] First, samples that were not eliminated in S401 to S403, along with images, the aforementioned screening records, and optional semantic or instance-level ground truth annotations, are submitted to manual or multimodal models for review. The review here mainly checks the consistency between the images and text.

[0054] Secondly, image preprocessing is performed. If a significant camera tilt is detected, the scene is first horizontally corrected, and then the spatial relationships are determined based on the corrected image. Subsequently, the consistency between the questions, options, and reference answers is verified by comparing them with the images, and one of the following three handling states is given: a) Passed, indicating that it is consistent with the image evidence and requires no modification; b) Passing after revision indicates that it has been approved after minor revisions; c) Deleted, indicating an uncorrectable defect; The scope of the light revision is limited to the question text, option descriptions, reference answers or reference explanations, and does not change the core semantics of the sample.

[0055] S405, Summary of samples that passed the review.

[0056] The samples that were processed as approved and those that were revised and approved are summarized into samples that passed the review, and the processing status and revision content are recorded.

[0057] The above embodiments, through the hierarchical verification mechanism of S401 to S404, employ scenario-adaptive hierarchical verification technology, which can reduce problems such as scenario premise errors, unclear target references, and unverifiable answers, thereby improving the stability of sample quality.

[0058] In one embodiment of this application, S500, generating a structured annotation field containing a list of target units, a set of relationships, and segmented inference instructions for the reviewed sample can be achieved through the following steps: S501, construct structured generation input.

[0059] Read the sample images that have passed the review, as well as the confirmed or revised question text, options, and final answers; The generated prompts, constraint-generated target unit list, relation set, and segmented inference explanations must logically support the final answer, and the revised question and answer content shall prevail.

[0060] S502, invokes the multimodal generation model to perform conditional generation.

[0061] The image and question-answer pairs are input into the multimodal generative model, and structured annotation fields are generated based on a predefined prompt template. The constrained output contains only a list of target units, a set of relations, and a segmented inference description, and conforms to a predefined data format.

[0062] S503 generates and parses three types of structured fields.

[0063] (1) Target Unit List: List the entity names related to the answer, following the principle of minimum sufficiency, and only retain the objects necessary to derive the answer; (2) Relation set: It describes the spatial or functional relationship between entities in the form of a triple of “head entity - relation predicate - tail entity”; (3) Segmented Inference Explanation: Output four segments of text in sequence: perception, relation, logic and conclusion. The perception segment describes visual facts, the relation segment describes entity relationships, the logic segment explains the reasoning process, and the conclusion segment must be consistent with the final answer.

[0064] S504, Perform a format and consistency check on the generated results.

[0065] The generated results are parsed and verified for field integrity, the order and number of paragraphs in the segmented inference explanations, entity consistency, and the consistency between the conclusion and the final answer. If the verification fails, the results are regenerated or sent to the subsequent consistency closed-loop verification stage.

[0066] S505 is stored in conjunction with sample records.

[0067] The validated structured annotation fields are bound and stored with the sample's question and answer content and image identifiers for subsequent consistency closed-loop verification and encapsulation output.

[0068] To address the issue of conflicts between objects, relationships, reasoning descriptions, and answers that easily occur in automatically generated annotations, in a specific embodiment of this application, S600 performs a consistency closed-loop check, which can be achieved through the following steps: S601 performs logical verification on the structured annotation fields to detect whether there are relationship contradictions, missing target units, or inference conclusions that are inconsistent with the reference answer; S602 If an inconsistency is detected, a correction prompt is generated and fed back to the structured annotation field generation step, triggering a regeneration process until the verification is passed or the preset iteration limit is reached.

[0069] S603 If a sample fails the verification even after reaching the preset iteration limit, it is determined to be an unqualified sample and is removed; if the sample passes the verification, it proceeds to the subsequent packaging and output steps.

[0070] The above embodiments, by constructing a closed-loop correction mechanism of generation, review, feedback, and regeneration, ensure that the target unit, relation description, inference description, and answer support each other, effectively solving the problem of self-contradictory automatically generated content and improving the interpretability and traceability of data.

[0071] In a specific embodiment of this application, in S700, the verified sample is packaged and output, which can be achieved by the following steps: S701 performs quality sampling and storage archiving on samples that pass closed-loop verification, and generates build logs containing the screening ratio and task type distribution. S702 converts the archived samples into data files in a preset format, forming a dataset that can be used for model training, validation, or testing. For example, the final sample size can reach tens of thousands of questions, thousands of images, and thousands of categories of object or relationship descriptions.

[0072] Based on the same technical concept, some other embodiments of this application provide a multi-source question-answering sample self-calibration construction system 100 for complex visual scenes, such as... Figure 3 As shown, it includes: The multi-source image access module 110 acquires multi-source visual images, filters them, and generates an image pool. The candidate question-answer generation module 120 generates candidate question-answer samples containing question text and reference answers based on the image pool; The no-image solvability screening module 130 calls at least two independent language models to perform no-image solvability screening on candidate question-answer samples, removes candidate samples that can be answered without visual information, and obtains the retained samples; The scene adaptation verification module 140 performs scene adaptation screening, visual verification and lightweight revision on the retained samples, verifies the scene type, target visibility and problem judgment, and performs processing on the samples to pass, pass after revision or delete, and obtains the samples that pass the verification. The structure field generation module 150 generates structured annotation fields containing a list of target units, a set of relations, and segmentation inference instructions for samples that have passed the review. The cross-consistency verification module 160 performs a closed-loop consistency verification on the structured annotation fields. If there is any inconsistency, it will feed back and regenerate the structured annotation fields until the verification is passed or the iteration limit is reached. The sample encapsulation module 170 encapsulates and outputs the verified samples.

[0073] The specific implementation techniques of each module / unit in the above examples of this application can be referred to the steps of the multi-source question answering sample self-calibration construction method for complex visual scenes in the above embodiments, which will not be repeated here.

[0074] This application provides a specific application example of a method for constructing a self-calibrated multi-source question-answering sample for complex visual scenarios.

[0075] This embodiment uses a kitchen countertop scene image as an example for illustration. The image contains multiple target objects such as a coffee machine, kettle, mug, toaster, cutting board, fruit plate, knife rack, condiment bottle, and tissue box. There are rich spatial relationships between the targets, which meets the data construction requirements of complex visual scenes.

[0076] First, the system acquires multi-source visual images from public datasets and authorized image libraries, and filters candidate images. The system counts the number of targets, spatial relationship types, and occlusion rates of key regions in each image. When an image is detected to contain more than ten independent target instances with various spatial relationships such as placement, adjacency, front-back, and left-right, and the occlusion rates of each major target are all below a preset threshold, the image is retained in the image pool as a sample to be constructed.

[0077] Subsequently, the system generates candidate question-and-answer samples based on the image. For example, it generates attribute recognition questions such as "What color is the mug on the counter?", spatial relationship judgment questions such as "Is the fruit plate located to the right of the coffee machine?", function and purpose inference questions such as "Can the cutting board in the picture be used for cutting vegetables?", abnormal state recognition questions such as "Is there an overturned condiment bottle on the counter?", and premise invalidation recognition questions such as "Is the microwave oven in the picture placed next to the coffee machine?". Corresponding reference answers are generated simultaneously. For the last question, since there is no microwave oven in the image, the reference answer is "There is no microwave oven in the image; this premise is invalid."

[0078] Next, the system invokes two independent language models to perform a graph-free solvability screening. Without inputting an image, only the question text is entered to obtain the answer. When both language models can correctly answer a question, it is determined that the question can be answered without relying on visual information and is deleted. For example, both language models can answer "Can a cutting board be used for cutting vegetables?" correctly based on common sense, so the sample is deleted; while neither language model can accurately determine "Is the fruit plate located to the right of the coffee machine?", so the sample is retained and proceeds to the next processing step.

[0079] For retained samples, the system further performs scene adaptation verification. First, it determines whether the question conforms to the actual semantics of a kitchen scene. For example, if the question involves targets that clearly do not belong to a kitchen scene, such as traffic lights or cars, the scene is deemed mismatched and the sample is deleted. For samples that pass the scene screening, target visibility and scale checks, as well as question decidability checks, are performed. When the target is fully visible and its size meets the recognition requirements, and the question can be uniquely answered based on the current image, the sample proceeds to the next processing stage; otherwise, the corresponding sample is deleted.

[0080] The system then performs visual verification and minor revisions. For example, if the initial label describes a container as a "white cup," the visual verification reveals that it is actually a "white ceramic mug." The system then revises the label accordingly, while retaining the rest of the information.

[0081] After visual verification, the system automatically generates corresponding structured annotation fields. The target unit list includes objects such as a coffee machine, kettle, white ceramic mug, fruit plate, cutting board, and knife rack; the relationship set includes spatial relationships such as "the mug is located on the surface of the work surface," "the fruit plate is located to the right of the coffee machine," and "the knife rack is located behind the work surface"; the segmented inference description includes the target recognition process, spatial relationship analysis process, logical reasoning process, and final answer generation process, so that the final answer has complete visual evidence support.

[0082] Subsequently, the system performs a consistency closed-loop check on the generated structured annotation fields, checking for logical conflicts between the target unit, relation set, inference description, and reference answer. When an inference description is detected to be inconsistent with the reference answer, the system automatically generates a correction prompt and sends it back to the structured annotation generation module to regenerate the corresponding fields; when all fields are consistent, the sample is deemed to have passed the consistency check.

[0083] Finally, the samples that pass the consistency check are randomly sampled, archived, and formatted for encapsulation, forming structured visual question-and-answer data containing images, question text, reference answers, target unit lists, relation sets, and segmented inference explanations, which is then output to the training dataset. The data samples constructed through this process effectively reduce interference from linguistic priors, increase the samples' reliance on visual evidence, and enhance the interpretability of the reasoning process and the consistency of data annotation.

[0084] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A method for self-calibrating and constructing multi-source question-answering samples for complex visual scenes, characterized in that, include: Acquire multi-source visual images and filter them to generate an image pool; Based on the image pool, candidate question-and-answer samples containing question text and reference answers are generated; At least two independent language models are invoked to perform graph unsolvability screening on the candidate question-answer samples, and candidate samples that can be answered without visual information are eliminated to obtain the retained samples; The retained samples are subjected to scene adaptation review and visual revision to verify scene type, target visibility and problem identifiability, and samples that pass the review are obtained. Generate structured annotation fields for the reviewed samples, including a list of target units, a set of relations, and segmented inference instructions; A consistency closed-loop check is performed on the structured annotation fields. If they are inconsistent with the reference answer, feedback is provided and the data is regenerated until the check is passed or the iteration limit is reached. The validated samples will be packaged and output.

2. The method according to claim 1, characterized in that, The process of acquiring multi-source visual images and filtering them to generate an image pool includes: Acquire multi-source visual images; Identify the number of targets, spatial relationship types, and occlusion rate of key regions in each image; Images whose target quantity is greater than a set threshold, whose spatial relationship type belongs to a preset relationship set, and whose key area occlusion rate is lower than a set occlusion threshold are retained to form the image pool.

3. The method according to claim 1, characterized in that, The generation of candidate question-answer samples containing question text and reference answers based on the image pool includes: For each image in the image pool, generate multiple candidate question-and-answer samples and corresponding reference answers; The task types of the multiple candidate question-answer samples cover attribute recognition, spatial relationship judgment, functional purpose inference, abnormal state detection, and premise invalidation recognition.

4. The method according to claim 3, characterized in that, The generation of candidate question-answer samples containing question text and reference answers based on the image pool includes: Generate question text according to the set task type distribution ratio; If the question text is a multiple-choice question, construct distracting options that do not match the content of the image; If the question text is an open-ended question, generate an answer that is supported by image evidence and meets the preset length requirement.

5. The method according to claim 1, characterized in that, The process involves calling at least two independent language models to perform a graph-free solvability screening on the candidate question-answer samples, eliminating candidate samples that can be answered without visual information, and obtaining the retained samples, including: Without inputting an image, the at least two independent language models are invoked to answer the question text of the candidate question-answering samples respectively; Compare the responses from at least two independent language models with the reference answer; If the answers from at least two independent language models are consistent with the reference answer, then the sample is determined to be a sample that can answer without visual information and is removed, and the remaining sample is taken as the retained sample.

6. The method according to claim 1, characterized in that, The process of performing scene adaptation verification and visual revision on the retained samples verifies the scene type, target visibility, and problem determineability to obtain samples that pass the verification, including: Perform scenario type and common sense compatibility screening on the retained samples; Parallel target visibility and scale checks, as well as problem determineability checks, are performed on the samples that pass the screening. For samples that were not eliminated in the above checks, a visual review and minor revision were performed, and a status of approval, approval after revision, or deletion was given based on the image evidence. Samples whose processing status is "passed" or "passed after revision" are identified as the samples that have passed the review.

7. The method according to claim 1, characterized in that, The consistency closed-loop verification includes: The structured annotation fields are logically verified to detect whether there are relationship contradictions, missing target units, or inference conclusions that are inconsistent with the reference answer.

8. The method according to claim 1, characterized in that, The feedback then generates the structured annotation fields, including: Based on the inconsistency type detected by the consistency closed-loop verification, a corresponding correction prompt is generated; The correction prompt is fed back to the step of generating structured annotation fields, triggering the regeneration of the structured annotation fields.

9. The method according to claim 1, characterized in that, The process of packaging and outputting the validated samples includes: sampling, archiving, and formatting the samples that have passed closed-loop validation to form data files that can be used for training, validation, or testing.

10. A self-calibration construction system for multi-source question-answering samples in complex visual scenes, characterized in that, include: The multi-source image access module acquires multi-source visual images and filters them to generate an image pool. The candidate question-answer generation module generates candidate question-answer samples containing question text and reference answers based on the image pool; The no-image solvability screening module calls at least two independent language models to perform no-image solvability screening on the candidate question-answer samples, eliminating candidate samples that can be answered without visual information, and obtaining the retained samples; The scene adaptation verification module performs scene adaptation verification and visual revision on the retained samples, verifies the scene type, target visibility and problem determination, and obtains samples that pass the verification. The structure field generation module generates structured annotation fields for the reviewed samples, including a list of target units, a set of relations, and segmentation inference instructions. The cross-consistency verification module performs a consistency closed-loop verification on the structured annotation fields. If it is inconsistent with the reference answer, it will be fed back and regenerated until the verification is passed or the iteration limit is reached. The sample encapsulation module encapsulates and outputs the validated samples.

Images