Question and answer data construction method and device

Abstract

A question and answer data construction method comprises the following steps: 1) reading samples containing image and text data in batches, constructing a unified instruction for each batch of samples, and calling a large language model to extract candidate questions which have discriminability and teaching value and can be answered from images to form multiple groups of candidate questions; (2) carrying out structured deduplication merging on the cross-batch candidate problems obtained in the step (1) to obtain a deduplication problem pool; 3) for each sample, identifying an available image mode of the sample, screening candidate problems matched with the mode from the problem pool, and randomly extracting candidates of which the number is greater than a target value q; 4) constructing a request only based on image answering, calling the multi-modal large language model by taking the sample image as input, and generating a question-answer pair; if reliable answers of the candidate questions cannot be obtained only through the images, replacing the candidate questions with new candidates and retrying until q question and answer pairs capable of being answered are accumulated and obtained; 5, the generated question and answer pairs are subjected to structured verification and safe disking.The method supports parallel processing and breakpoint continuous running so as to meet the large-scale data construction.The method has the advantages of being good in universality and expandability and standardized in result.

Description

Technical Field

[0001] This invention relates to the field of data intelligence and information processing, specifically to a method and apparatus for automatically constructing a visual question answering (VQA) dataset from unimodal or multimodal image and text data, and also covers apparatus, electronic devices and computer-readable storage media for implementing the method. Background Technology

[0002] Visual question-answering datasets typically consist of images, image-related questions, and corresponding answers, forming a crucial foundation for training and evaluating multimodal understanding models. Current data construction methods largely rely on manual questioning and annotation, which suffers from high costs, difficulties in scaling, uneven question quality, high cross-source duplication, and the fact that some questions cannot be answered directly from images. Furthermore, real-world production environments commonly feature multimodal data in the form of "images + text reports." A systematic and standardized technical solution remains lacking for how to universally and scalably convert such raw data into high-quality VQA datasets while ensuring question diversity, discriminability, and answerability.

[0003] To address the aforementioned issues, a general automated construction method is urgently needed that can: extract high-value questions from batches of samples without relying on specific domain terminology; perform structured deduplication and merging of cross-batch questions; filter answerable questions and generate answers based solely on image information at the sample-by-sample level; and possess parallelization and breakpoint resume capabilities to meet the needs of large-scale data production. Summary of the Invention

[0004] This invention aims to overcome the shortcomings of existing technologies, such as high cost of VQA data construction, high repetition, unanswerable questions, and uncontrollable quality, and provides a method and apparatus for constructing question-and-answer data.

[0005] The first aspect of this invention relates to a method for constructing question-and-answer data, comprising the following steps:

[0006] (1) Read samples containing images and text data in batches, construct a unified instruction for each batch of samples, call the large language model to extract candidate questions that are discriminative, have teaching value and can be answered from images, and form multiple sets of candidate questions;

[0007] (2) Perform structured deduplication and merging on the cross-batch candidate problems obtained in step (1) to obtain a deduplicated problem pool;

[0008] (3) For each sample, identify its available image modality, filter candidate questions that match the modality from the question pool, and randomly select a number of candidates that are greater than the target value q;

[0009] (4) Construct a request that answers based solely on the image, take the sample image as input and call the multimodal large language model to generate question-answer pairs; if the candidate question cannot be reliably answered based solely on the image, replace it with a new candidate and retry until q answerable question-answer pairs are accumulated;

[0010] (5) Perform structured verification and secure disk write of the generated question-answer pairs; wherein, the method supports parallel processing and breakpoint resume to meet the needs of large-scale data construction.

[0011] Preferably, the structured deduplication and merging in step (2) specifically includes: collecting and merging several candidate question groups in sequence until the number of questions in the group reaches a threshold K, and handing the group over to the large language model for structured merging; taking each merged group as a new round of candidate groups, repeating the grouping and merging process until only one group remains, thereby obtaining the final deduplicated question pool; and recording the iteration status after each round of merging to support breakpoint continuation.

[0012] Preferably, step (4) includes:

[0013] (4.1) For binary judgment questions, the positive and negative proportions of the answers should be kept basically balanced;

[0014] (4.2) Set the maximum number of retries. Replace or retry when a candidate question is unanswerable or fails the structured validation.

[0015] (4.3) Only retain answers that can be directly supported by images, and discard question-answer pairs that rely on external knowledge or textual materials to answer.

[0016] Preferably, the image processing in step (4) includes: limiting the size of the image and performing lossy compression without affecting key visual information, and submitting it with the request in the form of data encoding; the request text clearly states the constraint of "answering only based on the image", and includes a list of candidate questions and necessary task descriptions.

[0017] Preferably, the structured verification and secure disk write in step (5) includes: verifying the integrity, type and value range of the fields; removing non-compliant entries; saving the results in a detail or log file format written one by one, and using atomic write and temporary file renaming strategies to improve the reliability of concurrent writes when necessary.

[0018] The present invention also relates to an apparatus for implementing the question-answer data construction method of the present invention, comprising: The data acquisition module is used to read samples containing image and text data in batches; The question generation module is used to construct unified instructions and call a large language model to refine candidate questions; The deduplication and merging module is used to group by a threshold K and perform structured deduplication to form a problem pool; The question-answering generation module is used to filter candidate questions at the sample level, construct requests that answer only based on images, and generate question-answer pairs, while controlling answerability and balance. The storage and export module is used to perform structured verification and secure disk write-to-disk operations, and provides support for parallel processing and breakpoint resume.

[0019] The present invention also relates to an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the question-and-answer data construction method described in the present invention.

[0020] The present invention also relates to a computer-readable storage medium storing a computer program that, when executed by a processor, implements the question-and-answer data construction method of the present invention.

[0021] The technical solution provided by this invention includes two stages: 1. Question set generation: The raw data is read in batches, and question instructions are constructed based on the text data of each batch of samples. The large language model is called to extract questions that are discriminative, have teaching value and whose answers can be obtained from image observation, resulting in multiple sets of candidate questions. For cross-batch candidate questions, they are grouped according to the rule of "accumulating to a threshold and then merging", and each group is structured to remove duplicates and merge them, finally forming a deduplicated question pool.

[0022] 2. Question-Answer Pair Generation: For each sample, first determine its available image modal set; select candidate questions that match its image modality from the question pool, and randomly select a subset with a number greater than the target value as candidates; construct the instruction of "answering only based on image" and image input, and call the multimodal large language model to generate question-answer pairs; if a candidate question cannot obtain a reliable answer based solely on image information, it is replaced with a new candidate question until the target number is reached; at the same time, control the balance of positive and negative answer ratios for binary judgment questions; finally, perform structured verification and store the results on disk.

[0023] The present invention has the following beneficial effects: Universality and scalability: It does not depend on domain-specific terminology and is compatible with single-modal or multimodal images and general text data; the question generation and merging strategies can be reused in different domains; Quality controllable: By using answerability constraints based solely on images, cross-batch deduplication, and structured validation, unanswerable and duplicate questions are reduced, improving the discriminative power and coverage of questions; Large-scale production: Supports multi-process parallelism and breakpoint resume, combined with engineering measures such as image encoding and compression to improve throughput and stability, suitable for large-scale data construction; Results standardization: Output a unified structured format to facilitate subsequent training and evaluation. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the overall process of the method of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be further described in conjunction with the accompanying drawings and embodiments. It should be understood that the following embodiments are only used to illustrate the present invention and are not intended to limit the present invention.

[0026] Example 1

[0027] This embodiment relates to a method for constructing question-and-answer data, referring to... Figure 1 It includes the following steps:

[0028] I. Data Format and Overall Process

[0029] Data format: Each sample contains one or more images (which may be one or more image modalities) and textual information related to the sample (such as image descriptions, structured summaries, or hierarchical information).

[0030] Overall process: It is divided into two stages: "question set generation" and "question-answer pair generation". Structured format conventions and validation strategies are adopted in each stage to ensure consistency of results.

[0031] II. First Stage: Problem Set Generation

[0032] 1. Batch Reading: Assuming a total sample size of N, samples are read in batches according to a batch threshold B, forming a batch data list. To ensure scalability, the batch size B can be configured based on resource and throughput requirements.

[0033] 2. Question Instruction Construction: For each batch of data, key information from the text samples within that batch is concatenated according to sample order to construct a unified instruction. The large language model is required to output questions with the following properties:

[0034] Discriminativeness: It can produce significantly different answers between different samples; Pedagogical value: Covers key knowledge points or typical phenomena; Answers can be obtained from images: the answers can be obtained by observing the images, without relying on external knowledge bases; It covers multiple question types, including existence, positional relationship, and attribute description.

[0035] 3. Structured Output and Parsing: The output is defined as a structured collection (containing fields such as question text, question category, and reference image modality). The returned results are subjected to secure parsing, removing outliers and retaining only complete and valid question entries.

[0036] 4. Iterative Deduplication and Merging: Suppose the accumulated candidate questions are grouped into n groups. Grouping follows the rule of "merging from left to right until the number of questions in a group reaches a threshold K." Each group is then processed by the large language model for deduplication and merging, outputting a structured result. The merged results of each group are used as the new candidate groups, and the above process is repeated until only one group remains, resulting in a deduplicated question pool. This merging process can be executed in parallel, and the results of each round are recorded as a stage log for easy resumption of execution from a breakpoint.

[0037] III. Second Stage: Question-Answer Pair Generation

[0038] 1. Sample available modality identification and candidate selection: For each sample, identify its available image modality set; select questions that require such modalities from the question pool as candidate question set, and randomly select a subset with a number greater than the target value q to improve the success rate.

[0039] 2. Image-based response generation: Construct instructions that include task description and a list of candidate questions, explicitly requiring that responses be generated based solely on images; compress and encode sample images appropriately and submit them as image input to the multimodal large language model; obtain the generated question-and-answer results.

[0040] 3. Answerability and Balance Control: If the answer to a candidate question cannot be given based solely on image information, or if the answer is uncertain, a new candidate question is substituted and the process is retried until q answerable question-answer pairs are obtained; for binary judgment questions, the ratio of positive to negative answers is controlled to be roughly balanced to avoid category bias.

[0041] 4. Structured Validation and Disk Storage: Perform structured field integrity and type validation on the generated results and remove non-compliant entries; save the results in log files or detail files written one by one to reduce the risk of conflicts during concurrent writes; introduce atomic write and temporary file renaming strategies when necessary to ensure reliability.

[0042] IV. Parallelization and Resuming from Breakpoints

[0043] Parallelization: The question generation and merging process, as well as the sample-level question-answer generation process, can be executed in parallel using multiple processes. The number of processes can be adaptively set based on the number of processor cores and external service limits.

[0044] Resume execution from breakpoint: Record batch results and iteration status in stages; resume execution from the latest completed batch and iteration round when restarting to avoid duplicate calculations.

[0045] V. Parameters and Engineering Considerations

[0046] Key parameters: batch size B, merge threshold K, number of questions and answers per sample q, maximum number of retries, etc.

[0047] Image processing: Size constraints and lossy compression are applied to input images to balance fidelity and efficiency; multiple images can be input sequentially or side-by-side.

[0048] Reliability: The parsing process is compatible with possible wrapper characters, prefixes and suffixes, etc., to ensure that the results can be correctly deserialized; abnormal and failed samples are recorded for subsequent compensation processing.

[0049] This invention targets single-modal or multimodal raw data consisting of "images + text materials," employing a two-stage process to automatically generate Visual Question Answering (VQA) data: The first stage reads samples in batches, constructs unified instructions based on the text materials within each batch, and uses a large language model to extract discriminative, pedagogically valuable questions that can be answered from images. Candidate questions across batches are grouped and structurally deduplicated according to a "cumulative to threshold before merging" rule, resulting in a deduplicated question pool. The second stage identifies usable image modalities at the sample level, filters matching questions from the question pool, randomly selects more than a target number of candidates, constructs a "question based solely on image" request, and generates question-answer pairs based on the image input. If a candidate question is unanswerable, it is replaced until the target number is reached, and answer balancing control is performed for binary judgment questions. The results are securely stored on disk after structured verification. This solution offers advantages such as versatility, controllable quality, parallelism, and breakpoint resume capability, significantly reducing manual annotation costs and adapting to data construction needs of different sources and scales.

[0050] Example 2

[0051] This embodiment relates to a medical image intelligent recognition method that applies the question-and-answer data construction method described in Embodiment 1: In the data construction phase, the method described in Example 1 is used to generate a visual question-and-answer dataset in batches from medical imaging data and its accompanying reports. The medical images include, but are not limited to, single-modal or multi-modal images such as X-ray films, CT scan images, and MRI images. The accompanying reports include text information such as image descriptions, diagnostic conclusions, and lesion characteristics. Through the question generation and deduplication merging process in Example 1, a variety of clinically relevant questions are extracted, including lesion identification questions (such as "Does a nodule exist in the image?"), location questions (such as "Which lung lobe is the lesion located in?"), morphological description questions (such as "Are the nodule edges smooth?"), and density feature questions (such as "Is the lesion density high or low relative to the surrounding tissues?"). Through the generation constraint of answering only based on images, it is ensured that all question-and-answer pairs can be answered directly from the images themselves, without relying on external information such as patient history or laboratory tests. The generated structured question-answering dataset can be directly used for supervised or instructional fine-tuning of a multimodal medical language model. The training objective is to enable the model to learn the ability to "receive medical image input → identify visual features in the image → answer clinical observation questions". The structured format of the dataset (including fields such as image path, question text, answer text, and question type) can be seamlessly integrated with mainstream deep learning training frameworks. In clinical applications, the intelligent consultation model trained on the dataset can be deployed in the following medical scenarios: assisting radiologists in quickly understanding image features and obtaining key information such as lesion location, size, and shape through question-and-answer interaction; providing preliminary image interpretation references for primary healthcare institutions, reducing reliance on expert resources from high-level hospitals; and using it for image interpretation practice in standardized residency training, deepening the understanding of typical image manifestations through question-and-answer interaction. With advantages in quality and efficiency, the method of this invention ensures that question-answer pairs are answerable based on images through answerability constraints, guarantees question diversity through deduplication and recursion, and avoids answer bias through balance control. The constructed dataset can provide high-quality learning samples for model training. Compared with the traditional medical image question-answering dataset, which relies entirely on medical experts to annotate each question one by one, the method of this invention can significantly reduce the workload of expert annotation and shorten the data preparation cycle. At the same time, it meets the stability requirements of large-scale medical dataset construction through parallelization and breakpoint resume mechanism.

[0052] This embodiment applies the question-and-answer dataset obtained in Embodiment 1 to intelligent medical image recognition and consultation, achieving good technical results.

[0053] Example 3

[0054] This embodiment relates to an apparatus for implementing the question-answering data construction method of Embodiment 1, comprising: The data acquisition module is used to read samples containing image and text data in batches; The question generation module is used to construct unified instructions and call a large language model to refine candidate questions; The deduplication and merging module is used to group by a threshold K and perform structured deduplication to form a problem pool; The question-answering generation module is used to filter candidate questions at the sample level, construct requests that answer only based on images, and generate question-answer pairs, while controlling answerability and balance. The storage and export module is used to perform structured verification and secure disk write-to-disk operations, and provides support for parallel processing and breakpoint resume.

[0055] Example 4

[0056] This embodiment relates to an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the question-and-answer data construction method described in Embodiment 1.

[0057] Example 5

[0058] This embodiment relates to a computer-readable storage medium storing a computer program that, when executed by a processor, implements the question-and-answer data construction method of Embodiment 1.

Claims

1. A method for constructing question-and-answer data, comprising the following steps: (1) Read samples containing images and text data in batches, construct a unified instruction for each batch of samples, call the large language model to extract candidate questions that are discriminative, have teaching value and can be answered from images, and form multiple sets of candidate questions; (2) Perform structured deduplication and merging on the cross-batch candidate problems obtained in step (1) to obtain a deduplicated problem pool; (3) For each sample, identify its available image modality, filter candidate questions that match the modality from the question pool, and randomly select a number of candidates that are greater than the target value q; (4) Construct a request that answers based solely on the image, take the sample image as input and call the multimodal large language model to generate question-answer pairs; if the candidate question cannot be reliably answered based solely on the image, replace it with a new candidate and retry until q answerable question-answer pairs are accumulated; (5) Perform structured validation and secure disk write-to-database operations on the generated question-and-answer pairs; whereby, The method supports parallel processing and breakpoint resume to meet the needs of large-scale data construction.

2. The method for constructing question-and-answer data as described in claim 1, characterized in that, The structured deduplication and merging in step (2) specifically includes: collecting and merging several candidate question groups in sequence until the number of questions in the group reaches the threshold K, and handing the group over to the large language model for structured merging; taking each merged group as a new round of candidate groups, repeating the grouping and merging process until only one group remains, thereby obtaining the final deduplicated question pool; and recording the iteration status after each round of merging to support breakpoint continuation.

3. The method for constructing question-and-answer data as described in claim 1, characterized in that, Step (4) includes: (4.1) For binary judgment questions, the positive and negative proportions of the answers should be kept basically balanced; (4.2) Set the maximum number of retries. Replace or retry when a candidate question is unanswerable or fails the structured validation. (4.3) Only retain answers that can be directly supported by images, and discard question-answer pairs that rely on external knowledge or textual materials to answer.

4. The method for constructing question-and-answer data as described in claim 1, characterized in that, The image processing in step (4) includes: limiting the size of the image and lossy compression without affecting the key visual information, and submitting it with the request in the form of data encoding; the request text clearly states the constraint of "answering only based on the image", and includes a list of candidate questions and necessary task descriptions.

5. The method for constructing question-and-answer data as described in claim 1, characterized in that, The structured verification and secure disk write-to-disk step (5) includes: verifying the integrity, type and value range of fields; removing non-compliant entries; saving the results in a detailed or log file format written one by one, and using atomic write and temporary file renaming strategies to improve the reliability of concurrent writes when necessary.

6. A question-answering data construction apparatus, comprising: The data acquisition module is used to read samples containing image and text data in batches; The question generation module is used to construct unified instructions and call a large language model to refine candidate questions; The deduplication and merging module is used to group by a threshold K and perform structured deduplication to form a problem pool; The question-answering generation module is used to filter candidate questions at the sample level, construct requests that answer only based on images, and generate question-answer pairs, while controlling answerability and balance. The storage and export module is used to perform structured verification and secure disk write-to-disk operations, and provides support for parallel processing and breakpoint resume.

7. An electronic device comprising a processor and a memory, the memory storing a computer program, the processor being configured to execute the computer program to implement the question-and-answer data construction method of the present invention.

8. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method for constructing question-and-answer data according to any one of claims 1-5.

Images