Medical visual question and answer model construction method based on feedback loop

Abstract

The invention discloses a medical visual question and answer model construction method based on a feedback loop. The method comprises the following steps: step 1, generating initial candidates; step 2, performing multi-dimensional candidate evaluation; 3, iterative optimization of feedback driving is carried out; step 4; and 5, efficient fine adjustment of model parameters: performing fine adjustment on a pre-trained multi-modal large language model by using the enhanced training data set constructed in the step 4 to obtain a final medical visual question and answer model. According to the method, a feedback loop mechanism is introduced, and cooperative and iterative self-correction and optimization are carried out on generated text interpretation (Ratio) and a visual localization region of interest (RoI), so that the fact reliability, the logic consistency and the visual strong relevance of a final output result are systematically ensured.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence computer vision question answering, and particularly relates to a method for constructing a medical vision question answering model based on feedback loop. Background Technology

[0002] Medical Visual Question Answering (Med-VQA) is an interdisciplinary research area at the intersection of artificial intelligence and medical image analysis. It aims to enable computer systems to automatically reason and answer questions based on medical images and natural language processing. This technology has significant potential for assisting diagnosis and improving the efficiency of medical work, and has therefore attracted widespread attention.

[0003] Currently, the technical implementation path of Med-VQA has evolved from early classification models to generative models centered on Multimodal Large Language Models (MLLM). These models can generate free-form answers based on image and text information, and by introducing techniques such as "Chain-of-Thought" (CoT), guide the model to generate a series of intermediate reasoning steps before providing the final answer, thereby improving the interpretability and logical coherence of the output content to a certain extent.

[0004] However, despite the progress made in generative methods based on thought chains, existing generative methods still suffer from reliability issues. First, the model-generated answers often contain descriptions that do not match the image content, a phenomenon known as "hallucination." This problem is particularly critical in medical diagnostic scenarios, as it can lead to erroneous conclusions or reduce the verifiability of results. Second, there is a lack of tight binding between the model's textual reasoning and the visual evidence it relies on. Even if the model mentions the correct lesion, its internal attention mechanism may not precisely focus on the corresponding area in the image. When the model cannot accurately focus on the relevant area during the response process, it is prone to generating content based on language patterns or statistical associations rather than on real visual evidence, resulting in answers that deviate from the image facts. This disconnect between text and vision makes the model's reasoning process difficult to verify effectively, reducing the credibility of the results. Furthermore, the reasoning process of existing thought chains is mostly a linear, unidirectional generation process. The model generates answers step by step from the initial input along a fixed reasoning path, lacking the ability to actively backtrack, cross-validate, or dynamically correct during the generation process. Once factual deviations or logical errors occur in the early stages, these errors are continuously propagated and amplified, ultimately leading to a significant decline in the quality of the entire answer. Summary of the Invention

[0005] To overcome the shortcomings of existing one-way generation processes, such as the easy propagation and solidification of early deviations, the lack of self-verification in the reasoning process, and the discrepancy between visual positioning and text / image, this invention provides a medical visual question-answering model construction method based on feedback loops. It constructs a mechanism capable of intermediate checks and dynamic adjustments during the generation process, introducing a feedback loop mechanism to collaboratively and iteratively self-correct and optimize the generated text interpretation (Rationale) and the visually located Region of Interest (RoI), thereby systematically ensuring the factual reliability, logical consistency, and strong visual correlation of the final output results.

[0006] The technical solution adopted by this invention to solve its technical problem is:

[0007] A method for constructing a medical visual question-answering model based on feedback loops, the method comprising the following steps:

[0008] Step 1, Initial Candidate Generation: For the input medical image and the corresponding clinical question, an initial generation module generates multiple combinations in parallel, each containing a candidate text interpretation (Rationale) and a candidate Region of Interest (RoI). Each candidate combination includes a text interpretation and corresponding visual localization information, which are used to explore multiple reasoning paths for the same question.

[0009] Step 2, Multidimensional Candidate Evaluation: An independent verification module is used to comprehensively score each candidate combination generated in Step 1 based on the logicality and factuality of the textual explanation and its alignment with the visual positioning.

[0010] Step 3, Feedback-driven Iterative Optimization: Based on the scoring results of Step 2, if there are candidate combinations that meet the preset quality threshold, proceed to the next step; if none of the candidate combinations meet the threshold, aggregate all unqualified candidate combinations, format their information into a new negative correction instruction, and feed it back to the generation module of Step 1. The generation module then uses this signal to perform a new, more targeted generation. This process is repeated until a qualified candidate combination is produced or the maximum number of iterations is reached.

[0011] Step 4: Enhance training data synthesis: Select the best text interpretation-RoI combination (R) that passed the final validation in Step 3. f B f This is used as a high-quality training sample and stored in a new augmented training dataset, where candidate RoIs (B) are included. fThese will be directly and explicitly labeled on the original medical images to clearly indicate the lesions or anatomical structures of interest for the text interpretation, thus forming high-quality training samples with visual annotations; Steps 1 to 3 are repeated for all image-question pairs in the original dataset.

[0012] Step 5: Efficient fine-tuning of model parameters: Using the augmented training dataset constructed in Step 4, fine-tune a pre-trained multimodal large language model to obtain the final medical visual question answering model.

[0013] Furthermore, in step 1, the initial candidate generation process is as follows: A pair of medical images (V) and a clinical question (Q) are input, and a multimodal large language model is used as the generation module to generate N candidate combinations {(R1,B1),(R2,B2),...,(R...} in parallel. N B N )}, where R i For the interpretation of the i-th candidate text, B i Let be the bounding box coordinates of the i-th candidate RoI. This parallel generation process aims to explore multiple different inference paths for the same problem.

[0014] Furthermore, step 2 includes the following sub-steps:

[0015] 2.1 Logical Consistency Assessment: Examining Candidate Text Interpretations in R i Does it contain logical contradictions or reasoning that does not conform to clinical common sense?

[0016] 2.2 Factual Accuracy Assessment: Validating Candidate Text Interpretation R i Does the conclusion align with the medical knowledge relevant to this issue?

[0017] 2.3 Visual Alignment Evaluation: Evaluating Candidate Text Interpretation R i Whether the lesions or anatomical structures described in the text are related to the candidate RoI(B) i The area defined in image V corresponds to this region;

[0018] 2.4 Overall Score: Combining the evaluation results of the above three sub-steps, a quantitative confidence score is given for each candidate combination.

[0019] Step 3 includes the following sub-steps:

[0020] 3.1 All candidate combinations with confidence scores below the threshold {(R failed B failed Aggregate;

[0021] 3.2 Format these failed attempts and integrate them into a new prompt, which has a structure similar to: "The following are previous failed attempts; please avoid repeating the incorrect formatting and logic: [Failed Attempt 1: Explanation (R)" failed1 ), Positioning (B) failed1 )]...”;

[0022] 3.3 This new instruction containing the negative correction signal, along with the original image and problem, is re-entered into the generation module of step 1 to initiate a new round of higher-quality generation.

[0023] Step 4 involves combining the best text interpretation that passed verification in step 3 with the RoI (RoI). f B f The candidate RoIs (B) are selected and stored as high-quality training samples in the augmented training dataset. f During the storage process, the data will be explicitly labeled on the original medical images in the form of bounding boxes to clearly indicate the lesions or anatomical structures of interest in the text interpretation, thereby forming training samples with visual annotations and providing a data foundation for subsequent model fine-tuning.

[0024] Step 5 includes the following sub-steps:

[0025] 5.1 Each high-quality sample (R) synthesized in step 4 f B f This is used to construct a specific training instruction. This instruction incorporates the explicitly labeled image features (V), questions (Q), and text interpretations (R). f These factors, along with the input, guide the model to generate the final answer.

[0026] 5.2 When training the linear projection layer, every four adjacent visual tokens are concatenated together to reduce the number of input tokens, thereby improving training and inference efficiency;

[0027] 5.3 During training, different task identifiers are used to distinguish different tasks, enabling the model to better understand and perform various visual language tasks;

[0028] 5.4 The LoRa (LoR) technique is used to update only a small portion of the parameters in the multimodal large language model;

[0029] 5.5 The standard cross-entropy loss function is used for training, and the entire question-answering model is optimized through gradient backpropagation.

[0030] The beneficial effects of this invention are mainly reflected in the introduction of a feedback loop mechanism to perform collaborative and iterative self-correction and optimization of the generated text interpretation (Rationale) and the visually located Region of Interest (RoI), thereby systematically ensuring the factual reliability, logical consistency and strong visual correlation of the final output results. Attached Figure Description

[0031] Figure 1 This is a flowchart of a medical visual question-answering model construction method based on feedback loops.

[0032] Figure 2 This is a schematic diagram of the efficient fine-tuning process of model parameters. Detailed Implementation

[0033] The present invention will now be further described with reference to the accompanying drawings.

[0034] Reference Figure 1 and Figure 2 A feedback loop-based method for building a medical visual question answering model is proposed. By iteratively optimizing the original image-question pairs, a high-quality augmented training dataset with textual explanations and visually located regions of interest (RoIs) is automatically synthesized. This dataset is then used to fine-tune a downstream multimodal large language model, thereby achieving a highly reliable and interpretable medical visual question answering task.

[0035] The medical visual question-answering model construction method based on feedback loop in this embodiment includes the following steps:

[0036] (1) Initial candidate generation process: For the input medical images and clinical questions, the initial generation module generates multiple initial combinations in parallel, which include candidate text interpretation and visual localization.

[0037] In this embodiment of the invention, in the specific operation of step (1), the initial generation module can employ a high-performance multimodal large language model (such as Gemini). This module receives external input medical images V and clinical questions Q. Based on this input, the initial generation module generates N candidate combinations in parallel {(R1,B1),(R2,B2),...,(R...B1)}. N B N )}, where the preferred value of N is 3. In this candidate combination, R i The interpretation of the i-th candidate text is a natural language text describing a reasoning process; B i Let R be the i-th candidate region of interest, which can be specifically represented as the coordinates of one or more bounding boxes, used to mark R on the image V. iThe key area described. This step is designed to explore multiple different reasoning paths for the same problem.

[0038] (2) Multidimensional candidate evaluation process: Each candidate combination generated in step (1) is comprehensively evaluated and scored.

[0039] In an embodiment of the present invention, in step (2), the verification module performs a quality evaluation on the N candidate combinations generated in step (1). To ensure the independence and objectivity of the verification results and to avoid the model potentially exhibiting a consistent tendency in its output results based on the correlation of its internal parameters or generation preferences, the verification module is undertaken by another high-performance multimodal large language model (such as ChatGPT). The evaluation criteria may specifically include: logical consistency, i.e., evaluating the text interpretation R... i Does the text contain self-contradictory or clinically illogical statements? Is the factual accuracy assessed, i.e., is the text interpretation R? i Whether the final conclusion aligns with the standard medical answer or accepted medical knowledge for the question; and visual alignment, i.e., assessing the text interpretation R. i Whether the specific lesions or imaging features mentioned are related to RoI(B) i The visual regions defined in image V are highly consistent. Based on the above criteria, the verification module outputs a quantified confidence score for each candidate combination.

[0040] (3) Feedback-driven iterative optimization process: Based on the score in step (2), information is aggregated for unqualified candidate combinations and fed back to step (1) as a negative correction signal to start a new round of optimization generation until a combination that meets the quality standard is produced.

[0041] In this embodiment of the invention, in step (3), the feedback control module checks the confidence scores of all candidate combinations. If all scores are lower than a preset quality threshold, it indicates that all initial candidates are unqualified. At this time, the module will process all these unqualified candidate combinations {(R failed B failed Aggregate the data and construct a negative correction instruction. For example, this instruction could be a new prompt that reads: "The following are previous failed attempts; please avoid repeating their incorrect formatting and logic: [Failed Attempt 1: Explanation (R)" failed1 ), Positioning (B) failed1 The new instruction containing the correction signal, along with the original image and the problem, is re-input into the generation module of step (1) to initiate a new round of generation aimed at correcting known errors. This feedback loop process can be set to a maximum number of iterations K, in which, in this embodiment, the preferred value of K is 3.

[0042] (4) Efficient fine-tuning process of model parameters: Use the high-quality combination finally determined in step (3) to construct an enhanced training dataset and fine-tune a pre-trained multimodal large language model.

[0043] In this embodiment of the invention, in step (4), after a high-quality optimal combination (Rf, Bf) passes verification, this combination is used as a training sample and stored in a brand new augmented training dataset. The process described in (1) to (3) is repeated for all image-question pairs in the original dataset to construct a complete augmented training dataset. Subsequently, as... Figure 2 As shown, this augmented training dataset is used to fine-tune a pre-trained multimodal large language model, which consists of three parts: an EVA model as a visual encoder for extracting image features; a linear projection layer for projecting visual features onto the same dimensional space as text features to achieve cross-modal information alignment; and an LLaMA2-chat(7B) model as the language backbone.

[0044] The linear projection layer employs a method to enhance local information representation when processing visual features. Specifically, before entering the linear projection layer, every four adjacent feature vectors (e.g., within a 2×2 neighborhood) of the two-dimensional grid features output by the visual encoder are concatenated into a new feature vector with a higher dimension. This transforms the originally long visual feature sequence into a shorter sequence with richer local information per token. Subsequently, the linear projection layer maps this high-dimensional visual feature sequence to the same text feature space as the language model, achieving multimodal alignment. The language backbone is designed as a unified interface capable of handling multiple vision-language tasks. This is achieved by embedding unique task identifiers in the input instructions; for example, [vqa] indicates that the model performs a visual question answering task, while [caption] indicates that the model performs an image description task. To ensure that the model has strong multimodal instruction compliance capabilities before fine-tuning, the initial weights of the model in this embodiment use the third-stage pre-trained weights of MiniGPT-v2 as the initial weights.

[0045] The model fine-tuning process is divided into two stages: Stage 1: General medical domain training. To enable the model to understand general medical images, this invention uses multiple publicly available medical image datasets for initial training, including ROCO, MIMIC-CXR, and OpenI. In this stage, the task identifier embedded in the training instructions is [caption]. Stage 2: Med-VQA task fine-tuning based on augmented data. In this stage, this invention performs augmented synthesis on multiple publicly available Med-VQA benchmark datasets (e.g., VQA-RAD, SLAKE, PathVQA) according to steps (1) to (3), forming a pair of inference text and visual regions (R...f B f The model is fine-tuned using an augmented training dataset, with the task identifier [vqa] embedded, constructed as a structured training instruction, such as: "[INST]". {Image Feature Placeholder} [vqa]Question: {Question Q}Let's think step by step. {Text Explanation Rf}Therefore, the final answer is:[ / INST]". The part up to [ / INST] in this instruction constitutes the model's input, and the target the model is trained to generate is the standard answer to the question. The training hyperparameters for this stage can be set as follows: input image resolution uniformly 448×448 pixels, maximum input text length of 1024 tokens, AdamW optimizer, and learning rate set to 1×10⁻⁶. -5 The training rounds consist of 50 epochs.

[0046] Throughout the fine-tuning process, to achieve efficient parameter training, the parameters of the visual encoder EVA are kept frozen, with only the parameters of the linear projection layer and the LoRA module in the language backbone being updated. Training is performed using the standard cross-entropy loss function. This loss function calculates the difference between the logits distribution of the model's output at the last token position and the one-hot encoding of the standard answer. The above training process of this invention can be completed on a single NVIDIA A6000 graphics processor.

[0047] The embodiments described in this specification are merely examples of implementations of the inventive concept and are for illustrative purposes only. The scope of protection of this invention should not be considered limited to the specific forms described in these embodiments; rather, it extends to equivalent technical means conceived by those skilled in the art based on the inventive concept.

Claims

1. A method for constructing a medical visual question-answering model based on feedback loops, characterized in that, The method includes the following steps: Step 1, Initial Candidate Generation: For the input medical image and the corresponding clinical question, an initial generation module generates multiple combinations containing candidate text explanations and candidate regions of interest in parallel. Each candidate combination includes a text explanation and corresponding visual localization information, which is used to explore multiple reasoning paths for the same question. Step 2, Multidimensional Candidate Evaluation: An independent verification module is used to comprehensively score each candidate combination generated in Step 1 based on the logicality and factuality of the textual explanation and its alignment with the visual positioning. Step 3, Feedback-driven iterative optimization: Determine the scoring results of Step 2. If there are candidate combinations that meet the preset quality threshold, proceed to the next step. If none of the candidate combinations meet the threshold, aggregate all unqualified candidate combinations and format their information into a new negative correction instruction. Feedback is sent to the generation module of Step 1, and the generation module performs a new, more targeted generation based on this signal. This process is repeated until a qualified candidate combination is produced or the maximum number of iterations is reached. Step 4: Enhance training data synthesis: Select the best text interpretation-RoI combination (R) that passed the final validation in Step 3. f B f This is used as a high-quality training sample and stored in a new augmented training dataset, where candidate RoIs (B) are included. f These will be directly and explicitly labeled on the original medical images to clearly indicate the lesions or anatomical structures of interest in the text interpretation, thus forming high-quality training samples with visual annotations. Repeat steps 1 through 3 for all image-problem pairs in the original dataset; Step 5: Efficient fine-tuning of model parameters: Using the augmented training dataset constructed in Step 4, fine-tune a pre-trained multimodal large language model to obtain the final medical visual question answering model.

2. The method for constructing a medical visual question-answering model based on feedback loops as described in claim 1, characterized in that, In step 1, the initial candidate generation process is as follows: A pair of medical images V and a clinical question Q are input. A multimodal large language model is used as the generation module to generate N candidate combinations {(R1,B1),(R2,B2),...,(R...} in parallel. N B N )}, where R i For the interpretation of the i-th candidate text, B i The bounding box coordinates of the i-th candidate RoI are given.

3. The method for constructing a medical visual question-answering model based on feedback loops as described in claim 2, characterized in that, Step 2 includes the following sub-steps: 2.1 Logical Consistency Assessment: Examining Candidate Text Interpretations in R i Does it contain logical contradictions or reasoning that does not conform to clinical common sense? 2.2 Factual Accuracy Assessment: Validating Candidate Text Interpretation R i Does the conclusion align with the medical knowledge relevant to this issue? 2.3 Visual Alignment Evaluation: Evaluating Candidate Text Interpretation R i Whether the lesion or anatomical structure described in the image corresponds to the region bounded by the candidate RoI in image V; 2.4 Overall Score: Combining the evaluation results of the above three sub-steps, a quantitative confidence score is given for each candidate combination.

4. The method for constructing a medical visual question-answering model based on feedback loops as described in claim 3, characterized in that, Step 3 includes the following sub-steps: 3.1 All candidate combinations with confidence scores below the threshold {(R failed B failed Aggregate; 3.2 Format these failed combination messages and integrate them into a new instruction, Prompt; 3.3 This new instruction containing the negative correction signal, along with the original image and problem, is re-entered into the generation module of step 1 to initiate a new round of higher-quality generation.

5. The method for constructing a medical visual question-answering model based on feedback loops as described in claim 4, characterized in that, Step 4 involves combining the best text interpretation that passed verification in step 3 with the RoI (RoI). f B f The candidate RoIs (B) are selected and stored as high-quality training samples in the augmented training dataset. f During the storage process, the data will be explicitly labeled on the original medical images in the form of bounding boxes to clearly indicate the lesions or anatomical structures of interest in the text interpretation, thereby forming training samples with visual annotations and providing a data foundation for subsequent model fine-tuning.

6. The method for constructing a medical visual question-answering model based on feedback loops as described in claim 5, characterized in that, Step 5 includes the following sub-steps: 5.1 Each high-quality sample (R) synthesized in step 4 f B f This is used to construct a specific training instruction, which takes the explicitly labeled image features V, question Q, and text interpretation R as input. f These are used as inputs to guide the model in generating the final answer; 5.2 When training the linear projection layer, every four adjacent visual tokens are concatenated together to reduce the number of input tokens, thereby improving training and inference efficiency; 5.3 During training, different task identifiers are used to distinguish different tasks, enabling the model to better understand and perform various visual language tasks; 5.4 The LoRa (LoR) technique is used to update only a small portion of the parameters in the multimodal large language model; 5.5 The standard cross-entropy loss function is used for training, and the entire question-answering model is optimized through gradient backpropagation.

Images