A method and device for generating medical image question-answering thinking chain data in multiple stages

Abstract

This invention discloses a multi-stage method and device for generating thought chain data for medical image question-answering. By introducing a multi-stage model collaborative reasoning mechanism, the invention sequentially completes the extraction of key image content, the generation of enhanced image descriptions, and the construction of thought chains, thereby improving the accuracy, pertinence, and interpretability of medical image question-answering tasks. It addresses issues such as inaccurate image detail recognition, opaque reasoning processes, and generalized question-answering content in existing models.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal large models, and specifically relates to a method and device for generating medical image question-answering thinking chain data in multiple stages. Background Art

[0002] With the continuous advancement of artificial intelligence technology, especially the integration of multi-modal large models (MLLMs) in language processing and image understanding, medical image analysis and question-answering systems have gradually become important tools for assisting clinical decision-making. Existing medical image-assisted diagnosis methods generally rely on computer vision models to extract and classify medical images. Although they have achieved some success in tasks such as structure recognition and lesion detection, they have significant limitations when processing complex natural language problems or performing explainable reasoning, making it difficult to meet the comprehensive requirements of image and text understanding and interactive reasoning in medical scenarios.

[0003] Large Language Models (LLMs) have demonstrated excellent capabilities in text generation, semantic understanding, and Chain-of-Thought (CoT) reasoning in recent years and have been widely used in a variety of natural language processing tasks. To improve the reasoning capabilities of medical image question answering (VQA), some studies have attempted to integrate language models with visual models to achieve joint image-text modeling. However, existing methods generally have the following technical deficiencies: (1) It is difficult to accurately identify the key image details required to answer medical questions, resulting in a lack of specificity; (2) There is a lack of a clear chain-of-thought generation mechanism, resulting in poor interpretability of the output results; (3) The model's ability to integrate language information with image information is insufficient, affecting the coherence of the reasoning process and the reliability of the conclusions.

[0004] More critically, there is currently a lack of high-quality annotated data in the field of medical multimodality, especially structured training samples that simultaneously include images, questions, answers, and multi-step thought chains. This type of data plays a fundamental role in building large multimodal models with deep reasoning capabilities and medical interpretability. However, due to the high professionalism of medical images and the high cost of manually constructing thought chains, existing public data resources are difficult to meet training needs. At the same time, current pre-training models with strong language reasoning capabilities are mainly based on large-scale pure language data and cannot be directly migrated for medical image-text joint reasoning tasks, further restricting the development of high-performance multimodal medical models.

[0005] Therefore, there is an urgent need for a new data construction method that can fully utilize the advantages of pre-trained strong inference models in inference generation, and combine the ability of large multimodal models to understand the content of medical images, to automatically construct high-quality question-answering data containing multimodal information and thought chains, thereby providing effective support for training medical multimodal inference models. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the present invention provides a method and apparatus for generating multi-stage thought chain data for medical image question-answering. By introducing a multi-stage model collaborative reasoning mechanism, the present invention sequentially completes the extraction of key image content, the generation of enhanced image descriptions, and the construction of thought chains, thereby improving the accuracy, pertinence, and interpretability of medical image question-answering tasks. It addresses issues such as inaccurate image detail recognition, opaque reasoning processes, and generalized question-answering content in existing models.

[0007] In a first aspect, an embodiment of the present application provides a method for generating medical image question-answering thought chain data in multiple stages, comprising the following steps:

[0008] Obtain the open-source medical visual question answering (VQA) dataset to obtain triplet data containing medical images, questions, and ground truth answers.

[0009] Extracting keypoint information in medical image question answering using explicit reasoning language models.

[0010] Based on the extracted key point information, enhanced image description is generated through a multimodal medical large model.

[0011] Generate initial thought chain text and answers based on enhanced image description.

[0012] The generated initial thought chain text and answers are automatically post-processed to generate the final thought chain text and answers.

[0013] The final thought chain text and answers, as well as the original medical images and questions, are integrated into the medical image question-answering thought chain data.

[0014] In one possible implementation, keypoint information extraction is performed as follows: the question and corresponding answer from the medical visual question-answering task are embedded in a first prompt word template as input text, which is then fed into an explicit reasoning language model (i.e., a pre-trained language model with language understanding and strong reasoning capabilities). By analyzing the semantic content of the input question-answer pair, the image keypoint information necessary to answer the question is inferred.

[0015] In one possible implementation, the first prompt word template needs to include placeholders for both the question and the answer, and through instructions, the model is focused on extracting the key image elements required to answer the question-answer pair, while specifying a unified output format for subsequent processing.

[0016] Through natural language inference, the explicit language model analyzes the implicit logic between questions and answers, extracting the most relevant image observations for the question-answering scenario. The output is a structured list of key points, or image keypoint information, containing areas, organ locations, and structural features of particular interest within the image.

[0017] In one possible implementation, enhanced image description generation is performed as follows: the obtained image keypoint information is paired with the corresponding medical image data, both are embedded in a second prompt word template, and then input into a multimodal medical model capable of integrating image and text. Upon receiving this input, the multimodal medical model generates a clearly structured, semantically rigorous, and medically compliant image description, known as an enhanced image description.

[0018] In a possible implementation, the second prompt word template needs to include both image key point information and placeholders for medical image data, and instruct the model to output image description text according to the image key point information through instructions.

[0019] In one possible implementation, the initial thought chain text and answer are generated as follows: the generated image description text, along with the corresponding question and standard answer, are embedded in a third prompt word template and input into an explicit reasoning language model. Based on the input multi-source information, the explicit reasoning language model outputs the initial thought chain text and answer through a language generation mechanism. The thought chain text reflects the reasoning logic path from the image representation to the medical conclusion.

[0020] In a possible implementation, the third prompt word template needs to include both the image description text and the placeholders for the corresponding question and standard answer, and through instructions, the model is instructed to output the initial thought chain text and answer based on the description text, the corresponding question and the standard answer. The output format in the instructions needs to be limited to " <think> …< / think> <answer> …< / answer> ”, “ <think> …< / think> "The middle package thinking chain text," <answer> …< / answer> "The middle package standard answer.

[0021] In one possible implementation, the automated post-processing specifically includes the following processing steps:

[0022] (1) Keyword detection and marking: A set of keywords or phrases with explicit input prompt features is preset to automatically detect the thought chain text based on string matching and regular expressions, identify the sentences containing such content, and locate and mark them as abnormal sentences, providing target areas for subsequent processing.

[0023] (2) Content rewriting and semantic reconstruction: For detected abnormal sentences, predefined rule templates are used to rewrite them. The rule templates replace explicit wording that contains input source hints. The goal of rewriting is to eliminate explicit references to the input source in the text, so that the reasoning process simulates the language expression under multimodal input.

[0024] (3) Semantic consistency verification: After the content is rewritten, to ensure that the text semantics have not shifted, the original text and the processed text are verified for semantic consistency. The semantic similarity calculation method based on the embedding vector is used to ensure that the continuity of the reasoning logic and the accuracy of the conclusion are not affected during the modification process.

[0025] In a second aspect, an embodiment of the present application provides a device for generating medical image question-answering thought chain data in multiple stages, including:

[0026] Data acquisition module: Obtain the open source medical visual question answering (VQA) dataset to obtain triple data containing medical images, questions, and standard answers.

[0027] Key point information extraction module: uses explicit reasoning language model to extract key point information in medical image question answering.

[0028] The question and corresponding answer in the medical visual question answering task are embedded in the first prompt word template as input text and fed into the explicit reasoning language model. By analyzing the semantic content of the input question and answer pair, the image key points information necessary to answer the question are inferred.

[0029] Image description generation module: Based on the extracted key point information, enhanced image description is generated through a multimodal medical large model.

[0030] The module generates initial thought chain text and answers: This module embeds the generated image description text, along with the corresponding question and standard answer, into a third prompt word template and feeds this into the explicit reasoning language model. Based on the multi-source input, the explicit reasoning language model outputs the initial thought chain text and answer through a language generation mechanism. This thought chain text reflects the logical reasoning path from image representation to medical conclusion.

[0031] Automated post-processing module: performs keyword detection and tagging, content rewriting and semantic reconstruction, and semantic consistency verification on the generated initial thought chain text and answers to generate the final thought chain text and answers.

[0032] The final thought chain text and answers, as well as the original medical images and questions, are integrated into the medical image question-answering thought chain data.

[0033] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the implementation of the first aspect as described above.

[0034] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the implementation of the first aspect above.

[0035] The beneficial effects of the present invention are as follows:

[0036] The present invention introduces a multi-stage model collaborative reasoning mechanism to sequentially complete the extraction of key image content, the enhanced generation of image descriptions, and the construction of thought chains to obtain the final thought chain text and answers. This can effectively suppress the generation bias caused by the language input structure, strengthen the multimodal reasoning expression effect of the model output, and improve the professionalism, naturalness, and usability of the final thought chain text, thereby providing higher-quality reasoning data support for subsequent model training or evaluation. It further improves the naturalness, logic, and clinical acceptability of the model-generated reasoning text, making the final output thought chain text closer to the professional medical reasoning process, and enhancing its practical value in auxiliary diagnosis and medical decision support systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments described in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0038] Figure 1 A complete flow chart of the method for generating medical image question-answering thought chain data in multiple stages of the present invention;

[0039] Figure 2 A comparison chart of an image description directly generated using HealthGPT in an embodiment of the present invention and an enhanced image description generated by the method of the present invention;

[0040] Figure 3 This is a comparison diagram of the thought chain directly generated using HealthGPT in an embodiment of the present invention and the thought chain generated by the method of the present invention. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0042] like Figure 1As shown, a method for generating medical image question-answering thought chain data in multiple stages is implemented as follows:

[0043] Step 1: Obtain the open-source medical visual question answering (VQA) dataset to obtain triplet data containing medical images, questions, and standard answers.

[0044] To implement the multi-stage method for generating thought chain data for medical image question answering, we first need to obtain a high-quality medical visual question answering dataset as the raw input data. This example uses the PathVQA dataset as the basis for subsequent key point extraction, image description generation, and thought chain construction.

[0045] PathVQA is a medical visual question-answering dataset focused on pathology. It contains approximately 32,799 question-answer pairs and 4,998 pathology images, covering a variety of imaging modalities and disease types. The dataset is available from the Hugging Face platform and is provided in Parquet format. The dataset includes the image file path, the question text, and the corresponding ground truth answer.

[0046] By acquiring the PathVQA dataset and converting it into JSON format, we obtain triple data containing medical images, questions, and standard answers, providing a solid data foundation for subsequent steps such as key point extraction, image description generation, and thought chain construction.

[0047] When using other medical VQA datasets, it is necessary to extract the image file path, question text, and corresponding standard answer according to their original format (such as CSV, Parquet, or HDF5), and uniformly convert them into a triple structure in JSON format to ensure the consistency and scalability of multi-source data in subsequent processing flows.

[0048] Step 2: Utilize explicit reasoning language model to extract key point information in medical image question answering.

[0049] The question and corresponding answer in the medical visual question-answering task are embedded in the first prompt word template as input text and fed into an explicit inference language model (i.e., a pre-trained language model with language understanding and strong reasoning capabilities). This example uses DeepSeek R1 as the explicit inference language model. Although this model does not directly process images, it has strong reasoning capabilities. It can analyze the semantic content of the input question-answer pair and infer the image key points necessary to answer the question.

[0050] In this embodiment, the first prompt word template needs to include placeholders for both the question and the answer, and through instructions, the model is focused on extracting the key image elements required to answer the question-answer pair, while specifying a unified output format for subsequent processing.

[0051] Using natural language inference, the explicit reasoning language model parses the implicit logic between questions and answers, extracting the most relevant image observations for the specific question-answering scenario. The output is a structured list of key points, or image keypoint information, containing areas, organ locations, and structural features of particular interest. These key points have clear medical relevance, reflecting image details that are likely to be critically examined during a professional diagnosis.

[0052] The extracted key point information serves as a language guidance signal and will be used in the subsequent image description generation stage to guide the multimodal medical model to focus on image areas closely related to the problem, thereby improving the pertinence, accuracy and clinical readability of the generated image description.

[0053] Step 3: Generate enhanced image description based on the multimodal medical model.

[0054] The image keypoint information output in step 2 is paired with the corresponding medical image data, embedded in a second prompt word template, and fed into a multimodal medical model capable of integrating image and text. This model supports the collaborative modeling of image content and language cues, guiding the model to precisely focus on clinically relevant information areas during image analysis based on provided key regions or structural features.

[0055] In this embodiment, the second prompt word template needs to include both image key point information and placeholders for medical image data, and instruct the model to output image description text according to the image key point information through instructions.

[0056] After receiving the input, the multimodal medical model leverages its multimodal medical capabilities to generate a clearly structured, semantically rigorous, and medically compliant image description. This description, combined with the model's perception of medical images and guidance from language prompts, comprehensively covers key image structures, suspected lesions, and their spatial distribution, thereby enhancing the completeness of image information and clinical interpretability.

[0057] The image description generation process demonstrates the multimodal model's ability to model the semantics of medical image content and translate language. Guided by key point information, the generated results are highly professional and relevant to reasoning. This image description, as a structured image understanding result, together with the medical image, question, and standard answer, forms the input context for the subsequent reasoning model. Figure 2 This is a comparison chart of the image description directly generated using HealthGPT in an embodiment of the present invention and the enhanced image description generated by the method of the present invention; by introducing this description, the language model's ability to perceive the semantic background of the image is effectively enhanced, thereby improving the performance of the thought chain text in terms of logical consistency and medical credibility.

[0058] Step 4: Generate initial thought chain text and answers based on the enhanced image description.

[0059] The image description text generated in step 3 is embedded together with the corresponding question and standard answer into the third prompt word template and input into the explicit reasoning language model. As an example, this embodiment uses DeepSeek R1 as the explicit reasoning language model to generate the initial thought chain text and answer for medical image question answering. Its format is " <think> …< / think> <answer> …< / answer> This input data provides a complete reasoning context for DeepSeek R1 by fusing image descriptions with medical context information.

[0060] In this embodiment, the third prompt word template needs to contain the image description text and the placeholders for the corresponding questions and standard answers, and through instructions, the model outputs the initial thought chain text and answers based on the description text, the corresponding questions and standard answers. The output format in the instructions must be limited to " <think> …< / think> <answer> …< / answer> ”, “ <think> …< / think> "The middle package thinking chain text," <answer> …< / answer> "The middle package standard answer.

[0061] Based on the multi-source input, DeepSeek R1 uses a language generation mechanism to output initial thought chain text and answers. This thought chain text reflects the logical reasoning path from image representation to medical conclusion, covering key reasoning steps such as image feature recognition, medical significance determination, and clinical tendency assessment, and is both explainable and traceable.

[0062] Through this step, the device can output the initial thought chain text and answer, clearly derive the image basis and medical judgment process behind the standard answer, and thus provide more credible and verifiable reasoning support for medical image question-answering tasks. Figure 3 This is a comparison diagram of the thought chain directly generated using HealthGPT in an embodiment of the present invention and the thought chain generated by the method of the present invention.

[0063] In a possible implementation, the explicit reasoning language model used for extracting key point information and the explicit reasoning language model used for generating the initial thought chain text and answer based on the enhanced image description are selected according to actual needs and can be the same or different.

[0064] Step 5: Automatically post-process the generated initial thought chain text and answers to generate the final thought chain text and answers.

[0065] Because DeepSeek R1 has a certain degree of generative freedom when generating thought chain text, when it receives image descriptions as indirect input, it may be influenced by the explicit cues of the input text structure, resulting in expressions inconsistent with the intended reasoning process. For example, the model may reference surface information such as "image description" or "standard answer" in the output content, generating statements such as "as stated in the image description" or "according to the standard answer." This type of content exposes the linguistic properties of the input source, weakening the model's ability to "judge based on images" in reasoning. This is inconsistent with the design intent of simulating a real multimodal reasoning process and may also affect the naturalness and credibility of the thought chain text.

[0066] Therefore, this step automatically processes the thought chain text generated in step 4 to improve its expression quality and logical consistency. The specific processing flow includes the following:

[0067] (1) Keyword detection and marking: A set of keywords or phrases with explicit input prompt features (such as "image description" and "standard answer") are preset to automatically detect the thought chain text based on string matching and regular expressions, identify sentences containing such content, and locate and mark them as abnormal sentences, providing target areas for subsequent processing.

[0068] (2) Content rewriting and semantic reconstruction: For abnormal sentences detected, predefined rule templates are used to rewrite them. The rule templates need to replace explicit wording containing input source hints (such as image descriptions or standard answers), such as "as stated in the image description" or "according to the standard answer". The goal of rewriting is to eliminate explicit references to the input source in the text and make the reasoning process simulate the language expression under multimodal input. For example, "as stated in the image description" is rewritten as "observed in the image" to simulate the natural reasoning language under real multimodal input.

[0069] (3) Semantic consistency verification: After the content is rewritten, to ensure that the text semantics have not shifted, the original text and the processed text are verified for semantic consistency. The semantic similarity calculation method based on the embedding vector is used to ensure that the continuity of the reasoning logic and the accuracy of the conclusion are not affected during the modification process.

[0070] Obtaining the final thought chain text and answer through the above post-processing process can effectively suppress the generation bias caused by the language input structure, enhance the multimodal reasoning expression effect of the model output, and improve the professionalism, naturalness and usability of the final thought chain text, thereby providing higher-quality reasoning data support for subsequent model training or evaluation.

[0071] Through the above steps, the naturalness, logic and clinical acceptability of the model-generated reasoning text are further improved, making the final output thought chain text closer to the professional medical reasoning process, enhancing its practical value in auxiliary diagnosis and medical decision support systems.

[0072] Step 6: Integrate the final thought chain text and answers, as well as the original medical images and questions, into medical image question answering thought chain data.

[0073] Optionally, the explicit reasoning language model refers to a large reasoning language model that can explicitly output a complete thought chain, including DeepSeek R1, Tencent Hunyuan T1, Kimi K1.5, Spark X1, etc.

[0074] Optionally, the multimodal medical big model refers to a multimodal big model specially trained on medical data, including HuatuoGPT-Vision, LLaVA-Med, Med-Flamingo, etc.

[0075] In one embodiment, the present invention provides a device for generating medical image question-answering thought chain data in multiple stages, comprising:

[0076] Data acquisition module: Obtain the open source medical visual question answering (VQA) dataset to obtain triple data containing medical images, questions, and standard answers.

[0077] Extract image file paths, question texts, and corresponding standard answers from open-source medical visual question answering (VQA) datasets in their original formats (e.g., CSV, Parquet, or HDF5) and convert them into a JSON-formatted triple structure to ensure consistency and scalability of multi-source data in subsequent processing flows.

[0078] Key point information extraction module: uses explicit reasoning language model to extract key point information in medical image question answering.

[0079] The question and corresponding answer in the medical visual question answering task are embedded in the first prompt word template as input text and fed into the explicit reasoning language model. By analyzing the semantic content of the input question and answer pair, the image key points information necessary to answer the question are inferred.

[0080] The first prompt word template needs to contain placeholders for both the question and the answer, and through instructions, the model should focus on extracting the key image elements required to answer the question and answer pair, while specifying a unified output format for subsequent processing.

[0081] The first prompt word template needs to contain placeholders for both the question and the answer, and through instructions, the model should focus on extracting the key image elements required to answer the question and answer pair, while specifying a unified output format for subsequent processing.

[0082] Image description generation module: Based on the extracted key point information, enhanced image description is generated through a multimodal medical large model.

[0083] The obtained image keypoint information is paired with the corresponding medical image data, embedded in a second prompt word template, and fed into a multimodal medical model capable of integrating image and text. Upon receiving this input, the multimodal medical model generates a clearly structured, semantically rigorous, and medically compliant image description—the enhanced image description.

[0084] The second prompt word template needs to contain both image key point information and placeholders for medical image data, and through instructions, the model outputs image description text based on the image key point information.

[0085] The module generates the initial thought chain text and answers: This module embeds the generated image description text, the corresponding question, and the standard answer into a third prompt word template and feeds it into the explicit reasoning language model. Based on the multi-source input, the explicit reasoning language model outputs the initial thought chain text and answers through a language generation mechanism. The thought chain text reflects the logical reasoning path from image representation to medical conclusion.

[0086] The third prompt word template needs to contain the image description text and the placeholders for the corresponding questions and standard answers. The model is instructed to output the initial thought chain text and answers based on the description text, the corresponding questions and the standard answers. The output format in the instruction must be limited to " <think> …< / think> <answer> …< / answer> ”, “ <think> …< / think> "The middle package thinking chain text," <answer> …< / answer> "The middle package standard answer.

[0087] Automated post-processing module: performs keyword detection and tagging, content rewriting and semantic reconstruction, and semantic consistency verification on the generated initial thought chain text and answers to generate the final thought chain text and answers.

[0088] (1) Keyword detection and marking: A set of keywords or phrases with explicit input prompt features is preset to automatically detect the thought chain text based on string matching and regular expressions, identify the sentences containing such content, and locate and mark them as abnormal sentences, providing target areas for subsequent processing.

[0089] (2) Content rewriting and semantic reconstruction: For detected abnormal sentences, predefined rule templates are used to rewrite them. The rule templates replace explicit wording that contains input source hints. The goal of rewriting is to eliminate explicit references to the input source in the text, so that the reasoning process simulates the language expression under multimodal input.

[0090] (3) Semantic consistency verification: After the content is rewritten, to ensure that the text semantics have not shifted, the original text and the processed text are verified for semantic consistency. The semantic similarity calculation method based on the embedding vector is used to ensure that the continuity of the reasoning logic and the accuracy of the conclusion are not affected during the modification process.

[0091] To verify the effectiveness of the proposed method for generating medical image question-answering thought chain data in multiple stages, the LLaVA-v1.5 model was fine-tuned using 1000 data sets generated by this method. Experimental verification was conducted on the PathVQA test set. The experimental results are shown in Table 1.

[0092] surface

[0093]

[0094] As can be seen from the table, the LLaVA-v1.5 model, after fine-tuning the data generated by the method of the present invention, not only outperforms the original model in terms of closure issues and overall performance, but also surpasses the Med-Flamingo large multimodal model with a larger number of parameters and designed specifically for medical multimodal tasks, verifying the effectiveness of the present invention.

[0095] In another embodiment provided in the present application, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the methods described in the present application is implemented.

[0096] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the methods described in the present application.

[0097] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0098] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0099] Each embodiment in this specification is described in a related manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referenced to each other.

[0100] The above description is merely illustrative of certain exemplary embodiments of the present invention. It goes without saying that those skilled in the art will be able to modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and description are illustrative in nature and should not be construed as limiting the scope of protection of the claims.

Claims

1. A multi-stage method for generating medical image question-answering thought chain data, characterized in that: The steps are as follows: Obtain an open-source medical visual question-answering dataset, which contains triples of medical images, questions, and standard answers. Extract key point information in medical image question answering using explicit reasoning language models; Based on the extracted key point information, enhanced image description is generated through a multimodal medical large model; Enhanced image description generation is performed as follows: the obtained image key point information is paired with the corresponding medical image data, and the two are embedded in a second prompt word template. The template is then input into HealthGPT, a multimodal medical model capable of integrating image and text. After receiving the input, HealthGPT generates a clear, semantically rigorous image description text that conforms to medical terminology standards, namely the enhanced image description. Generate initial thought chain text and answers based on enhanced image description; Automatically post-process the generated initial thought chain text and answers to generate the final thought chain text and answers; The final thought chain text and answers, as well as the original medical images and questions, are integrated into the medical image question-answering thought chain data.

2. The method for generating medical image question-answering thought chain data in multiple stages according to claim 1, characterized in that: The key point information extraction is as follows: The questions and corresponding answers in the medical visual question-answering task are embedded in the first prompt word template as input text and input into the explicit reasoning language model; by analyzing the semantic content of the input question-answer pair, the image key point information necessary to answer the question is inferred.

3. The method for generating medical image question-answering thought chain data in multiple stages according to claim 2, characterized in that: The first prompt word template needs to contain placeholders for both the question and the answer, and instruct the model to focus on extracting the key image elements required to answer the question-answer pair, while also specifying a unified output format for subsequent processing; Through the natural language reasoning mechanism, the explicit reasoning language model analyzes the implicit logic between questions and answers and extracts the image observation elements most relevant to the question-answering scenario; The output result is a list of structured key points, namely image key point information, which includes the areas, organ parts, and structural features that should be focused on in the image.

4. The method for generating medical image question-answering thought chain data in multiple stages according to claim 1, characterized in that: The second prompt word template needs to include both image key point information and placeholders for medical image data, and through instructions, the model outputs image description text based on the image key point information.

5. The method for generating medical image question-answering thought chain data in multiple stages according to claim 1, characterized in that: The initial thought chain text and answer generation are specifically as follows: the generated image description text, the corresponding question, and the standard answer are embedded in the third prompt word template and input into the explicit reasoning language model; the explicit reasoning language model outputs the initial thought chain text and answer through a language generation mechanism based on the input multi-source information; the thought chain text reflects the reasoning logic path from image representation to medical conclusion.

6. The method for generating medical image question-answering thought chain data in multiple stages according to claim 5, characterized in that: The third prompt word template needs to contain the image description text and the placeholders for the corresponding questions and standard answers, and through instructions, the model outputs the initial thought chain text and answers based on the description text, the corresponding questions and standard answers. The output format in the instructions must be limited to " <think> …< / think> <answer> …< / answer> ”, " <think> …< / think> "The middle package thought chain text," <answer> …< / answer> "The middle package standard answer.

7. The method for generating medical image question-answering thought chain data in multiple stages according to claim 1, characterized in that: Automated post-processing includes the following processing steps: (1) Keyword detection and marking: A set of keywords or phrases with explicit input prompt features is preset, and the text of thought chains is automatically detected using string matching and regular expressions. Sentences containing such content are identified, and they are located and marked as abnormal sentences, providing target areas for subsequent processing. (2) Content rewriting and semantic reconstruction: For detected abnormal sentences, predefined rule templates are used to rewrite them. The rule templates replace explicit wording that contains input source hints. The goal of rewriting is to eliminate explicit references to the input source in the text, so that the reasoning process simulates the language expression under multimodal input. (3) Semantic consistency verification: After the content is rewritten, in order to ensure that the text semantics have not shifted, the original text and the processed text are semantically verified for consistency; a semantic similarity calculation method based on embedded vectors is used to ensure that the continuity of the reasoning logic and the accuracy of the conclusions are not affected during the modification process.

8. A device for generating medical image question-answering thought chain data in multiple stages, characterized in that: include: Data acquisition module: Obtain open source medical visual question answering datasets to obtain triple data containing medical images, questions, and standard answers; Key point information extraction module: uses explicit reasoning language models to extract key point information in medical image question answering; The question and corresponding answer in the medical visual question answering task are embedded in the first prompt word template as input text and input into the explicit reasoning language model. By analyzing the semantic content of the input question and answer pair, the image key point information necessary to answer the question is inferred. Image description generation module: Based on the extracted key point information, it generates enhanced image descriptions through the multimodal medical large model HealthGPT; The module generates an initial thought chain text and answers: This module embeds the generated image description text, the corresponding question, and the standard answer into a third prompt word template and inputs it into an explicit reasoning language model. The explicit reasoning language model, based on the input multi-source information, outputs the initial thought chain text and answer through a language generation mechanism. The thought chain text reflects the reasoning logic path from image representation to medical conclusion. Automated post-processing module: performs keyword detection and tagging, content rewriting and semantic reconstruction, and semantic consistency verification on the generated initial thought chain text and answers to generate the final thought chain text and answers.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images