Multi-modal medical image report generation method and system based on medical thinking chain

Abstract

The invention relates to a multi-modal medical image report generation method and system based on a medical thinking chain. The method comprises the following steps: acquiring an existing original disordered unstructured medical image report data set; preprocessing the image reports in the data set to obtain a high-quality data set; processing the medical image report in the high-quality data set based on a large language model, and extracting hierarchical medical attribute question and answer pairs; the hierarchical medical mathematical question and answer pairs are connected in series and reconstructed into chain reasoning question and answer pairs; and performing fine adjustment on an existing medical image report generation model by utilizing the chain type reasoning question and answer pair data, and generating a medical image report by utilizing the medical image report generation model after fine adjustment. Compared with the prior art, the method has the advantages that the utilization rate of the multi-modal features and the recognition capability of intention understanding are explicitly enhanced, the generalization, reliability and accuracy of the reasoning capability are improved, and the reliability of the generated medical image report is higher.

Description

Technical Field

[0001] The present invention relates to the field of medical image report generation, and in particular to a multimodal medical image report generation method and system based on a medical thinking chain. Background Art

[0002] The task of generating medical image reports is an important research direction in the field of medical artificial intelligence. It aims to use computer technology to automatically generate text reports corresponding to medical images, thereby reducing the workload of doctors and improving diagnostic efficiency and accuracy. With the development of medical imaging technology, especially the widespread application of imaging equipment such as X-rays, CT scans and MRI, doctors are faced with more and more imaging data that need to be interpreted and recorded. The traditional imaging report writing process usually relies on the professional knowledge and experience of radiologists. Manually generating reports is not only time-consuming, but also prone to human errors. Therefore, the task of generating medical image reports usually involves extracting important visual information from medical images and automatically converting this information into natural language descriptions that meet clinical needs. The core of this task is how to enable computers to accurately understand the content of medical images and express these contents in a way that conforms to the medical context, so as to assist doctors in diagnosis and treatment decisions. Therefore, when completing this task, it is necessary to fully consider the extraction of image features and pathological inference to ensure that the generated report can truly reflect the key medical information in the image.

[0003] In the field of medical image report generation, a large number of studies have attempted to improve the accuracy and quality of report generation through different technical means. Some studies have focused on attaching text labels to medical images. Most of these studies focus on generating fully structured or semi-structured text (such as labels or templates) rather than natural text. Although these studies have made progress in specific areas, the content of the reports they generate is limited by predefined topics and it is difficult to meet the diverse needs of doctors in actual medical scenarios. In addition, another part of the research focuses on the application of image description generation technology, which usually uses an encoder-decoder architecture to extract image features through CNN and then generate text through RNN. Shin et al. (2016) used a CNN-RNN framework to predict chest radiograph labels (such as location, severity). However, this method often fails to meet the needs of generating reports with accurate diagnoses when dealing with complex medical data. In contrast, recent studies have focused more on generating natural text reports. Zhang et al. (2017) proposed a semi-structured pathology report generation method, while this study is the first to propose a model for generating truly natural language reports, which are usually long and cover multiple topics. The innovation of this study lies in the combination of visual features and semantic information. Through a multi-task learning framework, a joint attention mechanism and a hierarchical LSTM model, it can more accurately locate abnormal areas in images and generate corresponding descriptions.

[0004] In recent years, researchers have begun to combine multi-task learning, knowledge graphs, contrastive learning and other technologies to enhance the model's learning and application of clinical features. For example, knowledge graphs are widely used to inject prior knowledge in the medical field into the model to help better identify and describe abnormalities in images. Other studies use multi-task learning combined with disease classification tasks to improve the ability to distinguish features. Although these methods have improved the performance of report generation to a certain extent, due to the lack of pathological inference analysis between local image disease features and overall disease conditions, the model still has deficiencies in diagnostic accuracy and handling of disease imbalances. The generated medical reports are prone to errors in which the overall condition does not match the local medical features.

[0005] With the development of large language models (LLMs), researchers have begun to explore the application of LLMs to medical imaging report generation tasks. Traditional report generation methods usually rely on manually annotated labels or templates. Although they can be automated to a certain extent, they are difficult to cope with the complex information and diverse diagnostic needs in the reports. With the advancement of deep learning and natural language processing technologies, especially the emergence of large pre-trained models such as LLaMA2-7B, these models have demonstrated strong text generation capabilities and can generate high-quality reports with appropriate prompts. However, the task of medical imaging report generation is much more complex than ordinary image description generation tasks. Reports usually need to cover multiple structured medical observations, including detailed descriptions of normal and abnormal conditions. Traditional medical impact report generation methods have difficulty in gradually analyzing and diagnosing the condition based on different levels of medical features in the image, and it is difficult to reason about the condition based on the detailed features in the image.

[0006] In summary, the existing medical report generation methods have the following problems:

[0007] First, a complete diagnostic report usually contains a variety of heterogeneous information (such as impressions, findings, and labels), and it is technically difficult to generate this information under a unified framework. Existing methods can often only generate partial structured content and it is difficult to cover all the details in a complex report.

[0008] Secondly, the performance of existing technologies in locating abnormal areas in images and generating correct descriptions for them is not ideal. Although traditional visual attention mechanisms can locate abnormalities in sub-regions of images, they often cannot accurately identify and describe these abnormalities due to the lack of sufficient semantic information, and cannot accurately match local abnormalities in images with pathological inferences and disease diagnosis. Especially for complex or rare diseases, the generated reports may lack sufficient clinical accuracy, which may lead to misdiagnosis or missed diagnosis.

[0009] In addition, in traditional structured and semi-structured medical report template generation methods, they often rely on predefined templates or high-frequency words, which makes the generated reports lack diversity and naturalness. Although this template-based generation method can improve efficiency to a certain extent, it also limits the personalization and richness of the report. Especially when facing different patients or complex cases, the report may appear too mechanical and cannot fully reflect the doctor's actual diagnostic thinking.

[0010] Finally, in recent years, large language models (LLMs) have been widely used in medical image report generation tasks due to their powerful language capabilities, but they also face various significant challenges, especially the model "hallucination" problem. The hallucination problem refers to the generation of seemingly reasonable but actually wrong content by the model during the generation process. This problem is particularly serious in medical scenarios, because erroneous reports may directly affect clinical diagnosis and treatment decisions, leading to potentially serious consequences, making it difficult for the model to be truly applied in real medical scenarios. Summary of the invention

[0011] The purpose of the present invention is to provide a multimodal medical imaging report generation method and system based on the medical thought chain, a method for constructing fine-grained instruction pairs by simulating the human cognitive process, and applying the concept of "chain of thought" (CoT) from reasoning scenarios to training scenarios, constructing chain training data containing the potential meaning of disease reasoning, and helping the model to better grasp how to analyze and evaluate medical images from details to the whole.

[0012] The purpose of the present invention can be achieved by the following technical solutions:

[0013] A method for generating a multimodal medical imaging report based on a medical thinking chain comprises the following steps:

[0014] Obtain existing raw unordered and unstructured medical imaging report datasets;

[0015] Preprocess the image reports in the dataset to obtain a high-quality dataset;

[0016] Process medical imaging reports in high-quality datasets based on large language models to extract hierarchical medical attribute question-answer pairs;

[0017] Reconstruct the hierarchical medical mathematics question-answer pairs into chained reasoning question-answer pairs;

[0018] The chained reasoning question-answering data is used to fine-tune the existing medical imaging report generation model, and the fine-tuned medical imaging report generation model is used to generate medical imaging reports.

[0019] The pre-processing comprises the following steps:

[0020] Regulations for deleting data in the dataset where the resolution of medical images is lower than a preset resolution threshold;

[0021] Deleting data provisions that lack a preset portion of text content or the length of the portion of text content is less than a preset length;

[0022] Medical imaging reports are protected from privacy and redundancy, and content related to doctor-patient information, duplicate medical records, and content related to the patient's historical medical conditions are deleted.

[0023] The hierarchical medical attribute question-answer pairs are divided into six dimensions, which are modality, organ, size, abnormal location, symptom, and overall health status from low to high levels.

[0024] The method for extracting hierarchical medical attribute question-answer pairs is specifically as follows: based on specific prompt words, a large language model is used to perform end-to-end semantic segmentation on the original disorganized medical image report, and questions corresponding to six different dimensions, namely, modality, organ, size, abnormal location, symptom, and overall health status, are extracted. According to each question, the information of each dimension is independently queried and retrieved in the original image report as the answer to the question, thereby hierarchizing the original disordered and unstructured medical image report text into question-answer pairs of six dimensions.

[0025] The method of serially reconstructing hierarchical medical mathematics question-answer pairs into chained reasoning question-answer pairs is specifically as follows:

[0026] For each original question except the lowest level in the extracted question-answer pair, the answers to all lower-level questions with lower levels than the original question are taken as prefixes and concatenated in series with the original question to reconstruct the question in a chain manner and combine with the answer corresponding to the original question to obtain a chained reasoning question-answer pair.

[0027] A multimodal medical imaging report generation system based on medical thinking chain, comprising:

[0028] Raw data acquisition module: used to obtain the existing raw disordered and unstructured medical imaging report data set;

[0029] Preprocessing module: used to preprocess the image reports in the data set to obtain a high-quality data set;

[0030] Hierarchical question-answer pair extraction module: used to process medical imaging reports in high-quality data sets based on a large language model and extract hierarchical medical attribute question-answer pairs;

[0031] Chained reasoning question-answer pair reconstruction module: used to reconstruct hierarchical medical mathematics question-answer pairs into chained reasoning question-answer pairs;

[0032] Model fine-tuning and report generation module: used to fine-tune the existing medical imaging report generation model using chained reasoning question-answering data, and generate medical imaging reports using the fine-tuned medical imaging report generation model.

[0033] The preprocessing module performs the following steps:

[0034] Regulations for deleting data in the dataset where the resolution of medical images is lower than a preset resolution threshold;

[0035] Deleting data provisions that lack a preset portion of text content or the length of the portion of text content is less than a preset length;

[0036] Medical imaging reports are protected from privacy and redundancy, and content related to doctor-patient information, duplicate medical records, and content related to the patient's historical medical conditions are deleted.

[0037] The hierarchical medical attribute question-answer pairs are divided into six dimensions, which are modality, organ, size, abnormal location, symptom, and overall health status from low to high levels.

[0038] The hierarchical question-answer pair extraction module performs the following steps: based on specific prompt words, using a large language model to perform end-to-end semantic segmentation on the original disorganized medical image report, extracting questions corresponding to six different dimensions of modality, organ, size, abnormal location, symptom, and overall health status, and independently querying and retrieving information of each dimension in the original image report as the answer to the question according to each question, thereby hierarchizing the original disordered and unstructured medical image report text into question-answer pairs of six dimensions.

[0039] The chained reasoning question-answering reconstruction module performs the following steps:

[0040] For each original question except the lowest level in the extracted question-answer pair, the answers to all lower-level questions with lower levels than the original question are taken as prefixes and concatenated in series with the original question to reconstruct the question in a chain manner and combine with the answer corresponding to the original question to obtain a chained reasoning question-answer pair.

[0041] The system tailors a method for simulating the cognitive process of human doctors to construct fine-grained instruction pairs for the field of medical imaging, and applies the concept of thought chain from reasoning scenarios to training scenarios, constructs a chain data set with implicit pathological inference and disease analysis thinking processes, helps the model learn the thinking mode of human doctors during the training process, further enhances the reasoning ability of the model, and significantly reduces the occurrence of hallucinations. This method constructs more refined and structured training data by simulating the cognitive process of human doctors when analyzing medical images. The present invention converts complex and unstructured medical reports into hierarchical question-answer pairs through hierarchical semantic segmentation. These question-answer pairs represent different levels of medical information, such as the modality of the image, the organs involved, the location of the abnormality, symptoms, and overall health status. By integrating these hierarchical question-answer pairs into the training process of the model, the method proposed by the present invention can gradually guide the model from superficial image information analysis to deep pathological conditions, thereby improving the accuracy and reliability of the model-generated reports.

[0042] Through the multimodal medical imaging report generation system based on the medical thinking chain proposed in the present invention, a medical imaging diagnosis report containing a variety of heterogeneous information (such as impressions, findings and labels) can be generated under a unified framework, focusing on local medical features or abnormal features in medical images, analyzing and inferring potential causes and pathological conditions, and accurately matching local image abnormalities with pathological inferences and disease diagnosis, thereby making a diagnostic analysis of the patient's overall condition and generating a comprehensive medical imaging diagnosis report.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] (1) Different from the traditional medical image report generation task which requires heavy image labeling work, the present invention does not require a large amount of human resources and feature engineering overhead, and avoids the lack of data diversity caused by constructing image reports based on image label templates. The present invention simulates the shallow to deep processing paradigm of different neurons and synapses in the human cerebral cortex for image information, combines the thinking steps and reasoning process of human diagnosis and analysis in real medical scenarios, and directly uses a large language model to hierarchically extract and reconstruct the original unstructured and disordered medical report text content into chain-structured, hierarchical feature attribute disease analysis and reasoning chain data. Specifically, the system designs six different dimensions in medical images from shallow to deep: modality, organ, size, abnormal location, symptoms, and overall health status. And use powerful large language models such as ChatGPT and Gemini to perform end-to-end semantic segmentation on the original disorganized medical image reports, and extract six different dimensions of question and answer pairs.

[0045] (2) Different from the traditional CNN-RNN framework and the existing method for completing the task of generating medical image reports based on a large language model, the present invention not only overcomes the fatal defect that the traditional CNN-RNN framework is difficult to achieve complex and diverse free text generation, but also significantly alleviates the serious model hallucination problem that is easily generated by the existing LLMs-based method. The existing methods are prone to ignoring the local disease attribute characteristics in medical images, ignoring the causal relationship and correspondence between local characteristics and overall disease assessment, or misdetecting local disease attribute characteristics that do not exist. These serious model hallucination problems will lead to missed diagnosis, misdiagnosis, and wrong diagnosis, making it difficult for existing LLMs to be actually applied to real medical scenarios, and it is difficult to accurately help doctors perform overall medical image assessment and auxiliary diagnosis and treatment. The present invention connects the local shallow medical attribute characteristics of medical images with the overall deep disease assessment summary in series to construct a disease reasoning thinking chain. On the one hand, it can significantly alleviate the model hallucination problem and avoid misdiagnosis and wrong diagnosis caused by the model's fabrication. The overall medical report on the disease obtained based on the reasoning thinking chain is well-founded and reliable. On the other hand, it can help the model focus on each local detail feature in the image, and analyze each level of feature attributes one by one in a structured manner, so as to avoid missing any abnormal part in the image and avoid missed diagnosis. The hierarchical chain data structure included in the system effectively captures the correlation between medical feature attributes across levels, promotes feature adaptation and expression capabilities between heterogeneous attributes, and provides a new learning paradigm for generating a more robust chain disease reasoning diagnosis model.

[0046] (3) Different from the application method of the reasoning stage of the traditional thinking chain, the present invention proposes a method to embed the concept of the reasoning thinking chain into the training data. This method simulates the human cognitive process and the information fusion strategy of the neural synapses in the cerebral cortex, realizes the dynamic convergence and integration of feature attributes from different levels, explicitly enhances the utilization rate of multimodal features and the recognition ability of intention understanding, and improves the generalization, reliability and accuracy of the system's reasoning ability.

[0047] (4) The present invention overcomes the defect of the lack of high-quality data sets for the previous medical image report generation task, and novelly designs and constructs a large-scale high-quality medical image report generation task data set. This data set is based on the existing public medical image report generation data set, and according to the image size resolution, text content length, text sensitive information, etc., it selects high-quality images and comprehensive report text content, and removes redundancy and privacy of medical and patient contents to construct a high-quality image report data set. On the basis of this data set, the key technology proposed by the present invention is applied to extract hierarchical medical attribute question-answer pairs, and chain thinking question-answer pair data that implies the process of disease reasoning and diagnosis is constructed in series, thereby overcoming the lack of diagnostic reasoning ability of the model in the previous medical image report generation task from point to surface, from shallow to deep, and from local detail features to infer the overall disease condition, so that the model can simulate the thinking and cognitive process of human doctors when analyzing medical images in real medical scenarios, gradually observe and analyze from the surface features of the image, and finally make an accurate and comprehensive evaluation and summary of the patient's condition in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of the method of the present invention;

[0049] Figure 2 is a flow chart of the pre-processing steps of the present invention;

[0050] Figure 3 It is a schematic diagram of the hierarchical medical attribute question-answer pair extraction process of the present invention;

[0051] Figure 4 It is a schematic diagram of the reconstruction process of the chained reasoning question and answer pair of the present invention. DETAILED DESCRIPTION

[0052] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0053] The limited availability of training data and privacy issues in the medical field greatly hinder the accuracy and generalization of models. In the task of medical imaging report generation, one way to improve the accuracy of generated content and mitigate hallucinations is to increase the robustness and generalization of the model by improving the quality and diversity of training data. However, traditional data augmentation methods, such as image flipping, rotation, scaling, and introducing synthetic noise, are not suitable for the multimodal medical field due to the high fidelity required for medical image interpretation. Any changes that distort the medical reality of the image or destroy the feature alignment between the image and text may lead to misdiagnosis or overlook critical patient-specific details. In recent years, many studies have expanded the data by directly rewriting the text using large language models, but the data diversity increased by simple rewriting is very limited.

[0054] Research in cognitive neuroscience shows that humans typically process graphic information from the surface to the deep layers, seamlessly understanding hierarchical information. However, artificial intelligence models lack this ability and must gradually analyze each small area in the image by reducing the receptive field. In addition, medical texts written by humans are usually unorganized, which complicates the process of associating visual features with unstructured free text. Therefore, the present invention applies the cognitive process of human image recognition to data organization in a multimodal model. In a broad sense, the integration of text information can help AI models simulate human cognitive processes. According to this integration process, the present invention proposes a hierarchical text enhancement method based on a chain hierarchy of medical attribute importance. This hierarchical semantic segmentation of medical text is inspired by the cognitive steps taken by doctors when evaluating medical images and diagnosing, involving sequential recognition and integration of information across various levels.

[0055] The cognitive process of doctors evaluating medical images usually starts with global information about the image, such as the modality and the human system depicted, and then gradually focuses on specific organs, followed by fine-grained attributes, such as the shape or location of organs or abnormalities. After thoroughly understanding this information, the human brain integrates the symptoms presented in the image and finally generates an unstructured image report. However, from the perspective of a large model, medical reports are usually highly disordered. Even for humans, it is a challenge to quickly associate a lengthy medical report with every fine-grained semantic detail in the image. Therefore, based on this cognitive process, the present invention specifically designs specific prompts for the task of generating medical image reports, so that powerful large language models such as ChatGPT and Gemini can generate structured question-answer pairs from the unstructured and chaotic medical report text written by original humans end-to-end, focusing on fine-grained medical image problems at different dimensions and levels. Subsequently, the system simulates the thinking chain paradigm and transfers the paradigm from the reasoning process to the fine-tuning process of the model. This involves integrating low-level information clues into high-level questions to generate new question-answer pairs and performing supervised fine-tuning (SFT) of the model. These question-answer pairs provide a more fine-grained description of the dimensions and are often used as high-quality training data to help models understand images, thereby alleviating the model hallucination problem.

[0056] Specifically, this embodiment provides a multimodal medical imaging report generation method based on the medical thinking chain, such as Figure 1 As shown, the following steps are included:

[0057] S1, obtain the existing raw unordered and unstructured medical imaging report dataset.

[0058] S2, preprocess the image reports in the dataset to obtain a high-quality dataset.

[0059] like Figure 2 As shown, the preprocessing includes the following steps:

[0060] S21, deleting data rules in which the resolution of medical images in the data set is lower than a preset resolution threshold;

[0061] S22: To ensure the completeness of the report content, delete the data items that lack the Finding and Impression sections or have less than 10 words in length in all medical inclusions;

[0062] S23, de-privacy and redundancy of medical imaging reports, deletes sentences related to doctor-patient information and duplicate medical records, and uses the powerful language capabilities of the large language model to filter and delete sentences related to the patient's historical medical condition.

[0063] S3, processes medical imaging reports in high-quality datasets based on a large language model and extracts hierarchical medical attribute question-answer pairs.

[0064] This embodiment divides the hierarchical medical attribute question-answer pairs into six dimensions, which are modality, organ, size, abnormal location, symptom, and overall health status from low to high levels.

[0065] In order to transform the disorganized image reports into high-quality question-answer pairs that simulate human cognition, a powerful large language model is used to perform semantic segmentation on the image reports. Figure 3 As shown in the figure, with the help of the specific prompt words (prompt) designed by us, we use powerful large language models such as ChatGPT and Gemini to perform end-to-end semantic segmentation on the original messy medical image report, and use large language models to perform end-to-end semantic segmentation on the original messy medical image report, extracting questions corresponding to six different dimensions of modality, organ, size, abnormal location, symptoms, and overall health status, and independently query and retrieve information of each dimension in the original image report as the answer to the question according to each question, thereby hierarchizing the original disordered and unstructured medical image report text into six-dimensional question-answer pairs. The definition of the above six dimensions is established based on the common focus points in medical image reports, and is also established to simulate the human doctor's analysis process of medical image reports from shallow to deep observation and analysis angles.

[0066] S4, reconstructs the hierarchical medical mathematics question-answer pairs into chained reasoning question-answer pairs.

[0067] In order to further simulate the human cognitive process and the diagnosis and treatment thinking steps of human doctors, this paper proposes a question-answer pair reconstruction method based on chain thinking. Figure 4 As shown in Figure 1, the method of reconstructing hierarchical medical mathematics question-answer pairs into chained reasoning question-answer pairs is as follows:

[0068] For each original question except the lowest level in the extracted question-answer pair, the answers to all lower-level questions with lower levels than the original question are taken as prefixes and concatenated in series with the original question to reconstruct the question in a chain manner and combine with the answer corresponding to the original question to obtain a chained reasoning question-answer pair.

[0069] According to the above construction process, this embodiment collected 10,000 images with medical reports from the public medical imaging report datasets MIMIC dataset and OpenI dataset, and constructed 60,000 hierarchical question-answer pairs and 60,000 chained question-answer pairs. The newly constructed chained question-answer pair data, combined with the original medical images and reports, is used for supervised fine-tuning of the model.

[0070] S5, fine-tuning the existing medical imaging report generation model using chained reasoning question-answering data, and generating a medical imaging report using the fine-tuned medical imaging report generation model.

[0071] This embodiment also provides a multimodal medical imaging report generation system based on a medical thinking chain, including:

[0072] Raw data acquisition module: used to obtain the existing raw disordered and unstructured medical imaging report data set;

[0073] Preprocessing module: used to preprocess the image reports in the data set to obtain a high-quality data set;

[0074] Hierarchical question-answer pair extraction module: used to process medical imaging reports in high-quality data sets based on a large language model and extract hierarchical medical attribute question-answer pairs;

[0075] Chained reasoning question-answer pair reconstruction module: used to reconstruct hierarchical medical mathematics question-answer pairs into chained reasoning question-answer pairs;

[0076] Model fine-tuning and report generation module: used to fine-tune the existing medical imaging report generation model using chained reasoning question-answering data, and generate medical imaging reports using the fine-tuned medical imaging report generation model.

[0077] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0078] The multimodal medical image report generation system based on the medical thinking chain demonstrated in this embodiment introduces a fine-grained medical image attribute thinking chain in the medical image report generation task for the first time, effectively promoting the development of medical image report generation tasks in the real world, significantly improving the disease reasoning and diagnosis ability of LLMs in this task, and greatly alleviating the problem of medical model hallucination. At the same time, by utilizing the powerful language ability of LLMs, hierarchical feature attribute question and answer pairs are extracted from the original medical image report, and the cause and condition reasoning and diagnosis chain thinking data is constructed. The heterogeneous modal information is preprocessed and feature extracted by combining different units, realizing the organic chain fusion of multimodal features. Finally, based on the medical thinking chain, the accuracy of multimodal medical image report generation is greatly enhanced, the generalization performance and accuracy of the model are improved, and the model hallucination problem is alleviated. The method proposed by the present invention can provide a complete and effective unified framework for medical image report generation training and reasoning, and provide reliable guarantee for subsequent application in human real scenes for auxiliary diagnosis and treatment.

[0079] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning, or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A multimodal medical imaging report generation method based on medical thinking chain, characterized in that: The following steps are involved: Obtain existing raw unordered and unstructured medical imaging report datasets; Preprocess the image reports in the dataset to obtain a high-quality dataset; Process medical imaging reports in high-quality datasets based on large language models to extract hierarchical medical attribute question-answer pairs; Reconstruct the hierarchical medical mathematics question-answer pairs into chained reasoning question-answer pairs; The chained reasoning question-answering data is used to fine-tune the existing medical imaging report generation model, and the fine-tuned medical imaging report generation model is used to generate medical imaging reports.

2. According to claim 1, a multimodal medical imaging report generation method based on medical thinking chain is characterized in that: The pre-processing comprises the following steps: Regulations for deleting data in the dataset where the resolution of medical images is lower than a preset resolution threshold; Deleting data provisions that lack a preset portion of text content or the length of the portion of text content is less than a preset length; Medical imaging reports are protected from privacy and redundancy, and content related to doctor-patient information, duplicate medical records, and content related to the patient's historical medical conditions are deleted.

3. According to claim 1, a multimodal medical imaging report generation method based on medical thinking chain is characterized in that: The hierarchical medical attribute question-answer pairs are divided into six dimensions, which are modality, organ, size, abnormal location, symptom, and overall health status from low to high levels.

4. According to claim 3, a multimodal medical imaging report generation method based on medical thinking chain is characterized in that: The method for extracting hierarchical medical attribute question-answer pairs is specifically as follows: based on specific prompt words, a large language model is used to perform end-to-end semantic segmentation on the original disorganized medical image report, and questions corresponding to six different dimensions, namely, modality, organ, size, abnormal location, symptom, and overall health status, are extracted. According to each question, the information of each dimension is independently queried and retrieved in the original image report as the answer to the question, thereby hierarchizing the original disordered and unstructured medical image report text into question-answer pairs of six dimensions.

5. According to claim 3, a multimodal medical imaging report generation method based on medical thinking chain is characterized in that: The method of serially reconstructing hierarchical medical mathematics question-answer pairs into chained reasoning question-answer pairs is specifically as follows: For each original question except the lowest level in the extracted question-answer pair, the answers to all lower-level questions with lower levels than the original question are taken as prefixes and concatenated in series with the original question to reconstruct the question in a chain manner and combine with the answer corresponding to the original question to obtain a chained reasoning question-answer pair.

6. A multimodal medical imaging report generation system based on medical thinking chain, characterized in that: include: Raw data acquisition module: used to obtain the existing raw disordered and unstructured medical imaging report data set; Preprocessing module: used to preprocess the image reports in the data set to obtain a high-quality data set; Hierarchical question-answer pair extraction module: used to process medical imaging reports in high-quality data sets based on a large language model and extract hierarchical medical attribute question-answer pairs; Chained reasoning question-answer pair reconstruction module: used to reconstruct hierarchical medical mathematics question-answer pairs into chained reasoning question-answer pairs; Model fine-tuning and report generation module: used to fine-tune the existing medical imaging report generation model using chained reasoning question-answering data, and generate medical imaging reports using the fine-tuned medical imaging report generation model.

7. A multimodal medical imaging report generation system based on medical thinking chain according to claim 6, characterized in that: The preprocessing module performs the following steps: Regulations for deleting data in the dataset where the resolution of medical images is lower than a preset resolution threshold; Deleting data provisions that lack a preset portion of text content or the length of the portion of text content is less than a preset length; Medical imaging reports are protected from privacy and redundancy, and content related to doctor-patient information, duplicate medical records, and content related to the patient's historical medical conditions are deleted.

8. The multimodal medical imaging report generation system based on medical thinking chain according to claim 6 is characterized in that: The hierarchical medical attribute question-answer pairs are divided into six dimensions, which are modality, organ, size, abnormal location, symptom, and overall health status from low to high levels.

9. A multimodal medical imaging report generation system based on medical thinking chain according to claim 8, characterized in that: The hierarchical question-answer pair extraction module performs the following steps: based on specific prompt words, using a large language model to perform end-to-end semantic segmentation on the original disorganized medical image report, extracting questions corresponding to six different dimensions of modality, organ, size, abnormal location, symptom, and overall health status, and independently querying and retrieving information of each dimension in the original image report as the answer to the question according to each question, thereby hierarchizing the original disordered and unstructured medical image report text into question-answer pairs of six dimensions.

10. A multimodal medical imaging report generation system based on medical thinking chain according to claim 8, characterized in that: The chained reasoning question-answering reconstruction module performs the following steps: For each original question except the lowest level in the extracted question-answer pair, the answers to all lower-level questions with lower levels than the original question are taken as prefixes and concatenated in series with the original question to reconstruct the question in a chain manner and combine with the answer corresponding to the original question to obtain a chained reasoning question-answer pair.

Images