Medical image report automatic generation method and device based on artificial intelligence

Abstract

The invention discloses a medical image report automatic generation method and device based on artificial intelligence, relates to the technical field of artificial intelligence and medical image crossing, and aims to shorten the report generation period and improve the report generation efficiency while improving the accuracy and consistency of report diagnosis. The method comprises the following steps: carrying out cross-modal image space alignment processing on a PET image and a CT image; a pre-trained PET focus segmentation model is adopted, a high-metabolism focus part is identified and segmented in the processed PET image, and focus iconography parameters are acquired; segmenting an organ image in the processed CT image by adopting a pre-trained CT image organ segmentation model, and determining focus position information; sorting the lesion iconography parameters and the lesion position information to generate an initial report, and processing the initial report by adopting the fine-tuned large language model to obtain a medical image report and outputting the medical image report.

Description

Technical Field

[0001] This application relates to the field of interdisciplinary technology of artificial intelligence and medical imaging, and in particular to a method and apparatus for automatically generating medical image reports based on artificial intelligence. Background Technology

[0002] A PET / CT (Positron Emission Tomography / Computed Tomography) report is a crucial document in tumor diagnosis. It integrates information from both PET and CT imaging techniques to present a comprehensive medical image report. PET imaging uses radiolabeled tracers, taking advantage of the high metabolic activity and tracer uptake of tumor cells to reflect the metabolic function of human tissues. CT imaging, on the other hand, uses X-ray beams to perform tomographic scanning, clearly revealing the anatomical structures of internal organs. A tumor PET / CT report combines the advantages of both imaging methods, not only demonstrating the metabolic activity of the tumor but also clarifying its anatomical location, size, shape, and relationship with surrounding tissues and organs. This provides comprehensive and critical information for tumor diagnosis, staging, and treatment evaluation.

[0003] In related technologies, the generation of traditional tumor PET / CT reports mainly relies on experts manually integrating metabolic and anatomical information from multiple lesions and organs. After obtaining the PET / CT image data, experts need to carefully analyze the metabolic manifestations of each lesion in the images, such as the level of metabolic activity and the range of metabolic distribution. At the same time, they need to combine the anatomical structures presented by the CT images to determine whether the tumor has invaded surrounding tissues, whether there is lymph node metastasis, and whether there is distant metastasis. In some specific scenarios, experts can also use neural network models to analyze the image data. Then, based on their own professional knowledge and experience, or with the help of NLP (Natural Language Processing) tools or large language models, experts transform this complex image information into textual descriptions and write a report.

[0004] However, the applicant recognizes that NLP tools or large language models sometimes fail to accurately express image information and struggle to precisely correlate image features with structured reports, leading to potentially inaccurate or incomplete report information. This necessitates expert intervention, but different experts have varying levels of experience, knowledge, and judgment criteria, potentially leading to different conclusions when observing the same image data. Significant inter-observer variability can easily affect the accuracy and consistency of the report's diagnosis, resulting in poor quality medical image reports. Furthermore, with the increasing number of cancer patients, the amount of image data that experts need to process is constantly rising, making the manual integration of information extremely labor-intensive, thus extending the report generation cycle and reducing report generation efficiency. Summary of the Invention

[0005] In view of this, this application provides an automated method and apparatus for generating medical image reports based on artificial intelligence. The main purpose is to solve the problem that current NLP tools or large language models sometimes fail to accurately express image information and are difficult to accurately associate image features with structured reports, resulting in potentially inaccurate or incomplete report information. This requires manual intervention from experts, but different experts have different experience, knowledge levels, and judgment standards. When observing the same image data, they may draw different conclusions. The large differences between observers can easily affect the accuracy and consistency of the report diagnosis, resulting in poor quality of the generated medical image reports. In addition, with the increase in the number of cancer patients, the amount of image data that experts need to process is constantly increasing. The workload of manually integrating information is huge, which leads to a longer report generation cycle and low report generation efficiency.

[0006] According to a first aspect of this application, an artificial intelligence-based method for automatically generating medical image reports is provided, the method comprising:

[0007] Acquire PET and CT images to be analyzed, and perform cross-modal image space alignment processing on the PET and CT images;

[0008] A pre-trained PET lesion segmentation model is used to identify and segment high-metabolic lesion portions in the processed PET images, and to obtain the lesion imaging parameters output by the PET lesion segmentation model for the high-metabolic lesion portions.

[0009] A pre-trained CT image organ segmentation model is used to segment organ images in the processed CT images and map the hypermetabolic lesion portion to the organ images to determine the lesion location information.

[0010] The imaging parameters and location information of the lesions are organized to generate an initial report. The initial report is then processed using a fine-tuned large language model to obtain a medical imaging report, which is then output.

[0011] According to a second aspect of this application, an artificial intelligence-based automated medical image report generation device is provided, the device comprising:

[0012] The alignment module is used to acquire PET and CT images to be analyzed and perform cross-modal image space alignment processing on the PET and CT images.

[0013] The analysis module is used to identify and segment high-metabolic lesion portions in the processed PET images using a pre-trained PET lesion segmentation model, and to obtain the lesion imaging parameters output by the PET lesion segmentation model for the high-metabolic lesion portions.

[0014] The analysis module is also used to segment organ images in the processed CT images using a pre-trained CT image organ segmentation model, and to map the hypermetabolic lesion portion to the organ image to determine the lesion location information.

[0015] The generation module is used to organize the imaging parameters and location information of the lesion, generate an initial report, and process the initial report using a fine-tuned large language model to obtain and output a medical imaging report.

[0016] According to a third aspect of this application, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the first aspects above.

[0017] According to a fourth aspect of this application, a readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0018] Using the above technical solution, this application provides an automated method and apparatus for generating medical image reports based on artificial intelligence. This application acquires PET and CT images to be analyzed, performs cross-modal image space alignment processing on the PET and CT images, uses a pre-trained PET lesion segmentation model to identify and segment high-metabolic lesions in the processed PET images, and obtains the lesion imaging parameters output by the PET lesion segmentation model for the high-metabolic lesion portions. It then uses a pre-trained CT image organ segmentation model to segment organ images in the processed CT images and maps the high-metabolic lesions to the organ images to determine lesion location information. The lesion imaging parameters and lesion location information are then processed to generate an initial report. Finally, a fine-tuned large language model is used to process the initial report to obtain and output a medical image report. By intelligently segmenting PET and CT images and extracting biomarkers, the accuracy of image information extraction is improved. Furthermore, the fine-tuned large language model can automatically generate and output realistic medical image reports, reducing manual intervention. This improves the accuracy and consistency of report diagnosis while shortening the report generation cycle and increasing report generation efficiency.

[0019] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0020] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0021] Figure 1 This illustration shows a flowchart of an automated medical image report generation method based on artificial intelligence, according to an embodiment of this application.

[0022] Figure 2 This illustration shows a flowchart of another AI-based automated medical image report generation method provided in an embodiment of this application.

[0023] Figure 3 This illustration shows a schematic diagram of an artificial intelligence-based automated medical image report generation device provided in an embodiment of this application;

[0024] Figure 4 A schematic diagram of the device structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation

[0025] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0026] This application provides an artificial intelligence-based method for automatically generating medical image reports, such as... Figure 1 As shown, the method includes:

[0027] 101. Obtain the PET and CT images to be analyzed, and perform cross-modal image space alignment processing on the PET and CT images.

[0028] The method provided in this application can be applied to a report generation system, offering a convenient upload interface for doctors and other staff. It supports receiving PET and CT images of different formats and sources, and through image analysis of these images, determines the imaging parameters and location information of lesions. Then, following the standardized format of medical imaging reports, it initially generates an initial report containing key information. The report generation system incorporates a finely tuned large language model, trained on a large amount of medical imaging report text data. This model possesses deep medical expertise and language comprehension capabilities, deeply processing the initial report, refining the language, and supplementing necessary medical terminology and diagnostic suggestions to make the report richer, more accurate, and more professional. Ultimately, it generates a medical imaging report that is authentic in style and conforms to medical standards. Furthermore, the report generation system also features report output and interactive functions. The generated medical imaging report can be output in various ways, such as being directly displayed on the system's user interface for doctors to view in real time; or exported to common document formats such as PDF (Portable Document Format) and Word, facilitating printing, archiving, or sharing by doctors. Meanwhile, the report generation system allows doctors to review and modify the generated reports. Doctors can adjust and improve the content of the reports based on their clinical experience and professional knowledge to ensure the accuracy and reliability of the reports. This not only improves the flexibility and practicality of the report generation system, but also enhances doctors' trust in and willingness to use the system.

[0029] In this embodiment, to generate a medical imaging report, it is necessary to first acquire the PET and CT images to be analyzed. PET images are positron emission tomography (PET) scans, which detect the distribution of radioactive tracers in the body to reflect the metabolic activity of tissues and are commonly used for the early diagnosis and treatment evaluation of diseases such as tumors. CT images are computed tomography (CT) scans, which use X-rays to perform tomographic scanning of the human body to obtain detailed anatomical information of the internal structures. The PET and CT images to be analyzed can be manually uploaded by doctors or other staff, or they can be obtained through the connected hospital's PACS system. This application does not limit the specific method of acquiring the PET and CT images to be analyzed.

[0030] Subsequently, the report generation system performs cross-modal image spatial alignment processing on the PET and CT images. This process matches the spatial locations of images from different imaging devices (PET and CT) so that they can be analyzed and processed within the same coordinate system. For example, suppose a patient suspected of having lung cancer undergoes both PET and CT scans. The PET image shows abnormally active metabolism in a certain area of ​​the lung, but the location of this area in the CT image deviates from that in the PET image. After cross-modal image spatial alignment processing, the metabolically abnormal area in the PET image accurately corresponds to the corresponding lung anatomical structure in the CT image. This clarifies the relationship between the metabolically abnormal area and surrounding tissues, improving the accuracy of image information extraction and avoiding analytical errors caused by mismatched image spatial locations. This provides more reliable information for generating subsequent medical imaging reports.

[0031] 102. Using a pre-trained PET lesion segmentation model, the high-metabolic lesion portion is identified and segmented in the processed PET images, and the lesion imaging parameters output by the PET lesion segmentation model for the high-metabolic lesion portion are obtained.

[0032] In this embodiment, the PET lesion segmentation model is a deep learning model trained on a large number of PET images. It can automatically identify different tissue structures and lesion areas in PET images. Therefore, after initial processing of the PET images, the report generation system inputs the processed PET images into the pre-trained PET lesion segmentation model. Using this model, high-metabolic lesion regions are identified and segmented in the processed PET images. High-metabolic lesions refer to areas in PET images where metabolic activity is significantly higher than surrounding normal tissue, typically associated with diseases such as tumors. After segmenting the high-metabolic lesions, the PET lesion segmentation model calculates and outputs lesion imaging parameters based on pixel values, volume, and other information of the lesion region, enabling the report generation system to obtain these indicators. The imaging parameters of the lesion are quantitative indicators calculated by the PET lesion segmentation model based on the characteristics of high metabolic lesions, such as SUVmean (mean Standardized Uptake Value), SUVmax (Maximum Standardized Uptake Value), TLG (Total Lesion Glycolysis), and MTV (Metabolic Tumor Volume), to assess the metabolic degree and malignancy of the lesion.

[0033] In this way, the pre-trained PET image recognition model can quickly and accurately identify and segment high-metabolic lesions and output lesion imaging parameters, greatly improving the efficiency and accuracy of image information extraction, reducing the subjectivity and error of manual identification and segmentation, and helping to improve the accuracy and consistency of the generated reports.

[0034] 103. Using a pre-trained CT image organ segmentation model, organ images are segmented in the processed CT images, and the high-metabolic lesion portion is mapped to the organ image to determine the lesion location information.

[0035] In this embodiment of the application, the CT image organ segmentation model is a deep learning model trained on a large number of sample CT images. It can automatically identify different organs and tissue structures in CT images. Therefore, after the initial processing of CT images, the report generation system will input the processed CT images into the pre-trained CT image organ segmentation model. The pre-trained CT image organ segmentation model will segment organ images in the processed CT images. The organ images refer to the images of various organs presented in the CT images, such as lungs, liver, kidneys, etc.

[0036] Then, the CT image organ segmentation model maps the high-metabolic lesion to the organ image. By calculating the spatial relationship between the lesion and the organ, the location information of the lesion is determined, such as which lobe or segment of the lung the lesion is located in, thereby improving the efficiency and accuracy of information extraction and reducing the tediousness and error of manual operation.

[0037] 104. Organize the imaging parameters and location information of the lesions, generate an initial report, and process the initial report using a fine-tuned large language model to obtain and output a medical imaging report.

[0038] In this embodiment, the report generation system organizes the obtained lesion imaging parameters and lesion location information to form an initial report. To ensure the generated report conforms to medical professional style and standardized text content, this embodiment employs a fine-tuned large language model. This fine-tuned large language model is trained using a large amount of text data related to medical imaging reports, building upon a general large language model. The report generation system processes the initial report using this fine-tuned large language model. By learning from a large amount of medical imaging report text, the large language model can understand the information in the initial report and, based on medical professional knowledge and language standards, refine, supplement, and improve the initial report, generating a medical imaging report with a realistic style and accurate content. This generated medical imaging report is then output for doctors' reference and use.

[0039] In this way, by processing the information and fine-tuning the large language model, it is possible to automatically generate and output medical imaging reports with realistic styles, which greatly reduces the workload and time of manual report writing and improves the efficiency of report generation. At the same time, the fine-tuned large language model can ensure the standardization and accuracy of the report's language, improve the accuracy and consistency of the report's diagnosis, and help improve the quality and efficiency of medical services.

[0040] The method provided in this application acquires PET and CT images to be analyzed, performs cross-modal image space alignment processing on the PET and CT images, uses a pre-trained PET lesion segmentation model to identify and segment high-metabolic lesions in the processed PET images, and obtains the lesion imaging parameters output by the PET lesion segmentation model for the high-metabolic lesion portions. A pre-trained CT image organ segmentation model is used to segment organ images in the processed CT images, and the high-metabolic lesion portions are mapped to organ images to determine lesion location information. The lesion imaging parameters and lesion location information are processed to generate an initial report, and a fine-tuned large language model is used to process the initial report to obtain and output a medical image report. By intelligently segmenting and extracting biomarkers from PET and CT images, the accuracy of image information extraction is improved. Furthermore, the fine-tuned large language model can automatically generate and output realistic medical image reports, reducing manual intervention. This improves the accuracy and consistency of report diagnosis while shortening the report generation cycle and increasing report generation efficiency.

[0041] Furthermore, as a refinement and extension of the specific implementation methods of the above embodiments, and in order to fully illustrate the specific implementation process of this embodiment, this application provides another method for automatically generating medical image reports based on artificial intelligence, such as... Figure 2 As shown, the method includes:

[0042] 201. Obtain the PET and CT images to be analyzed, and perform cross-modal image space alignment processing on the PET and CT images.

[0043] The technical solution of this application embodiment can be applied to a report generation system. For details of the report generation system, please refer to the description in step 101 of the above embodiment, which will not be repeated here.

[0044] In medical imaging diagnosis, PET and CT images each play unique roles. PET images detect the distribution of radioactive tracers in the human body, reflecting the metabolic activity of tissues, and are often used for the early diagnosis, treatment evaluation, and disease monitoring of diseases such as tumors. CT images, on the other hand, use X-rays to perform tomographic scans of the human body, clearly presenting anatomical information of internal structures, such as organ morphology and location. However, due to the differences in PET and CT scanning equipment and imaging principles, the acquired images differ in their spatial coordinate systems, which can affect the subsequent comprehensive analysis of the two types of image information. Therefore, in this embodiment, the report generation system performs cross-modal image spatial alignment processing on PET and CT images to ensure that the spatial positions of the PET and CT images are consistent, guaranteeing accurate correspondence of the information from the two images in subsequent processing.

[0045] To achieve spatial consistency between PET and CT images, this application employs a two-stage registration process: First, a mutual information-driven rigid registration algorithm aligns the CT image to the image coordinate system of the PET image. In this process, the PET image is set as a fixed image, while the CT image is a moving image. Mutual information is an indicator used to measure the statistical dependence between two images. By maximizing the mutual information between the PET and CT images, the algorithm finds the optimal rigid transformation parameters, such as translation and rotation, to gradually bring the CT image closer to the PET image in space. Simultaneously, considering the potential global misalignment caused by differences in patient scanning posture, the algorithm corrects for such misalignment during registration. For example, if the patient's body posture differs slightly during PET and CT scans, causing a deviation in the overall image position, the algorithm adjusts the transformation parameters to eliminate this deviation, ensuring that the CT and PET images are initially aligned in overall spatial position.

[0046] After rigid registration, the post-processing stage begins, resampling the PET images and CT images aligned to the PET image coordinate system into a preset 3D space. The purpose of this preset 3D space is to standardize the input resolution and voxel spacing of the PET and CT images. In practical applications, the resolution and voxel spacing of images acquired by different scanning devices may vary, which can affect the accuracy of subsequent image analysis and processing. By resampling all images into a unified space, including segmentation masks (if applicable), it ensures that all images have the same spatial scale. For example, assuming the resolution of the PET image is 2mm × 2mm × 2mm and the resolution of the CT image is 1mm × 1mm × 1mm, after resampling, the resolution of both is adjusted to a preset 1.5mm × 1.5mm × 1.5mm, and the voxel spacing is also standardized accordingly. This allows for operation based on a unified spatial standard in subsequent lesion identification, segmentation, and other processing, avoiding errors caused by inconsistent resolution and voxel spacing. This completes cross-modal image spatial alignment processing of PET and CT images, providing a consistent spatial basis for subsequent image analysis.

[0047] In this way, by performing cross-modal image spatial alignment processing, the spatial differences between PET images and CT images can be eliminated, enabling the information from the two types of images to correspond accurately. This provides a reliable guarantee for subsequent comprehensive analysis and further improves the accuracy of the generated medical image reports.

[0048] 202. Using a pre-trained PET lesion segmentation model, the high-metabolic lesion portion is identified and segmented in the processed PET image, and the lesion imaging parameters output by the PET lesion segmentation model for the high-metabolic lesion portion are obtained.

[0049] In order to identify and analyze PET images, a PET lesion segmentation model is pre-trained in this embodiment. The PET lesion segmentation model is used to segment high-metabolic lesions. High-metabolic lesions are usually associated with diseased tissues such as tumors, and their metabolic activity is significantly higher than that of the surrounding normal tissue. On PET images, they are areas with increased uptake of radioactive tracers. The specific training process of the PET lesion segmentation model is as follows:

[0050] To train a high-performance PET lesion segmentation model, it is first necessary to acquire multiple sample PET images with lesion annotations. In this embodiment, the multiple sample PET images come from annotated PET images from multiple centers (specifically, 1308 PET images). These images contain manually annotated lesions to ensure the accuracy and reliability of the data. After acquiring the sample images, preprocessing is required. Preprocessing includes normalization (CT window truncation, Z-score normalization) and voxel normalization. CT window truncation can highlight the display of specific tissue structures, while Z-score normalization can convert the image data into a distribution with a mean of 0 and a standard deviation of 1, making the data between different images comparable. Voxel normalization can unify the voxel size of the images, avoiding the impact of inconsistent voxel sizes on the model's training effect.

[0051] Next, following the pre-defined model architecture, multiple pre-processed PET images were used for model training. The pre-defined model architecture can be nnUNet (no-new-Net, a deep learning framework for medical image segmentation). The nnUNet architecture has the advantages of automatically configuring the number of channels, patch size, and training time, and can automatically adjust model parameters according to the characteristics of the input data, improving training efficiency and performance. During model training, the model parameters need to be optimized based on a pre-defined model training loss function to obtain a pre-trained PET lesion segmentation model. The loss function is a combination of Dice Loss and Cross-Entropy Loss. Dice Loss primarily measures the overlap between the model's predictions and the ground truth annotations, while Cross-Entropy Loss measures the difference between the model's predicted probability distribution and the true distribution. By combining these two loss functions, the model's segmentation accuracy and classification accuracy can be comprehensively considered, improving model performance.

[0052] Meanwhile, the model training also employs a 5-fold cross-validation strategy, dividing the dataset into five parts and using four parts as the training set and one part as the validation set in rotation for five training and validation iterations, thus providing a more comprehensive evaluation of the model's performance. Furthermore, training using a GPU (A100) significantly accelerates the training process, which takes approximately 120 hours. In addition, data augmentation techniques, including affine transformations, random blurring, rotation, and intensity perturbations, are employed to increase data diversity and improve the model's generalization ability.

[0053] After training the PET lesion segmentation model through the above process, the report generation system can then use this model. The PET lesion segmentation model can be pre-trained and stored in the report generation system so that it can be directly invoked when a report generation requirement arises. When invoking the PET lesion segmentation model, the report generation system inputs the processed PET image into the model. Based on this, the model identifies and segments high-metabolic lesions from the PET image. Subsequently, the high-metabolic lesion portion (i.e., the segmentation mask output by the model) and the processed PET image are input into the feature extraction module of the PET lesion segmentation model. This allows the model to map the high-metabolic lesion portion onto the processed PET image, enabling it to extract and output lesion imaging parameters for the high-metabolic lesion portion. For each segmented region (lesion or organ), the PET lesion segmentation model outputs a series of lesion imaging parameters, including SUVmean, SUVmax, TLG, and MTV. The extraction process for each indicator is described below:

[0054] Based on the PET lesion segmentation model, the average standard uptake value (SUVmean) of all voxels in the high-metabolic lesion portion can be calculated, and this average SUVmean can be output as one of the imaging parameters of the lesion. The average SUVmean is the average standard uptake value (SUV) of all voxels within the three-dimensional volume of a lesion, reflecting the average level of metabolic activity of the lesion as a whole. It is less affected by lesion heterogeneity and is suitable for efficacy evaluation and dynamic change monitoring.

[0055] Based on the PET lesion segmentation model, the single voxel value with the highest standardized uptake value is determined in the high-metabolic lesion region, and this single voxel value is output as one of the lesion imaging parameters. The single voxel value is also known as SUVmax, which is the highest SUV value among the single voxels within the lesion region. It is commonly used to identify the most active area of ​​the tumor, has high sensitivity, and is frequently used for screening or guiding the selection of puncture targets. Doctors can use the SUVmax value to determine the most active site of the tumor and perform targeted puncture biopsies or treatments.

[0056] The above two indicators are lesion-level indicators. In this embodiment, the following two regional-level indicators will also be calculated, including:

[0057] Based on a PET lesion segmentation model, multiple voxels with standard uptake values ​​exceeding a preset threshold are identified in high-metabolic lesion regions. The metabolically active volume (MTV) of the lesion, composed of these multiple voxels, is then statistically analyzed and output as one of the lesion imaging parameters. The metabolically active volume (MTV) refers to the total metabolically active volume of the lesion composed of all voxels above a preset SUV threshold (e.g., 2.5 or 40% SUVmax), typically measured in cm. 3 MTV (tumor burden) is a direct quantitative indicator of the extent of disease and can be used for efficacy evaluation and prognostic analysis. By monitoring changes in MTV, we can understand changes in tumor size and metabolic activity, providing a reference for developing treatment plans.

[0058] Furthermore, based on the PET lesion segmentation model, the product of the average standard uptake value and the metabolically active volume of the lesion is calculated, and this product is used as a comprehensive index. This comprehensive index is then output as one of the imaging parameters of the lesion. The comprehensive index, also known as TLG, is the product of the average standard uptake value and the metabolically active volume of the lesion, measured in grams. It is the product of SUV and volume, comprehensively considering both metabolic intensity and lesion volume, and is often used for comprehensive efficacy assessment, superior to simply using SUVmax or MTV. For example, when evaluating treatment effectiveness, changes in TLG can more comprehensively reflect changes in the metabolism and volume of the lesion, providing doctors with more accurate diagnostic information.

[0059] In this way, through the above process, the PET lesion segmentation model can accurately identify and segment high-metabolic lesion parts and output a series of valuable lesion imaging parameters, providing important support for the generation of medical imaging reports and clinical diagnosis.

[0060] 203. Using a pre-trained CT image organ segmentation model, organ images are segmented in the processed CT images, and the high-metabolic lesion portion is mapped to the organ image to determine the lesion location information.

[0061] To identify and analyze CT images, this embodiment of the application pre-trains a CT image organ segmentation model. CT images clearly present the anatomical structure of internal organs, and the CT image organ segmentation model is used to accurately segment various organs from CT images, thereby assisting in determining the correspondence between hypermetabolic lesions and organs, as well as the specific location of the lesions. The specific training process of the CT image organ segmentation model is as follows:

[0062] First, multiple sample CT images labeled with organ parts are acquired. These sample CT images come from labeled CT images from multiple centers (specifically, 1308 sample CT images). These images all contain manually labeled organs to ensure the accuracy and reliability of the data, providing a solid foundation for model training. Next, similar to the training process of the PET image recognition model above, the multiple sample CT images need to be preprocessed. Preprocessing includes normalization (CT window truncation, Z-score standardization) and voxel standardization, which can be referred to the description in step 202 above, and will not be repeated here.

[0063] Subsequently, similar to the training process of the PET image recognition model above, the model needs to be trained using multiple preprocessed sample CT images according to the preset model architecture. The preset model architecture can also be nnUNet. The nnUNet architecture has the advantages of automatically configuring the number of channels, patch size, and training time, and can automatically adjust the model parameters according to the characteristics of the input data, improving the training efficiency and performance of the model. It will not be elaborated here. In addition, during the model training process, similar to the training process of the PET image recognition model, the model parameters need to be optimized based on the preset model training loss function to obtain the pre-trained CT image organ segmentation model. The loss function can also be a combination of Dice Loss and Cross-Entropy Loss. During training, a 5-fold cross-validation training strategy is adopted, dividing the dataset into 5 parts, using 4 parts as the training set and 1 part as the validation set in turn, and performing 5 training and validation cycles to more comprehensively evaluate the model performance. Moreover, using a GPU (A100) for training can greatly speed up the training process. The entire training process takes about 120 hours. Furthermore, similar to the training process of the PET image recognition model above, data augmentation techniques can be used, including affine transformation, random blurring, rotation, intensity perturbation, etc., to increase the diversity of data and improve the generalization ability of the model.

[0064] After training the CT image organ segmentation model through the above process, the report generation system can then use this model. The CT image organ segmentation model can be pre-trained and stored in the report generation system so that it can be directly invoked when a report generation requirement arises. When invoking the CT image organ segmentation model, the report generation system inputs the processed CT image into the model. This allows the model to overlap the center of the hypermetabolic lesion with the organ image mask, mapping the hypermetabolic lesion to the organ image and determining the organ to which it belongs, thus obtaining the lesion-related organ information. For example, through model analysis, it can be determined whether a hypermetabolic lesion is located in the liver, lung, or another organ. After determining the organ to which the lesion belongs, the CT image organ segmentation model also identifies the three-dimensional coordinate information of the hypermetabolic lesion. The three-dimensional coordinate information is represented in the form of (x, y, z), and this three-dimensional coordinate (x, y, z) is relative to the standard CT coordinate system. At the same time, the three-dimensional coordinate information needs to be processed with reference to a preset location coding mapping table. The preset location coding mapping table combines organ anatomical region atlases (such as the CT-SEG 27-organ standard) and can automatically generate a textual description of the lesion location based on the three-dimensional coordinate information, such as "left upper lobe of the lung" or "segment VII of the liver". These textual descriptions can intuitively reflect the specific location of the lesion in the human body. The report generation system will use the textual description as lesion location information, providing strong support for the generation of medical imaging reports and clinical diagnosis.

[0065] 204. According to the preset organizational standards, hierarchical organizational processing is performed on the information of organs related to the lesion and the location information of the lesion to obtain structured summary information.

[0066] In this embodiment, to facilitate the automatic generation of medical image reports, the report generation system performs hierarchical organization processing on the lesion-related organ information and lesion location information according to preset organizational standards to obtain structured summary information. Specifically, the report generation system can be equipped with a structured summary generator, which organizes features hierarchically according to region-modality-tracer classification and generates a standardized tabular summary as structured summary information. The specific process of generating structured summary information is as follows:

[0067] First, information is categorized based on human anatomical regions. For example, the human body is divided into major regions such as the head, neck, chest, abdomen, and pelvis. The chest region is further subdivided into more specific sub-regions such as the left lung and right lung. After determining the organ and location information of the hypermetabolic lesion, the lesion information is categorized according to these anatomical regions. For example, if the lesion is located in the "left upper lobe of the lung," it is categorized under the left lung sub-region of the chest region. This method of classification by region conforms to the clinical diagnostic thinking habits of doctors, facilitates the rapid location of the approximate location of the lesion, and enables a holistic assessment of the lesions in the relevant areas.

[0068] Building upon the regional hierarchy, the structured summary generator further subdivides the data according to image modalities. Different image modalities (such as CT images and PET images) have different imaging principles and characteristics, providing different types of information. For example, CT images primarily present information about human anatomical structures, while PET images focus on reflecting the metabolic activity of tissues. For lesion information within each region, the structured summary generator labels its corresponding image modality. For instance, within the left lung sub-region of the chest, it distinguishes between lesion information from CT images and lesion information from PET images to quickly filter out lesion information under specific modalities, perform comparative analysis, and improve diagnostic accuracy.

[0069] The structured summary generator further categorizes tracers by type at the modality level. Different tracers (such as FDG) have different targeting and metabolic properties, reflecting the physiological and pathological processes of different tissues. For example, FDG is a commonly used PET tracer, primarily used to reflect tissue glucose metabolism. For lesion information in PET images, the structured summary generator labels the type of tracer used to subsequently determine the metabolic characteristics of lesions under different tracers, providing more detailed information for the automated development of disease diagnosis, staging, and treatment plans. For example, by comparing the uptake of lesions under different tracers, the malignancy and metabolic activity of tumors can be determined.

[0070] After obtaining the hierarchical feature organization through the above process, the structured summary generator designs a standardized table structure based on the results of the hierarchical feature organization. The column headings of the table should include key information such as region, modality, tracer, lesion-related organ, lesion location information, and lesion imaging parameters. For example, the first column of the table could be "Region," the second "Modality," the third "Tracer," the fourth "Lesion-Related Organ," and the fifth "Lesion Location Information." Subsequent columns can be set with lesion imaging parameters according to actual needs, such as SUVmean, SUVmax, TLG, MTV, etc. Then, the information after hierarchical feature organization is filled in according to the table structure. For each lesion, its corresponding region, modality, tracer, lesion-related organ, lesion location information, and lesion imaging parameters are accurately filled into the corresponding cells of the table. For example, for a lesion located in the "left upper lobe of the lung," enter "Chest - Left Lung" in the "Region" column, "PET" in the "Modal" column, "FDG" in the "Tracer" column, "Lung" in the "Lesion-Related Organ" column, and "Lesion Location Information" in the "Lesion Upper Lobe of the Left Lung" column. If there are lesion imaging parameters, enter their values ​​in the corresponding indicator columns. Finally, optimize the format of the generated table to ensure it is clear and easy to read. For example, adjust column width, font size and color, set appropriate border styles, etc., and output the standardized table summary as structured summary information for subsequent generation of medical imaging reports, improving the accuracy of the generated medical imaging reports and contributing to the improvement of the quality of medical imaging reports.

[0071] 205. Obtain the preset report template, add the structured summary information and lesion imaging parameters to the preset report template, and use the added preset report template as the initial report.

[0072] In this embodiment, the report generation system obtains a preset report template, combines the structured summary information with the preset report template, and integrates information such as scanning protocol, lesion distribution, and statistical indicators to form a complete initial report. This provides a foundation for subsequently generating accurate and diagnostic medical imaging reports. The specific process for generating the initial report is as follows:

[0073] The report generation system pre-designs a standardized clinical template as a default report template. Based on the routine procedures and requirements of medical imaging diagnosis, the default report template is divided into different information areas, such as a patient basic information area, a scan information area, an imaging manifestation description area, and a diagnostic conclusion area. Each area has corresponding format and content specifications to ensure the consistency and standardization of the report. For example, the patient basic information area includes fields such as patient name, gender, age, and medical record number; the imaging manifestation description area reserves space for describing lesion characteristics. The report generation system matches structured summary information according to its category with the corresponding area in the default report template. For example, it fills the imaging manifestation description area with information about the organs related to the lesion and the location of the lesion, describing in detail the organ and specific location of the lesion; it embeds the lesion's imaging parameters in the imaging manifestation description area or a dedicated statistical indicator display area in the form of tables or text, ensuring that the filled information is accurate and conforms to the format requirements of the clinical template.

[0074] In another optional implementation, the report generation system also extracts the protocol information for this scan from the imaging examination system, including the scanning equipment model, scanning parameters (such as tube voltage, tube current, slice thickness, interslice spacing, etc.), and the type and dosage of tracer used (for PET scans). The extracted scanning protocol information is integrated into the scan information area of ​​a preset report template, clearly and accurately listing all scanning parameters and related information according to the template's format requirements. For example, the scan information area might specify, "This examination used [equipment model] for scanning, tube voltage was [X] kV, tube current was [X] mAs, slice thickness was [X] mm, and interslice spacing was [X] mm; if it was a PET scan, [tracer name] was used, and the dosage was [X] mCi," to ensure that the subsequently generated medical imaging report accurately records the specific details of the scan, providing a reference for subsequent medical diagnosis.

[0075] In another optional implementation, the report generation system will also comprehensively analyze the distribution of lesions in different regions of the human body based on structured summary information. This includes statistically analyzing the number, size, and morphology of lesions in each region (such as the head, neck, chest, abdomen, and pelvis), and analyzing the correlation between lesions in different regions. For example, if lesions are found simultaneously in the chest and abdomen, the system can analyze whether there is a potential relationship such as metastasis. Next, the analysis results of the lesion distribution are integrated into the image description area or a dedicated lesion distribution display area of ​​the preset report template in the form of text descriptions or charts. The text descriptions can be concise and clear, highlighting key information, such as "This examination found [X] lesions in the chest [specific organ], with the largest lesion diameter approximately [X] mm; and [X] lesions in the abdomen [specific organ], with irregular shapes, suggesting possible metastasis." It should be noted that if charts are used, anatomical diagrams can be used to mark the location and number of lesions in the corresponding areas, making the lesion distribution more intuitive.

[0076] In another optional implementation, the report generation system also summarizes and organizes the lesion imaging parameters (SUVmean, SUVmax, TLG, MTV, etc.) extracted from PET images, embeds the summarized statistical indicators in the form of tables or text into the corresponding areas of the preset report template, and provides a brief interpretation of the statistical indicators. For example, the preset report template lists the SUVmean, SUVmax, TLG, and MTV values ​​of each lesion, with the following text explaining, "In this examination, the average SUVmean of the lesions is [X], and the maximum SUVmax is [X], indicating that the overall metabolic activity of the lesions is high; the TLG and MTV values ​​reflect the metabolic intensity and volume of the lesions, and larger TLG and MTV values ​​may indicate a heavier tumor burden," thus enabling the subsequently generated medical imaging report to display key statistical information.

[0077] In this way, the above process can generate a rich initial report, which can then be polished to obtain a richer medical imaging report with a style close to the real report, reducing the burden of manually generating reports.

[0078] 206. The initial report is processed using a finely tuned large language model to obtain and output a medical imaging report.

[0079] To make the generated medical imaging report resemble a real clinical report and improve its professionalism and readability, this embodiment of the application uses a fine-tuned large language model to process the initial report. The specific process of fine-tuning the large language model is as follows:

[0080] First, the original large language model is determined, which can be the Qwen large model. To avoid the high computational resource consumption and overfitting problems caused by full parameter fine-tuning, this embodiment of the application uses the LoRA technique to insert trainable parameter blocks only into the attention module of the original large language model. The LoRA technique, through low-rank decomposition, significantly reduces the number of trainable parameters while maintaining model performance, thus improving fine-tuning efficiency.

[0081] Subsequently, fine-tuning data is acquired, which includes multiple summary samples, each corresponding to a medical report. In practical applications, the fine-tuning data can consist of clinical image summaries and reference reports, and can specifically include 1308 cases.

[0082] Next, the large language model with inserted trainable parameter blocks was fine-tuned using fine-tuning data. The fine-tuning process aimed to achieve an output style close to that of a real report, from inputting an image structure summary. Simultaneously, the large language model with inserted trainable parameter blocks was continuously optimized based on a preset fine-tuning loss function (cross-entropy loss + BLEU / ROUGE guided scoring) until the fine-tuning loss function converged. The current state of the large language model was then used as the fine-tuned large language model. The entire fine-tuning process was completed on four A100 GPUs and required approximately 48 hours of training.

[0083] After fine-tuning the large language model through the above process, the report generation system can store the fine-tuned large language model for easy retrieval when needed. Specifically, upon receiving the initial report, the report generation system processes the initial report using the fine-tuned large language model to refine it. The fine-tuned large language model, based on its learned real-world reporting style and language patterns, optimizes the sentences in the initial report to produce and output a medical imaging report, making the report more fluent, accurate, and professional. This medical imaging report can include anatomical regional imaging findings, diagnostic conclusions, and impressions, providing doctors with clear and comprehensive diagnostic evidence.

[0084] In another optional implementation, the report generation system can also determine medical recommendations for the medical imaging report, including follow-up suggestions. These recommendations are output simultaneously with the medical imaging report to guide the patient's subsequent treatment and management. For example, in a lung cancer patient's medical imaging report, if the report shows a suspicious nodule in the lung, and certain characteristics of the nodule (such as size, shape, and metabolic activity) suggest a possible malignancy, the medical recommendations provided by the report generation system can include detailed follow-up suggestions. For instance, it might recommend that the patient undergo a follow-up chest CT scan within the next three months to observe changes in the nodule. If the nodule increases in size, becomes irregular in shape, or exhibits increased metabolic activity during the follow-up examination, a biopsy is further recommended to clarify the pathological diagnosis. Furthermore, the medical recommendations can also include lifestyle modifications, such as quitting smoking and avoiding exposure to harmful gases and dust, as well as dietary recommendations, such as increasing the intake of foods rich in vitamins and antioxidants. The report generation system will output this detailed medical recommendation along with the medical imaging report, providing comprehensive and targeted guidance for the patient's subsequent treatment and management, helping doctors to better develop personalized treatment plans.

[0085] Furthermore, to facilitate doctors' modifications and additions to the report content based on actual circumstances, the report generation system can also generate editable draft reports from medical imaging reports. These draft reports retain the core information of the report but are presented in an easily editable format. Doctors can directly manipulate the drafts; for example, if a doctor feels the description of lesions in the report is not detailed enough, or wants to add special findings discovered during image review, they can modify and improve the draft report to better meet actual clinical needs. Additionally, the system can obtain mapping results obtained by mapping hypermetabolic lesions to organ images. The editable draft report and the mapping results are then output for doctors to manually review. Doctors can use their clinical experience and professional knowledge to assess the accuracy of the mapping results and further review the content of the editable draft report, ensuring the accuracy and reliability of the report.

[0086] Furthermore, with a data link established between the report generation system and the hospital's imaging system (PACS, Picture Archiving and Communication Systems), the report generation system can also transmit medical imaging reports to the hospital's imaging system. As a crucial information management platform within the hospital, the hospital's imaging system can centrally manage these reports. When doctors need to access patient imaging reports, they can simply log into the hospital's imaging system to conveniently and quickly obtain the required reports, eliminating the need to search through different systems or folders, significantly improving work efficiency.

[0087] Practical applications demonstrate that the report generation system proposed in this application significantly improves both report generation efficiency and quality. Compared to traditional manual processes, the time from image input to report output is reduced by 83%, greatly improving hospital efficiency and reducing patient waiting time. Regarding diagnostic consistency, the medical image reports generated by the system achieve a 92.4% consistency rate with expert physician diagnoses (Cohen's κ = 0.82), indicating high accuracy and reliability and providing strong diagnostic support for doctors. Furthermore, cross-institutional report consistency is improved by 45% (ROUGE-L score 0.78), with reports generated by different medical institutions using this system exhibiting greater uniformity in content and format, facilitating communication and collaboration among doctors and improving the overall quality of medical services. Moreover, the report generation system offers good deployment flexibility, adapting to the report generation needs of solid tumors such as prostate cancer, lung cancer, and lymphoma, as well as hematological malignancies. Regardless of the type of tumor, the system can generate accurate and standardized medical image reports based on its specific imaging characteristics and diagnostic requirements. Furthermore, the report generation system also has a built-in interpretability verification module (SHAP value visualization). Taking lung cancer reports as an example, when the report generation system generates a report and provides a diagnostic conclusion, the interpretability verification module will use SHAP value visualization to show the contribution of each imaging feature (such as nodule size, density, metabolic activity, etc.) to the diagnostic conclusion. Doctors can intuitively understand the model's decision-making process by viewing these visualization results, understand why the report generation system comes to such a diagnostic conclusion, thereby increasing their trust in the report and better applying the reports generated by the report generation system to clinical practice.

[0088] The method provided in this application acquires PET and CT images to be analyzed, performs cross-modal image space alignment processing on the PET and CT images, uses a pre-trained PET lesion segmentation model to identify and segment high-metabolic lesions in the processed PET images, and obtains the lesion imaging parameters output by the PET lesion segmentation model for the high-metabolic lesion portions. A pre-trained CT image organ segmentation model is used to segment organ images in the processed CT images, and the high-metabolic lesion portions are mapped to organ images to determine lesion location information. The lesion imaging parameters and lesion location information are processed to generate an initial report, and a fine-tuned large language model is used to process the initial report to obtain and output a medical image report. By intelligently segmenting and extracting biomarkers from PET and CT images, the accuracy of image information extraction is improved. Furthermore, the fine-tuned large language model can automatically generate and output realistic medical image reports, reducing manual intervention. This improves the accuracy and consistency of report diagnosis while shortening the report generation cycle and increasing report generation efficiency.

[0089] Furthermore, as Figure 1 In a specific implementation of the method, this application provides an automated medical image report generation device based on artificial intelligence, such as... Figure 3 As shown, the device includes: an alignment module 301, an analysis module 302, and a generation module 303.

[0090] The alignment module 301 is used to acquire PET images and CT images to be analyzed, and to perform cross-modal image space alignment processing on the PET images and the CT images.

[0091] The analysis module 302 is used to identify and segment high-metabolic lesion portions in the processed PET image using a pre-trained PET lesion segmentation model, and to obtain the lesion imaging parameters output by the PET lesion segmentation model for the high-metabolic lesion portions.

[0092] The analysis module 302 is also used to segment organ images in the processed CT images using a pre-trained CT image organ segmentation model, and to map the hypermetabolic lesion portion to the organ image to determine the lesion location information.

[0093] The generation module 303 is used to organize the imaging parameters and location information of the lesion, generate an initial report, and process the initial report using a fine-tuned large language model to obtain and output a medical imaging report.

[0094] In specific application scenarios, the alignment module 301 is used to align the CT image to the image coordinate system of the PET image using a mutual information-driven rigid registration algorithm, and to correct global misalignment caused by differences in scanning position during the alignment process; and to resample the PET image and the CT image aligned to the image coordinate system of the PET image into a preset three-dimensional space to complete the cross-modal image space alignment processing of the PET image and the CT image.

[0095] In specific application scenarios, the device also includes:

[0096] The training module is used to acquire multiple sample PET images with lesion portions marked, preprocess the multiple sample PET images, and train a model using the preprocessed multiple sample PET images according to a preset model architecture. During the model training process, the model parameters are optimized based on a preset model training loss function to obtain the pre-trained PET lesion segmentation model. The module also acquires multiple sample CT images with organ portions marked, preprocesses the multiple sample CT images, and trains a model using the preprocessed multiple sample CT images according to the preset model architecture. During the model training process, the model parameters are optimized based on the preset model training loss function to obtain the pre-trained CT image organ segmentation model.

[0097] In a specific application scenario, the analysis module 302 is used to input the processed PET image into the PET lesion segmentation model, and based on the PET lesion segmentation model, identify and segment the high-metabolic lesion portion in the PET image; input the high-metabolic lesion portion and the processed PET image into the PET lesion segmentation model, so that the PET lesion segmentation model maps the high-metabolic lesion portion onto the processed PET image, so that the PET lesion segmentation model extracts and outputs the lesion imaging parameters for the high-metabolic lesion portion.

[0098] In specific application scenarios, the analysis module 302 is used to calculate the average standard uptake value of all voxels in the high-metabolic lesion portion based on the PET lesion segmentation model, and output the average standard uptake value as one of the lesion imaging parameters; and / or, based on the PET lesion segmentation model, determine the single voxel value with the highest standard uptake value in the high-metabolic lesion portion, and output the single voxel value as one of the lesion imaging parameters; and / or, based on the PET lesion segmentation model, determine multiple voxels in the high-metabolic lesion portion with standard uptake values ​​higher than a preset threshold, and statistically analyze the metabolically active lesion volume composed of the multiple voxels, outputting the metabolically active lesion volume as one of the lesion imaging parameters; and / or, based on the PET lesion segmentation model, calculate the product of the average standard uptake value and the metabolically active lesion volume, and output the product as a comprehensive index as one of the lesion imaging parameters.

[0099] In a specific application scenario, the analysis module 302 is used to overlap the center of the hypermetabolic lesion with the organ image mask so that the hypermetabolic lesion is mapped to the organ image and to determine the organ to which the hypermetabolic lesion belongs, thereby obtaining the organ information related to the lesion; to identify the three-dimensional coordinate information of the hypermetabolic lesion, and to process the three-dimensional coordinate information with reference to a preset position encoding mapping table to obtain the lesion location information.

[0100] In specific application scenarios, the device includes:

[0101] A fine-tuning module is used to determine the original large language model, insert trainable parameter blocks into the attention module of the original large language model using LoRA technology; acquire fine-tuning data, wherein the fine-tuning data includes multiple summary samples, each of which corresponds to a medical report; fine-tune the large language model with the inserted trainable parameter blocks by referring to the fine-tuning data, and continuously optimize the large language model with the inserted trainable parameter blocks based on a preset fine-tuning loss function during the fine-tuning training process until the fine-tuning loss function converges, and take the large language model in the current state as the fine-tuned large language model.

[0102] In a specific application scenario, the generation module 303 is used to perform hierarchical organization processing on the lesion-related organ information and the lesion location information according to a preset organization standard to obtain structured summary information; obtain a preset report template, add the structured summary information and the lesion imaging parameters to the preset report template, and use the added preset report template as the initial report.

[0103] In specific application scenarios, the generation module 303 is also used to determine a medical recommendation plan with the medical image report, and output the medical recommendation plan and the medical image report simultaneously; and / or, use the medical image report to generate an editable report draft, and obtain the mapping result obtained by mapping the hypermetabolic lesion part with the organ image, and output the editable report draft and the mapping result for doctors to manually review; and / or, when a data link is established with the hospital imaging system, transmit the medical image report to the hospital imaging system so that the hospital imaging system can manage the medical image report.

[0104] The apparatus provided in this application acquires PET and CT images to be analyzed, performs cross-modal image space alignment processing on the PET and CT images, uses a pre-trained PET lesion segmentation model to identify and segment high-metabolic lesions in the processed PET images, and acquires the lesion imaging parameters output by the PET lesion segmentation model for the high-metabolic lesion portions. It also uses a pre-trained CT image organ segmentation model to segment organ images in the processed CT images, maps the high-metabolic lesion portions to the organ images, determines the lesion location information, organizes the lesion imaging parameters and lesion location information, generates an initial report, and processes the initial report using a fine-tuned large language model to obtain and output a medical image report. By intelligently segmenting and extracting biomarkers from PET and CT images, the accuracy of image information extraction is improved. Furthermore, the fine-tuned large language model can automatically generate and output realistic medical image reports, reducing manual intervention. This improves the accuracy and consistency of report diagnosis while shortening the report generation cycle and increasing report generation efficiency.

[0105] It should be noted that other corresponding descriptions of the functional units involved in the AI-based automated medical image report generation device provided in this application embodiment can be found in the following references. Figure 1 and Figure 2 The corresponding descriptions in [the document] will not be repeated here.

[0106] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0107] The above embodiments and the technical features in the embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0108] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

[0109] In an exemplary embodiment, see Figure 4The invention also provides a computer device including a bus, a processor, a memory, and a communication interface. It may also include an input / output interface and a display device, wherein the various functional units can communicate with each other via the bus. The memory stores a computer program, and the processor executes the program stored in the memory to perform the artificial intelligence-based automated medical image report generation method described in the above embodiments.

[0110] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the AI-based automated medical image report generation method.

[0111] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented in hardware or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0112] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application.

[0113] Those skilled in the art will understand that the modules in the apparatus of the implementation scenario can be distributed within the apparatus of the implementation scenario as described, or they can be located in one or more apparatuses different from this implementation scenario, with corresponding changes. The modules of the above-described implementation scenario can be combined into one module, or they can be further divided into multiple sub-modules.

[0114] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of the implementation scenario.

[0115] The above disclosures are only a few specific implementation scenarios of this application. However, this application is not limited to these. Any variations that can be conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. A method for automatically generating medical image reports based on artificial intelligence, characterized in that, include: Acquire PET and CT images to be analyzed, and perform cross-modal image space alignment processing on the PET and CT images; A pre-trained PET lesion segmentation model is used to identify and segment high-metabolic lesion portions in the processed PET images, and to obtain the lesion imaging parameters output by the PET lesion segmentation model for the high-metabolic lesion portions. A pre-trained CT image organ segmentation model is used to segment organ images in the processed CT images and map the hypermetabolic lesion portion to the organ images to determine the lesion location information. The imaging parameters and location information of the lesions are organized to generate an initial report. The initial report is then processed using a fine-tuned large language model to obtain a medical imaging report, which is then output.

2. The method according to claim 1, characterized in that, The cross-modal image space alignment process for the PET images and the CT images includes: Using a mutual information-driven rigid registration algorithm, the CT image is aligned to the image coordinate system of the PET image, and global misalignment caused by differences in scanning position is corrected during the alignment process; The PET image and the CT image aligned to the image coordinate system of the PET image are resampled into a preset three-dimensional space to complete the cross-modal image space alignment process of the PET image and the CT image.

3. The method according to claim 1, characterized in that, Before identifying and segmenting hypermetabolic lesions in the processed PET image using a pre-trained PET lesion segmentation model, and before obtaining the lesion imaging parameters output by the PET lesion segmentation model for the hypermetabolic lesion portion, the method further includes: Multiple PET images of sample lesions are acquired, the multiple PET images of sample lesions are preprocessed, and the model is trained using the preprocessed multiple PET images of sample lesions according to a preset model architecture. During the model training process, the model parameters are optimized based on a preset model training loss function to obtain the pre-trained PET lesion segmentation model. Multiple sample CT images labeled with organ parts are acquired, the multiple sample CT images are preprocessed, and the model is trained using the preprocessed multiple sample CT images according to the preset model architecture. During the model training process, the model parameters are optimized based on the preset model training loss function to obtain the pre-trained CT image organ segmentation model.

4. The method according to claim 1, characterized in that, The method employs a pre-trained PET lesion segmentation model to identify and segment high-metabolic lesion portions in the processed PET images, and obtains the lesion imaging parameters output by the PET lesion segmentation model for the high-metabolic lesion portions, including: The processed PET image is input into the PET lesion segmentation model, and the hypermetabolic lesion portion is identified and segmented in the PET image based on the PET lesion segmentation model. The hypermetabolic lesion portion and the processed PET image are input into the PET lesion segmentation model, so that the PET lesion segmentation model maps the hypermetabolic lesion portion to the processed PET image, so that the lesion imaging parameters are extracted and output based on the PET lesion segmentation model for the hypermetabolic lesion portion.

5. The method according to claim 4, characterized in that, The step of extracting and outputting imaging parameters of the hypermetabolic lesion based on the PET lesion segmentation model includes: Based on the PET lesion segmentation model, the average standard uptake value of all voxels in the high-metabolic lesion portion is calculated, and the average standard uptake value is used as one of the imaging parameters of the lesion and output; and / or, Based on the PET lesion segmentation model, the single voxel value with the highest standardized uptake value is determined in the hypermetabolic lesion portion, and the single voxel value is used as one of the imaging parameters of the lesion and output; and / or, Based on the PET lesion segmentation model, multiple voxels with standard uptake values ​​exceeding a preset threshold are identified in the high-metabolic lesion portion, and the metabolically active volume of the lesion composed of these multiple voxels is statistically analyzed. This metabolically active volume is then output as one of the lesion imaging parameters; and / or, Based on the PET lesion segmentation model, the product of the average standard uptake value and the metabolically active volume of the lesion is calculated, and the product is used as a comprehensive index. The comprehensive index is then used as one of the imaging parameters of the lesion and output.

6. The method according to claim 1, characterized in that, The step of mapping the hypermetabolic lesion portion to the organ image to determine the lesion location information includes: The center of the hypermetabolic lesion is overlapped with the organ image mask so that the hypermetabolic lesion is mapped onto the organ image, and the organ to which the hypermetabolic lesion belongs is determined to obtain the organ information related to the lesion. The three-dimensional coordinate information of the hypermetabolic lesion is identified, and the three-dimensional coordinate information is processed with reference to a preset location encoding mapping table to obtain the lesion location information.

7. The method according to claim 1, characterized in that, Before organizing the imaging parameters and location information of the lesion to generate an initial report, and processing the initial report using a fine-tuned large language model to obtain and output a medical imaging report, the method further includes: The original large language model is determined, and a trainable parameter block is inserted into the attention module of the original large language model using the LoRA technique; Acquire fine-tuning data, wherein the fine-tuning data includes multiple summary samples, each of which corresponds to a medical report; The large language model with the inserted trainable parameter block is fine-tuned and trained with reference to the fine-tuning data. During the fine-tuning training process, the large language model with the inserted trainable parameter block is continuously optimized based on the preset fine-tuning loss function until the fine-tuning loss function converges. The large language model in the current state is then used as the fine-tuned large language model.

8. The method according to claim 1, characterized in that, The process of organizing the imaging parameters and location information of the lesion to generate an initial report includes: According to preset organizational standards, the organ information related to the lesion and the location information of the lesion are hierarchically organized to obtain structured summary information; Obtain a preset report template, add the structured summary information and the lesion imaging parameters to the preset report template, and use the preset report template after adding the information as the initial report.

9. The method according to claim 1, characterized in that, The method further includes: Determine a medical recommendation plan based on the medical imaging report, and output the medical recommendation plan and the medical imaging report simultaneously; and / or, The medical imaging report is used to generate an editable draft report, and a mapping result obtained by mapping the hypermetabolic lesion portion to the organ image is obtained. The editable draft report and the mapping result are then output for manual review by a physician; and / or, When a data link is established with the hospital's imaging system, the medical image report is transmitted to the hospital's imaging system so that the hospital's imaging system can manage the medical image report.

10. An automated medical image report generation device based on artificial intelligence, characterized in that, include: The alignment module is used to acquire PET and CT images to be analyzed and perform cross-modal image space alignment processing on the PET and CT images. The analysis module is used to identify and segment high-metabolic lesion portions in the processed PET images using a pre-trained PET lesion segmentation model, and to obtain the lesion imaging parameters output by the PET lesion segmentation model for the high-metabolic lesion portions. The analysis module is also used to segment organ images in the processed CT images using a pre-trained CT image organ segmentation model, and to map the hypermetabolic lesion portion to the organ image to determine the lesion location information. The generation module is used to organize the imaging parameters and location information of the lesion, generate an initial report, and process the initial report using a fine-tuned large language model to obtain and output a medical imaging report.

Images