Multi-modal medical examination report automatic generation method and system

Through automatic analysis of medical examination images, using deep learning and large language models to generate detailed reports, the subjectivity and omission problems in traditional report generation methods are solved, and the efficiency and accuracy of reports are improved.

CN119092032APending Publication Date: 2024-12-06Artificial Intelligence and Robotics Innovation Center of Hong Kong Institute of Innovation, Chinese Academy of Sciences +1
View PDF 0 Cites 10 Cited by

Patent Information

Application Number
CN202410945875.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The traditional medical examination report generation method is highly subjective and easily missed, which affects the accuracy of diagnosis and the effectiveness of treatment.

Method used

Using the automatic generation method of multimodal medical examination reports, the pre-processed examination images are used to identify and segment lesions, and pre-trained deep learning models and large language models are called to generate detailed reports that meet medical standards, including lesion descriptions and diagnostic opinions. and treatment advice.

Benefits of technology

It greatly improves the efficiency and accuracy of inspection reports, reduces the burden of manual records by doctors, and improves the professionalism and integrity of reports.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119092032A_ABST
    Figure CN119092032A_ABST
Patent Text Reader

Abstract

The invention provides a multi-modal medical examination report automatic generation method and system, and the method comprises the steps: carrying out the lesion recognition and segmentation of a preprocessed examination image, and obtaining a segmentation map of an abnormal labeling region; a pre-trained deep learning model is called to carry out recognition based on the marked abnormal region segmentation map, and a preliminary text prompt is obtained, and the preliminary text prompt comprises the focus position, the focus morphological characteristics, the focus lesion type and the focus clinical significance of the marked abnormal region segmentation map; a pre-trained large language model is called to prompt the preliminary text, a medical examination report is generated, and the medical examination report comprises focus description, diagnosis suggestions and treatment suggestions. According to the invention, the efficiency and accuracy of medical examination report generation are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for automatically generating a multimodal medical examination report. Background Art

[0002] In the medical field, accurate and detailed examination reports are the key basis for diagnosis and treatment. Examination reports not only provide doctors with a comprehensive overview of the condition, but also provide important references for subsequent treatment decisions, condition monitoring and rehabilitation plans.

[0003] However, traditional examination report writing relies on doctors' manual recording and summarization, which is a time-consuming and labor-intensive process and may contain subjective biases and omissions, affecting the accuracy of diagnosis and the effectiveness of treatment.

[0004] It can be seen from this that the medical examination report generation method in the relevant technology is highly subjective and prone to missing technical problems. Summary of the invention

[0005] The present invention provides a method and system for automatically generating multimodal medical examination reports, which are used to solve the defects of the medical examination report generation method in the prior art, such as strong subjectivity and easy omission of content, and realize automatic processing of multiple medical images and generate detailed reports that meet medical standards to assist doctors in diagnosis and decision-making.

[0006] The present invention provides a method for automatically generating a multimodal medical examination report, comprising the following steps: performing lesion recognition and segmentation on a preprocessed examination image to obtain a segmentation map of annotated abnormal regions; calling a pre-trained deep learning model to perform recognition based on the segmentation map of annotated abnormal regions to obtain a preliminary text prompt, wherein the preliminary text prompt includes the lesion location, lesion morphological characteristics, lesion pathology type and clinical significance of the lesion in the segmentation map of annotated abnormal regions; calling a pre-trained large language model to generate a medical examination report based on the preliminary text prompt, wherein the medical examination report includes a lesion description, a diagnostic opinion and a treatment suggestion.

[0007] According to a method for automatically generating a multimodal medical examination report provided by the present invention, before the lesion identification and segmentation of the preprocessed examination image to obtain a segmentation map of the marked abnormal area, the method also includes: acquiring the examination image, wherein the examination image includes an X-ray image, a computer tomography image, and a magnetic resonance imaging image; and denoising, standardizing, and enhancing the examination image to obtain the preprocessed examination image.

[0008] According to a method for automatically generating a multimodal medical examination report provided by the present invention, the lesion identification and segmentation of the preprocessed examination image to obtain a marked abnormal area segmentation map includes: calling a pre-trained deep learning model to identify the lesion of the preprocessed examination image to obtain at least one abnormal lesion area; annotating the lesion information corresponding to the abnormal lesion area to the abnormal lesion area to obtain a marked abnormal examination image; and segmenting the marked abnormal examination image according to the abnormal area to obtain a marked abnormal area segmentation map.

[0009] According to a method for automatically generating a multimodal medical examination report provided by the present invention, after calling a pre-trained large language model to generate a medical examination report based on the preliminary text prompt, the method further includes: obtaining user feedback information on the medical examination report; and fine-tuning and reinforcing learning the pre-trained large language model based on the feedback information to update the pre-trained large language model.

[0010] According to a method for automatically generating a multimodal medical examination report provided by the present invention, after calling a pre-trained deep learning model to perform recognition based on the annotated abnormal area segmentation map to obtain a preliminary text prompt, the method further includes: obtaining historical clinical records and a preset medical knowledge base corresponding to the preprocessed examination images; supplementing the preliminary text prompt based on the historical clinical records and the preset medical knowledge base to obtain a supplementary text prompt.

[0011] The present invention provides a multimodal medical examination report automatic generation system, comprising the following modules: an image processing module, used to perform lesion recognition and segmentation on preprocessed examination images to obtain a marked abnormal area segmentation map; a prompt generation module, used to call a pre-trained deep learning model to perform recognition based on the marked abnormal area segmentation map to obtain a preliminary text prompt, wherein the preliminary text prompt includes the lesion location, lesion morphological characteristics, lesion pathology type and clinical significance of the lesion in the marked abnormal area segmentation map; a large model generation module, used to call a pre-trained large language model to generate a medical examination report based on the preliminary text prompt, wherein the medical examination report includes a lesion description, a diagnostic opinion and a treatment suggestion.

[0012] According to a multimodal medical examination report automatic generation system provided by the present invention, the multimodal medical examination report automatic generation system also includes: a feedback and optimization module, which is used to obtain user feedback information on the medical examination report; based on the feedback information, fine-tuning and reinforcement learning are performed on the pre-trained large language model to update the pre-trained large language model.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for automatically generating a multimodal medical examination report as described in any one of the above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for automatically generating a multimodal medical examination report.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned methods for automatically generating a multimodal medical examination report.

[0016] The multimodal medical examination report automatic generation method and system provided by the present invention obtain a segmentation map of annotated abnormal areas by performing lesion recognition and segmentation on preprocessed examination images; a pre-trained deep learning model is called to perform recognition based on the segmentation map of annotated abnormal areas to obtain preliminary text prompts; a pre-trained large language model is called to generate a medical examination report based on the preliminary text prompts, wherein the medical examination report includes a lesion description, a diagnostic opinion, and a treatment suggestion. Thus, artificial intelligence technology can be used to learn and extract rules from massive medical data, automatically analyze examination images, and generate detailed reports that meet medical standards, thereby greatly improving the efficiency and accuracy of examination reports, and further solving the technical problems of the medical examination report generation method in the related art being highly subjective and prone to omission of content. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of the method for automatically generating a multimodal medical examination report provided by the present invention.

[0019] Figure 2 It is a schematic diagram of the image segmentation model provided by the present invention.

[0020] Figure 3 It is a structural schematic diagram of the multimodal medical examination report automatic generation system provided by the present invention.

[0021] Figure 4 It is a schematic diagram of the internal structure of the system provided by the present invention.

[0022] Figure 5 It is a schematic diagram of the external operation of the system provided by the present invention.

[0023] Figure 6 It is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] In the medical field, accurate and detailed examination reports are the key basis for diagnosis and treatment. Examination reports not only provide doctors with a comprehensive overview of the condition, but also provide important references for subsequent treatment decisions, condition monitoring, and rehabilitation plans. However, traditional examination report writing relies on doctors' manual recording and summarization, which is time-consuming and labor-intensive, and may contain subjective biases and omissions, affecting the accuracy of diagnosis and the effectiveness of treatment.

[0026] With the development of artificial intelligence technology, especially the breakthroughs in deep learning and natural language processing technology, it has become possible to automatically generate medical examination reports. Artificial intelligence systems can learn and extract patterns from massive amounts of medical data, automatically analyze examination images, and generate detailed reports that meet medical standards.

[0027] The multimodal medical examination report generation system greatly improves the efficiency and accuracy of examination reports through intelligent analysis and report generation of patient examination images. This system can automatically process a variety of medical images and generate detailed reports that meet medical standards to assist doctors in diagnosis and decision-making.

[0028] refer to Figure 1 , Figure 1 is a flow chart of the method for automatically generating a multimodal medical examination report provided by the present invention, such as Figure 1 As shown, the method includes the following steps.

[0029] Step 101 , performing lesion identification and segmentation on the pre-processed inspection image to obtain a segmentation map of annotated abnormal regions.

[0030] In the embodiment of the present invention, after the patient is examined, the examination images (such as X-rays, CT, MRI, etc.) are input into the image processing module for pre-processing and lesion segmentation.

[0031] First, the input examination images are preprocessed, including denoising, standardization and enhancement, to improve image quality and processing effect. Then, a deep learning-based segmentation model is used to identify and segment the preprocessed images, and generate segmentation maps with abnormal areas marked, such as bleeding, neoplasms, mucosal abnormalities, airway abnormalities and other abnormal areas.

[0032] Step 102, calling a pre-trained deep learning model to perform recognition based on the annotated abnormal region segmentation map to obtain a preliminary text prompt, wherein the preliminary text prompt includes the lesion location, lesion morphological characteristics, lesion pathology type and clinical significance of the lesion in the annotated abnormal region segmentation map.

[0033] In the embodiment of the present invention, after completing the lesion segmentation of the image, key features in the segmentation map of the annotated abnormal region are extracted, including key information such as the location, size, and shape of the lesion.

[0034] The above preliminary text prompts include the specific location of the lesion, morphological characteristics, possible lesion types and their clinical significance. The system uses predefined prompt templates to convert image features into structured text prompts to ensure that the prompt content is standardized and meets the clinical surgical report standards. The system then combines the information from the medical knowledge base and historical clinical records to further explain and supplement the lesion characteristics.

[0035] Step 103, calling the pre-trained large language model to generate a medical examination report based on the preliminary text prompt, wherein the medical examination report includes a lesion description, a diagnosis opinion, and a treatment suggestion.

[0036] In the embodiment of the present invention, the above preliminary text prompt is input into the large language model, and the model trained with medical professional knowledge automatically generates an examination report that meets medical standards based on the prompt text, including lesion description, diagnosis opinion, treatment suggestion, etc. The accuracy and professionalism of the report content are ensured through semantic analysis and optimization.

[0037] The pre-training process of the above-mentioned large language model (LLM) specifically includes steps such as data collection and preprocessing, model selection and training, template and framework design, and performance evaluation and optimization.

[0038] For example, collect a large amount of medical literature, research papers, clinical guidelines, case reports, etc., which should cover various diseases, symptoms, diagnostic methods and treatment plans. Obtain actual case data from the hospital's electronic medical record system, including patients' personal information (need to be anonymized to protect privacy), medical history, physical examination results, laboratory test results, imaging examination reports, doctor's diagnosis and treatment recommendations, etc. Clean the collected data to remove noise and errors, and annotate them appropriately, especially for key parts of the examination report, such as abnormal findings, diagnostic conclusions, etc.

[0039] Select or design a large language model architecture suitable for processing text data, such as the GPT series of models; use the general domain corpus to pre-train the model so that it has basic language understanding and generation capabilities. Subsequently, use the medical domain corpus to further train the model so that it can understand and generate medical-related text. For the examination report generation task, use the annotated medical examination data and templates to fine-tune the model to improve its ability to generate accurate examination reports.

[0040] Design or utilize existing medical examination report templates, which should contain the basic structure and common terms of the report, and design logic to fill the text generated by the model into the corresponding positions of the template to ensure the completeness and consistency of the report.

[0041] Through the above steps of the embodiment of the present application, the pre-processed inspection image is subjected to lesion identification and segmentation to obtain a segmentation map of the marked abnormal area; the pre-trained deep learning model is called to identify based on the segmentation map of the marked abnormal area to obtain a preliminary text prompt, wherein the preliminary text prompt includes the lesion location, lesion morphological characteristics, lesion type and clinical significance of the lesion in the segmentation map of the marked abnormal area; the pre-trained large language model is called to generate a medical examination report based on the preliminary text prompt, wherein the medical examination report includes a description of the lesion, a diagnostic opinion and a treatment suggestion. The efficiency and accuracy of the inspection report are improved, thereby solving the technical problem that the method for generating medical inspection reports in the related art is highly subjective and easy to omit content.

[0042] According to a method for automatically generating a multimodal medical examination report provided by the present invention, before performing lesion identification and segmentation on the preprocessed examination image to obtain a segmentation map of annotated abnormal regions, the method further includes: Acquiring examination images, wherein the examination images include X-ray images, computerized tomography images, and magnetic resonance imaging images; The inspection image is subjected to denoising, standardization and enhancement processing to obtain a preprocessed inspection image.

[0043] Denoising is the process of reducing or eliminating noise in an image. Noise may come from a variety of factors, such as defects in the device itself, interference during transmission, unstable shooting environment, etc. The presence of noise will interfere with the useful information in the image and affect the accuracy and efficiency of subsequent processing.

[0044] In the embodiment of the present invention, image denoising includes spatial domain filtering (such as mean filtering, median filtering, Gaussian filtering, etc.) and frequency domain filtering (such as low-pass filtering, band-pass filtering, etc.). These methods reduce noise by smoothing the image or suppressing high-frequency noise components.

[0045] Normalization is the process of adjusting image data to a specific range or distribution so that subsequent processing can be performed on a uniform and comparable basis.

[0046] In the embodiment of the present invention, for grayscale images such as medical images, standardization generally includes grayscale normalization, that is, scaling the grayscale value of the image to a specified range (such as 0-255). In addition, it may also include contrast enhancement, histogram equalization and other techniques to improve the visual effect and contrast of the image. Standardization helps to reduce the differences between images from different sources and improve the robustness and generalization ability of subsequent processing algorithms.

[0047] According to a multimodal medical examination report automatic generation method provided by the present invention, lesion recognition and segmentation are performed on pre-processed examination images to obtain a segmentation map of annotated abnormal areas, including: Calling a pre-trained deep learning model to perform lesion recognition on the pre-processed examination image to obtain at least one abnormal lesion area; Annotate the lesion information corresponding to the abnormal lesion area to the abnormal lesion area to obtain an annotated abnormal examination image; The anomaly inspection image marked is segmented according to the abnormal area to obtain a segmentation map of the anomaly area marked.

[0048] refer to Figure 2 , Figure 2 It is a schematic diagram of the image segmentation model provided by the present invention.

[0049] In an embodiment of the present invention, a large amount of medical image data with lesion annotations is collected, and the annotations usually include the boundaries or regions of the lesions. The annotated data set is used to train a preset deep learning model (e.g., U-Net, Mask R-CNN, DeepLab, etc.). During the training process, the model learns how to classify pixels in the input image as lesions or background, and optimizes the model performance by adjusting hyperparameters, using regularization, data enhancement, etc., to obtain a pre-trained deep learning model.

[0050] In an embodiment of the present invention, the preprocessed inspection image is input into a pretrained deep learning model, which outputs the probability or category of each pixel belonging to a lesion to obtain an abnormal lesion area; and the lesion area is segmented to obtain a segmentation map of the marked abnormal area.

[0051] According to a method for automatically generating a multimodal medical examination report provided by the present invention, after calling a pre-trained large language model to generate a medical examination report based on preliminary text prompts, the method further includes: Obtain user feedback on medical examination reports; Based on the feedback information, the pre-trained large language model is fine-tuned and reinforced to update the pre-trained large language model.

[0052] In the embodiment of the present invention, after generating a medical examination report, the doctor will evaluate the quality of the report content, further review and modify it, and the pre-trained large language model will record the doctor's feedback information. According to the doctor's feedback, the pre-trained large language model is fine-tuned and reinforced to optimize the model performance and continuously improve the quality of report generation.

[0053] For example, we can organize and analyze the collected feedback from doctors to distinguish which are specific errors that need to be corrected and which are suggestions for adjusting the report style or format. For error correction, we need to be able to accurately locate the specific steps or algorithm links in generating the report so that targeted improvements can be made. For improvement suggestions, we need to evaluate their rationality and feasibility to determine whether they should be included in the subsequent optimization plan.

[0054] The pre-trained large language model fine-tunes the generative model based on doctor feedback. This includes adjusting the model's parameters, optimizing the algorithm structure, or introducing new features to improve the model's accuracy and robustness. The application of reinforcement learning technology enables the model to automatically learn how to generate reports that better meet doctors' expectations in the process of continuous trial and error. The model is given corresponding rewards or penalties based on the doctor's feedback, thereby guiding the model to evolve in a better direction. In addition, historical data and doctor feedback will be used to train more advanced models, such as deep learning models, to further improve the quality and efficiency of report generation.

[0055] According to a method for automatically generating a multimodal medical examination report provided by the present invention, after calling a pre-trained deep learning model to perform recognition based on a segmentation map of annotated abnormal regions and obtaining a preliminary text prompt, the method further includes: Obtaining historical clinical records corresponding to the preprocessed examination images and a preset medical knowledge base; The preliminary text prompt is supplemented based on historical clinical records and a preset medical knowledge base to obtain a supplementary text prompt.

[0056] In the embodiment of the present invention, the preliminary text prompt includes the specific location of the lesion, morphological characteristics, possible lesion type and its clinical significance. The image features are converted into structured text prompts using predefined prompt templates to ensure that the prompt content is standardized and meets the clinical surgical report standards. Combined with the information of the medical knowledge base and historical clinical records, the lesion characteristics are further explained and supplemented.

[0057] The multimodal medical examination report automatic generation system provided by the present invention is described below. The multimodal medical examination report automatic generation system described below and the multimodal medical examination report automatic generation method described above can refer to each other.

[0058] refer to Figure 3 , Figure 3 It is a structural diagram of the multimodal medical examination report automatic generation system provided by the present invention, which includes an image processing module 301, a prompt generation module 302, a large model generation module 303 and a feedback and optimization module 304.

[0059] The image processing module 301 is used to identify and segment the pre-processed examination image to obtain a segmentation map of the annotated abnormal area; the prompt generation module 302 is used to call the pre-trained deep learning model to identify based on the segmentation map of the annotated abnormal area to obtain a preliminary text prompt, wherein the preliminary text prompt includes the lesion location, lesion morphological characteristics, lesion type and clinical significance of the lesion in the segmentation map of the annotated abnormal area; the large model generation module 303 is used to call the pre-trained large language model to generate a medical examination report based on the preliminary text prompt, wherein the medical examination report includes a description of the lesion, a diagnostic opinion and a treatment suggestion. The feedback and optimization module 304 is used to obtain user feedback information on the medical examination report; based on the feedback information, the pre-trained large language model is fine-tuned and reinforced to learn to update the pre-trained large language model.

[0060] This paper proposes a multimodal medical examination report generation system, which aims to provide an efficient and accurate information processing solution through automatic analysis of examination images and report generation. The system integrates image processing, text generation and model optimization, etc., which can significantly improve the generation efficiency and quality of examination reports and reduce the workload of doctors. Specifically, the system mainly includes the following modules: Image processing module: This module uses a deep learning model to process the inspection image, including lesion segmentation and feature extraction. First, the input inspection image is preprocessed by denoising, contrast enhancement and other preprocessing operations to improve the accuracy and stability of the segmentation model. Then, the pre-trained segmentation model is used to automatically identify the abnormal lesion area in the inspection image, generate the corresponding segmentation map, and extract the specific location and morphological characteristics of the lesion. The segmentation map is further analyzed to extract detailed feature information of the lesion, such as size, shape, boundary features, etc., to provide basic data for the prompt generation module.

[0061] Prompt generation module: After the lesion segmentation is completed, the system generates preliminary text prompts based on the segmentation map. These prompts include detailed information such as the specific location of the lesion, morphological characteristics, possible lesion types, etc., providing a basis for the next report generation. First, the segmentation results and extracted features are sorted and summarized to form structured information. Then, based on the structured information, concise and clear text prompts are generated to describe the detailed characteristics of the lesion.

[0062] Large model generation module: This module uses a pre-trained large language model to automatically generate a detailed inspection report based on the text prompts provided by the prompt generation module. First, the generated text prompts are input into the large language model, and then the language model is used to generate a complete inspection report. Finally, the generated report is initially optimized and proofread to ensure that the language is fluent and the information is accurate.

[0063] Feedback and optimization module: Continuously optimize and adjust the generated model through doctors' feedback. The system records doctors' modification opinions and evaluations, and uses these data to fine-tune the model and strengthen learning, and continuously improve the quality of report generation. Feedback collection: Collect doctors' evaluation and modification suggestions on generated reports. Model fine-tuning: Based on feedback information, fine-tune the model to improve its generation capabilities.

[0064] refer to Figure 4 , Figure 4 It is an internal schematic diagram of the system provided by the present invention, which specifically includes image preprocessing (Step 0), image segmentation model (Step 1), text prompt generation (Step 2), large model (Step 3), report generation, expert feedback (Step 4), and reinforcement learning (Step 5).

[0065] The detailed steps of the internal and external processes of this system are as follows: Internal Process: Step S0: After the patient is examined, the examination images (such as X-rays, CT, MRI, etc.) are input into the image processing module for preprocessing and lesion segmentation. The input examination images are first subjected to preprocessing steps, including denoising, standardization, and enhancement, to improve image quality and processing effects. Then, the deep learning segmentation model is used to identify and segment the lesions of the preprocessed images, and a segmentation map with abnormal areas marked is generated, such as the segmentation tasks of abnormal areas such as bleeding, neoplasms, mucosal abnormalities, and airway abnormalities.

[0066] Step S1: After completing the lesion segmentation of the image, use the image processing module to extract key features in the segmentation map, including key information such as the location, size, and shape of the lesion.

[0067] Step S2: Use the prompt generation module to generate preliminary text prompts based on the extracted features. These prompts include the specific location of the lesion, morphological characteristics, possible lesion types and their clinical significance. The system uses predefined prompt templates to convert image features into structured text prompts to ensure that the prompt content is standardized and meets the clinical surgical report standards. The system then combines the information from the medical knowledge base and historical clinical records to further explain and supplement the lesion characteristics.

[0068] Step S3: Input the above prompt text into the large language model, and let the model trained with medical expertise automatically generate an examination report that meets medical standards based on the prompt text, including lesion description, diagnosis opinion, treatment suggestions, etc. And through semantic analysis and optimization, ensure the accuracy and professionalism of the report content.

[0069] Step S4: After the examination report is generated, the doctor will evaluate the quality of the report content, further review and modify it, and the system will record the doctor's feedback information.

[0070] Step S5: The feedback and optimization module fine-tunes and performs reinforcement learning on the generated model based on the doctor’s feedback, optimizes the model performance, and continuously improves the quality of report generation.

[0071] refer to Figure 5 , Figure 5 It is a schematic diagram of the external working of the system provided by the present invention, specifically including a doctor (Step 0), image interaction (Step 1) and image preprocessing (Step 2).

[0072] Step S0: The doctor inputs the examination report image into the system through instructions.

[0073] Step S1: The system parses the doctor's instructions, extracts keywords, and calls the corresponding modules for image processing and report generation.

[0074] Step S2: After the report is generated, it is fed back to the doctor in various forms such as voice, image and text to assist in surgical decision-making and operation. These steps describe in detail how the system works in the patient's examination report and show how the technology integrates data from multiple modalities to help doctors generate examination reports.

[0075] It should be noted that the present invention is applicable to a variety of scenarios of medical imaging diagnosis and examination report generation, and has broad application prospects in the following fields, in particular: medical imaging analysis, examination process recording and report generation, medical education and training, and telemedicine and diagnosis.

[0076] Specifically, the above-mentioned multimodal medical examination report automatic generation device provided by the present invention can implement all the method steps implemented by the above-mentioned multimodal medical examination report automatic generation method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0077] Figure 6 is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the multimodal medical examination report automatic generation method, which includes: performing lesion recognition and segmentation on the pre-processed examination image to obtain a segmentation map of the annotated abnormal region; calling a pre-trained deep learning model to perform recognition based on the segmentation map of the annotated abnormal region to obtain a preliminary text prompt, wherein the preliminary text prompt includes the lesion location, lesion morphological characteristics, lesion pathology type and clinical significance of the lesion in the segmentation map of the annotated abnormal region; calling a pre-trained large language model based on the preliminary text prompt to generate a medical examination report, wherein the medical examination report includes a lesion description, a diagnosis opinion and a treatment suggestion. 。

[0078] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0079] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal medical examination report automatic generation method provided by the above methods, which method includes: performing lesion identification and segmentation on the preprocessed examination image to obtain a marked abnormal area segmentation map; calling a pre-trained deep learning model to perform identification based on the marked abnormal area segmentation map to obtain a preliminary text prompt, wherein the preliminary text prompt includes the lesion location, lesion morphological characteristics, lesion pathology type and clinical significance of the lesion in the marked abnormal area segmentation map; calling a pre-trained large language model based on the preliminary text prompt to generate a medical examination report, wherein the medical examination report includes a lesion description, a diagnostic opinion and a treatment recommendation.

[0080] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multimodal medical examination report automatic generation method provided by the above-mentioned methods, the method comprising: performing lesion identification and segmentation on the preprocessed examination image to obtain a marked abnormal area segmentation map; calling a pre-trained deep learning model to perform identification based on the marked abnormal area segmentation map to obtain a preliminary text prompt, wherein the preliminary text prompt includes the lesion location, lesion morphological characteristics, lesion pathology type and clinical significance of the marked abnormal area segmentation map; calling a pre-trained large language model to generate a medical examination report based on the preliminary text prompt, wherein the medical examination report includes a lesion description, a diagnostic opinion and a treatment recommendation.

[0081] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0082] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatically generating a multimodal medical examination report, characterized in that: include: Perform lesion identification and segmentation on the preprocessed examination images to obtain a segmentation map of the marked abnormal area; Calling a pre-trained deep learning model to perform recognition based on the segmentation map of the annotated abnormal region to obtain a preliminary text prompt, wherein the preliminary text prompt includes the lesion location, lesion morphological characteristics, lesion pathology type and clinical significance of the lesion in the segmentation map of the annotated abnormal region; The pre-trained large language model is called to generate a medical examination report based on the preliminary text prompt, wherein the medical examination report includes a lesion description, a diagnosis opinion, and a treatment suggestion.

2. The method for automatically generating a multimodal medical examination report according to claim 1, characterized in that: Before performing lesion identification and segmentation on the preprocessed examination image to obtain a segmentation map of annotated abnormal regions, the method further includes: Acquiring examination images, wherein the examination images include X-ray images, computer tomography images, and magnetic resonance imaging images; The inspection image is subjected to denoising, standardization and enhancement processing to obtain a preprocessed inspection image.

3. The method for automatically generating a multimodal medical examination report according to claim 1, characterized in that: The step of performing lesion identification and segmentation on the preprocessed examination image to obtain a segmentation map of annotated abnormal regions includes: Calling a pre-trained deep learning model to perform lesion recognition on the pre-processed examination image to obtain at least one abnormal lesion area; marking the lesion information corresponding to the abnormal lesion area to the abnormal lesion area to obtain a marked abnormal inspection image; The anomaly-labeled inspection image is segmented according to the abnormal region to obtain an anomaly-labeled region segmentation map.

4. The method for automatically generating a multimodal medical examination report according to claim 1, characterized in that: After calling the pre-trained large language model to generate a medical examination report based on the preliminary text prompt, the method further includes: Obtaining user feedback information regarding the medical examination report; Fine-tune and reinforce the learning of the pre-trained large language model based on the feedback information to update the pre-trained large language model.

5. The method for automatically generating a multimodal medical examination report according to claim 1, characterized in that: After the calling of the pre-trained deep learning model to perform recognition based on the annotated abnormal region segmentation map to obtain a preliminary text prompt, the method further includes: Acquiring historical clinical records and a preset medical knowledge base corresponding to the preprocessed examination images; The preliminary text prompt is supplemented based on the historical clinical record and the preset medical knowledge base to obtain a supplementary text prompt.

6. A multimodal medical examination report automatic generation system, characterized in that: include: An image processing module is used to identify and segment the lesions on the pre-processed examination images to obtain a segmentation map of the marked abnormal area; A prompt generation module is used to call a pre-trained deep learning model to perform recognition based on the segmentation map of the annotated abnormal region to obtain a preliminary text prompt, wherein the preliminary text prompt includes the lesion location, lesion morphological characteristics, lesion pathology type and clinical significance of the lesion in the segmentation map of the annotated abnormal region; The large model generation module is used to call the pre-trained large language model to generate a medical examination report based on the preliminary text prompt, wherein the medical examination report includes a lesion description, a diagnosis opinion, and a treatment suggestion.

7. The multimodal medical examination report automatic generation system according to claim 6, characterized in that: The multimodal medical examination report automatic generation system also includes: A feedback and optimization module is used to obtain user feedback information on the medical examination report; based on the feedback information, fine-tune and reinforce the learning of the pre-trained large language model to update the pre-trained large language model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for automatically generating a multimodal medical examination report as described in any one of claims 1 to 5 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for automatically generating a multimodal medical examination report as described in any one of claims 1 to 5 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for automatically generating a multimodal medical examination report as described in any one of claims 1 to 5 is implemented.

Citation Information

Cited By

  • Medical image multi-stage analysis system

    CN119831938A

  • Image report generation method, system and equipment based on multi-granularity knowledge fusion

    CN119851852A

  • Image Report Generation Method, System and Device Based on Multi-Granularity Knowledge Fusion

    CN119851852B

  • Medical image analysis system based on mixed detection mask large model

    CN119863433A

  • Focus positioning method based on large model

    CN119941649A