Radiology report generation system and method

Abstract

The present disclosure relates to a radiology report generation system and method. This radiology report generation system is configured to: obtain an analysis result for a target medical image by using an artificial intelligence analysis model; extract at least one similar image to the target medical image from a catalog set composed of pairs of medical images and radiology reports; determine at least one radiology report corresponding to the at least one similar image as a reference report; and generate a radiology report from the analysis result for the target medical image by using the reference report as a guideline.

Description

Video report generation system and method

[0001] The present disclosure relates to radiology report generation.

[0002] Recently, as artificial intelligence (AI) technology has been actively introduced into the medical field, research is being conducted on AI-based medical image analysis technology, such as the Lunit INSIGHT solution, which analyzes medical images and visually presents the analysis results.

[0003] Radiologists are assigned imaging interpretation tasks through a worklist, identify abnormalities identified through medical image analysis, and then create radiology reports. Recently, interest in technologies that automatically generate reports using generative AI has been growing.

[0004] The present disclosure provides an artificial intelligence-based image report generation system and method.

[0005] The present disclosure relates to an interface screen providing a video report.

[0006] A system for generating an image report according to some embodiments comprises a memory and a processor for executing commands stored in the memory. The processor is configured to obtain analysis results for a target medical image using an artificial intelligence analysis model, extract at least one similar image similar to the target medical image from a catalog set consisting of medical image-image report pairs, determine at least one image report corresponding to the at least one similar image as a reference report, and generate an image report from the analysis results for the target medical image using the reference report as a guideline.

[0007] The processor may be configured to determine whether findings corresponding to predetermined finding labels are present in the image report, and generate a finding label set comprised of finding labels extracted from the image report. The finding label set may be provided as a separate report distinct from the image report, or may be included in a designated section of the image report.

[0008] The above processor may be implemented to obtain clinical information through user input or linkage with a database of a medical institution, and to modify the image report using the clinical information or the analysis result of the clinical information.

[0009] The above processor may be implemented to store the final image report, which has undergone user editing or user confirmation for the image report, in a designated location.

[0010] The processor may be implemented to determine whether to add the image report to the catalog set, and to add the image report-the target medical image pair to the catalog set based on the determination result.

[0011] The above analysis results may include lesion information detected in the target medical image.

[0012] The analysis results may further include additional information extracted from the target medical image. The additional information may include at least one of detailed information about the detected lesion, information about findings detected other than the detected lesion, quality information about the target medical image, or information about metadata of the target medical image.

[0013] The above processor may be implemented to generate the additional information through visual question answering, which produces answers to questions from the target medical image.

[0014] The above processor may be implemented to select a set of questions related to the target medical image or the analysis results of the target medical image from a question bank consisting of questions, and to extract answers to each question included in the set of questions to generate the additional information.

[0015] The above analysis results may further include quantitative information about objects of interest present in the target medical image.

[0016] The above processor may be implemented to link the non-text form analysis results for the target medical image to the image report.

[0017] A method of operating an image report generation system according to some embodiments, comprising: obtaining analysis results for a target medical image using an artificial intelligence analysis model; extracting at least one similar image similar to the target medical image from a catalog set consisting of medical image-image report pairs, determining at least one image report corresponding to the at least one similar image as a reference report; and generating an image report from the analysis results for the target medical image by utilizing the reference report as a guideline.

[0018] The above operating method may further include a step of determining whether there are findings corresponding to predetermined finding labels in the image report, and a step of generating a finding label set composed of finding labels extracted from the image report.

[0019] The above method of operation may further include a step of obtaining clinical information through user input or linkage with a database of a medical institution, and a step of modifying the image report using the clinical information or an analysis result of the clinical information.

[0020] The above method of operation may further include a step of saving a final image report that has undergone user editing or user confirmation for the image report to a designated location.

[0021] The above operating method may further include a step of determining whether to add the image report to the catalog set, and a step of adding the image report-the target medical image pair to the catalog set based on the determination result.

[0022] The analysis result may include at least one of lesion information detected in the target medical image, additional information extracted from the target medical image, or quantitative information about an object of interest present in the target medical image. The additional information may include at least one of detailed information about the detected lesion, information about findings detected other than the detected lesion, quality information about the target medical image, or information about metadata of the target medical image.

[0023] The step of obtaining the above analysis results may include selecting a set of questions related to the target medical image or the analysis results of the target medical image from a question bank consisting of questions, and extracting answers to each question included in the set of questions to generate the above additional information.

[0024] The above method of operation may further include a step of linking a non-text form analysis result for the target medical image to the image report.

[0025] According to some embodiments, a computer program stored in a computer-readable recording medium comprises instructions that cause a processor to execute the instructions to obtain an analysis result for a target medical image using an artificial intelligence analysis model, extract at least one similar image similar to the target medical image from a catalog set consisting of medical image-image report pairs, determine at least one image report corresponding to the at least one similar image as a reference report, and generate an image report from the analysis result for the target medical image by using the reference report as a guideline.

[0026] According to an embodiment, by generating an image report using information on lesions detected in a medical image, detailed information such as lesion location information, lesion severity, information on various findings other than the detected lesion, information on the quality and metadata of the medical image, and quantitative information on objects of interest such as lesions, an image report with high reliability and accuracy and containing abundant analysis information can be automatically generated.

[0027] According to an embodiment, by using a reference report as a guideline to generate an image report, an image report with guaranteed reliability and accuracy can be generated, and errors caused by hallucinations, which are pointed out as a problem in generative artificial intelligence models, can be reduced.

[0028] According to an embodiment, by managing reference reports to be referenced when creating a video report as a catalog set, a personalized video report can be created based on the user's video report writing style or user preference.

[0029] In an embodiment, by formalizing an image report into a set of finding labels, the clinical validity of the image report can be evaluated using the set of finding labels, thereby increasing the reliability of the image report generated based on artificial intelligence.

[0030] According to an embodiment, by linking auxiliary images that visually provide lesion information to a text image report, the user's understanding of the image report can be improved, and consequently, reading efficiency can be improved.

[0031] According to an embodiment, by automatically generating an image report based on the analysis results of a medical image, the user's reading time and workload can be reduced, thereby improving reading efficiency, and as a result, memory resources and computing resources of a medical imaging system for managing medical images waiting to be read can be reduced.

[0032] FIG. 1 is a diagram illustrating the concept of an image report generation system according to one embodiment.

[0033] Figure 2 is a configuration diagram of an image report generation system according to one embodiment.

[0034] Figure 3 is an example of an image report according to one embodiment.

[0035] FIG. 4 is a diagram illustrating a method for generating a set of opinion labels according to one embodiment.

[0036] FIG. 5 is a drawing illustrating a method for generating an image report according to one embodiment.

[0037] FIG. 6 is a diagram illustrating a visual question answering (VQA) of an additional information extractor according to one embodiment.

[0038] Figure 7 is an example of a method for providing an image report according to one embodiment.

[0039] Figure 8 is a flowchart of a method for generating an image report according to one embodiment.

[0040] Figure 9 is a flowchart of a method for modifying an image report according to one embodiment.

[0041] Figure 10 is a flowchart of a catalog set management method according to one embodiment.

[0042] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted for clarity of explanation, and similar parts are designated with similar reference numerals throughout the specification.

[0043] In the description, when a part is said to "include" a component, this does not exclude other components, but rather implies that other components may be included, unless otherwise specifically stated. Furthermore, terms such as "part," "unit," and "module" used in the specification mean a unit that processes at least one function or operation, which may be implemented using hardware, software, or a combination of hardware and software.

[0044] The device or terminal of the present disclosure is a computing device configured and connected so that at least one processor can perform the operations of the present disclosure by executing instructions. A computer program includes instructions that are described to cause the processor to perform the operations of the present disclosure and can be stored on a non-transitory computer-readable storage medium. The computer program can be downloaded over a network or sold in product form.

[0045] The medical images of the present disclosure may be images taken of a patient's body part using various modalities or pathology images. For example, the modalities may be X-ray, MRI (magnetic resonance imaging), ultrasound, CT (computed tomography), MMG (mammography), DBT (digital breast tomosynthesis), endoscopy, positron emission tomography (PET), etc., and the medical images obtained through these may include X-ray images, MRI images, ultrasound images, CT images, MMG images, DBT images, endoscopic images, PET images, etc.

[0046] A user of the present disclosure may be, but is not limited to, a medical professional such as a doctor, nurse, clinical pathologist, radiologist, sonographer, or medical imaging specialist (radiologist).

[0047] The artificial intelligence model (AI model) of the present disclosure is a model that learns at least one task and can be executed by a processor. The task learned by the AI ​​model may refer to a task to be solved through learning or a task to be performed through learning. The AI ​​model may be implemented as a computer program that runs on a computing device, may be downloaded over a network, or may be sold as a product. Alternatively, the AI ​​model may be connected to various devices over a network.

[0048] FIG. 1 is a diagram illustrating the concept of an image report generation system according to one embodiment.

[0049] Referring to FIG. 1, an image report generation system (simply referred to as a system) (1) is a computing device implemented to analyze a medical image using an artificial intelligence model and automatically generate an image report (radiology report) using the analysis results. The image report generated by the system (1), information derived from the image report (e.g., a set of finding labels), and the analysis results of the medical image used to generate the image report (e.g., lesion information) may be provided to a user terminal (200) via a network. The medical image may be an image captured using various modalities of a patient's body part or a pathology image. For example, the medical image may be an X-ray image, an MRI image, an ultrasound image, a CT image, an endoscopic image, a PET image, etc., which are classified according to an imaging device, and may include a chest X-ray image, an MMG image, a DBT image, etc., which are classified according to a photographed part. In the description, the image report may be simply referred to as a report.

[0050] System (1) can additionally use the patient's clinical information provided along with the medical image to generate an image report. The clinical information can be used for various purposes in the image report generation process. System (1) can acquire various clinical information of the patient by linking with various databases of a medical institution, such as a Picture Archiving and Communication System (PACS), Electronic Medical Record (EMR), Electronic Health Record (EHR), etc. In addition, the information generated by System (1) can be implemented to be stored in a designated database of the medical institution.

[0051] The user terminal (200) can provide a user interface that displays necessary information on the screen by linking with the system (1) or a database storing medical data. The user terminal (200) can display information generated by analyzing medical images in the system (1) through a dedicated viewer.

[0052] The system (1) may be a server device, and the user terminal (200) may be a client terminal installed in a medical institution, and these may be interconnected via a network. The system (1) may be a local server connected to a network within a specific medical institution. Alternatively, the system (1) may be a cloud server and may be interconnected with terminals (medical staff terminals) of multiple medical institutions with access rights. The system (1) may be a cloud server and may be interconnected with individual patient terminals with access rights.

[0053] System (1) can generate an image report using the results of medical image analysis using an artificial intelligence model. System (1) can generate an image report from the results of analysis of medical images using a language model. The results of analysis of medical images can include information on lesions detected in the medical images, various findings other than specific lesions, quality and metadata of the medical images (e.g., modality, shooting information, etc.), and quantitative information extracted from the medical images (e.g., size, volume, ratio, number, etc. of lesions).

[0054] System (1) can be implemented to generate an image report using analysis results of medical images, by referencing image reports of images similar to the medical image. Since system (1) generates an image report using reports of similar images as guidelines, it can ensure the reliability and accuracy of the image report, and also reduce errors caused by hallucinations, which are pointed out as a problem in generative artificial intelligence models.

[0055] System (1) can generate an image report including the non-text analysis results by linking the non-text analysis results of medical images with the text image report. The non-text analysis results can include, for example, auxiliary images that visually provide the analysis results including lesion information, and the auxiliary images can be generated as, for example, DICOM Secondary Capture (SC). Through this, system (1) can generate an image report with improved explainability using an artificial intelligence model.

[0056] The system (1) can generate a final medical report through user confirmation of an initial medical report. The initial medical report can be edited by the user, and a final medical report can be generated through user confirmation of the modified medical report. The user editing process can be performed selectively. Accordingly, the initial medical report can be saved as a final medical report through user confirmation. Alternatively, the system (1) can regenerate the medical report by modifying the initial medical report using additionally acquired clinical information after generating the initial medical report. The user confirmation process can be omitted, and the initial medical report can be saved as the final medical report.

[0057] System (1) can be implemented to generate a finding label set consisting of predefined finding labels from an image report. The finding label set is data indicating whether a predefined clinical finding exists in the analysis results of a medical image and can be used to evaluate the clinical validity of the image report. Image reports generated using a Large Language Model (LLM) express the same finding in various phrases. These image reports themselves lack consistency, making it difficult to quantitatively evaluate the clinical performance of the system (1) from the image report output of the system (1). Therefore, the content of the image report is formatted into predefined finding labels to generate a finding label set, which can be used to evaluate the performance of the image report generation, such as reliability, reproducibility, sensitivity, and specificity. In addition, through the finding label set, a user can quickly recognize key findings detected in a medical image.

[0058] The image report and finding label set generated by the system (1) can be transmitted to the user terminal (200) or can be transmitted to a designated device. The finding label set can be provided as a separate report distinct from the image report, or can be included in a designated section of the image report. Alternatively, the finding label set can be provided as part of the analysis results of the medical image, for example, can be included in DICOM SC. In the description, the image report and the finding label set are generally described separately, but this description does not exclude the inclusion of the finding label set in the image report, and the method of providing the finding label set can vary depending on the settings.

[0059] Clinical information can be used for various purposes in the image report generation process. System (1) can utilize clinical information to generate analysis results used in image report generation or modify image reports generated using clinical information. Clinical information can be acquired through user input or by linking with a database that stores clinical information. System (1) can provide an interface through which a user can input clinical information. For example, a user can input clinical information using an input device such as a keyboard or mouse, or can input clinical information using a microphone. System (1) can actively acquire clinical information necessary for image report generation. That is, system (1) can inquire about the presence of clinical information necessary for image report generation based on the analysis results of medical images and receive clinical information input from the user. For example, in response to the detection of a specific finding in a medical image, system (1) can inquire about the presence of clinical information related to the specific finding and generate an image report reflecting the entered clinical information. Alternatively, the system (1) may, in response to detection of a specific finding in a medical image, search for the presence of clinical information related to the specific finding in a database of a medical institution.

[0060] The clinical information used to generate an image report can vary and include multimodal data such as text and images. For example, clinical information may include text data recording the patient's symptoms, test results (e.g., blood test results, functional test results, etc.), age, gender, and reason for examination. Clinical information may also include additional images beyond the target image for which the image report is generated. These additional images may include past images captured with the same type of imaging device as the target image, or images captured with a different type of imaging device.

[0061] For convenience of explanation, a simple form of communication between a user terminal (200) and a system (1) is described. However, the procedure for the user terminal (200) to receive an image report generated by the system (1) and to modify and confirm the same can be implemented through the linkage of various devices. For example, the user terminal (200) can display an image report and receive user input through a viewer that displays medical image-related data. The viewer can be installed and executed on a computing device within a workstation, for example, and can include a PACS viewer that links with a PACS (Picture Archiving and Communication System), but is not necessarily limited thereto.

[0062] In the following, the operation of the video report generation system is described in detail.

[0063] FIG. 2 is a configuration diagram of an image report generation system according to one embodiment, FIG. 3 is an example of an image report according to one embodiment, and FIG. 4 is a diagram explaining a method for generating an opinion label set according to one embodiment.

[0064] Referring to FIG. 2, the system (1) may include a report generator (100) that is executed by at least one processor and generates an image report (20) based on an analysis result for a medical image (10), a lesion detector (110) that generates an analysis result for the medical image (10), a reference repository (120) that stores reference medical image-report pairs, and a reference retrieval module (130) that extracts a report mapped to a similar image of the medical image (10) from the reference repository (120). Here, the report generator (100) may generate a finding label set (30) composed of predefined finding labels from the image report (20). The system (1) may include an interface device (not shown) for providing the image report and the finding label set to the outside and obtaining an input related to the image report.

[0065] Meanwhile, the system (1) may further include an additional information extractor (140) that provides additional analysis results for the medical image (10), and a measurer (150) that extracts quantitative information of an object of interest (e.g., size, volume, ratio, number, etc. of a lesion) from the medical image (10).

[0066] The lesion detector (110), the additional information extractor (140), and the measuring device (150) can be implemented as an artificial intelligence analysis model that outputs an analysis result for at least one of a medical image (10) and clinical information, and can be designed to utilize the analysis results of other models. The analysis result of at least one of the lesion detector (110), the additional information extractor (140), and the measuring device (150) can be used to generate an image report. In order to explain a method of analyzing a medical image (10) or information extracted from a medical image (10) separately, the artificial intelligence analysis model is divided into the lesion detector (110), the additional information extractor (140), and the measuring device (150), and it is not necessary to be clearly distinguished physically or logically, and can be implemented as at least one artificial intelligence model. In addition, although the system (1) is described as including an artificial intelligence analysis model that analyzes a medical image, the lesion detector (110), the additional information extractor (140), or the measuring device (150) need not be a dedicated model for the system (1).

[0067] The system (1) may further include a report reviser (160) that modifies the image report (20) generated by the report generator (100) to generate a modified report (20-1).

[0068] In addition, the system (1) may further include a pre-checker (170) that verifies the suitability of the medical image (10) before inputting the medical image (10) into a lesion detector (110), etc., thereby filtering out unsuitable medical images for analysis by an artificial intelligence model, thereby ensuring the reliability of the analysis results and consequently ensuring the reliability of the image report.

[0069] In addition to the lesion detector (110), the additional information extractor (140), and the measuring device (150), the components constituting the system (1) are blocks named to distinguish and explain operations, and do not need to be clearly separated physically or logically, and can be implemented as at least one artificial intelligence model.

[0070] The report generator (100) is implemented to receive analysis results for a medical image (10) and an image report (reference report) of a medical image similar to the medical image (10), and to use the reference report as a guideline to generate an image report corresponding to the analysis results. The report generator (100) may be implemented as a large language model that receives the analysis results and the reference report as prompts to generate an image report, but is not necessarily limited thereto. The report generator (100) is implemented to link with an external language model, and can transmit commands through an interface with the external language model, thereby obtaining the generated image report.

[0071] The analysis result for the medical image (10) may include information on a pre-specified lesion, additional information extracted from the medical image (10), quantitative information (e.g., size, volume, ratio, number, etc.) on an object of interest (e.g., lesion / organ / medical device), and may be obtained using at least one of a lesion detector (110), an additional information extractor (140), and a measuring device (150). Here, the additional information includes detailed information on the detected lesion (e.g., location information, detailed information (e.g., severity)), information on various findings other than the detected lesion, information on the quality and metadata of the medical image, and may be obtained through a Visual Question Answering (VQA) process that produces answers to questions from a given medical image.

[0072] The lesion detector (110) may be an AI model trained to perform a task of detecting abnormal findings (e.g., lesions) in medical images. The lesion detector (110) may be a medical image analysis model including a computer-aided diagnosis (CAD) model. The probability of the presence of predefined abnormal findings may be generated as a continuous value in the range of 0 to 1, and may be converted into a binary value indicating the presence of an abnormal finding exceeding a threshold value. The lesion detector (110) may output a list of lesions detected in the medical image (10) and an auxiliary image representing the detected lesion information in the form of a heat map, etc., as lesion information.

[0073] The analysis results of medical images provided by the lesion detector (110) can be provided in various formats, for example, SC (Secondary Capture), GSPS (Grayscale Softcopy Presentation State), SR (Structured Report) in DICOM format, etc. SC creates an image separate from the original medical image and displays the analysis results (e.g., lesion score indicating the presence or absence of a lesion, lesion location, etc.) as a heatmap, contour, etc., and can be provided separately from the original medical image. GSPS displays lesion information by overlaying it on the original medical image, and can turn the overlaid lesion information on / off. For example, the lesion detector (110) can analyze a chest X-ray image and detect lesion information from the chest X-ray image. The lesion detector (110) can detect pre-specified types of lesions such as nodules, pneumothorax, pleural effusion, consolidation, cardiomegaly, atelectasis, pneumoperitoneum, calcification, fibrosis, mediastinal widening, tuberculosis, and acute bone fracture from a chest X-ray image, and can also calculate major medical indicators including an abnormality score.

[0074] The lesion information, which is the analysis result of the lesion detector (110), is provided to the report generator (100), and may be provided to other artificial intelligence analysis models to assist in the analysis of other artificial intelligence analysis models. For example, the lesion list may be provided to the additional information extractor (140), through which the additional information extractor (140) may be used to recognize lesions detected in the medical image, select questions to obtain additional information about the detected lesions, select questions to detect findings other than the detected lesions, or obtain clinical information related to the detected lesions. Similarly, the lesion list may be provided to the measuring device (150), through which the measuring device (150) may be used to recognize lesions detected in the medical image, and measure quantitative information about the lesions. In addition, an auxiliary image (e.g., SC image) output from the lesion detector (110) may be provided to the additional information extractor (140) or the measuring device (150).

[0075] The additional information extractor (140) is a model that complements the lesion detector (110) that detects a specific lesion, and may be an AI model trained to extract additional information (e.g., location, etc.) about the lesion detected by the lesion detector (110) or to extract various additional information other than the detected lesion. The additional information extractor (140) may include a visual question-answering-based language model. The training method of the additional information extractor (140) may vary, and for example, it may be trained through contrastive learning using multimodal data including medical images and clinical information of a patient.

[0076] The additional information extractor (140) can be implemented based on a foundation model. A foundation model is an unsupervised or self-supervised model trained on a large amount of data with little or no annotated data. It typically learns useful information using large amounts of data that do not require labeling. For example, a foundation model can be trained using paired images and text descriptions (e.g., medical images and image reports) for each patient. Instead of regressing answers (annotations) from a given image, the foundation model can be trained by comparing data. For example, given an image and a text description, the training goal is to find the most informative representation (embedding space) for comparing the data. In particular, contrastive learning can be used, where features of a correct image and image report pair are made more similar to each other than features of a random pair.

[0077] The additional information extractor (140) can receive a medical image (10) as input and extract detailed information related to the medical image. At this time, the additional information extractor (140) can receive clinical information together with the medical image and extract additional information from multi-modal data. In addition, the additional information extractor (140) can receive analysis results from a lesion detector (110) or a measuring device (150) and use the same to analyze additional information related to the medical image (10). The additional information extractor (140) can extract detailed information related to the medical image through a VQA (Visual Question Answering) process and, for this purpose, can be implemented to obtain answers to questions from the medical image (10) or clinical information. The additional information extractor (140) can output additional information including question and answer pairs regarding the medical image (10) as an analysis result. The analysis results provided by the additional information extractor (140) can be determined in various ways depending on the questions.

[0078] The additional information extractor (140) can select questions to be used to obtain additional information based on the properties of the medical image, body part, patient information, or lesion information or quantitative information detected in the medical image. The additional information extractor (140) can select a set of questions related to the medical image and its analysis results from a question bank consisting of questions, or can generate a set of questions required for generating an image report. The analysis results for the medical image can be obtained through linkage with a lesion detector (110) or a measuring device (150), or can be obtained through self-analysis.

[0079] The additional information that the additional information extractor (140) can provide may be determined in various ways depending on the questions. For example, if questions are determined to analyze the location or lesion status (e.g., severity, size, etc.) of a detected lesion, or to search and analyze medical devices (e.g., catheters, pacemakers, etc.), surgical sites, clinical findings, etc., the additional information extractor (140) may perform analysis on medical images (10) or clinical information to obtain answers to each question. In addition, if questions are determined regarding the quality of the medical image (10) or the metadata of the medical image (10), the additional information extractor (140) may perform analysis such as quality measurement and metadata verification on the medical image (10) to obtain answers to each question. For example, the additional information extractor (140) may analyze the medical image (10) or clinical information to obtain answers to each question in Table 1, and output additional information composed of answers to the questions as the analysis results. In Table 1, 'FINDING_NAME' may be various findings that can be detected in medical images, and in particular, lesions or medical devices that were not detected by the lesion detector (110) may be described.

[0080] 질문 예시Is the image acquired with adequate quality for interpretation?Is this a frontal view chest x-ray?Is this a frontal view chest x-ray from a pediatric case?Is there a {FINDING_NAME} in the upper-left lung zone?Is there a {FINDING_NAME} in the upper-right lung zone?Is there a {FINDING_NAME} in the lower-right lung zone?Is there a {FINDING_NAME} in the lower-left lung zone?Is the {FINDING_NAME} severe?

[0081] The set of questions for further analysis of the medical image (10) can be configured in various ways. For example, the question bank can be configured with various questions, such as, 'Is the nodule located in the right upper lung area?', 'Is a rib fracture detected at the left fourth rib?', 'Is the image quality adequate?', 'Is this a frontal view chest X-ray?', etc. Thereafter, the additional information extractor (140) can select a set of questions for obtaining further detailed information from the medical image (10) from the question bank based on the list of lesions detected in the medical image (10). For example, if a 'nodule' is detected in a medical image (10) and a 'rib fracture' is not detected, a question related to the detected lesion (e.g., 'Is the nodule located in the right upper lung area?', etc.) may be selected, and a question related to the undetected lesion (e.g., 'Was a rib fracture detected at the fourth rib on the left?', etc.) may not be selected. Once a set of questions about the medical image is determined, the additional information extractor (140) can analyze the given information and output a text answer for each question. The additional information extractor (140) can obtain an answer to the question through a vision language model based on visual instruction tuning.

[0082] The measuring device (150) may be an AI model trained to analyze quantitative information (e.g., size, volume, ratio, number, etc.) for each measurement item of an object of interest (lesion, organ, medical device, etc.) in a medical image (10). At this time, the measuring device (150) may directly measure quantitative information of the object of interest in the original medical image (10), or may measure quantitative information of the object of interest existing in the medical image from an auxiliary image (e.g., SC image) including lesion information. The items to be measured by the measuring device (150) may be determined according to the object of interest, the characteristics of the medical image, the body part, etc., or may be determined according to lesion information or additional information detected in the medical image. The lesion information or additional information detected in the medical image may be obtained through linkage with the lesion detector (110) or the additional information extractor (140), or may be obtained through self-analysis. The quantitative information on the object of interest may be a measurement value to be included in an image report for the object of interest such as a lesion, organ, or medical device. Quantitative information about the object of interest may include, for example, lesion size, organ volume, cardiothoracic ratio, and endotracheal tube to carina distance.

[0083] The measuring device (150) receives analysis results from the lesion detector (110) or the additional information extractor (140), identifies objects of interest (e.g., nodules) requiring quantitative measurement from the analysis results, and then measures measurement items (e.g., size) specified for the objects of interest. The measuring device (150) can provide metric measurement values ​​for the objects of interest. For example, the measuring device (150) can generate quantitative information composed of an object of interest, a measurement value, and a measurement unit, for example, <Nodule: 3.0, mm>.

[0084] Meanwhile, a lesion detector (110), additional information extractor (140), or measuring device (150) implemented to analyze medical images can provide various analysis results of medical images, and can output analysis results displayed visually along with analysis results in text format. For example, the lesion detector (110) can provide analysis results of medical images in DICOM SC, GSPS, SR, etc.

[0085] The text-type analysis results output from the lesion detector (110), the additional information extractor (140), or the measuring device (150) are input to the report generator (100) and used to generate an image report, and the non-text-type analysis results can be provided in conjunction with the image report. For example, the system (1) can be implemented to provide text such as lesions described in the image report and auxiliary images (e.g., SC images) that visually display the lesions in conjunction with each other. Various methods for providing the image report and the visually displayed analysis results together can be implemented. For example, when the user clicks or hovers over the lesion name in the image report displayed through the viewer of the user terminal (200), the SC image including the lesion information can be displayed together with the image report.

[0086] The reference repository (120) stores medical image-image report pairs that serve as guidelines when generating image reports. Medical image-image report pairs may be annotated with information related to the presence / location of findings. The reference repository (120) may manage a catalog set consisting of N predefined medical image-image report pairs. At this time, each medical image may be converted into a feature representation vector (FVR) through an encoder and stored. The medical images and reports stored in the reference repository (120) may be referred to as reference images and reference reports.

[0087] Any image report stored in the reference repository (120) is referenced as a guideline when generating an image report. Accordingly, the user can customize the image report generated by referencing the reference report by adding or modifying the catalog set. For example, the user can add an image report written in a format adopted by a medical institution to the catalog set, add an image report written about a lesion / finding commonly detected in the medical institution to the catalog set, add an image report about an image in which a rare case is detected to the catalog set, or add an image report written in the user's own report style to the catalog set. In addition, the user can add an image report generated and confirmed by the report generator (100) to the catalog set. In this way, an image report that can be referenced by the report generator (100) when generating an image report can be added by the user.

[0088] In another embodiment, the catalog set can be managed without user intervention, or with minimal user intervention. For example, a device designated to manage the reference repository (120), such as a reference searcher (130), can determine whether it is useful to add a final image report to the catalog set. The reference searcher (130) can determine the usefulness of a new image report by measuring the information gain obtained by adding the new image report to the catalog set. The reference searcher (130) can approximate the information gain using a trained machine learning algorithm, such as a neural network, that outputs a score for the information gain. In another embodiment, the reference searcher (130) can embed the image reports into feature representation vectors and evaluate the similarity between the new image report and the reference reports stored in the catalog set. The reference searcher (130) can then add a new image report to the catalog set if it is determined that the new image report has low similarity to the reference reports stored in the catalog set, based on the similarity distance between the new image report and the reference report. According to another embodiment, the reference searcher (130) may determine whether to add an image report generated by the report generator (100) to the catalog set based on catalog set features including the number of cases included in the catalog set and the feature distribution of the cases included in the catalog set (e.g., distribution of lesions and other findings).

[0089] There may be various ways to add a new image report to the catalog set. The reference searcher (130) adds the image report generated by the report generator (100) to the catalog set until the catalog set is filled with M cases. After the catalog set is filled with M cases, the reference searcher (130) determines whether the image report newly generated by the report generator (100) is different from the cases included in the catalog set, and based on the determination result, discards cases that are similar to other cases among the M cases included in the catalog set and adds a new case. That is, since similar medical image-image report pairs do not need to exist repeatedly in the catalog set, the reference searcher (130) can add a rare case to the catalog set if a case including a relatively rare lesion or finding is found based on the feature distribution of the cases included in the catalog set.

[0090] The reference searcher (130) can identify at least one image similar to a medical image (10) among the reference images stored in the reference storage (120), and determine a report mapped to the similar image as a reference report. The reference searcher (130) can provide the reference report to the report generator (100), or can provide a pair of similar image-reference report to the report generator (100). The information provided to the report generator (100) can be referred to as guideline information, and guideline settings such as the number of reference reports and whether to provide similar images can be determined according to the report generator (100). Specifically, when a target image (10) for which an image report is to be generated is input, the reference searcher (130) can generate a feature representation vector of the target image (10) through an encoder. The reference searcher (130) can determine K reference image-report pairs similar to the target image by comparing the similarity between the vector of the target image (10) and the vectors of the reference images stored in the reference storage (120). Depending on the guideline settings, the K reference image-report pairs can be output as guideline information, or the K reports can be output as guideline information. The similarity can be calculated in various ways, including cosine similarity.

[0091] Since the K guidelines are the most similar data to the target image, they can serve as a reference in the image report generation process. That is, the report generator (100) can use the K reference reports as guidelines to generate an image report with a comprehensive conclusion similar to that written by an actual diagnostician, thereby reducing errors caused by hallucinations, which are pointed out as a problem in generative artificial intelligence models.

[0092] The report generator (100) can obtain analysis results for the medical image (10). The analysis results for the medical image (10) can be provided from a lesion detector (110), an additional information extractor (140), and a measuring device (150). The analysis results for the medical image (10) can include lesion information detected in the medical image (10), additional information composed of answers obtained by analyzing the medical image (10) or clinical information in response to a question, and quantitative information on an object of interest included in the medical image (10). The report generator (100) can obtain a reference report that has been prepared in advance for an image similar to the medical image (10). The reference report can be provided from a reference searcher (130).

[0093] The report generator (100) can generate an image report from the analysis results of a medical image (10). The report generator (100) can be an AI model (e.g., LLM, Large Language Model) implemented to generate an image report with a standardized description method and structure for each section, similar to image reports written by medical experts. The report generator (100) can generate an image report for a medical image (10) in a structured format based on the analysis results of the medical image (10) by referencing at least one reference report.

[0094] The format of the image report generated by the report generator (100) can be varied, and the freedom to adjust the amount of text included in the image report, the description style, the content, etc. can be provided through configuration. In addition, the formats such as a full version report including detailed analysis results and a short version report including concise analysis results for essential items can be specified through configuration, and the report generator (100) can generate an image report in the specified format. Meanwhile, the report generator (100) can pre-generate reports in multiple formats that can be generated based on the analysis results for the medical image (10) and first provide a report in the format selected by the user. Thereafter, when the user selects a new format, the report in the corresponding format can be quickly provided.

[0095] The report generator (100) may provide an interface through which a user can input settings for generating an image report. The user can create a user-defined format and personalize the image report generation method (text volume, description method, description content, etc.) through the interface provided via the user terminal (200). Furthermore, the user can select a desired format from among various image report formats provided through the interface provided via the user terminal (200). To this end, the report generator (100) provides various image report formats that can be generated based on the characteristics of each medical image (e.g., modality, body part captured, etc.), and can generate an image report in a format selected by the user from among the provided image report formats.

[0096] Referring to FIG. 3, the image report (20) may be structured, for example, into a header section (310), a finding section (320), and a clinical impression section (330). The header section (310) may include, for example, items of INDICATION, COMPARISON, and TECHNIQUE. The header section (310) may be generated to include quality and metadata of the medical image (10) (e.g., modality information, shooting information such as view position, etc.), comparison information with past images (COMPARISON), etc. The finding section (320) may be generated to include specific lesion information analyzed in the medical image (10), and detailed descriptions of various impressions. The clinical impression section (330) may describe a comprehensive conclusion, summary, differential diagnosis (DDx), recommended action, etc. for the medical image (10).

[0097] The report generator (100) can write text in the header section (310) using the quality and metadata (e.g., shooting information such as modality information, view position, etc.) of the medical image (10) included in the analysis result. The quality of the medical image (10) may be included in the answer obtained by the additional information extractor (140) analyzing the medical image (10), for example, when a question inquiring about the quality of the medical image (e.g., 'Is the image acquired with adequate quality for interpretation?') is given. The metadata of the medical image (10) may be included in the answer obtained by the additional information extractor (140) analyzing the metadata of the medical image (10), for example, when a question inquiring about the quality of the medical image (e.g., 'Is this a frontal view chest x-ray?') is given.

[0098] The report generator (100) can record in the findings section (320) a description generated using specific lesion information, various findings, and quantitative information on objects of interest (lesions, organs, medical devices) included in the analysis results. The findings section (320) can be generated to include findings segmented by each anatomical structure and other items present in the image.

[0099] The report generator (100) can infer comprehensive conclusions, summaries, differential diagnoses (DDx), recommended actions, etc. based on the analysis results for medical images (10) and record them in the clinical estimation section (330).

[0100] The report generator (100) generates a command to generate an image report using the analysis results and reference reports for the medical image (10), and applies the command to a language model (e.g., LLM) to obtain the image report. This prompting process can be implemented to sequentially apply predetermined questions to the language model, or to apply adaptively generated questions based on the analysis results to the language model.

[0101] For example, a command for generating an image report may include an instruction to generate an image report from the analysis results for the medical image (10) and the reference report, such as 'The input image contains the following findings: [finding 1 located at X1 with severity of Y1, finding 2 located at X2, with measurement of 3mm… ]. A similar x-ray sample has the following radiology report: {reference radiology report text goes here}. Please generate a radiology report.'

[0102] For another example, a command for generating an image report may include a system prompt and a user prompt as shown in Table 2. The report generator (100) may create a system prompt that implies to the language model that it is an expert radiologist and assigns the role of generating an image report by collating a large amount of given information. Based on the analysis results of the medical image (10), the report generator (100) may generate sentences that provide information on lesions detected in the medical image (10), additional information analyzed in the medical image (10) (e.g., quality, metadata, various findings), quantitative information on objects of interest included in the medical image (10), sentences that provide K reference reports, and sentences that command the collation of the given information to generate an image report in a specific format to create the user prompt.

[0103] System prompt: “You are an expert radiologist and will generate reports following ACR guidelines. Your replies should only contain the requested information and nothing else.” User prompt: “This is a vector of abnormal scores from an AI CAD engine: {FINDING_1: SCORE_1, FINDING_2: SCORE_2, … , FINDING_N: SCORE_N}.” Statement: “The imaging acquisition quality is adequate. This is a frontal view chest X-ray. The upper-right lung zone is normal, … . There is no medical device to provide measurements.” Statement: “The reference report”: “This is the first similar reference radiology report: {REFERENCE REPORT 1 from Retrieval Module}. This is the second similar reference radiology report: {REFERENCE REPORT 2 from Retrieval Module}.” Statement: “Please write the radiology report given the information.”

[0104] The language model receives a command sentence composed of sentences as shown in Table 2 and generates an image report. In particular, the language model can generate an image report of a medical image (10) by referring to an image report of an image similar to a medical image (10). At this time, the language model can display and output reference information including at least one of the content, feature, information, and source of the referenced reference report in the image report. In addition, when the language model generates an image report by referring to the patient's history information (e.g., EMR / EHR data, past image reports, etc.), the reference information including the patient's history information can be displayed and output in the image report. The report generator (100) can determine whether or not there are findings corresponding to predetermined finding labels in the final image report, and can generate a finding label set (30) composed of finding labels extracted from the final image report. The opinion label set (30) may be composed of only opinion labels extracted from the image report, or may be displayed by distinguishing between labels that exist in the image report and labels that do not exist among all predefined opinion labels.

[0105] The finding label set (30) is data indicating whether predefined clinical findings exist in the analysis results of medical images and can be used to evaluate the clinical validity of an image report. The finding label set (30) can be provided as a separate report distinct from the image report (20) or can be included in a designated section (e.g., finding label section) of the image report (20).

[0106] The report generator (100) can use a language model to extract opinion labels corresponding to the content of a video report, and thereby generate an opinion label set (30) composed of opinion labels extracted from among all predefined opinion labels. For example, the report generator (100) can simply extract labels corresponding to opinions explicitly present in each sentence of a video report, or can extract opinion labels based on the meaning of text included in the video report. If the existence of a specific opinion in the text included in the video report is uncertain, the report generator (100) can extract a label for the specific opinion and add information indicating that it is uncertain to the label.

[0107] The finding label included in the finding label set (30) indicates that the finding exists in the image report, and the finding label may further include additional information such as the location or uncertainty of the finding. Meanwhile, finding labels that can be extracted from the image report may be defined for each finding item. Finding items may be classified by certain criteria (e.g., anatomical section, medical device). The finding label set (30) may be structured by finding items to which the finding labels belong. That is, the finding label set (30) may be structured so that finding labels extracted from the image report can be identified among all labels defined for each finding item.

[0108] The report generator (100) can define in advance the observation items and observation labels to be extracted from the image report for each type of image, and can generate a consistent observation label set (30) using the observation items and observation labels specified for the type of medical image.

[0109] Referring to Fig. 4, the generation of a finding label set in the case where the medical image is a chest X-ray image (10A) is described. The finding label set for the chest X-ray image can be generated based on finding items classified into, for example, Medical Devices, Airways and Lungs, Pleural, Cardiomediastinum and hila, Diaphragm, Bony Structures, Soft Tissues, Upper Abdomen, etc., and finding labels that can be extracted from each finding item.

[0110] The report generator (100) can generate an image report (20A) using the analysis results of a chest X-ray image (10A), and extract finding labels corresponding to the text of the image report (20A) to generate a finding label set (30A). The text of the image report (20A) can be mapped to each finding item, and the finding item mapping table (21) is a table presented to explain the process of generating a finding label set (30A) from the image report (20A), and the finding item mapping table (21) does not necessarily have to be generated to generate a finding label set.

[0111] The report generator (100) can extract corresponding finding labels, for example, [Sternotomy Wire], [Surgical clips], [Cardiac prosthetic valve], from text about a medical device (e.g., The patient is status post median sternotomy, CABG, and mitral valve replacement).

[0112] The report generator (100) can extract corresponding finding labels, for example, [Interstitial opacity, uncertain], [Consolidation, uncertain], [Atelectasis], from text about the airway and lungs (e.g., Mild pulmonary edema is noted. Left basilar opacification likely reflects atelectasis). In this case, the report generator (100) can clearly determine the presence of atelectasis from the text, but it may be difficult to determine whether interstitial opacity or consolidation is present. In this case, additional information indicating uncertainty (e.g., uncertain) may be described in the finding label. The report generator (100) can numerically estimate the uncertainty of the opinion label and add the uncertainty score to the opinion label.

[0113] The report generator (100) can extract a corresponding finding label, for example, [Pleural effusion], from text about the pleura (e.g., Small bilateral pleural effusions are present. / Small amount of pleural effusion on both sides).

[0114] The report generator (100) can extract corresponding finding labels, for example, [Cardiomegaly], [Aortic atherosclerosis], from text about atrial fibrillation and hilum (e.g., The heart is mildly enlarged, CTR 0.61. Calcification of the aortic knob noted. / The heart is mildly enlarged and CTR is 0.61. Calcification of the aortic arch was observed).

[0115] The report generator (100) extracts predefined opinion labels, but if the sentences in the image report contain opinions that do not correspond to the opinion labels or are unknown, the information contained in the sentences can be recorded in the other (others) section.

[0116] Even if the image reports are automatically generated, the generative language model allows for the same findings to be expressed in various ways. For example, image reports may express cardiomegaly in various texts, as shown in Table 3. These sentences can be summarized by a single expression, the finding label [Cardiomegaly]. Therefore, users can quickly recognize key findings detected in medical images through the finding label set. Furthermore, when evaluating the clinical validity of the system (1), quantitative analyses, such as sensitivity and specificity, can be performed based on the consistently output finding label set without additional analysis of the high-degree-of-freedom image reports, and reliability and reproducibility can be evaluated.

[0117] Video report example observation labelHeart size is enlarged.Cardiac silhouette is widened.Enlarged cardiomediastinal contour is noted.Widened cardiac width suggestive of cardiomegaly.Cardiothoracic ratio is measured up to 0.65, representing cardiomegaly.cardiomegaly

[0118] Referring back to FIG. 2, the image report and finding label set generated by the report generator (100) can be stored in a designated device and provided to a user terminal. Here, the initial image report generated by the report generator (100) is provided to the user via the user terminal (200), and a final image report can be generated through user confirmation. In addition, the initial image report generated by the report generator (100) can be edited by the user, and a final image report can be generated through user confirmation of the modified image report. In particular, the final image report for the medical image (10) can be added to the catalog set of the reference repository (120). The final image report can be added to the catalog set at the user's request, or can be selectively added according to the current status of the catalog set (number of cases, feature distribution of cases, etc.). The report generator (100) can provide an interface that can personalize the format of the image report by adjusting the amount of text included in the image report, the description method, the description content, etc. through settings. Alternatively, the report generator (100) may provide an interface for selecting a full version report and a short version report, or may provide various report formats that can be provided, and may generate and provide an image report in a format selected by the user.

[0119] Image reports can be provided, for example, in the form of DICOM basic text SR. When a user selects a patient study case for medical images from the worklist provided through the viewer, the viewer can output an image report, which is a pre-populated report generated by the system (1). The user can review the image report and, through the provided interface, edit the structure or content of the image report to create a final image report.

[0120] The finding label set may be provided as a separate report distinct from the image report, or may be included in a designated section of the image report (e.g., finding label section). Alternatively, the finding label set (30) may be provided as included in the DICOM SC generated by the lesion detector (110).

[0121] Meanwhile, the report generator (100) can check the non-text analysis results output from the lesion detector (110), the additional information extractor (140), or the measuring device (150), and link the non-text analysis results (e.g., SC images) with the image report, thereby generating an image report including the non-text analysis results. The report generator (100) can link text related to various findings such as lesions, organs, and medical devices included in the image report with the non-text analysis results output from the artificial intelligence analysis model that provided the findings. The image report including the non-text analysis results can be displayed through the viewer of the user terminal (200), and the non-text analysis results corresponding to the text of the image report can be provided explanatory. Therefore, the image report including the non-text analysis results can be called an explainable report.

[0122] The initial image report generated by the report generator (100) can be modified, and the report modifier (160) is described as taking on this role. Here, the report modifier (160) is a component introduced to explain how the initial image report is modified, and can be implemented by being included in the report generator (100).

[0123] The report modifier (160) can provide an interface through which a user can edit a video report generated by the report generator (100). This allows the user to personalize the video report.

[0124] The report modifier (160) can generate a personalized report (20-1) by personalizing a video report based on preset user style information. The report modifier (160) can input a video report that can identify a user's preferred report style (writing style, sentence length, terminology used, etc.) and can post-convert the video report into a user-style report based on this. Meanwhile, by adding a video report created in the user's preferred report style to a catalog set before post-conversion, the report generator (100) can generate a video report by referencing the user's style when generating the video report.

[0125] The report modifier (160) may include a language model implemented to modify an image report based on additional clinical information to generate an image report (20-1). The clinical information may include, for example, text data recording the patient's symptoms, test results (e.g., blood test results, functional test results, etc.), the patient's age, gender, reason for testing, etc. The clinical information may include, for example, additional images other than the target image for which the image report is to be generated. The additional images may include past images captured with the same type of imaging device as the target image, images captured with a different type of imaging device, etc.

[0126] The report modifier (160) can add the results of additional analysis based on clinical information (e.g., comparison results with past images) to the image report, and can modify the sentences of the image report using clinical information (e.g., the patient's symptoms or test results). The clinical information added to the image report or the analysis results added based on clinical information can be recorded in a designated section (e.g., the header section) of the image report, and reference information can be displayed in the image report to indicate the use of the clinical information. Meanwhile, if the additional clinical information is the patient's past images, the report modifier (160) can link the comparison images of the past images and the target images to the image report. The comparison images can be provided together with the image report through a viewer. For example, the past images can be provided in the form of thumbnails in the DICOM SC, which is the analysis result for the target image, or the target image and the past images can be provided together in a split form.

[0127] The pre-verifier (170) can verify whether the medical image (10) is suitable for analysis by an artificial intelligence model before inputting the medical image (10) into a lesion detector (110), etc. The pre-verifier (170) can verify suitability before inputting the medical image (10) by using the lesion detector (110), the additional information extractor (140), and the measuring device (150), which are artificial intelligence models that perform analysis on the medical image (10). Alternatively, the pre-verifier (170) can be implemented in the additional information extractor (140) and implemented as VQA to check the image quality.

[0128] The pre-verifier (170) can verify whether the quality of the medical image (10) is suitable for analysis by an artificial intelligence model (blur, motion artifact, improper patient positioning, etc.). The pre-verifier (170) can verify whether information about the modality through which the medical image (10) was acquired and / or information included in the metadata of the medical image (10) meet predetermined criteria. The pre-verifier (170) can verify whether the medical image (10) has a distribution similar to the learning data used to train the lesion detector (110), the additional information extractor (140), and the measuring device (150).

[0129] If the pre-verifier (170) determines that the medical image is unsuitable, the medical image may not be input into the artificial intelligence model. The pre-verifier (170) may output a notification indicating that the medical image is unsuitable for image interpretation using the artificial intelligence model. The notification may include a reason why the medical image is unsuitable for image interpretation using the artificial intelligence model. In addition, if the medical image is determined to be unsuitable for image interpretation using the artificial intelligence model, the pre-verifier (170) may generate a signal requesting acquisition of a new medical image of the patient, and a new medical image may be acquired based on the signal.

[0130] Alternatively, even if the medical image is determined to be unsuitable by the pre-verifier (170), the medical image may be input into the artificial intelligence model to obtain the analysis result, but the report generator (100) may generate an image report on the premise that the reliability of the analysis result of the artificial intelligence model for the medical image is low. To this end, the pre-verifier (170) may provide the verification result for the medical image (10) to the report generator (100). The report generator (100) may describe the reason why the medical image is unsuitable for interpretation in the image report. The report generator (100) may notify that the reliability of the image report created based on the unsuitable medical image is low, or may add a guidance phrase to the image report indicating that the medical image has limitations in generating the image report.

[0131] FIG. 5 is a drawing illustrating a method for generating an image report according to one embodiment.

[0132] Referring to FIG. 5, the lesion detector (110), the additional information extractor (140), and the measuring device (150) are artificial intelligence analysis models that output analysis results for a medical image (10B), and can be designed to utilize the analysis results of other models in various ways. For example, the lesion detector (110) can be implemented to serve as a basic analysis model, and the analysis results of the lesion detector (110) can be utilized by the additional information extractor (140) and the measuring device (150). Alternatively, the lesion detector (110), the additional information extractor (140), and the measuring device (150) can be designed as a sequential pipeline to sequentially utilize the analysis results of the previous model, or can be implemented to exchange analysis results with other models.

[0133] The lesion detector (110) can output a list of lesions detected in the medical image (10B) [lesion 1, lesion 2, …] and an auxiliary image (e.g., SC image) (11B) representing the detected lesion information as a heat map, etc., as lesion information. The lesion information, which is the analysis result of the lesion detector (110), can be provided to assist in the analysis of other artificial intelligence analysis models.

[0134] For example, a list of lesions [lesion 1, lesion 2, …] detected in a medical image (10B) may be provided to an additional information extractor (140) and a measuring device (150). At this time, an image (11B) including lesion information may also be provided to the additional information extractor (140) or the measuring device (150).

[0135] The additional information extractor (140) can recognize lesions detected in the medical image (10B) based on the lesion list, select questions to obtain additional information about the detected lesion from the question set, or obtain clinical information related to the detected lesion. For example, if a 'nodule' is detected in the medical image (10B), the additional information extractor (140) can select questions such as 'Is the nodule located in the right upper lung area?' from the question set, and not select questions about undetected lesions. Through this, the additional information extractor (140) can obtain additional information, i.e., location information and / or detailed information, about the lesion detected in the medical image (10B) through lesion-related questions, and output additional information including the same.

[0136] Alternatively, the additional information extractor (140) may obtain additional information about the medical image (10B) by selecting a question about whether a lesion other than the specific lesion detected in the medical image by the lesion detector (110) exists in the medical image (10B).

[0137] The measuring device (150) can be used to recognize lesions detected in a medical image (10B) based on a list of lesions and measure quantitative information about the lesions. For example, if a 'nodule' is detected in the medical image (10B), the measuring device (150) can measure the measurement item (e.g., size) of the nodule, which is an object of interest in the medical image (10B), and generate quantitative information <nodule: 3.0, mm>.

[0138] Meanwhile, the reference searcher (130) may also receive the analysis results of at least one of the lesion detector (110), the additional information extractor (140), and the measuring device (150), and utilize the analysis results of the artificial intelligence analysis model to detect similar images of the medical image (10B). Alternatively, the reference searcher (130) may utilize the analysis results of the artificial intelligence analysis model to determine the most similar image report among the image reports of the similar images. For example, the reference searcher (130) may compare the similarity between the lesions detected in the medical image (10B) and the lesions described in the Findings section, and determine at least one image report with a high similarity as the reference report.

[0139] The report generator (100) can receive analysis results from at least one of a lesion detector (110), an additional information extractor (140), and a measuring device (150), and generate an image report (20B) having a standardized description method and structure for each section. At this time, the report generator (100) can generate an image report (20B) for the medical image (10B) in a structured format based on the analysis results for the medical image (10B) by referring to at least one reference report.

[0140] FIG. 6 is a diagram illustrating a visual question answering (VQA) of an additional information extractor according to one embodiment.

[0141] Referring to FIG. 6, the additional information extractor (140) can analyze additional information such as location information and detailed information about detected lesions, information about various findings other than detected lesions, quality and metadata of medical images, etc. by producing answers to questions from a given medical image through visual question answering (VQA).

[0142] To train the additional information extractor (140), question-and-answer pairs can be extracted from text data containing image reports. Medical images can be embedded as input tokens of the additional information extractor (140) through image encoding. Questions generated from the text data can be embedded as input tokens of the additional information extractor (140) through a text tokenizer. The additional information extractor (140) can be trained to generate correct answers when given medical images and questions through visual guidance tuning.

[0143] Figure 7 is an example of a method for providing an image report according to one embodiment.

[0144] Referring to FIG. 7, the report generator (100) generates an image report based on analysis results for medical images, and can provide an image report linked to analysis results in a non-text format.

[0145] The user terminal (200) displays an image report (20C) through an interface (400) provided by the viewer, and can provide text (e.g., mass opacity) of lesions, etc. described in the image report (20C) and an auxiliary image (e.g., SC image) (11C) that visually displays the lesion in question by linking them.

[0146] There are various ways to present the image report and the analysis results visually displayed together. For example, when a user clicks or hovers over a lesion name in the image report (20C), an SC image (11C) containing the lesion information can be displayed along with the image report. This allows the user to visually identify the lesion in the upper left lung region.

[0147] Figure 8 is a flowchart of a method for generating an image report according to one embodiment.

[0148] Referring to Fig. 8, the system (1) generates an analysis result for a target medical image using an artificial intelligence analysis model (S110). The analysis result for the target medical image may include pre-specified lesion information, and the lesion information may include a list of detected lesions, an auxiliary image (e.g., SC image) that represents the detected lesion information in the form of a heat map, etc. The analysis result for the target medical image may include additional information extracted from the target medical image, and the additional information may include detailed information on the detected lesion, information on various findings other than the detected lesion, information on the quality and metadata of the medical image, etc. The additional information may be obtained through a visual question-and-answer (VQA) process that generates answers to questions from a given medical image. In addition, the analysis result for the target medical image may include quantitative information (e.g., size, volume, ratio, number, etc.) on objects of interest such as lesions.

[0149] The system (1) extracts at least one similar image similar to a target medical image from a catalog set consisting of medical image-image report pairs, and determines an image report corresponding to the similar image as a reference report (S120).

[0150] System (1) uses a reference report as a guideline to generate an image report from the analysis results of a target medical image (S130). System (1) can generate an image report with a standardized description method and structure for each section. System (1) can link the analysis results of the target medical image in a non-text format (e.g., an SC image representing detected lesion information as a heat map, etc.) to a text image report. Through this, when a user clicks or hovers a lesion name in the image report with a mouse, an SC image including the lesion information can be displayed together with the image report.

[0151] The system (1) determines whether findings corresponding to predefined finding labels are present in an image report and generates a finding label set composed of finding labels extracted from the image report (S140). The finding label set is data indicating whether predefined clinical findings exist in the analysis results of a medical image and can be used to evaluate the clinical validity of the image report. The finding label set can be provided as a separate report distinct from the image report or can be included in a designated section of the image report (e.g., finding label section).

[0152] Figure 9 is a flowchart of a method for modifying an image report according to one embodiment.

[0153] Referring to Fig. 9, the system (1) generates an initial image report using the analysis results for the medical image (S210).

[0154] The system (1) obtains clinical information through user input or linkage with a database, and modifies the initial imaging report using the clinical information or the analysis results of the clinical information (S220). The clinical information may include, for example, text data recording the patient's symptoms, test results (e.g., blood test results, functional test results, etc.), the patient's age, gender, reason for testing, etc. The clinical information may include, for example, additional images other than the target image for which the imaging report is to be generated. The additional images may include past images captured with the same type of imaging device as the target image, images captured with a different type of imaging device, etc. The system (1) may add the results of additional analysis based on the clinical information (e.g., comparison results with past images) to the imaging report, or modify the sentences of the imaging report using clinical information (e.g., patient's symptoms or test results). The medical institution's database may include a picture archiving and communication system (PACS), an electronic medical record (EMR), an electronic health record (EHR), etc.

[0155] The system (1) provides a modified image report to the user terminal (200) (S230).

[0156] The system (1) stores the final image report, which has undergone user editing or user confirmation, in a designated location (S240). The final image report can be added to a catalog set consisting of medical image-image report pairs, based on user selection or the system (1)'s decision.

[0157] Figure 10 is a flowchart of a catalog set management method according to one embodiment.

[0158] Referring to Fig. 10, the system (1) generates a new image report using the analysis results for the medical image (S310).

[0159] The system (1) determines whether to add a new image report to a catalog set referenced when generating an image report (S320). According to one embodiment, the system (1) can determine whether it is useful to add a new image report to the catalog set. The system (1) can determine the usefulness of the new image report by measuring the information gain obtained by adding the new image report to the catalog set. According to another embodiment, the system (1) can embed image reports into feature representation vectors and evaluate the similarity between the image reports stored in the catalog set and the new image report. Then, the system (1) can add a new image report determined to have low similarity with the reference reports stored in the catalog set to the catalog set based on the similarity distance between the new image report and the reference report, etc. According to another embodiment, the system (1) can determine whether to add a new image report to the catalog set based on catalog set features including the number of cases included in the catalog set and the feature distribution of the cases included in the catalog set (e.g., distribution of lesions and other findings).

[0160] Based on the judgment result, the system (1) adds a new image report-medical image pair to the catalog set (S330). If the catalog set consists of M cases, the system (1) may discard one case from among the M cases included in the catalog set and add a new image report. The system (1) may discard a case similar to another case from among the M cases included in the catalog set. The system (1) may discard a case with the lowest information gain from among the M cases included in the catalog set.

[0161] The modules constituting the system (1) of the present disclosure may include one or more processors, a memory for loading a computer program executed by the processor, a storage device for storing the computer program and various data, a communication interface, and, in addition, various other components may be further included. The processor may be various types of processors that process instructions included in the computer program, and may be configured to include, for example, at least one of a CPU (Central Processing Unit), an MPU (Micro Processor Unit), an MCU (Micro Controller Unit), a GPU (Graphics Processing Unit), or any other type of processor well known in the technical field of the present disclosure. The memory stores various data, commands, and / or information. The memory may be implemented to store commands so that the commands described to execute the operations of the present disclosure are processed by the processor. The memory may be, for example, a ROM (read only memory), a RAM (random access memory), etc. The storage device may non-temporarily store a computer program and various data. The storage device may be configured to include non-volatile memory such as Read Only Memory (ROM), Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), flash memory, a hard disk, a removable disk, or any form of computer-readable recording medium well known in the art to which the present disclosure pertains. The communication interface may be a wired / wireless communication module that supports wired / wireless communication.A computer program comprises instructions that are executed by a processor and are stored in a non-transitory computer readable storage medium, the instructions causing the processor to perform the operations of the present disclosure.

[0162] The embodiments of the present disclosure described above are not implemented only through devices and methods, but may also be implemented through a program that realizes a function corresponding to the configuration of the embodiments of the present disclosure or a recording medium on which the program is recorded.

[0163] Although the embodiments of the present disclosure have been described in detail above, the scope of the present disclosure is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concepts of the present disclosure defined in the following claims also fall within the scope of the present disclosure.

Claims

1. Memory, and A processor that executes instructions stored in the above memory, The above processor Obtain analysis results for target medical images using an artificial intelligence analysis model, In a catalog set consisting of medical image-image report pairs, at least one similar image similar to the target medical image is extracted, and at least one image report corresponding to the at least one similar image is determined as a reference report. An image report generation system implemented to generate an image report from analysis results for the target medical image by using the above reference report as a guideline.

2. In paragraph 1, The above processor It is implemented to determine whether there are findings corresponding to predetermined finding labels in the above image report, and to generate a finding label set composed of finding labels extracted from the above image report. A system for generating an image report, wherein the above set of opinion labels is provided as a separate report distinct from the image report or is included in a designated section of the image report.

3. In paragraph 1, The above processor An image report generation system implemented to obtain clinical information through user input or linkage with a medical institution's database, and to modify the image report using the clinical information or the analysis results of the clinical information.

4. In paragraph 1, The above processor A video report generation system implemented to save a final video report that has undergone user editing or user confirmation for the above video report to a designated location.

5. In paragraph 1, The above processor An image report generation system implemented to determine whether to add the image report to the catalog set, and to add the image report-the target medical image pair to the catalog set based on the determination result.

6. In paragraph 1, The above analysis results are At least one of lesion information detected in the target medical image, additional information extracted from the target medical image, or quantitative information on an object of interest present in the target medical image, The above additional information is An image report generation system comprising at least one of detailed information about the detected lesion, information about findings detected other than the detected lesion, quality information about the target medical image, or information about metadata of the target medical image.

7. In paragraph 6, The above processor An image report generation system, which is implemented to select a set of questions related to the target medical image or the analysis results of the target medical image from a question bank consisting of questions, and to extract answers to each question included in the question set to generate the additional information.

8. In paragraph 1, The above processor An image report generation system implemented to link non-text analysis results for the above target medical image to the image report.

9. As an operating method of the video report generation system, A step of obtaining analysis results for target medical images using an artificial intelligence analysis model; A step of extracting at least one similar image similar to the target medical image from a catalog set consisting of medical image-image report pairs, and determining at least one image report corresponding to the at least one similar image as a reference report, and A step of generating an image report from the analysis results for the target medical image using the above reference report as a guideline. A method of operation including:

10. In paragraph 9, A step of determining whether or not there are findings corresponding to predetermined finding labels in the above video report, and A step of generating a set of observation labels consisting of observation labels extracted from the above image report. A method of operation, further comprising:

11. In paragraph 9, A step of obtaining clinical information through user input or linking with a medical institution's database, and A step of modifying the image report using the above clinical information or the analysis results of the above clinical information. A method of operation, further comprising:

12. In paragraph 9, A step of saving the final video report that has undergone user editing or user confirmation for the above video report to a designated location. A method of operation, further comprising:

13. In paragraph 9, a step of determining whether to add the above video report to the above catalog set, and A step of adding the image report-target medical image pair to the catalog set based on the judgment result. A method of operation, further comprising:

14. In paragraph 9, The above analysis results are At least one of lesion information detected in the target medical image, additional information extracted from the target medical image, or quantitative information on an object of interest present in the target medical image, The above additional information is An operating method comprising at least one of detailed information about the detected lesion, information about findings detected other than the detected lesion, quality information about the target medical image, or information about metadata of the target medical image.

15. In paragraph 14, The steps for obtaining the above analysis results are An operating method comprising: selecting a set of questions related to the target medical image or the analysis results of the target medical image from a question bank consisting of questions, and extracting answers to each question included in the set of questions to generate the additional information.

Images