Generative al system for enhanced radiology reports with colorized MRI and illustrative overlays

Abstract

Embodiments disclosed herein include software processes executed by a computer for ingesting and analyzing MRI image data and generating MRI-related reports. The report generation software executed by a computer generates MRI reports using medical data and MRI imagery data. The computer creates a text summary. Additionally, the computer generates the MRI report to include medical illustrations that overlay and merge with the MRI imagery data to visually explain the MRI results. Generative machine-learning models integrate colorized MRI images with the illustrations. The computer selects key MRI slices, colorizes them, matches them with relevant illustrations, and recursively refines the combined images to improve clarity and informational content. This process enhances the interpretability of radiology reports for both medical professionals and patients.

Description

EXPTR01-PCT / 140068-0110 PATENTGENERATIVE Al SYSTEM FOR ENHANCED RADIOLOGY REPORTS WITH COLORIZED MRI AND ILLUSTRATIVE OVERLAYSCROSS REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 695,685, filed September 17, 2024, which is incorporated by reference in its entirety.TECHNICAL FIELD

[0002] This application generally relates to medical imaging and radiology, specifically to a system and method for improving radiology reports by adding key images, colorized images, illustration of images based upon the language in the radiologists’ reports, as well as combining colorized MRI images with illustrative overlays using generative artificial intelligence (Al) and machine-learning architecture operations.BACKGROUND

[0003] Radiology reports are critical for diagnosing and treating various medical conditions. Traditional reports, however, can be challenging for non-specialists to interpret. This invention addresses this problem by integrating colorized MRI images and illustrative overlays to create more intuitive and informative reports.SUMMARY

[0004] Disclosed herein are systems and methods capable of addressing the technological shortcomings and may also provide any number of additional or alternative benefits and advantages. Embodiments include systems and methods for ingesting and analyzing MRI image data and generating MRI-related reports. The embodiments implement machine-learning architecture trained to generate the MRI reports. Beneficially, the report generation software executed by a computing device generates MRI reports using medical data and MRI imagery data ingested from medical imaging devices and other inputted data from medical provider devices or medical resource databases. The computer includes the machinelearning architecture trained to generate the MRI reports based on the various types of inputs and using terms or phrasing that are easier to understand for laypeople without a medical background. Additionally, the computer generates the MRI report to include illustrations that visually explain the findings of the MRI results, making the complex details of the MRI report much more accessible and easier to understand for a layperson to review. A generative Al may enhance MRI images and integrate the MRI images with dynamically selected, generated, or otherwise curated medical illustrations. The operations include selecting key MRI slices of thePage l of 28EXPTR01-PCT / 140068-0110 PATENTMRI imagery, colorizing the MRI slices, matching the MRI images with appropriate illustrations, and iteratively refining the combined images to improve clarity and informational content.

[0005] Embodiments may include computing system(s) and computer-implemented method(s) for enhancing radiology reports having medical imaging data. Embodiments may include an illustration database and a computer having at least one processor. The illustration database includes non-transitory machine-readable storage medium configured to store a plurality of illustrations images having corresponding attributes related to a portion of human anatomy or a medical condition indicator. The computer having at least one processor may execute operations of obtain MRI imagery data containing one or more MRI images generated from an MRI imaging device. The computer may identify in the illustration database, a set of one or more illustrations based upon a set of one or more MRI slices of the one or more MRI images. The set of one or more illustrations being identified according to text of an input source report having the MRI imagery data. For each illustration, the computer identifies an illustration based upon at least one of an anatomical descriptor or a condition indicator in the input source report and associated with the illustration. For each MRI image, the computer may generate an output image by combining an MRI image with an illustration as identified in the illustration database using the one or more MRI slices of the MRI image. The computer may generate an output report having one or more output images.

[0006] The computer may generate a report summary based upon the text of the input source report. The computer generates the output report having the one or more output images and report summary.

[0007] When selecting the one or more MRI slices, for each MRI slice of the one or more MRI slices, the computer may generate a colorized instance of the MRI slice.

[0008] When generating the colorized instance, the computer may identify in the MRI slice a portion of an anatomy having a medical condition according to the condition indicator of the input source report. The computer may update a coloring of the portion of the anatomy having the medical condition.

[0009] When generating the colorized instance, the computer may receive a user input indicating a coloring a portion of an anatomy in the MRI slice having a medical condition.EXPTR01-PCT / 140068-0110 PATENT

[0010] When generating the output image, the computer may update the MRI imagery data containing the one or more MRI images based upon merging the MRI image and an overlay of the medical illustration.

[0011] The computer may generate a refined image according to a graph representation of one or more image features, including an anatomical accuracy value and an illustration clarity value.

[0012] When identifying the set of one or more illustrations, the computer may select from the illustration database at least one of an axial image or a sagittal image based upon herniation data of a condition indicator.

[0013] The computer may receive the input source report having the MRI imagery data in a healthcare message data structure having a standard format. The computer may extract the input source report and the MRI imagery data from the healthcare message data structure.

[0014] The computer may select the one or more MRI slices from the MRI imagery data according to one or more input from a user device. Each MRI slice may include an overlay corresponding to one or more condition indicators of the input source report.

[0015] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed.BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present disclosure can be better understood by referring to the following figures. The components in the figures are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the disclosure. In the figures, reference numerals designate corresponding parts throughout the different views.

[0017] FIG. 1 shows components of a system for generating MRI reports using machine-learning architectures, according to an example embodiment.

[0018] FIG. 2 shows operations of a process for receiving medical image data and medical records data for machine-generated output reports using machine-learning architectures of a medical reporting system, according to an example embodiment.EXPTR01-PCT / 140068-0110 PATENT

[0019] FIGS. 3A-3C show operations of a process for executing machine-learning architectures of a medical reporting system trained for generating machine-generated output reports, according to an example embodiment.

[0020] FIG. 4 shows operations of a process for outputting medical image data and medical records data for machine-generated output reports using machine-learning architectures of a medical reporting system, according to an example embodiment.

[0021] FIG. 5 shows operations of a process for a computer-implemented method for enhancing radiology reports having medical imaging data, according to an example embodiment.

[0022] FIG. 6 to FIG. 11 depict graphical user interfaces displaying portions of a source report and an output enhanced report generated by one or more machine-learning models of a machine-learning architecture, according to various example embodiments.

[0023] FIG. 7 depicts a sagittal view of a portion of the anatomy (e.g., T2 L4, L5), according to an example embodiment.

[0024] FIG. 8 depicts a sagittal view of a portion of the anatomy (e.g., T2 L5, SI), according to an example embodiment.

[0025] FIGS. 9A-9B depict the graphical user interfaces having machine-generated overlays for an enhanced version of the MRI images, according to example embodiments.

[0026] FIG. 10 depicts the graphical user interface displaying a machine-generated illustration for a sagittal view (side view) of the spine, according to an example embodiment.

[0027] FIG. 11 depicts the graphical user interface displaying a machine-generated illustration for an axial view (top-down view) of the spine at level L4-L5, according to an example embodiment.DETAILED DESCRIPTION

[0028] Reference will now be made to the illustrative embodiments illustrated in the drawings, and specific language will be used here to describe the same. It will nevertheless be understood that no limitation of the scope of the invention is thereby intended. Alterations and further modifications of the inventive features illustrated here, and additional applications of the principles of the inventions as illustrated here, which would occur to a person skilled in theEXPTR01-PCT / 140068-0110 PATENT relevant art and having possession of this disclosure, are to be considered within the scope of the invention.

[0029] Embodiments include systems and methods for ingesting and analyzing MRI image data and generating MRI-related reports. The embodiments implement machinelearning architecture trained to generate the MRI reports. Beneficially, the report generation software executed by a computing device generates MRI reports using medical data and MRI imagery data ingested from medical imaging devices and other inputted data from medical provider devices or medical resource databases. The computer includes the machine-learning architecture trained to generate the MRI reports based on the various types of inputs and using terms or phrasing that are easier to understand for laypeople without a medical background. The computer may take text from an MRI report, which is often complex and full of medical jargon, and create a summary that highlights the important information in a simple and concise manner. Additionally, the computer generates the MRI report to include illustrations that visually explain the findings of the MRI results, making the complex details of the MRI report much more accessible and easier to understand for a layperson to review.

[0030] In some embodiments, a computer executes machine-learning architecture for enhancing radiology reports using generative Al machine-learning models of the machinelearning architecture trained and programmed to integrate colorized MRI images with curated medical illustrations. The computer selects key MRI slices, colorizes them, matches them with relevant illustrations, and recursively refines the combined images to improve clarity and informational content. This process enhances the interpretability of radiology reports for both medical professionals and patients.

[0031] FIG. 1 shows components of a system 100 for generating MRI reports using machine-learning architectures. The system 100 includes a reporting system 102, client devices 103, medical imaging devices (e.g., MRI device 105), and a Picture Archiving and Communication System (PACS) 120. The components of the system 100, including the reporting system 102 and the PACS 120, may communicate with one another via one or more networks 107. The reporting system 102 includes a cloud computing system 104 having hardware and software for hosting virtualized computing devices, including a virtual machine (VM) 106 hosted within the cloud computing system 104. The reporting system 102 may further include a Virtual Private Network (VPN) containing hardware and software components of the reporting system 102 and a VPN gateway 108 for remotely or securelyEXPTR01-PCT / 140068-0110 PATENT accessing the components of the reporting system 102 via the one or more networks 107. The reporting system 102 further includes hardware and software components for executing an MRI report generation software (or report generator 110), which includes software programming of one or more machine-learning architectures. The reporting system 102 further includes a reporting system database 112 for storing various types of information related to generating the MRI reports, such as medical or image data records, operational logs, and medical illustrations, among other types of data. Embodiments may comprise additional or alternative components or omit certain components from what is shown in FIG. 1, yet still fall within the scope of this disclosure.

[0032] The one or more networks 107 may include various hardware and software components of one or more public or private networks for interconnecting the various components of the system 100. Non-limiting examples of such networks may include Local Area Network (LAN), Wireless Local Area Network (WLAN), Metropolitan Area Network (MAN), Wide Area Network (WAN), and the Internet. The communication over the network 107 may be performed in accordance with various communication protocols, such as Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), and IEEE communication protocols. The VPN gateway 108 includes software programming for securely communicating with computing devices and software components that are internal or external to the reporting system 102 infrastructure. The operations of the VPN gateway 108 include, for example, encrypting and decrypting data packets sent or received via a VPN of the one or more networks 107.

[0033] The PACS 120 includes hardware and software components for storing various types of data. The PACS 120 generally includes a medical imaging technology for healthcare organizations to securely store, retrieve, distribute, and present medical images electronically, which may include storing medical images (e.g., MRI images, x-ray images) in a digital storage system. The PACS 120 includes an authorized, external computing system infrastructure for capturing, storing, and analyzing certain types of image data for the reporting system 102. For instance, the PACS 120 includes hardware and software components for communicating with the reporting system 102 and sending image data and reports to the reporting system 102 via the one or more networks 107, which may include a VPN and the VPN gateway 108.

[0034] In some embodiments, the reporting system 102 includes a cloud computing system 104 that includes various hardware and software components, such as hypervisors orEXPTR01-PCT / 140068-0110 PATENT virtualized computing resources, for hosting and executing various software applications or computing services of the reporting system 102. The cloud computing system 104 may include, for example, virtualized or “bare metal” instances of servers, routers, firewalls, databases, gateway devices, or other types of computing resources. In the example system 100, the reporting system 102 includes a VM 106 as computing device that executes certain operations for managing or handling the operations and interactions of the reporting system 102, though the operations and features of the VM 106 may be performed any form of computing device.

[0035] The VM 106 within the cloud computing system 104 (or other computing device) is used to execute various operations for interacting with or otherwise operating the report generator 110 and other functionality of the reporting system 102. The VM 106 of the reporting system 102 executes, for example, software programming for an interface engine (e.g., Mirth®) that handles receiving reports in a healthcare data messaging format, such as Health Level 7 (HL7) and image data from the MRI device 105 or the client device 103 of a care provider. The interface engine receives these HL7 messages containing various types of MRI image data or medical data and processes, such healthcare data to prepare the MRI reports using the machine-learning architecture of the report generator 110.

[0036] The interface engine includes software programming executed by the VM 106 or other computing device, such as the open-source Mirth® software programming. The interface engine is programmed or otherwise designed for healthcare applications, enabling the integration and exchange of clinical data between various components of the reporting system 102 (e.g., client devices 103, MRI device 105, VM 106, PACS 120). The interface engine of the VM 106 receives the incoming reports as HL7 messages from the PACS 120 or other data source, processes them, and creates a final report. The VM 106 then sends or returns the final report or other outputs back to the PACS 120 or other destination (e.g., VM 106) via the one or more networks 107.

[0037] In some embodiments, the VM 106 includes various logical communications and processing channels or pipelines for sending and receiving certain types of data or instructions via the one or more networks 107. The VM 106 may receive the HL7 messages and return an acknowledgement message, to and from the PACS 120 or other data source using a source channel, where the HL7 messages include input reports. The VM 106 receives the input reports in a text format according to the HL7 standard, where the VM 106 may receiveEXPTR01-PCT / 140068-0110 PATENT image data (e.g., MRI imagery), among other types of data, in conjunction with the input message.

[0038] In a processing channel, the VM 106 executes operations that, for example, extract the text (or other types of information) from report text of the input report and generate a summary of the input report. The VM 106 may execute one or more machine-learning architectures when performing the operations of the data processing channel, such as executing a neural network architecture for extracting feature vectors in the report text or a large language model (LLM) for generating the text summary for the input report. In some implementations, the VM 106 outputs the device-generated summary and one or more medical illustrations as appended to the original input report. The VM 106 or other component of the reporting system 102 may execute or otherwise utilize various custom Java packages to fulfill the tasks described herein with respect to the processing channel.

[0039] In an output channel (sometimes referred to as a return channel), the programming of the interface engine executed by the VM 106 instructs the VM 106 to send the updated output report to the PACS 120 or other destination device (e.g., client device 103). The interface engine instructs the VM 106 to send the updated report back to the PACS 120 or other device and receives an acknowledgment from the PACS 120 or other device that confirms successful transmission.

[0040] The report generator 110 includes software programming of one or more machine-learning architectures for generating reports using input medical data, image data, and illustrations data. The medical data and image data may be received from, for example, the client devices 103, MRI device 105, or PACS 120. The report generator 110 may query the reporting system database 112 for particular illustrations relevant to a patient.

[0041] To add illustrations and summarize the MRI reports, the report generator 110 executes or otherwise accesses one or more generative machine-learning models (sometimes referred to as “generative Al” models), such as a GPT model (e.g., GPT-4). The generative Al model processes the report data and generates clear and concise summaries that are easier to understand for a layman.

[0042] The report generator 110 or other component of the reporting system 102 may receive an MRI report from the PACS 120 as an HL7 message. The software of the report generator 110 extracts the various types of report data from the report in the HL7 message. TheEXPTR01-PCT / 140068-0110 PATENT report generator 110 converts the extracted report data to a text file having a word processing file format (e.g., DOCX, PDF, TXT, RTF). The report generator 110 may extract the text for processing. In some cases, for example, the report generator 110 extracts the text at or between certain bookmarks (e.g., at Sectionl; between StartSectionl and EndSectionl) indicating portions of the text file for further processing.

[0043] When the report generator 110 has the text from the original HL7 message containing the MRI report, the trained machine-learning models of the machine-learning architecture of the report generator 110 may generate the text of the report summary for the output reports. The report generator 110 may further select and extract a set one or more selected medical illustrations from the reporting system database 112 in accordance with one or more anatomical or anatomy indicators for portions of human anatomy (e.g., a portion of the spine indicated in the MRI report or output report) and one or more conditions indicators (e.g., location of herniation, severity of herniation). For each particular illustration selected from the reporting system database 112, the report generator 110 may generate one or more visual overlay elements as a representation of the medical conditions indicated by the one or more one or more condition indicators of the MRI report or output report. The report generator 110 updates and outputs the particular illustration having the visual overlay in the output report, in addition to the text of the output report.

[0044] In operation, the report generator 110 selects appropriate illustrations for both Sagittal and Axial views based upon the inputted image data. For this, the report generator 110 executes various machine-learning architectures to extract, identify, or otherwise determine medical information or condition indicators using the medical information of the input report, such as a herniation level, size, and curvature type information, from the report text. The report generator 110 may later use this information as parameters to select appropriate, relevant illustration images from the reporting system database 112. In some cases, special prompts are crafted to make sure all the required information is extracted and / or avoid unwanted information, such as bulges.

[0045] The report generator 110 receives or generates an Al prompt for a sagittal view. For instance, herniation condition information for sagittal view, such as spine region, curvature type, herniated levels, and herniation sizes, may be obtained by sending the report text to the machine-learning model with one or more Al prompts to an LLM or GPT or similar machine-EXPTR01-PCT / 140068-0110 PATENT learning model of the machine-learning architecture of the report generator 110. An example Al prompt may include:First review and understand the following radiology report. Then extract information from the given report and return the response in the following JSON format {"spine region": "L-Spine or T-Spine or C-Spine ", "curvature type ": "Normal or Straight or Reverse ", herniated levels: "List of herniated levels as comma separated string", herniation sizes: "List of herniation sizes as comma separated string corresponding to the same herniated level. " }}. IGNORE bulge, if no herniation is found then return empty strings.

[0046] The report generator 110 receives or generates an Al prompt for an axial view. For instance, herniation condition information for axial view, such as herniation direction, spine region, and herniation sizes, may be obtained by sending the report text to the machinelearning model with one or more Al prompts to the LLM or GPT or similar machine-learning model of the machine-learning architecture of the report generator 110. An example Al prompt may include:First review and understand the following radiology report. Then extract information about only one single biggest herniation from the report and return the response in the following JSON format {"spine region": "L-Spine or T-Spine or C-Spine", herniated level: "Herniated level", herniation direction: "List of herniation direction for herniated level as comma separated string. Possible herniation directions are 'Central', 'Left Central', 'Left Far Lateral', 'Left Neural Foraminal', 'Left Paracentral', 'Right Central', 'Right Far Lateral', 'Right Neural Foraminal', 'Right Paracentral'", "herniation size": "Size of the biggest herniation." }}. IGNORE bulge, if no herniation is found then return empty strings.

[0047] The report generator 110 may query the reporting system database 112 to identify and select a set of one or more illustration images. Once the report generator 110 has the information about herniation level, herniation direction, herniation sizes, curvature type, and spine region, the report generator 110 may use this information to identify and select the appropriate illustrations.

[0048] Optionally, before querying the reporting system database 112 for the illustration data, the report generator 110 or other component of the operation 202 confirms whether the report is about C-Spine or L-Spine. This is because the illustrations are added for these two types of reports. For this purpose, the report generator 110 checks a “spine region”EXPTR01-PCT / 140068-0110 PATENT field value extracted for Sagittal and Axial views. If the value of the spine region is one of C- Spine or L-Spine, then the report generator 110 may perform certain operations.

[0049] The report generator 110 queries the reporting system database 112 using the extracted data to find the appropriate illustration based on values of Herniation Level / Direction, Sizes, and curvature type. If no matching illustration is found in the reporting system database 112, a default illustration is used. Otherwise, the report generator 110 appends the corresponding illustration the source report. In some implementations, each illustration image is placed on an VM 106, whereas the reporting system database 112 contains data indicating a full path of each image.

[0050] The extracted report text is sent to a generative machine-learning model of the report generator 110 for summarization. The report generator 110 may perform certain operations for generating a machine-generated summary text.

[0051] The report generator 110 may receive or generate an Al prompt for the summarization. An example Al prompt may include:Simplify this radiology report for a layman. Format your response in XHTML 1.0 Strict DTD.

[0052] Because the outputted summary from the generative Al is often plain text, the report generator 110 may convert this text into a certain word processing text file format (e.g., DOCX, RTF, PDF) that is generally an easy-to-read and nice-looking format. In some cases, for this purpose, the prompt above directs the Al to get a response as XHTML that is later converted into the word processing text file format. For instance, the report generator 110 may convert the text of the summary from an XHTML summary to a certain word processing text file format (e.g., DOCX, RTF, TXT, PDF) by executing the XHTMLImport for docx4j Java package.

[0053] In some implementations, the report generator 110 may merge the Al report summary text, the selected illustrations, and the input source report. The report generator 110 may, for example, append the summarized word processing text file and the illustration images to the source report in a word processing text file format (e.g., DOCX, RTF, TXT, PDF), thereby generating the enhanced output report.

[0054] The reporting system database 112 is hosted in non-transitory machine-readable storage of the reporting system 102. The reporting system database 112 stores various types ofEXPTR01-PCT / 140068-0110 PATENT data, such medical data, image data, report data, transaction logs, and illustrations, among other types of data.

[0055] The reporting system database 112 may include illustration data related to illustrations that the report generator 110 may select for generating the outputted reports. For instance, the illustration data may include an illustrations table in the reporting system database 112 that contains possible combinations for sagittal and axial views. For a sagittal view, the illustrations table includes herniation levels, herniation sizes, spine regions, and curvature types. For the axial view, the illustrations table includes herniation directions and herniation sizes. When the VM 106 receives the response from the report generator 110 for either the sagittal or axial view, the VM 106 may query the illustrations table with the herniation information to retrieve the appropriate illustration. TABLE 1 shows an example structure of the illustrations table:TABLE 1

[0056] In some implementations, the reporting system database 112 includes transaction logs for troubleshooting and maintaining logs of transactions and processes. ForEXPTR01-PCT / 140068-0110 PATENT instance, the report generator 110 or other component of the system 100 (e.g., client devices 103, VM 106) references the reporting system database 112 (e.g., SQL Server) for diagnosing errors occurring in the reporting system 102.

[0057] The software of the VM 106 records various events and errors in one or more tables of the reporting system database 112 for auditing and logging purposes. Non-limiting example tables include an illustration results table (TABLE 2) and a logs table (TABLE 3).

[0058] The example structure of the illustration results table (TABLE 2) is shown below:TABLE 2

[0059] The example structure of the logs table (TABLE 3) is shown below:EXPTR01-PCT / 140068-0110 PATENTTABLE 3

[0060] Whenever a component of the system 100 (e.g., report generator 110) executes a query on the illustrations table of the reporting system database 112 to obtain illustration images, report generator 110 and reporting system database 112 capture, store, and log the parameters into the illustration results table (TABLE 2) and / or logs table (TABLE 3) of the reporting system database 112. This indicates, and helps later identify, which illustration the report generator 110 retrieved and included in a report and what the requested parameters were. The results table may also include entries that indicate when the report generator 110 query did not find or identify the illustration image for the query requested parameters. In these cases, the report generator 110 may be configured to reference and include optional default illustrations stored in the reporting system database 112 or other device of the reporting system 102

[0061] In some embodiments, a server or other computing devices in the reporting system 102 executes software programming of a webserver for hosting a website and webEXPTR01-PCT / 140068-0110 PATENT application (sometimes referred to as a “web app”), accessible to the client device 103 or other device of the system 100. The web app is programmed to query, retrieve, and display the data from the illustration results (TABLE 2) and logs table (TABLE 3), which may be user interface (e.g., web browser) or word processing file in a paginated format (e.g., DOCX, PDF, TXT, RTF). This functionality allows users to sort and filter the data directly from a web interface, eliminating the need to manually query the reporting system database 112.

[0062] Additionally or alternatively, the web app includes a “prompt playground” webpage where users can test LLM or GPT prompts against different LLM or GPT machinelearning models.

[0063] FIG. 2 shows operations of a process 200 for receiving medical image data and medical records data for machine-generated output reports using machine-learning architectures of a medical reporting system. As described in FIG. 2, the operations and features of the process 200 are executed by a computer (e.g., VM 106, server computer) of a medical data reporting system (e.g., reporting system 102), though embodiments are not so limited.

[0064] At operation 202, the computer receives an MRI report via one or more networks using software components of an interface engine for a source channel. The PACS or other data source sends the MRI report to the interface engine. The MRI report may be transmitted or received as a message structure in the HL7 message format standard. The source channel is set up to listen for these incoming messages and capture the MRI report at arrival.

[0065] At operation 204, the software source channel of the computer validates the data structure and content of the MRI report. For instance, the computer checks the structure and content of the HL7 message to make sure everything is correct.

[0066] At operation 206, once validated, the computer sends a confirmation or acknowledgment message back to the PACS system.

[0067] At operation 207, the computer saves the validated HL7 message, containing the MRI report, to a non-transitory machine-readable storage medium accessible to the computer, such as a database or local storage media of computer. The computer then executes software programming of a processing channel.

[0068] FIGS. 3A-3C show operations of a process 300 for executing machine-learning architectures of a medical reporting system trained for generating machine-generated outputEXPTR01-PCT / 140068-0110 PATENT reports. The operations and features of the process 300 are executed by a computer (e.g., VM 106, server computer) of the medical data reporting system (e.g., reporting system 102), though embodiments are not so limited.

[0069] At operation 302, the computer saves and stores an HL7 message of an MRI report in a directory folder of a non-transitory storage location and executes software programming of a processing channel of an interface engine or other software program. In some cases, the processing channel operations in the process 300 are triggered when the computer stores the HL7 message into the storage location.

[0070] At operation 304, software programming of the processing channel executed by the computer extracts the report (in DOCX format) from the HL7 message. At operation 306, the computer extracts the text between specified markers (or bookmarks) in the text file.

[0071] At operation 308, the computer generates a text summary by executing or accessing an LLM or similar machine-learning model of a generative Al software program. For instance, the text is summarized using a third-party generative Al machine-learning model software instance of the Azure® OpenAI® LLM.

[0072] At operation 310, the computer extracts sagittal and axial information from the extracted text using the textual generative Al. In some cases, a radiologist or other user operates an end-user device to indicate or selects one or more key MRI slice from the patient’s scan (in the MRI image report), where the selected slice indicates or highlights a primary area of concern (e.g., a herniation). Optionally, the computer or other device of the system colorizes the selected MRI slice(s), either manually or using generative Al, to emphasize critical features as according to the input report data, such as herniations and spinal curvature.

[0073] At operation 312, the computer determines or identifies a herniation or other medical condition in one or more condition indicators. If the computer identifies the herniation or other medical condition, the process 300 proceeds to operation 314 of FIG. 3B, otherwise the process 300 proceeds to operation 324 of FIG. 3C.

[0074] At operation 314, the computer queries a database containing medical illustrations. The generative Al or report generator may query and identify relevant illustration images for C-Spine or L-Spine MRI reports in the database. In some implementations, the computer executes an Al that extracts key descriptors, indicators, or parameters from the input source radiology report, including herniation levels, sizes, and directions. The computerEXPTR01-PCT / 140068-0110 PATENT matches the descriptors against the database of curated medical illustrations to find the most relevant illustration.

[0075] At operation 316, the computer determines whether the query successfully identified the one or more relevant illustration images in the database. At operation 318, the computer selects and retrieves the selected images from the database. In some cases, the computer may reference an illustrations table in the database to select and retrieve the illustrations from the database. Optionally, at operation 320, the computer may return one or more default illustrations when the computer does not identify the relevant illustrations.

[0076] As an example, according to operations 314-320, the computer executes a generative Al (or other type of machine-learning model) to generate or update an illustration image based upon the patient’s particular condition. The computer determines or selects the medical illustrations according to the key descriptors, indicators, or parameters as in the condition indicators for the patient’s condition. Using anatomy indicators and / or condition indicators in the source report, the computer may generate the output illustration using an illustration selected from the database or, alternatively, generate the output illustration using a default illustration selected from the database.

[0077] At operation 322, the computer merges or appends the selected relevant illustrations to a new or updated version of the original report in the text file. Turning to FIG. 3C, at operation 324, the computer merges or appends the Al-generated summary text the new or updated version of the original report in the text file having the relevant illustrations selected from the database. The computer may generate or otherwise output an enhanced output report based on the source text file and illustrations, as generated or selected using the source MRI report.

[0078] The computer may, for example, execute generative Al operations that combines the colorized MRI image with the selected illustration, overlaying anatomical details from the MRI onto the illustration and vice versa. In some implementations, the computer executes operations for selecting a most significant axial image depicting a comparatively largest herniation and a direction of the largest herniation. For axial images and sagittal images, the computer may implement similar operations or processes (e.g., generative Al operations) for colorizing and merging the axial or sagittal images with the relevant illustrations.EXPTR01-PCT / 140068-0110 PATENT

[0079] In some embodiments, the generative Al operations refine the combination through multiple iterations, creating two versions: one that emphasizes the illustration with MRI anatomical details, and another that emphasizes the MRI with illustrative elements. In some implementations, the computer may execute generative Al operations that recursively enhance images to balance anatomical accuracy and illustrative clarity. This process involves transformations to ensure both images retain their respective details while improving interpretability. In some implementations, the Al places the refined images on a graph to determine the most appropriate representations for inclusion in the final report. The computer selects the images that best balance anatomical accuracy and illustrative clarity, enhancing the understanding of radiology findings.

[0080] Optionally, at operation 326, the computer saves the enhanced report in a specific directory folder (e.g., processing-doc-files). At operation 328, the computer moves the original report file to another directory folder (e.g., ai-processed).

[0081] FIG. 4 shows operations of a process 400 for outputting medical image data and medical records data for machine-generated output reports using machine-learning architectures of a medical reporting system. As described in FIG. 4, the operations and features of the process 200 are executed by a computer (e.g., VM 106, server computer) of a medical data reporting system (e.g., reporting system 102), though embodiments are not so limited.

[0082] At operation 402, the computer saves and stores the final, enhanced output report into a directory folder (e.g., processing-doc-files folder) of a non-transitory storage location and executes software programming of an output channel (sometime referred to as a return channel) of an interface engine or other software program. In some cases, the output channel operations in the process 400 are triggered when the computer stores the output report into the storage location.

[0083] At operation 404, the software programming of the output channel executed by the computer retrieves or reads the enhanced output report from the storage location (e.g., processing-doc-files folder).

[0084] At operation 406, the computer updates the original HL7 message with the updated enhanced report content. In some cases, the computer replaces a segment or portion of the original HL7 message with the updated enhanced report.EXPTR01-PCT / 140068-0110 PATENT

[0085] At operation 408, the computer sends and returns the updated HL7 message back to the PACS or other source device. Optionally, software programming executed at a device of the PACS checks the updated HL7 message to validate accuracy and format.

[0086] At operation 410, the computer receives, via the output channel, a final acknowledgment message from a device of the PACS. The computer saves the final acknowledgement into an output acknowledgement directory folder in the storage location.

[0087] FIG. 5 shows operations of a process 500 for a computer-implemented method for enhancing radiology reports having medical imaging data. The operations and features of the process 500 are executed by a computer (e.g., VM 106, server computer) of the medical data reporting system (e.g., reporting system 102), though embodiments are not so limited.

[0088] At operation 510, the computer obtains MRI imagery data containing one or more MRI images generated from an MRI imaging device. At operation 520, the computer identifies, in an illustration database, a set of one or more illustrations based upon a set of one or more MRI slices of the one or more MRI images. The computer identifies the set of one or more illustration according to text of an input source report having the MRI imagery data. In some cases, for each illustration, the computer identifies an illustration based upon at least one of an anatomical descriptor or a condition indicator in the input source report and associated with the illustration.

[0089] At operation 530, for each MRI image, the computer generates an output image by combining or merging an MRI image with an illustration as identified in the illustration database using the one or more MRI slices of the MRI image. At operation 540, the computer generates an output report having one or more output images and other components, such as machine-generated text summary that a machine-learning model of a generative Al program generates based upon the text of the input source report.

[0090] FIGS. 6-11 depict a series of graphical user interfaces 600-1100 displaying portions of a source report and an output enhanced report generated by one or more machinelearning models of a machine-learning architecture.

[0091] The source report pertains to lower back pain following a patient’s injury (e.g., motor vehicle accident). The definitions of the input source report and / or output report may be based on preconfigured definitions of disc bulge, disc herniation, protrusion, and extrusion, which a computing device uses for training machine-learning models of theEXPTR01-PCT / 140068-0110 PATENT machine-learning architecture, such as a text-generating LLM. The definitions may be pulled from one or more training corpora of medical data and information, such as medical encyclopedias and medical journals, among others.

[0092] FIG. 6 depicts a sagittal view of a portion of the anatomy (e.g., T2 L3, L4). For instance, the graphical user interface 600 shows that at L3-L4, there is a right paracentral disc herniation superimposed on a disc bulge, the combination of which measures 5 mm indenting the ventral thecal sac and elevating the posterior longitudinal ligament; there are anterior and there are posterior osteophytes; the disc material extends posterior to the posterior osteophytes; there is a zone of hyperintensity within the disc consistent with an annular fissure; there is moderate right neural foraminal narrowing; there is mild left neural foraminal narrowing; there is moderate spinal stenosis to 0.8 cm; and there is narrowing of the right lateral recess and contact to the right L4 transiting nerve root.

[0093] FIG. 7 depicts a sagittal view of a portion of the anatomy (e.g., T2 L4, L5). For instance, the graphical user interface 700 shows that at L4-L5, there is a left paracentral / neural foraminal disc herniation superimposed on a disc bulge, the combination of which measures 4.5 mm indenting the ventral thecal sac and elevating the posterior longitudinal ligament; there are anterior and there are posterior osteophytes; however, the disc material extends posterior to the posterior osteophytes; there is a zone of hyperintensity within the disc consistent with an annular fissure; there is severe bilateral neural foraminal narrowing; there is a 2.5 mm grade 1 retrolisthesis of L4 on L5; there is moderate spinal stenosis to 0.8 cm; and there is narrowing of the lateral recesses bilaterally.

[0094] FIG. 8 depicts a sagittal view of a portion of the anatomy (e.g., T2 L5, SI). For instance, the graphical user interface 800 shows that at L5-S1, there are biforaminal disc herniations superimposed on a disc bulge, the combination of which measures 4 mm indenting the ventral thecal sac and elevating the posterior longitudinal ligament; there are anterior and there are posterior osteophytes; the disc material extends posterior to the posterior osteophytes; this is superimposed on a 2 mm grade 1 anterolisthesis of L5 on SI; there is moderate spinal stenosis to 0.7 cm; and there is severe bilateral neural foraminal narrowing.

[0095] FIGS. 9A-9B depict the graphical user interfaces 900a, 900b having machinegenerated overlays for an enhanced version of the MRI images. In FIG. 9A, the graphical user interface 900a displays the sagittal view of the portion of the anatomy (e.g., T2 L3, L4) in the graphical user interface 600 of FIG. 6 having the right paracentral disc herniationEXPTR01-PCT / 140068-0110 PATENT superimposed on the disc bulge. In the graphical user interface 900a of FIG. 9A, the software programming of a generative Al (or other type of machine-learning model) generates the one or more bolded or colorized overlays that, for example, follow and indicate contours of the anatomy (e.g., interior portion of the spinal column). The generative Al may further generate a condition overlay as a bolded or colorized overlay following and indicating contours of the medical condition (e.g., disc herniation).

[0096] For comparison, FIG. 9B depicts the graphical user interface 900b displaying the sagittal view of the comparable portion of the anatomy for a normal spine, without the herniation or other condition. In this graphical user interface 900b, the programming of the generative Al (or other type of machine-learning model) generates the one or more bolded or colorized overlays that, for example, follow and indicate contours of the anatomy (e.g., the interior portion of the normal spinal column).

[0097] FIG. 10 depicts the graphical user interface 1000 displaying a machinegenerated illustration for a sagittal view (side view) of the spine. The illustration image selection may be generated by the software programming of the generative Al executed by the computing device. The illustration includes additional image elements of portions of the anatomy, medical conditions, and annotations indicating some of the areas of interest as described in the input report (e.g., input radiology report). The generative Al may generate the illustration image according to anatomy indicators and / or condition indicators, where the generated illustration contains elements that depict or indicate a location of anatomy (e.g., vertebra, disk, spinal cord) or conditions and a severity of the condition (e.g., herniation). The generative Al generates the elements of the illustration of the graphical user interface 1000 based upon, for example, a corresponding level(s) of the hemiation(s) as indicated by the anatomy indicators and / or condition indicators, such that the output illustration provides a visual representation to improve the understanding of the reported findings for a layperson.

[0098] FIG. 11 depicts the graphical user interface 1100 displaying a machinegenerated illustration for an axial view (top-down view) of the spine at level L4-L5. The illustration includes additional image elements of portions of the anatomy at the portion of the spine, medical conditions, and annotations indicating some of the areas of interest as described in the input report (e.g., input radiology report). The generative Al may generate the illustration image according to anatomy indicators and / or condition indicators, where the generated illustration contains elements that depict or indicate a location of anatomy (e.g., left and rightEXPTR01-PCT / 140068-0110 PATENT sides of body for orientation, nucleus pulposus, annulus fibrosis, disk) or conditions and a severity of the condition (e.g., herniation). The generative Al generates the elements of the illustration of the graphical user interface 1100 based upon, for example, a corresponding level(s) of the herniation(s) as indicated by the anatomy indicators and / or condition indicators, such that the output illustration provides a visual representation to improve the understanding of the reported findings for the layperson.

[0099] The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

[0100] Embodiments implemented in computer software may be implemented in software, firmware, middleware, microcode, hardware description languages, or any combination thereof. A code segment or machine-executable instructions may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, attributes, or memory contents. Information, arguments, attributes, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0101] The actual software code or specialized control hardware used to implement these systems and methods is not limiting of the invention. Thus, the operation and behavior of the systems and methods were described without reference to the specific software code being understood that software and control hardware can be designed to implement the systems and methods based on the description herein.

[0102] When implemented in software, the functions may be stored as one or more instructions or code on a non-transitory computer-readable or processor-readable storageEXPTR01-PCT / 140068-0110 PATENT medium. The steps of a method or algorithm disclosed herein may be embodied in a processorexecutable software module which may reside on a computer-readable or processor-readable storage medium. A non-transitory computer-readable or processor-readable media includes both computer storage media and tangible storage media that facilitate transfer of a computer program from one place to another. A non-transitory processor-readable storage media may be any available media that may be accessed by a computer. By way of example, and not limitation, such non-transitory processor-readable media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other tangible storage medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer or processor. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-Ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and / or instructions on a non-transitory processor-readable medium and / or computer-readable medium, which may be incorporated into a computer program product.

[0103] The preceding description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the following claims and the principles and novel features disclosed herein.

[0104] While various aspects and embodiments have been disclosed, other aspects and embodiments are contemplated. The various aspects and embodiments disclosed are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims.

Claims

EXPTR01-PCT / 140068-0110 PATENTCLAIMSWhat is claimed is:

1. A computer-implemented method for enhancing radiology reports having medical imaging data, comprising: obtaining, by a computer, MRI imagery data containing one or more MRI images generated from an MRI imaging device; identifying, by the computer in an illustration database, a set of one or more illustrations based upon a set of one or more MRI slices of the one or more MRI images, the set of one or more illustrations being identified according to text of an input source report having the MRI imagery data, wherein for each illustration the computer identifies an illustration based upon at least one of an anatomical descriptor or a condition indicator in the input source report and associated with the illustration; for each MRI image, generating, by the computer, an output image by combining an MRI image with an illustration as identified in the illustration database using the one or more MRI slices of the MRI image; and generating, by the computer, an output report having one or more output images.

2. The method according to claim 1, further comprising generating, by the computer, a report summary based upon the text of the input source report, wherein the computer generates the output report having the one or more output images and report summary.

3. The method according to claim 1, wherein selecting the one or more MRI slices includes, for each MRI slice of the one or more MRI slices, generating, by the computer, a colorized instance of the MRI slice.

4. The method according to claim 3, wherein generating the colorized instance includes: identifying, by the computer, in the MRI slice a portion of an anatomy having a medical condition according to the condition indicator of the input source report; and updating, by the computer, a coloring of the portion of the anatomy having the medical condition.

5. The method according to claim 3, wherein generating the colorized instance includes receiving, by the computer, a user input indicating a coloring a portion of an anatomy in the MRI slice having a medical condition.EXPTR01-PCT / 140068-0110 PATENT6. The method according to claim 1, wherein generating the output image includes updating, by the computer, the MRI imagery data containing the one or more MRI images based upon merging the MRI image and an overlay of the medical illustration.

7. The method according to claim 6, further comprising generating, by the computer, a refined image according to a graph representation of one or more image features, including an anatomical accuracy value and an illustration clarity value.

8. The method according to claim 1, wherein identifying the set of one or more illustrations includes selecting, by the computer, from the illustration database at least one of an axial image or a sagittal image based upon herniation data of a condition indicator.

9. The method according to claim 1, further comprising: receiving, by the computer, the input source report having the MRI imagery data in a healthcare message data structure having a standard format; and extracting, by the computer, the input source report and the MRI imagery data from the healthcare message data structure.

10. The method according to claim 1, further comprising selecting, by the computer, the one or more MRI slices from the MRI imagery data according to one or more input from a user device, each MRI slice includes an overlay corresponding to one or more condition indicators of the input source report.

11. A system for enhancing radiology reports having medical imaging data, the system comprising: an illustration database configured to store a plurality of illustrations corresponding to a plurality of portions of human anatomy; and a computer comprising at least one processor configured to: obtain MRI imagery data containing one or more MRI images generated from an MRI imaging device; identify in the illustration database a set of one or more illustrations based upon a set of one or more MRI slices of the one or more MRI images, the set of one or more illustrations being identified according to text of an input source report having the MRI imagery data, wherein for each illustration the computer identifies an illustration based upon at least one of an anatomical descriptor or a condition indicator in the input source report and associated with the illustration;EXPTR01-PCT / 140068-0110 PATENT for each MRI image, generate an output image by combining an MRI image with an illustration as identified in the illustration database using the one or more MRI slices of the MRI image; and generate an output report having one or more output images.

12. The system according to claim 11, wherein the computer is further configured to generate a report summary based upon the text of the input source report, and wherein the computer generates the output report having the one or more output images and report summary.

13. The system according to claim 11, wherein, when selecting the one or more MRI slices, the computer is further configured to, for each MRI slice of the one or more MRI slices, generate a colorized instance of the MRI slice.

14. The system according to claim 13, wherein, when generating the colorized instance, the computer is further configured to: identify in the MRI slice a portion of an anatomy having a medical condition according to the condition indicator of the input source report; and update a coloring of the portion of the anatomy having the medical condition.

15. The system according to claim 13, wherein, when generating the colorized instance, the computer is further configured to receive a user input indicating a coloring a portion of an anatomy in the MRI slice having a medical condition.

16. The system according to claim 11, wherein, when generating the output image, the computer is further configured to update the MRI imagery data containing the one or more MRI images based upon merging the MRI image and an overlay of the medical illustration.

17. The system according to claim 16, wherein the computer is further configured to generate a refined image according to a graph representation of one or more image features, including an anatomical accuracy value and an illustration clarity value.

18. The system according to claim 11, wherein, when identifying the set of one or more illustrations, the computer is further configured to select from the illustration database at least one of an axial image or a sagittal image based upon herniation data of a condition indicator.

19. The system according to claim 11, wherein the computer is further configured to:EXPTR01-PCT / 140068-0110 PATENT receive the input source report having the MRI imagery data in a healthcare message data structure having a standard format; and extract the input source report and the MRI imagery data from the healthcare message data structure.

20. The system according to claim 11, wherein the computer is further configured to select the one or more MRI slices from the MRI imagery data according to one or more input from a user device, each MRI slice includes an overlay corresponding to one or more condition indicators of the input source report.

Images