Disease velocity representation using conversational artificial intelligence

Abstract

Methods and systems are herein provided for a disease velocity representation system. In one example, a disease velocity representation system comprises one or more processors; memory storing instructions executable by the one or more processors that when executed cause the one or more processors to: receive a user query relating to a patient; obtain patient data of the patient from one or more data repositories; generate, by one or more artificial intelligence (AI) models, a first response output to the user query based on the patient data, wherein the first response output is representative of disease velocity; and output the first response output to a display device.

Description

DISEASE VELOCITY REPRESENTATION USING CONVERSATIONAL ARTIFICIAL INTELLIGENCECROSS REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims priority to Indian Patent Application No. 202341079521, filed on November 23, 2023. The entire contents of the above-listed application are hereby incorporated by reference for all purposes.FIELD

[0002] Embodiments of the subject matter disclosed herein relate to healthcare management and more specifically, to disease velocity representation with conversational artificial intelligence (Al).BACKGROUND

[0003] Computerized healthcare management systems that capture department workflows and manage patient medical information can accumulate a large amount of information such as clinical notes, patient notes, in-patient and out-patient notes, radiology and pathology reports, etc. generated through treatment cycles during a healthcare journey of a patient. Treating clinician(s) that can enter the healthcare journey of the patient at various phases of a treatment cycle can desire to understand a longitudinal aspect of disease progression of the patient through various facets. For example, the clinician(s) can desire to know whether the patient is responding to a treatment, whether a disease is progressing or regressing, visualize tumor lesions that have dropped in size, view a summarization of a patient therapy regime and changes needed thereof, etc. However, with multiple data repositories storing the relevant information, including one or more electronic medical records (EMRs), picturing archiving and communication systems (PACSs), lab systems, and the like, relevant patient data that captures the longitudinal disease progression may be disparate and therefore manual analysis by clinician(s) may be a laborious task that may potentially lead to relevant information being missed.BRIEF DESCRIPTION

[0004] In one example, a disease velocity representation system comprises: one or more processors; memory storing instructions executable by the one or more processors that when executed cause the one or more processors to: receive a user query relating to a patient; obtain patient data of the patient from one or more data repositories; generate, by one or more artificial intelligence (Al) models, a first response output to the user query based on the patient data, wherein the first response output is representative of disease velocity; and output the first response output to a display device.

[0005] In another example, a method for a disease velocity representation system comprises receiving, from a healthcare management application, a first user query relating to a patient; obtaining patient data of the patient from one or more data repositories; determining a query type of the first user query; in response to determining the query type, deploying one or more artificial intelligence (Al) models to generate a response output to the first user query; and displaying the response output in a graphical user interface (GUI) of the healthcare management application.

[0006] In another example, a method for representing disease velocity comprises receiving a first user query relating to a patient; obtaining patient data of the patient from one or more electronic medical records (EMRs); generating, by a large language model (LLM), a first response to the first user query based on the patient data in response to determining that the first user query indicates a text-based response; displaying the first response in a user interface (UI) on a display device, wherein the first response is a text-based response; receiving a second user query, wherein the second user query is a follow up to the first user query; generating, by a foundation model, a second response to the second user query based on the first response and imaging data obtained from an imaging database in response to determining that the second user query indicates a nontext based response; and displaying the second response in the UI on the display device, wherein the second response comprises one or more of unsegmented image slices; one or more segmented image slices, a trend graph, and a disease velocity label.

[0007] It should be understood that the brief description above is provided to introduce in simplified form a selection of concepts that are further described in the detailed description. It is not meant to identify key or essential features of the claimed subject matter, the scope of which is defined uniquely by the claims that follow the detailed description. Furthermore, the claimedsubject matter is not limited to implementations that solve any disadvantages noted above or in any part of this disclosure.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present invention will be better understood from reading the following description of non-limiting embodiments, with reference to the attached drawings, wherein below:

[0009] FIG. 1 shows a block diagram of a disease velocity representation system;

[0010] FIG. 2 shows an example process flow for employing the disease velocity representation system using one or more artificial intelligence (Al) models;

[0011] FIG. 3 shows a block diagram of an example training system for an Al model;

[0012] FIG. 4 shows a high-level flowchart illustrating a method for determining disease velocity using conversational Al;

[0013] FIG. 5 shows a flowchart illustrating a method for generating and displaying representations of disease velocity using one or more Al models in response to user queries;

[0014] FIGS. 6A-6B shows a flowchart illustrating a method for region localization and lesion tracking in response using one or more Al models in response to user queries;

[0015] FIG. 7A shows a process diagram of a first user query and a corresponding first response generated by one or more Al models;

[0016] FIG. 7B shows a process diagram of a second user query and a corresponding second response generated by one or more Al models;

[0017] FIG. 8A shows a process diagram for identifying relevant medical images from an imaging database based on a response generated by an Al model for a user query;

[0018] FIG. 8B shows example images that can be stored in the imaging database;

[0019] FIG. 9 shows a process flow of identifying relevant medical images with a second Al model based on a generated response from a first Al model;

[0020] FIG. 10 shows a process flow of identifying relevant medical images;

[0021] FIG. 11 shows an example graphical user interface (GUI) from which user queries are entered and to which disease velocity representation are displayed; and

[0022] FIG. 12 shows examples of response outputs generated by one or more Al models displayed within a GUI in response to user queries.DETAILED DESCRIPTION

[0023] The following description relates to various embodiments of systems and methods for disease velocity presentation. In particular, systems and methods for disease velocity representation using one or more artificial intelligence (Al) models are provided. Hospitals and outpatient clinical medical facilities may provide computing systems with graphical user interfaces (GUIs) for displaying patient information to healthcare providers and other users. Such healthcare management systems may capture department workflows and manage patient information. In this way, a healthcare provider may view the most up-to-date patient information and retrieve data from electronic medical records (EMRs), imaging results, laboratory results, and so on. Further, alerts or other notifications may be automatically and / or manually generated to indicate status(es) of patients within the healthcare environment. However, such healthcare management systems may accumulate a large amount of information related to a healthcare journey of a patient during a treatment cycle, even when targeted to a particular disease type (e.g., cancer, cardiology, etc ).

[0024] In some examples, patients within the healthcare environment may have ongoing medical treatments for one or more diagnoses such that the patient data therefor includes multiple types of data over a span of time, including lab results, imaging results, provider encounter notes, prescriptions, provider orders, and more. With the various data repositories in which such patient data is stored, analysis of progress of disease state and / or analysis of disease velocity may be challenging. Healthcare providers may resort to manually monitoring information and sorting through multiple data repositories to gain an accurate representation of the disease progress so far and to analyze disease velocity for care management recommendations. For example, treating clinician(s) that can enter the healthcare journey of the patient at various phases of a treatment cycle can desire to understand a longitudinal aspect of disease progression of the patient through various facets. As a non-limiting example, a patient recently diagnosed with cancer may be first referred to a surgical specialist for surgical resection before the patient is seen by an oncologist. When the oncologist first sees the patient for a visit, the patient may have had multiple rounds of labs, multiple imaging scans, multiple patient encounters with other physicians, such as the surgical specialist, and may have even undergone surgical or other procedures. All the patient data relating to the patient’s course of treatment for their cancer may be stored in different data repositories, making it difficult for the oncologist to efficiently gain an understanding of the course of treatment so far, understand the progression of the disease so far, and understand a currentdisease velocity. Healthcare management systems may help providers to visualize patient data from multiple sources but may not adequately present relevant longitudinal data in a way that represents disease progress and velocity. For example, the clinician(s) can desire to know whether the patient is responding to a treatment, know whether a disease is progressing or regressing, visualize tumor lesions that have dropped in size, view a summarization of a patient therapy regime and changes needed thereof, etc. Under current methods, the oncologist may resort to manually sorting through multiple EMRs to find the information, which is time inefficient and may result in the oncologist missing information.

[0025] Methods and systems are provided herein for a disease velocity representation system for representing disease progress and velocity. The disease velocity representation system as herein described utilizes one or more Al models for analysis of patient data. For example, a first conversational Al model, such as a conversational large language model (LLM) may be deployed to generate a response to a user query regarding a treatment course of a patient. A second Al model, such as a foundation model, may be leveraged by the first Al model to identify relevant patient data and process the identified relevant patient data to generate representations of disease velocity. As a non-limiting example, the second Al model may be leveraged to identify relevant medical images based on a user query and output of the first Al model and to segment those identified relevant medical images to track a specified lesion across time.

[0026] In this way, the disease velocity representation system may ingest patient information from a variety of data repositories and based on a user query, output a representation of disease velocity, in the form of text, graph, medical image, or the like to the user as a response. By leveraging an Al model such as a foundation model to identify relevant patient data, extraneous patient data that is not relevant to the user query may be filtered out, thus increasing processing efficiency.

[0027] Turning now to the figures, FIG. 1 shows a block diagram of an exemplary computing system 100 comprising a disease velocity representation system 102 that employs one or more Al models. The disease velocity representation system 102 may be configured to ingest data in various formats from different sources, processes the data, and generates an output for display on a display device, as will be herein described.

[0028] The disease velocity representation system 102 may be configured as part of a computing device, such as a server, a personal computer, a workstation, a mobile device (e.g., acellular phone, a smart phone, a computing tablet, and so on), or any other type of computing device. For example, the disease velocity representation system 102 may be configured as a standalone application or as a portion of an application that incorporates other functionalities as well. For example, the disease velocity representation system 102 may be configured as part of a healthcare management application that a user (e.g., a clinician, technician, or the like) may launch on computing device.

[0029] The disease velocity representation system 102 includes a processor 104 which may be configured to execute machine-readable instructions stored in non-transitory memory 106. Processor 104 may be single core or multi-core, and the programs executed thereon may be configured for parallel or distributed processing. In some embodiments, the processor 104 may optionally include individual hardware components that are distributed throughout two or more devices, which may be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of the processor 104 may be virtualized and executed by remotely-accessible networked computing devices configured in a cloud computing configuration. The disease velocity representation system 102 further includes the non-transitory memory 106. It should be appreciated that the disease velocity representation system 102 may include additional memory devices, including volatile memory, mass storage, local memory, and so on.

[0030] In some examples, the non-transitory memory 106 may include components disposed at two or more devices, which may remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of the non-transitory memory 106 may include remotely-accessible networked storage devices configured in a cloud computing configuration. The processor 104 and the non-transitory memory 106 may be coupled, for example, via a communication bus.

[0031] An LLM module 108, a training module 110, an identification module 114, a segmentation module 116, a disease velocity module 118, and a data presentation module 120 may be stored in non-transitory memory 106, The LLM module 108 may include an LLM and instructions for implementing the LLM to generate responses to inputted user queries. For example, the LLM may be an LLM trained to ingest a text query, ingest obtained patient data (e.g., obtained from one or more data repositories), and generate a text-based response output to the text query based on the obtained patient data. As an example, a user query may be a request for a summary of a treatment course for a patient. The LLM, according to its training, may processobtained patient data for the patient and output a summary based on the patient data. It should be understood that while an LLM is herein described, other types of conversational AIs capable of ingesting user queries and patient data and outputting a response to the user query in a requested format. The LLM may be trained to extract clinically structured multi-modal data in near real-time in order to generate the response to the user query. Insights generated by the LLM and presented in the response may be based on reports (e.g., lab result reports, patient note reports, etc.) that are stored in the patient database, imaging data (e.g., an image on which tumor markings or deliniations can be performed), or a combination thereof. Additionally, the LLM may be trained to identify a context for contemplating and using other Al models, such as foundation models, for selection and visualization of anatomy-specific data and pathology-specific data.

[0032] Identification module 114 may store instructions for identifying a subset of relevant patient data from multi-modality data. For example, the identification module 114 may store instructions for identifying a subset of medical images of a patient that are relevant to a given user query and / or LLM output. Segmentation module 118 may store instructions for segmenting or otherwise processing medical images. For example, segmentation module 118 may store instructions for segmenting a lesion from the subset of relevant medical images and measuring the size of the segmented lesion over the course of time according to a segmentation algorithm or model, such as an edge computing algorithm, a segment anything model, or the like.

[0033] In some examples, the identification module 114 and the segmentation module 116 may be formed together as a second Al model. For example, a foundation model 112 may include the identification module 114 and the segmentation module 116. As an example, the foundation model 112 may be a self distillation no labels (DINO) model that incorporates both DINO and Segment Anything Model (SAM) capabilities. In an example scenario, based on the initial response provided by the LLM, a clinician may enter a second user query to the disease velocity representation system 102, wherein the second user query identifies a lesion of interest. The LLM may then leverage the foundation model 112 to identify from the obtained patient data, a subset of images that are relevant to the lesion of interest. The subset of images may be identified based both on the images in which the lesion of interest is present, but also the images that best show the lesion. In this way, the foundation model may be trained to identify, based on data from obtained image reports, the user query, and / or the like, images that are relevant to a user query. The LLM may additionally leverage the foundation model 112 to segment the lesion of interest from theidentified subset of images and track the size (e.g., measure) the lesion. Tracking the size of the lesion within the identified subset of images may thus indicate a progression of disease with respect to the lesion of interest over time, thereby quantifying responses to treatments and disease progression.

[0034] Training module 110 may comprise instructions for training and / or fine-tuning one or more of the Al models herein described. In some examples, a first set of instructions may be stored for training and / or fine-tuning the LLM and a second set of instructions may be stored for training and / or fine-tuning the foundation model. Training module may include instructions that, when executed by the processor 104, causes the disease velocity representation system 102 to conduct a method for fine-tuning a respective Al model, wherein the method includes generating a dataset of training pairs and test pairs and fine-tuning the Al model to execute its given functions. For example, the LLM may be fine-tuned on a dataset of pairs including training inputs of patient data, including imaging reports, images, lab results, physician notes (e.g., text data), and the like, as well as text based queries, and fine-tuning outputs of responses to queries based on the patient data. The foundation model may be fine-tuned on generated pairs, wherein the pairs are natural images, medical images, or a combination thereof for outputs of relevant images and / or segmented images.

[0035] The disease velocity module 118 may store instructions for determining a disease velocity based on the outputs from the LLM and / or the foundation model. For example, disease velocity may be indicated by a label, wherein a given disease state is “stable”, “slowly regressing”, “rapidly regressing”, “slowly progressing”, or “rapidly progressing”. Assigning of these labels may be based on quantitative metrics, in some examples, for example based on tracked or otherwise determined progression of lesion sizes as outputted by the one or more Al models.

[0036] The data presentation module 120 may store instructions for presenting the outputted information from the one or more Al models on the display device(s) 136. For example, the data presentation module 120 may store instructions for rendering graphs, text data, medical images, segmented lesions, and the like for display within a GUI, for example of the healthcare management application 130.

[0037] The disease velocity representation system 102 may be operably coupled to a healthcare management application 130, one or more data repositories 132, one or more user input devices 134, and one or more display devices 136. While not specifically shown in FIG. 1, thehealthcare management application 130 may also be operably coupled to the one or more data repositories 132, the one or more user input devices 134, and the one or more display devices 136.

[0038] The one or more user input devices 134 may comprise one or more of touchscreens, keyboards, mice, trackpads, motion sensing cameras, or other devices configured to enable a user to interact with and manipulate data within the disease velocity representation system 102. One or more display devices 136 may utilize virtually any type of display technology, such as mobile device screens, computer monitors, tablet screens, or the like. In some embodiments, each display device 136 may be combined with the processor 104, the non-transitory memory 106 with the modules thereof, and / or a given user device 134 in a shared enclosure, or may be peripheral display devices and may comprise a monitor, touchscreen, projector, or other display device known in the art, which may enable a user to view medical data, such as images, patient notes, processed and aggregated patient data, and the like in a GUI of the healthcare management application 130 or in a GUI of one or more of the one or more data repositories 132. Further, through a GUI, for example of the healthcare management application 130, displayed on a display device of auser input device, user queries may be inputted to the disease velocity representation system 102.

[0039] The disease velocity representation system 102 may be configured to obtain patient data of one or more patients from the one or more data repositories 132. The one or more data repositories 132 may comprise EMRs, imaging storage databases, such as picture archiving and communication systems (PACS), laboratory systems, and the like. The disease velocity representation system 102 may be further configured to receive user inputs, for example user queries, via a GUI of the healthcare management application 130 via the user input device(s) 134 and may output responses representative of disease velocity to the display device(s) 136.

[0040] The disease velocity representation system 102 as herein disclosed may be implemented in a suitable computing environment that allows the various methods herein described to be executed. While the embodiments have been described above in the general context of computer-executable instructions that can run on one or more computers, those skilled in the art will recognize that the embodiments can be also implemented in combination with other program modules or as a combination of hardware and software.

[0041] Generally, program modules include routines, programs, components, data structures, etc., that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the inventive methods can be practiced with other computersystem configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, Internet of Things (loT) devices, distributed computing systems, as well as personal computers, hand-held computing devices, microprocessor-based or programmable consumer electronics, and the like, each of which can be operatively coupled to one or more associated devices.

[0042] The illustrated embodiments of the embodiments herein can be also practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote memory storage devices.

[0043] Computing devices typically include a variety of media, which can include computer- readable storage media, machine-readable storage media, or communications media, which two terms are used herein differently from one another as follows. Computer-readable storage media or machine-readable storage media can be any available storage media that can be accessed by the computer and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer-readable storage media or machine-readable storage media can be implemented in connection with any method or technology for storage of information such as computer-readable or machine-readable instructions, program modules, structured data or unstructured data.

[0044] Computer-readable storage media can include, but are not limited to, random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disk read only memory (CD ROM), digital versatile disk (DVD), Blu-ray disc (BD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, solid state drives or other solid state storage devices, or other tangible or non-transitory media which can be used to store desired information. In this regard, the terms “tangible” or “non-transitory” herein as applied to storage, memory or computer-readable media, are to be understood to exclude only propagating transitory signals per se as modifiers and do not relinquish rights to all standard storage, memory or computer-readable media that are not only propagating transitory signals per se.

[0045] Computer-readable storage media can be accessed by one or more local or remote computing devices, e g., via access requests, queries or other data retrieval protocols, for a variety of operations with respect to the information stored by the medium.

[0046] Communications media typically embody computer-readable instructions, data structures, program modules or other structured or unstructured data in a data signal such as a modulated data signal, e.g., a carrier wave or other transport mechanism, and includes any information delivery or transport media. The term “modulated data signal” or signals refers to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in one or more signals. By way of example, and not limitation, communication media include wired media, such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media.

[0047] Namely, the computing system 100 in which the disease velocity representation system 102 is implemented may comprise a computer or other computing device that includes a processing unit, a system memory, and a bus. The bus couples the system components including, but not limited to, the system memory to the processing unit. The processing unit may be any of various commercially available processors. Dual microprocessors and other multi-processor architectures can also be employed as the processing unit.

[0048] Thus, the disease velocity representation system 102 as implemented in the computing system 100 implements specialized protocols for efficient DICOM data handling including, custom compression algorithms optimized for medical image sequences, parallel decompression and processing pipeline, memory-efficient streaming of large image datasets, integration with PACS systems using optimized network protocols, and real-time image quality assessment and enhancement to allow for efficient, quality image display within GUIs.

[0049] Turning now to FIG. 2, an example process flow 200 for deployment of the disease velocity representation system 102 described above is shown, according to an embodiment of the present disclosure. Various embodiments herein may perform lesion analysis or region of interest (ROI) analysis based on medical images by leveraging multi-modal data from text, non-imaging information (e.g., lab results, physician orders, etc.), and imaging information of a patient to identify meaningful time points in disease progression of a patient and combining the meaningful time points with illustrative information that can generate explainable concepts. Much of the information related to disease progression of a patient can be provided to a clinician in a clinical report, however, such information can become extensive and therefore result in the clinician missing information in the interest of time efficiency when reviewing. Thus, various embodiments herein can retrieve a second set of information from an imaging database to display the secondaryinformation in an illustrative manner. With continued reference to FIG. 1, process flow 200 illustrates additional aspects of disease velocity representation based on multi-modal data of a patient.

[0050] In the process flow 200, an EMR 202 may be accessed by both a healthcare management application 204 and an LLM 208. For example, in order to present longitudinal patient data according to protocols thereof, the healthcare management application 204 may obtain patient data from the EMR 202 (and other data repositories, if applicable) as shown by arrow 203. The healthcare management application 204 may display patient data in a GUI 206. The GUI 206 may comprise a query search bar into which a user may enter queries.

[0051] The LLM 208 may receive a user query from the healthcare management application 204, as indicated by arrow 205. The query may be a prompt related to a patient that can be entered by a clinician (e.g., an oncologist, a radiologist, a surgeon, a nurse practitioner, or other care provider). In response to receiving the user query, the LLM 208 may access the EMR 202 to obtain patient data particular to the patient indicated by the user query, as indicated by arrow 207. The LLM 208, based on the patient data, may extract clinically structured multi-modal data from the patient data and may generate a response to the user query and may output the response to the healthcare management application 204, for example by displaying the response within the GUI 206, as indicated by arrow 217.

[0052] When the user query indicates that a text-based response is to be generated, such as a request for a summary of a progression of a course of treatment for a disease, the LLM may generate a text-based response for output that includes a text summary generated based on the patient data which may include care provider notes, imaging reports, medical images, lab results, current and past orders, and the like.

[0053] When the user query indicates that a non-text-based response is to be generated, the LLM may generate a text-based portion of the response and may leverage or otherwise access the foundation model 212, as indicated by arrow 209. The foundation model 212 may access a PACS database 210 that stores medical images and corresponding imaging reports and may retrieve a set of images corresponding to the patient, as indicated by arrow 211. The foundation model 212 may identify a set of images 214 as indicated by arrow 213, wherein the set of images 214 may be the most relevant to the user query. Further, the foundation model 212 may segment the images in the set of images 214 (either automatically or in response to user input) and may perform one or moreimage processing algorithms to measure segmented lesion(s) segmented from the set of images 214. Based on the segmentation of the set of images 214, the foundation model 212 may generate a disease velocity output 216, as noted by arrow 215. The disease velocity output 216 may include one or more graphs 218 and / or text data 220, in some examples. The disease velocity output 216, as well as the segmented images of the set of images 214 may be outputted for display within the GUI 206 of the healthcare management application 204, as noted by arrow 219. While the process flow 200 shows the disease velocity output 216 being outputted directly to the healthcare management application 204, it should be understood that in some examples, the foundation model 212 may transmit the disease velocity output 216 back to the LLM 208 and then to the healthcare management application 204 similar to the first response.

[0054] As an example, in the first user query, a clinician may request a timeline of events in a patient’s treatment course. The LLM may receive the first user query, access patient data from the EMR and extract multi-modal data therefrom, and generate a first text response that is displayed on the GUI of the healthcare management device. In a second, follow up user query, the clinician may request a quantitative analysis of disease progression of a particular lesion of interest (e.g., “how has the patient’s lesion in the left temporal lobe changed?”). The LLM may receive the second user query and may leverage the foundation model to identify a subset of images stored in the PACS database that are relevant to the lesion of interest in the context of the patient’ s treatment course, and to segment the lesion of interest from the subset of images, measuring the size of the lesion of interest in the segmented images over time. In this way, volume data may be tracked longitudinally to generate multiple time points in the patient’s treatment journey. Then, based on the tracked longitudinal volume data, disease velocity representations, in the form of graphs, text data, etc. and in some examples the segmented images themselves may be outputted to the healthcare management system. Further, the disease velocity representation system may generate, based on the quantitative information extracted by the foundation model, may determine and display a disease velocity label on the GUI of the healthcare management application.

[0055] Turning now to FIG. 3, an example of an Al model training system 300 is shown. The Al model training system 300 herein described is exemplary in nature and may apply to either the first Al model (e.g., the LLM) or the second Al model (e.g., the foundation model) herein described. Further, in some examples, the disease velocity representation system 102 described above may include a first training system for the LLM and a second training system for thefoundation model, for example stored in the training module 110. For example, the LLM may be a conversational model trained to ingest patient medical data, extract from the patient medical data multi-modal data, and generate text responses to user queries based on the extracted multi-modal data. The foundation model may be trained to ingest various inputs, such as imaging data retrieved from an imaging database (e.g., a PACS), user queries, and outputs from the LLM to identify relevant images, segment lesion from images, and track lesions longitudinally within images across multiple time points. The Al model training system 300 may be implemented by a disease velocity representation system, such as disease velocity representation system 102 described with respect to FIG. 1, to train one or both of the LLM and the foundation model to perform the functions described above.

[0056] Al model 302 may be stored within an Al module 301 of the disease velocity representation system. For example, when Al model 302 is the LLM, the Al model 302 may be stored within the LLM module 108 described in FIG. 1. Al model training system 300 also includes training module 304, which includes a training dataset comprising a plurality of training pairs of data, divided into training pairs 306 and test pairs 308. Training module 304 may be a non-limiting example of training module 110 of disease velocity representation system 102 of FIG. 1. As an example, when the training system 300 is configured to train the LLM, the training pairs 306 and test pairs 308 may be pairs of text-based data, including text data, lab data, and the like, in various formats. The training data may be specially curated medical data intended to train the LLM to ingest various types of medical data, including lab results, physician reports, medication orders (e.g., prescriptions), radiology reports, and the like, and generate outputs based thereon. When the training system 300 is configured to train the foundation model, the training pairs 306 and test pairs 308 may be pairs of image data, such as specially curated medical image data and / or natural images intended to train the foundation model to ingest imaging data, identify data relevant to an input (e.g., an output from the LLM and / or a user query), segment the relevant data, and measure the segmented lesions.

[0057] A number of training pairs 306 and test pairs 308 may be selected to ensure that sufficient training data is available to prevent overfitting, whereby the Al model 302 learns to map features specific to samples of the training set that are not present in the test set.

[0058] Each pair of the training pairs 306 and the test pairs 308 comprises an input and a target. Al model training system 300 may include a training data generator 310, which may be used to generate the training pairs 306 and the test pairs 308 of the training module 304.

[0059] Once each pair is generated, the pair may be assigned to either the training pairs 306 or the test pairs 308. In an embodiment, the pair may be assigned to either the training pairs 306 or the test pairs 308 randomly in a pre-established proportion. For example, the pair may be assigned to either the training pairs 306 or the test pairs 308 randomly such that 90% of the pairs generated are assigned to the training pairs 306, and 10% of the pairs generated are assigned to the test pairs 308. Alternatively, the pair may be assigned to either the training pairs 306 or the test pairs 308 randomly such that 85% of the pairs generated are assigned to the training pairs 306, and 15% of the pairs generated are assigned to the test pairs 308. It should be appreciated that the examples provided herein are for illustrative purposes, and pairs may be assigned to the training pairs 306 dataset or the test pairs 308 dataset via a different procedure and / or in a different proportion without departing from the scope of this disclosure.

[0060] Al model training system 300 may include a validator 320 that validates the performance of the Al model 302 against the test pairs 308. The validator 320 may take as input a partially trained Al model 302 and a dataset of test pairs 308, and may output an assessment of the performance of the partially trained Al model 302 on the dataset of test pairs 308.

[0061] Once the Al model 302 has been validated, a trained Al model 322 (e.g., the validated Al model 302) may be used to generate an output based on data obtained from one or more data repositories 330, which may be the one or more data repositories 132 of FIG. 1 and a user query 331, which may be inputted to a healthcare management application via a GUI, as described above. Trained Al model 322 may be stored within an inference module 321 of the disease velocity representation system, in some examples.

[0062] Based on a training process as described with respect to FIG. 3, the LLM is specially trained using a novel multi-stage training process optimized for medical context understanding and disease progression analysis. The training process includes: 1) pre-training on a curated dataset of millions of medical documents to leam medical terminology and relationships; 2) fine-tuning on paired examples of patient records and expert-generated disease progression summaries; 3) additional fine-tuning using reinforcement learning with expert feedback to optimize for accuratedisease velocity assessment; and 4) implementation of attention mechanisms specifically weighted to identify temporal relationships and progression indicators in medical data.

[0063] In a similar fashion, based on a training process as described with respect to FIG. 3, the foundation model (e.g., DINO-SAM) undergoes specialized training including, as non-limiting examples: 1) transferring learning from natural images to medical imaging domains using domain adaptation techniques; 2) training on anatomically-labeled medical image datasets to learn anatomical parcellation; 3) integration of medical knowledge graphs during training to establish relationships between imaging findings and clinical conditions; and 4) implementation of custom loss functions that optimize for accurate lesion boundary detection and measurement consistency across time points.

[0064] Turning now to FIG. 4, a high-level flowchart illustrating a method 400 for determining disease velocity representation via one or more Al models is shown. Method 400 may be executed by a processor of a disease velocity representation system, such as the disease velocity representation system 102 of FIG. 1. Some operations of method 400 may be stored in a non- transitory memory of the disease velocity representation system (e.g., in LLM module 108, identification module 114, and segmentation module 116 of FIG. 1) and executed by a processor of the disease velocity representation system (e.g., the processor 104 of FIG. 1).

[0065] At 402, method 400 includes receiving one or more user input queries. As described above, a clinician or other user may input queries into a GUI of a healthcare management application regarding a given patient. An exemplary GUI will be described in greater detail below with respect to FIG. 11. The first Al model, such as a conversational LLM, may be operably coupled to the healthcare management application and may receive the inputted user queries. The inputted user queries may be requests for summaries of data, timelines of data, impressions or insights as to a patient’s disease progression, requests of analysis of progression of a particular, specified lesion, insights as to a patient’s response to treatment, or other query type related to a clinician’s sought insights about disease velocity of a patient. The one or more user input queries may be inputted with respect to a given patient.

[0066] At 404, method 400 includes obtaining patient data specific to the patient specified in the user queries. For example, one or more data repositories may store patient data, including patient demographics, current medications, provider orders, lab results, imaging reports, medical images, provider encounter notes, and the like. Each of the entries stored in the one or more datarepositories (e.g., EMRs, lab systems, etc.) may be date stamped. The patient data may be obtained by a first Al model (e.g., an LLM) in one or more data formats and the disease velocity representation system, if necessary, may translate data formats of various obtained data in order to bring the obtained data into a common form usable by the disease velocity representation system. In some examples, the LLM may receive data from the one or more data repositories for a plurality of patients and then in response to a user query regarding a patient, may specifically look at patient data specific to the patient.

[0067] At 406, method 400 includes generating one or more responses to the one or more user input queries representative of disease progression based on the obtained patient data. In some examples, the LLM may receive a user query and determine the type of response that is requested by the user query, as noted at 408. In particular, the LLM may determine whether leveraging of the foundation model is demanded based on the type of request in the user query. For example, when the user query includes a request for a text-based response, such as a summary, a list, or a timeline, the LLM may extract multi-modal data specific to the patient, as noted at 410, and may generate a text-based response, as noted at 412, without leveraging a second Al model (e.g., a foundation model). Further, certain types of disease processes, such as blood based cancers or other hematologic conditions, may be monitored by lab data. Lab data may be presented in text form and as such, for a user query regarding such a disease process, a text-based response may be generated by the LLM without leveraging the foundation model.

[0068] In other examples, when the user query includes a request for display of non-text- based data, such as imaging data, the LLM may output a text-based portion of a response output and, in addition, leverage the foundation model to identify relevant patient data (e.g., relevant imaging slices) and process the patient data (e.g., segment lesion(s) of interest, track size of the segmented lesions over multiple time points) to generate non-text-based portions of the response output, as noted at 414. The non-text-based portions of the response output generated by the foundation model may include the relevant imaging slices (annotated with segmentation details or not) and / or trend information based on the size tracking (e.g., in the form of a graph).

[0069] At 416, method 400 includes determining a disease velocity based on the output. For example, based on quantitative longitudinal data outputted by the foundation model, a disease velocity label, such as “stable”, “slowly progression”, “slow regression”, “rapid progression”, or“rapid regression”, may be assigned to the patient and displayed within the GUI of the healthcare management application.

[0070] Turning now to FIG. 5, a flowchart illustrating an exemplary method 500 for generating and displaying representations of disease velocity using one or more Al models in response to user queries is shown. The method 500 may be an example method for a use-case scenario of the disease velocity representation system described with respect to FIG. 1 and as such should not be construed as the only implementation scenario of the system. Some operations of method 500 may be stored in a non-transitory memory of the disease velocity representation system (e.g., in LLM module 108, identification module 114, and segmentation module 116 of FIG. 1) and executed by a processor of the disease velocity representation system (e.g., the processor 104 of FIG. 1).

[0071] At 502, method 500 includes receiving a first user query of a first type. As described above, a clinician or other user may input queries into a GUI of a healthcare management application. An exemplary GUI will be described in greater detail below with respect to FIG. 11. The first Al model, such as a conversational LLM, may be operably coupled to the healthcare management application and may receive the inputted user queries. The inputted user queries may be requests for summaries of data, timelines of data, impressions or insights as to a patient’s disease progression, requests of analysis of progression of a particular, specified lesion, insights as to a patient’s response to treatment, or other query type related to a clinician’s sought insights about disease velocity of a patient. In the scenario presented in the method 500, the first user query may be a first type of query in which a response is to be text based (e.g., a summary, a timeline, etc.) and thus, upon reception, the LLM may determine that leveraging of the foundation model is not needed. The first user query may be inputted with respect to a given patient.

[0072] At 504, method 500 includes obtaining patient data from one or more data repositories. The patient data may correspond to the given patient for which the first user query was entered. In some examples, patient data may be obtained by the LLM prior to receiving the query, for example when the LLM is continuously receiving data from one or more data repositories, and may particularly parse the data to obtain data specific to the given patient in response to the query. For example, one or more data repositories may store patient data, including patient demographics, current medications, provider orders, lab results, imaging reports, medical images, provider encounter notes, and the like. Each of the entries stored in the one or more data repositories (e.g.,EMRs, lab systems, etc.) may be date stamped. The patient data may be obtained by the LLM in one or more data formats and the disease velocity representation system, if necessary, may translate data formats of various obtained data in order to bring the obtained data into a common form usable by the disease velocity representation system.

[0073] At 506, method 500 includes extracting clinically structured multi-modal data from the patient data based on the first user query. The clinically structured multi-modal data may be a subset of the obtained patient data that is relevant to the user query. For example, the patient data may include all patient data from the one or more data repositories for the given patient. However, the first user query may be specific to a particular disease process (e.g., a diagnosis like cancer or neurodegenerative disease) and some of the patient data, such as data obtained prior to the patient being diagnosed with the particular disease process, may not be relevant to the user query. The data being multi-modal may refer to it comprising multiple types of data, such as imaging reports, lab results, provider notes, different types of imaging data (e.g., MRI vs CT vs PET vs x-ray, etc.), and the like. As noted above, extracting this multi-modal data may include selective data reformatting, wherein the disease velocity representation system may detect the data format in which data is received and, when needed, translate data into a format usable by the system.

[0074] At 508, method 500 includes generating a first output in response to the first user query. As described above, the LLM may be trained to ingest patient data and user queries and output responses to the user queries based on the patient data. The first output may thus be generated by the LLM as a response to the first user query. In some examples, the first user query may specify a particular text format in which the output is to be presented, such as an xml format and / or a form of the response, such as a timeline, a summary, a list, etc. The LLM may thus output its response in the requested text format and in the requested form.

[0075] In some examples, the first user query may be query in which the LLM is able to output a text-based response and thus may process the patient data and the query without needing to leverage another Al model (e.g., the foundation model). For example, the first user query may be a request for the LLM to create a timeline of progression for a specified lesion of interest. As will be noted below at 518, the first output may be displayed as a text output within the GUI.

[0076] At 510, method 500 includes determining if a second user query of a second type is received. The second type of user query may indicate a request for non-text-based data, in some examples. In some examples, the second user query may be entered by the user into the same GUIas the first user query. For example, in the scenario herein discussed, the LLM may output the first response as discussed above, and in response, the user may enter a second user query as a follow up that is more specific to the disease process or a lesion of interest included in the first output. If a second user query is received, method 500 proceeds to 512. If no second user query is received, method 500 proceeds to 518.

[0077] At 512, method 500 includes identifying a subset of data from the multi-modal data that is relevant to the second user query. For example, as will be further described with respect to FIGS. 6A-B, the second user query may be a request to show progression of the disease process or the lesion of interest. In the example where the disease process is shown in imaging data as herein described, the LLM may thus either use its first output or generate a second text output and may leverage the foundation model for the purpose of identifying relevant imaging slices and segmenting those imaging slices to track the progression of the lesion of interest therewithin. For example, the foundation model may take as input the second user query, the output from the LLM (either the first output or a second LLM output if the second query is significantly different), the multi-modal data, and imaging data acquired from an imaging database (e.g., PACS). The foundation model may determine relevant imaging protocols based on condition data presented in the first output and / or imaging reports and may determine relevant image slices based on identified anatomy parcel information. Merging these two information sets, the foundation model may identify the subset of data, for example a set of relevant imaging slices.

[0078] At 514, method 500 includes generating a second output in response to the second user query. The second output may be a non-text-based response, in some examples. Again as will be further described with respect to FIG. 6A-B, the second output may be the set of relevant imaging slices. In some cases, the set of relevant imaging slices may be segmented and the size of the lesion therein may be measured and tracked across multiple time points, as noted at 516. Segmentation and measurement of lesion size may be executed by the foundation model automatically in some examples or responsive to user input selecting a displayed image slice in other examples, as will be further described below. The second output, in some examples, may additionally include disease progression trends presented in graphical format.

[0079] At 518, method 500 includes displaying the first output and / or the second output within a GUI. In some examples, the first output may be displayed within the GUI automatically following its generation prior to receiving the second user query. Then, upon receiving the seconduser query, the LLM may generate a second LLM output and the foundation model may generate its output, as described above, both of which may be displayed in the GUI. For example, the first output may be displayed within a designated region of the GUI into which the user queries are entered upon generation of the output (e.g., prior to the second query being entered). The second output, in examples in which a second query is entered, may also be displayed within the same GUI in which the queries are entered. For example, images may be displayed in a pop-up UI within the GUI. Further, the first and / or second outputs may be used to generate a representation of disease velocity, for example with a label or color as previously discussed.

[0080] It should be understood that the examples described with respect to FIG. 5 are nonlimiting. For example, in some instances, the first user query may include a request for which the LLM has to leverage the foundation model. As an example, the first user query may be a request to show a progression of a lesion of interest in imaging data. In such an example, the LLM may output a text response similar to the first output described above and the foundation model may output a subset of imaging slices similar to the second output described above at the same time based on the text response and imaging data, as will be further described below.

[0081] Further, it should be understood that the disease velocity representation system is not limited to imaging data. For example, longitudinal lab data may also be presented in graphs. The types of data identified and presented may be based on the user query and the patient data for the given patient. For example, a patient with metastatic liver cancer with metastases to the brain may have imaging data that is most relevant to disease progression and response to treatment while, conversely, a patient with acute myeloid leukemia may have lab data that is most relevant to disease progression and response to treatment. Thus, for the patient with imaging data relevant to the given disease process, the LLM may leverage the foundation model as is described herein. Conversely, for the patient with no imaging data relevant to the disease progression, for example with relevant lab data only, the LLM may generate text-based response and corresponding graphs or trends of the lab data without leveraging the foundation model. In this way, the LLM may act as a gatekeep, whereby based on determination of the type of request and the type of relevant data, the system may only leverage the second Al model when necessary, thereby increasing overall system efficiency.

[0082] In this way, the disease velocity representation system may receive user queries and via implementation of one or more Al models, including the LLM and / or the foundation model asherein described, output responses in a conversational manner. Such responses may reduce time spent by the user in manually searching through various EMRs for the disparate medical data as well as increasing overall accuracy. In addition, the system may be configured to respond to queries in a computationally efficient manner, wherein the foundation model is leveraged for queries in which it is demanded and not for queries in which the LLM can generate the response itself. Additionally, as will be further described below, the foundation model may reduce overall processing power in accessing, displaying, and segmentation of medical images.

[0083] FIG. 6A and 6B shows a flowchart illustrating a method for identifying a subset of relevant medical images by a foundation model leveraged by an LLM in response to a user query. The method 600 may be an example method for a use-case scenario of the disease velocity representation system described with respect to FIG. 1. As such, some operations of method 600 may be stored in a non-transitory memory of the disease velocity representation system (e.g., in LLM module 108, identification module 114, and segmentation module 116 of FIG. 1) and executed by a processor of the disease velocity representation system (e.g., the processor 104 of FIG. 1).

[0084] Starting with FIG. 6A, at 602, method 600 includes obtaining an LLM output for a received user query. As an example, the LLM may generate an output in response to a user query, as discussed above. The LLM output may be a text-based output that includes information of relevant patient data. As a non-limiting example, the LLM output may be a list detailing a progression of a lesion of interest based on obtained patient data (e.g., imaging reports, provider notes, orders, etc.), in response to a user query requesting as such.

[0085] At 604, method 600 includes processing the LLM output. Processing the LLM output may include identifying region information expressed in the LLM output, as noted at 606, and identifying condition information expressed in the LLM output, as noted at 608.

[0086] For example, the information presented in the LLM output may include information about where particular lesions (e.g., masses, tumors, plaques, etc.) are located. As a non-limiting example, hepatic lesions, brain lesions, and similar may be identified in the LLM output by lobe, segment, and / or other location identifiers such as anterior / posterior, inferior / superior, and the like. These region identifiers may be extracted from the LLM output.

[0087] Further, the information presented in the LLM output may include information about conditions or features of the patient as seen in the imaging scan. As a non-limiting example,conditions like enhancement, mass effect, edema, and the like may be identified within the LLM output. The system may then use these identified features to identify relevant imaging protocols, as will be described further, according to an predefined look-up dictionary or other algorithm that is applied to the LLM output.

[0088] At 610, method 600 includes identifying relevant imaging protocol(s) based on condition information. For example, enhancing lesions may be best viewed in T2-FLAIR imaging, as opposed to other types of imaging and thus T2-FLAIR imaging may be identified as relevant. In particular, the system may be equipped with an algorithm, look-up table, or model through which the identified condition information extracted from the LLM output .

[0089] At 612, method 600 includes matching region information extracted from the LLM output to template anatomical regions. The template anatomical regions may be stored in a dictionary, similar to a look-up table. Each template anatomical region may be linked, in the dictionary, to relevant imaging slices of various medical imaging protocols. The system may process the region information from the LLM output to match each region information snippet to one of template anatomical regions.

[0090] At 614, method 600 includes localizing regions in protocol volume data and identifying relevant slices. As noted, each template anatomical region may be linked to relevant imaging slices. The template anatomical regions may comprise a parcellated case that may comprise all parcels associated with particular regions of the body, such as the brain, the liver, and the like. The system may then use this information as a template to identify where the identified regions extracted from the LLM output lie within particular sets of data.

[0091] At 616, method 600 includes determining, from obtained imaging data, a subset of imaging data relevant to the user query. For example, the foundation model may be coupled to an imaging database, such as a PACS, and may obtain imaging volume data therefrom. Based on the matching of the region information, the foundation model may identify image slices of the imaging volume data that is stored in the imaging database that are relevant to the extracted region information as well as corresponding to the identified relevant imaging protocols. In some examples, a single image slice from each image volume relevant to the user query may be identified, for example the slice through the largest dimension of a lesion included therein.

[0092] As an example, for a lesion of interest in the left temporal lobe of the brain, all imaging data that is not of the brain may be filtered out, brain imaging slices of the right half of the brainmay filtered out, and so on, going on to identify a set of slices within the relevant imaging protocols, that are relevant to the lesion of interest in the left temporal lobe. This filtering may be based on the system identifying where the identified anatomy parcels (as extracted from the LLM output) lie in the obtained imaging data (e.g., from PACS), specifically which image slices correspond to the identified anatomy parcels / regions. In some examples, such as when a patient has undergone multiple imaging scans of varying types (e.g., CTs, MRIs, PET scans, x-rays, and the like) over time, the set of slices may comprise slices from multiple imaging volumes from multiple time points and thus may be multi-modal imaging data.

[0093] In some examples, relevant imaging protocols may be identified in parallel with identification of relevant imaging slices (e.g., concurrently). In other examples, as is presented herein, relevant imaging protocols may be identified before or after identification of relevant imaging slices. For example, the relevant imaging protocols may be identified first, and thus the relevant imaging slices may be identified from the reduced bank of relevant imaging protocols, thereby reducing overall image processing.

[0094] In this way, the subset of imaging data that is identified may comprise imaging data relevant to the user query. For example, when a user query is a request relating to a disease process (rather than a particular lesion of interest), the subset of imaging data may comprise image slices of one or more lesions in one or more anatomy parcels. For example, a patient with metastatic melanoma may have a plurality of brain metastases. The user query may be a request to show a progression of the patient’s disease and in response the foundation model may identify a subset of image slices of relevant protocols, wherein the subset includes at least one slice for each of the plurality of brain metastases. As another example, when a user query is a request relating to a particular lesion of interest, the subset of imaging data may comprise image slices that include that particular lesion of interest. For example, for a patient with metastatic melanoma with multiple brain metastases, the user query may be a request to show a progression of a right parietal metastasis and thus the subset of imaging data may include one or more image slices from relevant protocols of that right parietal metastasis.

[0095] Continuing to FIG. 6B, at 618, method 600 includes tracking the lesion of interest across time based on the subset of imaging data. As an example, the subset of imaging data, as described above, may include a subset of image slices that include the lesion of interest. Tracking the lesion of interest across time may include segmenting the lesion in each image slice in the setof slices, as noted at 620. For example, the foundation model may be trained to segment the lesion of interest from each of the slices, for example based on edge identification, thresholding, or a segment anything model. The segmented lesion of interest in each of the slices may be measured at its widest dimension, as noted at 622. As the set of slices includes slices from imaging data at multiple time points, measurements of the segmented lesion may be performed across multiple time points, thereby tracking the lesion across time.

[0096] At 624, method 600 includes generating an output representative of disease velocity based on the measurements. For example, a graph representing the trend of size of the lesion of interest over time may be generated based on the measurements. Further, a disease velocity label may be determined based on the trend. For example, overall trends, most recent changes in lesion size (e.g., between the most recent two time points), and other kinds of statistics may be used to determine a disease velocity label such as “stable”, “slowly progressing”, “slowly regressing”, “rapidly progressing”, and “rapidly regressing”.

[0097] At 626, method 600 includes displaying the output and / or the annotated slices (e.g., with segmentation annotations) on a display device. For example, the output representative of disease velocity and the annotated slices may be displayed within a GUI of a healthcare management application. As an example, the output may be displayed in a first overlay or pop-up UI and the annotated slices may be displayed within the same or a separate overlay or pop-up UI.

[0098] In some examples, the foundation model may automatically segment the image slices and track the lesion of interest across time to generate the outputs. In other examples, the foundation model may first display, for example in an overlay or pop-up UI in the GUI, the subset of image slices, unsegmented and unannotated, prior to segmenting them, and then segment the image slices and track the lesion of interest across time in response to user selection of one of the displayed unsegmented image slices.

[0099] As an example, when a user query includes a request regarding general disease progression (as opposed to pointing to a particular lesion of interest), the foundation model may first output unsegmented identified image slices for the user to then review. The user may then select one of the slices, indicating a lesion of interest that is included within the selected slice (e.g., by selecting a particular region of interest of the selected slice). Then, in response to receiving user selection of the selected slice, the LLM may identify a subset of the subset of image slices that includes the selected slice and other slices that include the same lesion of interest. The foundationmodel may then segment the subset of the subset of image slices and size track the lesion of interest across time. Conversely, when a user query includes a request indicating a particular lesion of interest, the foundation model may automatically proceed to segmenting the slices containing a corresponding region of interest and, when indicated based on the user input, tracking the lesion over time given that, in such examples, the subset of image slices comprises only slices that include the lesion of interest indicated by the user query.

[0100] The methods herein, as presented above, may thus reduce processing demands on the system and computing device on which the system is run by first identifying relevant imaging protocols and relevant imaging slices to generate a set of image slices. Thus, rather than processing (e.g., segmenting and / or measuring) irrelevant or duplicative imaging data, by narrowing the data to include only relevant image slices, for example a single slice from each relevant image volume (e.g., the slice through the widest dimension of a lesion therein), the amount of data that has to be processed is reduced. Further, the amount of data that is ultimately presented to the user in the GUI may be reduced as well. For example, in a current workflow, the clinician may open the imaging database, find the patient, and manually sort through multiple image volumes, scrolling through slices to find the most relevant slices and may then execute processes such as segmentation and measurement algorithms on each selected slice. This is not only time consuming for the clinician but accessing, displaying, and processing all the image volumes the user clicks on increases the amount of processing power the user device uses. The disease velocity representation system herein disclosed reduces this processing power by identifying relevant slices and then segmenting and measuring lesions of only those relevant slices or a subset of those relevant slices (e.g., in examples when segmentation and size tracking is performed in response to user selection).

[0101] To reiterate, the disease velocity representation system herein described implements a novel hierarchical filtering architecture that significantly reduces computational overhead compared to conventional approaches. Specifically: 1) the LLM uses an attention-based prefiltering mechanism to identify relevant sections of patient records before full processing; 2) the foundation model employs a cascaded inference approach where initial low-resolution processing identifies regions of interest before applying computationally intensive high-resolution analysis (e.g., segmentation analysis); 3) integration between the LLM and foundation model uses an efficient API that minimizes data transfer overhead; 4) parallel processing pipelines allow simultaneous natural language understanding and image analysis; and 5) caching mechanismsstore intermediate results of commonly accessed anatomical regions and measurement calculations. These technical improvements reduce processing time, for example by 60-80%, compared to sequential processing of all available patient data while maintaining accuracy above 95% compared to manual expert analysis.

[0102] Turning now to FIG. 7A, a first example of a user query and corresponding LLM output are shown in a flow diagram 700. In particular, a user can input a user query 702 into a GUI of a healthcare management application, such as healthcare management application 130 of FIG. 1. The user query 702 may include a request for a specific format in which the response is to be given. For example, the user query 702 may specify a request such as “list all lesions detected for the patient in xml structured format”.

[0103] A corresponding response 704 may be generated by the LLM. In some examples, the corresponding response 704 may be generated by the LLM based on obtained patient data and the specific request in the user query 702. For example, the corresponding response 704 may list, for the specified patient, the identified lesions in the xml format, as requested.

[0104] FIG. 7B shows a second example of a user query and a corresponding LLM output in a flow diagram 710. Similar to the user query 702, a user query 712 may be inputted to a query search bar in the GUI of the healthcare management application. The user query 712 may include a request for a specific form in which the response is to be given. For example, the user query 712 may specific a request such as “create the timeline of progression of the lesion in the left temporal lobe in the form of a list”.

[0105] A corresponding response 714 may be generated by the LLM based on the user query 712 and the multi-modal data extracted from patient data obtained from the EMRs. In the specific example shown in FIG. 7B, the corresponding response 714 includes a numbered list detailing dates and imaging findings relevant to the specified lesion in the left temporal lobe. This kind of LLM output may reduce the time a provider uses when sorting through different data repositories, including both EMRs and PACS to find the pertinent information.

[0106] Further, because the corresponding responses 704 and 714 can be generated based on text data extracted from the patient data obtained from the data repositories (e.g., EMRs), the LLM may generate the responses 704 and 714 without leveraging the foundation model. In this way, by including more than one Al model each configured to respond to different user queries, overall processing demands for the system may be reduced by selectively leveraging the foundation modelonly when the user query includes a request that involves processing of medical images, as an example.

[0107] Turning now to FIG. 8A, an example process flow 800 for identifying relevant medical images from a PACS database based on a response generated by an LLM for a query is shown. The process flow 800 is herein described with reference to the components described with respect to FIG. 1.

[0108] In particular, the LLM may generate a first response 804 based on obtained patient data 806 in response to a user query 802. In particular, in the example shown, the user query 802 may be a request for a findings of a left temporal lobe mass. The first response 804 from the LLM may be text data detailing a list of findings, including dates of imaging and what the imaging shows with respect to the left temporal lobe mass. Because the query doesn’t specify a request for a list or a particular text format like xml as shown in FIGS. 7A and 7B described above, the LLM may also leverage the foundation model.

[0109] For example, the first response 804 and the may be fed into the foundation model. The foundation model may ingest imaging data from the imaging database 808 (e.g., PACS 808). The foundation model may identify, based on the first response 804, relevant images from the set of images obtained from the PACS 808. For example, as described with respect to FIGS. 6A-B, the region of the lesion, the type of lesion, including present of edema or enhancement, and the like may be used by the foundation model to identify relevant slices of the imaging scans.

[0110] In the example provided, the lesion of interest is a left temporal lobe mass with identified surrounding edema and local mass effect. From this information, the foundation model may determine a subset of image slices that are most relevant, including slices through the left temporal lobe (e.g., but not slices through the cerebellum or other regions not relevant to the lesion of interest) of one or more relevant sequences, such as a T2-Flair sequence and / or a Tl-contrast enhanced sequence, but not other less relevant sequences (such as diffusion-weighted imaging sequences). In this way, a subset of image slices 810 that are relevant to the particular user query 802 may be identified for further processing, such as segmentation. Thus, processing efficiency may be increased by reducing the amount of data that is processed in order to generate the ultimate output. In this instance, the output may be a second response including a set of segmented image slices with or without annotations, a list of findings (e.g., in the first response 804), and / or a disease velocity label.

[0111] FIG. 8B shows an example of a plurality of images 812 that may be stored in the PACS 808. As shown, the plurality of images 812 may include images of various body sections, including head images, torso images, and the like. As such, only a subset of the plurality of images 812 may be relevant to the identified lesion of interest. Thus, the subset of image slices 810 may be a selected group of slices of the plurality of images 812 determined by the foundation model to be relevant to the lesion of interest as described herein.

[0112] Turning now to FIG. 9, a process flow 900 for identifying relevant medical images via an LLM and foundation model, as herein presented, is shown. One or more embodiments described with reference to FIG. 9 can be enabled by one or more components presented above with respect to FIG. 1 and the methods described with respect to FIGS. 4-6. Repetitive description of like elements and / or processes employed in respective embodiments is omitted for sake of brevity.

[0113] In a first response 902 generated thereby, the LLM may identify features or conditions within imaging, based on imaging reports obtained as text data, such as, for example, a mass effect, enhancement, decrease in water, etc. for a patient. In some examples, the first response 902 may be only a portion of a response given by the LLM. In some examples, for each feature or condition identified, there may be an imaging protocol dictionary 906, which may be similar to a lookup table. The imaging protocol dictionary 906 may be used to determine the appropriate imaging protocol to display certain images, as illustrated at 904. For example, the imaging protocol dictionary 906 may be used to determine that edema can be best shown through T2-FLAIR weighted images, or that images associated with words ‘enhancement’ or ‘tumor’ appearing in the first response 902 generated by the LLM are to be shown as post-contrast images, or that a solid component can be shown through PET data, etc. While a look-up style method for matching is herein presented, it should be understood that other methods, such as regression models, Al models, and the like, may also be used without departing from the scope of this disclosure. A foundation model, such as DINO-v2, may select the appropriate imaging protocol for the appropriate images based on an order of priority. For example, for a solid component, PET data can take priority and an image can be shown through PET, through diffusion weighted imaging (DWI) or perfusion weighted imaging (PWI) if PET data is unavailable, or through dynamic contrast enhanced (DCE) if DWI and PWI are unavailable, and so on. By matching information from the first response 902 and imaging protocol dictionary (or other algorithm), the foundationmodel may identify relevant protocol imaging series 910 from available imaging sequences obtained from a PACS.

[0114] Before matching conditions to protocols, after matching conditions to protocols, or in parallel with matching condition protocols, the foundation model may utilize an anatomy parcel template image library 912 to extract relevant slices based on anatomy parcel information in the first response 902. For example, the anatomy parcel template image library 912 may comprise a parcellated case that can contain all parcels associated with the brain, the liver, or other anatomy region of the body. The foundation model may utilize this template to identify where the parcel lies on new data (e.g., imaging data obtained from PACS). The first response 902 includes anatomy parcel information 908 such as inferior left temporal, left occipital lobe, occipital horn of the left lateral ventricle, hepatic segment VI, and the right posterior medial parietal lobe. An inferior left temporal lobe can indicate a temporal lobe of the brain, and the foundation model can automatically parcellate for the user, the inferior left temporal lobe on an image based on anatomy parcel information 908. In this way, the foundation model may identify parcels 914 corresponding to particular slices that are relevant to the anatomy parcel information presented in the first response 902 from the LLM.

[0115] Thus, the foundation model may have information about relevant imaging protocols as well as relevant imaging slices, which can be merged and applied to the obtained images from PACS to identify a subset of relevant image slices 916. In this way, the foundation model may localize an ROI in an image slice for segmenting the parcel based on the first response 902, which can be coupled to identify the requested region from the anatomy parcel template.

[0116] Thereafter, the most relevant image slices can be presented to a clinician via a GUI, and the clinician can scroll through and observe images, in some instances to have the images segmented. In some examples, the foundation model may proceed to segment only images selected by the user through the GUI. In other examples, the foundation model may segment each identified image slice in the same parcel, either automatically in response to identification of the image slices or in response to selection of one of the image slices within the parcel. For example, segmentation may be performed for all image slices of the left temporal lobe either automatically when the foundation model identifies the subset of relevant image slices or in response to user selection of an image slice containing data of the left temporal lobe mass. Thus, the foundation model maysegmented image slices and may measure the size of the segmented lesions over multiple time points, generating segmented and measured image slices 918.

[0117] As a non-limiting example, a user query may be a request to show a progression of a left temporal lobe mass. The LLM may generate a first response, similar to the first response 902 that includes condition information and anatomy parcel information relevant to the lesion of interest (e.g., the left temporal lobe mass). Via the foundation model, the condition information may be processed to determine relevant protocol(s) and the anatomy parcel information may be processed via an anatomy parcel template image library to identify relevant image slices. This information may be merged to identify a subset of image slices that correspond to the relevant image slices and the relevant image protocol(s). Given that the user query requests a show of the progression of the particular lesion of interest, the foundation model may automatically segment a corresponding lesion in the left temporal lobe from the subset of image slices and present the segmented lesion and tracked measurement information thereof to the user. For example, the foundation model may segment the lesion over multiple time points as included within different image slices of the subset of relevant image slices. The segmented lesion may be measured over the multiple time points to determine quantitative disease progression.

[0118] As another non-limiting example, a user query may be a request to show a progression of overall disease. The LLM may generate a first response, similar to the first response 902 that includes condition information and anatomy parcel information relative to the disease, which may include information of multiple lesions in multiple regions of the body. Via the foundation model, the condition information may be processed to determine relevant protocol(s) and the anatomy parcel information may be processed via an anatomy parcel template image library to identify relevant image slices. This information may be merged to identify a subset of image slices that correspond to the relevant image slices and the relevant image protocol(s). Because the user query does not specify a specific lesion of interest, the subset of image slices may be presented to the user via the GUI as is and then, in response to receiving user input selecting one of the image slices or a lesion within an image slice, the foundation model may be leveraged to segment the selected image slice and, in some examples, other image slices that include the same lesion at different time points. The segmented lesion may be measured over the multiple time points to determine quantitative disease progression.

[0119] FIG. 10 shows an example portion 1000 of the process flow 900 of FIG. 9 for identifying relevant medical images by utilizing an LLM response in conjunction with the foundation model. In particular, FIG. 10 illustrates images corresponding to anatomy parcel template image library 912, which may be used by the foundation model to identify parcel(s) 914. Based on the identified parcel(s) 914 and identified relevant imaging protocols, based on conditions / features shown identified in the LLM response, the relevant image slices 916 may be identified. Based on the content of the corresponding user query, the foundation model may automatically or responsively (e.g., in response to user input) lesions from the relevant image slices 916 to output segmented and measured image slices 918.

[0120] As described above, the measurements of the image slices at multiple time points may be processed to determine a disease velocity. For example, measurements indicating a significant decline in the size of a given lesion, a disease velocity label of “rapidly regressing” may be assigned to the lesion and / or the disease process. Conversely, measurements indicating no significant change in the size of a given lesion, a disease velocity label of “stable” or “dormant” may be assigned to the lesion and / or the disease process. The disease velocity label may be determined for the lesion or disease process based on a look up table, a regression model, an Al model (e.g., a deep neural network or other machine learning algorithm), or the like that may ingest the various measurements of the segmented lesion and may determine trends. Additionally, the measurements may be processed to be presented in a graph or a chart or the like to be presented alongside the segmented image slices in the GUI.

[0121] Turning now to FIG. 11, an example GUI 1100 is shown. The GUI 1100 may be configured to present information of a healthcare management application, such as healthcare management application 130 of FIG. 1. The GUI 1100 may include a data presentation section 1102, which may include one or more panels, timelines, and the like for displaying aggregated obtained medical patient data. Elements within the panels, timelines, and the like may be selectable to launch pop-up windows to be displayed over a portion of the data presentation section 1102, as shown in FIG. 11.

[0122] The GUI 1100 may additionally include a query section 1104. The query section 1104 may include a user query row including a query search bar 1106 into which the user can enter (e.g., type) the query, and a fire query element 1108 that when selected triggers the LLM to obtain data from data repositories, extract multi-modal data therefrom, and generate a response to the query.

[0123] The query section 1104 may additionally include an LLM output row that includes a display element 1110 in which the response to the query generated by the LLM may be displayed. Further, the query section 1104 may include a run model element 1112 that when selected may trigger the LLM to leverage the foundation model. In response to the foundation model identifying one or more image slices (segmented or not), those image slices, as well as tracking information and a disease velocity label may be displayed in a pop-up window over the data presentation section 1102.

[0124] The GUI herein described and the computing device on which the GUI is rendered and displayed may be configured to implement an interactive visualization framework that is capable of handling real-time display of high-resolution medical images (e.g., as outputted by the foundation model), implements programs for displaying annotated images (e.g., following segmentation and measuring), and employs progressive loading techniques to maintain interface responsiveness while processing large datasets.

[0125] FIG. 12 shows example outputs that may be displayed within the GUI 1100. For example, a text output 1200 generated by the LLM in response to a user query may be displayed within the display element 1110. Segmented image slices 1202 may also be displayed within the GUI 1100, for example based on the text output 1200 and a corresponding user query as described above. Based on tracked measured segmented lesion data at multiple time points, a graph 1204 of size trends may also be displayed within the GUI 1100. In this way, the user may visualize presented disease velocity representations in various manners, including in text form, as in the text output 1200, in image form, as in the segmented image slices 1202, and / or in graphical form, as in the graph 1204. Such information may not only reduce time spent by the user in searching for the disparate information when analyzing a patient, but may also aid in clinical decision making. For example, based on disease velocity trends and presented text data indicating which treatments have occurred so far (e.g., surgical resections, different medications / chemotherapy agents, radiation, etc ), the clinician may more readily determine the next steps in the treatment course of the patient.

[0126] The technical benefit of the disease velocity representation system and the methods herein disclosed is that by inputting a query, a user can easily visualize progression of a disease and representations of disease velocity. The query is inputted into an LLM and based on the content of the query, the LLM may generate a text-based output based on identified relevant patient data and, in some examples, the LLM may leverage a foundation model capable of identifying relevantpatient data (e.g., relevant image slices) and processing that relevant patient data (e.g., segmenting and size tracking lesions therein). Based on the outputs from the LLM and / or the foundation model, a disease velocity label may be assigned to the patient’s disease process. In particular, the LLM may obtain disparate patient data from multiple data repositories (e.g., one or more EMRs, lab systems, medical devices, imaging databases, etc.) and aggregate that data in a single response output in a requested format. Further, for certain user queries, the LLM may leverage the foundation model that identifies, based on condition information and anatomy parcel information extracted from the LLM output, a subset of relevant image slices. In this way, the amount of information that is accessed, displayed, and processed by the user may be reduced as the foundation model filters out irrelevant data and processes and displays the relevant information. Being able to visualize representations of disease velocity, for example via a disease velocity label, a trend graph of lesion sizes over time, and the like, may decrease time spent by a user in searching for patient medical data to understand the progression of a patient’s disease status.

[0127] The disclosure also provides support for a disease velocity representation system, comprising: one or more processors, memory storing instructions executable by the one or more processors that when executed cause the one or more processors to: receive a user query relating to a patient, obtain patient data of the patient from one or more data repositories, generate, by one or more artificial intelligence (al) models, a first response output to the user query based on the patient data, wherein the first response output is representative of disease velocity, and output the first response output to a display device. In a first example of the system, when the user query relates to text-based data, the first response output is generated by a first Al model of the one or more Al models. In a second example of the system, optionally including the first example, the first Al model is a large language model (LLM). In a third example of the system, optionally including one or both of the first and second examples, the system further comprises: instructions stored in memory executable by the one or more processors, when the user query relates to both text-based and imaging-based data, to: generate, by a first Al model of the one or more Al models, a text-based response output to the user query based on the patient data, determine, by a second Al model of the one or more Al models, based on imaging data obtained from an imaging database and the text-based response output, a set of image slices relevant to the user query, and output the set of image slices to the display device, wherein the set of image slices are displayed in a graphical user interface (GUI) on the display device. In a fourth example of the system, optionally includingone or more or each of the first through third examples, the system further comprises: instructions stored in memory that when executed cause the one or more processors, when the user query specifies a lesion of interest, to: automatically segment each image slice of the set of image slices, and size track the lesion of interest across time through the set of image slices. In a fifth example of the system, optionally including one or more or each of the first through fourth examples, the system further comprises: instructions stored in memory that when executed cause the one or more processors, when the user query does not specify a lesion of interest, to: receive user selection of a selected image slice of the set of image slices, wherein the user selection indicates the lesion of interest, in response to user selection of the selected image, segment the selected image and other image slices in the set of image slices that include the lesion of interest, size track the lesion of interest across time through the set of image slices, generate a trend graph based on the size tracking of the lesion of interest, and output the segmented set of image slices and the trend graph for display within the GUI. In a sixth example of the system, optionally including one or more or each of the first through fifth examples, to determine the set of image slices, the second Al model is configured to: extract from the text-based response output condition information and anatomy parcel information, determine, based on the condition information, one or more relevant imaging protocols, and determine, based on the anatomy parcel information, one or more relevant image slices. In a seventh example of the system, optionally including one or more or each of the first through sixth examples, to determine the set of image slices, the second Al model is further configured to merge the determined one or more relevant imaging protocols and the determined one or more relevant image slices to identify the set of image slices. In a eighth example of the system, optionally including one or more or each of the first through seventh examples, the second Al model is a foundation model.

[0128] The disclosure also provides support for a method for a disease velocity representation system, comprising: receiving, from a healthcare management application, a first user query relating to a patient, obtaining patient data of the patient from one or more data repositories, determining a query type of the first user query, in response to determining the query type, deploying one or more artificial intelligence (al) models to generate a response output to the first user query, and displaying the response output in a graphical user interface (GUI) of the healthcare management application. In a first example of the method when the query type is a first query type indicating a text-based response, deploying the one or more Al models to generate the responseoutput comprises deploying a large language model (LLM), wherein the response output is a textbased response output. In a second example of the method, optionally including the first example when the query type is a second query type indicating a non-text-based response, deploying the one or more Al models to generate the response output comprises deploying an LLM to generate a text-based portion of the response output and leveraging a foundation model to generate a nontext-based portion of the response output. In a third example of the method, optionally including one or both of the first and second examples, generating the non-text-based portion of the response output comprises: obtaining imaging volume data from an imaging database, wherein the imaging volume data comprises multi-modal imaging data acquired across multiple time points, determining, based on the text-based portion of the response output generated by the LLM, condition information and anatomy parcel information, identifying, based on the condition information, one or more relevant imaging protocols, identifying, based on the anatomy parcel information and the one or more relevant imaging protocols, a subset of image slices from the obtained imaging volume data. In a fourth example of the method, optionally including one or more or each of the first through third examples, generating the non-text-based portion of the response output further comprises, when the first user query identifies a lesion of interest: automatically segmenting the subset of image slices, and size tracking the lesion of interest within the segmented subset of image slices over time. In a fifth example of the method, optionally including one or more or each of the first through fourth examples, generating the non-text-based portion of the response output further comprises, when the first user query does not identify a lesion of interest, displaying the subset of image slices within the GUI of the healthcare management application, receiving user selection of a selected image slice of the subset of image slices, wherein the user selection identifies the lesion of interest within the selected image, in response to receiving the user selection, segmenting a subset of the subset of image slices, wherein the subset of the subset of image slices comprises the selected image slice and other image slices in the subset of image slices that include the lesion of interest , and size tracking the lesion of interest within the subset of the subset of image slices over time. In a sixth example of the method, optionally including one or more or each of the first through fifth examples, the non-text-based portion of the response output comprises one or more of a one or more unsegmented image slices, one or more segmented image slices, a trend graph, and a disease velocity label. In a seventh example of the method, optionally including one or more or each of the first through sixthexamples, the method further comprises: receiving, from the healthcare management application, a second user query as a follow up to the first user query, wherein the response output to the first user query is a text-based response generated by an LLM of the one or more Al models, in response to determining that the second user query indicates a non-text-based response, leveraging, by the LLM, a foundation model of the one or more Al models to generate a second response output based on the text-based response, wherein the second response output is a non-text-based response, and displaying the response output and the second response output in the GUI of the healthcare management application, wherein the second response output comprises one or more of unsegmented image slices, one or more segmented image slices, a trend graph, and a disease velocity label.

[0129] The disclosure also provides support for a method for representing disease velocity, comprising: receiving a first user query relating to a patient, obtaining patient data of the patient from one or more electronic medical records (EMRs), generating, by a large language model (LLM), a first response to the first user query based on the patient data in response to determining that the first user query indicates a text-based response, displaying the first response in a user interface (UI) on a display device, wherein the first response is a text-based response, receiving a second user query, wherein the second user query is a follow up to the first user query, generating, by a foundation model, a second response to the second user query based on the first response and imaging data obtained from an imaging database in response to determining that the second user query indicates a non-text based response, and displaying the second response in the UI on the display device, wherein the second response comprises one or more of unsegmented image slices, one or more segmented image slices, a trend graph, and a disease velocity label. In a first example of the method, generating the second user query comprises: determining, based on the first response generated by the LLM, condition information and anatomy parcel information, identifying, based on the condition information, one or more relevant imaging protocols, identifying, based on the anatomy parcel information and the one or more relevant imaging protocols, a subset of image slices from the imaging data, when the user query identifies a lesion of interest: automatically segmenting the subset of image slices, size tracking the lesion of interest within the segmented subset of image slices over time, and displaying the segmented subset of image slices and a trend graph generated based on the size tracking within the UI, and when the user query does not identify the lesion of interest, displaying the subset of image slices within theUI, receiving user selection of a selected image slice of the subset of image slices, wherein the user selection identifies the lesion of interest within the selected image, in response to receiving the user selection, segmenting a subset of the subset of image slices, wherein the subset of the subset of image slices comprises the selected image slice and other image slices in the subset of image slices that include the lesion of interest, size tracking the lesion of interest within the subset of the subset of image slices over time, and displaying the segmented subset of the subset of image slices and the trend graph generated based on the size tracking within the UI. In a second example of the method, optionally including the first example, the first and second user queries are received from a healthcare management application adapted to aggregate and display patient medical data.

[0130] As used herein, an element or step recited in the singular and proceeded with the word “a” or “an” should be understood as not excluding plural of said elements or steps, unless such exclusion is explicitly stated. Furthermore, references to “one embodiment” of the present invention are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features. Moreover, unless explicitly stated to the contrary, embodiments “comprising,” “including,” or “having” an element or a plurality of elements having a particular property may include additional such elements not having that property. The terms “including” and “in which” are used as the plain-language equivalents of the respective terms “comprising” and “wherein.” Moreover, the terms “first,” “second,” and “third,” etc. are used merely as labels, and are not intended to impose numerical requirements or a particular positional order on their objects.

[0131] This written description uses examples to disclose the invention, including the best mode, and also to enable a person of ordinary skill in the relevant art to practice the invention, including making and using any devices or systems and performing any incorporated methods. The patentable scope of the invention is defined by the claims, and may include other examples that occur to those of ordinary skill in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insubstantial differences from the literal languages of the claims.

Claims

CLAIMS1. A disease velocity representation system, comprising: one or more processors; memory storing instructions executable by the one or more processors that when executed cause the one or more processors to: receive a user query relating to a patient; obtain patient data of the patient from one or more data repositories; generate, by one or more artificial intelligence (Al) models, a first response output to the user query based on the patient data, wherein the first response output is representative of disease velocity; and output the first response output to a display device.

2. The disease velocity representation system of claim 1, wherein when the user query relates to text-based data, the first response output is generated by a first Al model of the one or more Al models.

3. The disease velocity representation system of claim 2, wherein the first Al model is a large language model (LLM).

4. The disease velocity representation system of claim 1, further comprising instructions stored in memory executable by the one or more processors, when the user query relates to both text-based and imaging-based data, to: generate, by a first Al model of the one or more Al models, a text-based response output to the user query based on the patient data; determine, by a second Al model of the one or more Al models, based on imaging data obtained from an imaging database and the text-based response output, a set of image slices relevant to the user query; and output the set of image slices to the display device, wherein the set of image slices are displayed in a graphical user interface (GUI) on the display device.

5. The disease velocity representation system of claim 4, further comprising instructions stored in memory that when executed cause the one or more processors, when the user query specifies a lesion of interest, to: automatically segment each image slice of the set of image slices; and size track the lesion of interest across time through the set of image slices.

6. The disease velocity representation system of claim 4, further comprising instructions stored in memory that when executed cause the one or more processors, when the user query does not specify a lesion of interest, to: receive user selection of a selected image slice of the set of image slices, wherein the user selection indicates the lesion of interest; in response to user selection of the selected image, segment the selected image and other image slices in the set of image slices that include the lesion of interest; size track the lesion of interest across time through the set of image slices; generate a trend graph based on the size tracking of the lesion of interest; and output the segmented set of image slices and the trend graph for display within the GUI.

7. The disease velocity representation system of claim 4, wherein to determine the set of image slices, the second Al model is configured to: extract from the text-based response output condition information and anatomy parcel information; determine, based on the condition information, one or more relevant imaging protocols; and determine, based on the anatomy parcel information, one or more relevant image slices.

8. The disease velocity representation system of claim 7, wherein to determine the set of image slices, the second Al model is further configured to merge the determined one or more relevant imaging protocols and the determined one or more relevant image slices to identify the set of image slices.

9. The disease velocity representation system of claim 4, wherein the second Al model is a foundation model.

10. A method for a disease velocity representation system, comprising: receiving, from a healthcare management application, a first user query relating to a patient; obtaining patient data of the patient from one or more data repositories; determining a query type of the first user query; in response to determining the query type, deploying one or more artificial intelligence (Al) models to generate a response output to the first user query; and displaying the response output in a graphical user interface (GUI) of the healthcare management application.

11. The method of claim 10, wherein, when the query type is a first query type indicating a text-based response, deploying the one or more Al models to generate the response output comprises deploying a large language model (LLM), wherein the response output is a text-based response output.

12. The method of claim 10, wherein, when the query type is a second query type indicating a non-text-based response, deploying the one or more Al models to generate the response output comprises deploying an LLM to generate a text-based portion of the response output and leveraging a foundation model to generate a non-text-based portion of the response output.

13. The method of claim 12, wherein generating the non-text-based portion of the response output comprises: obtaining imaging volume data from an imaging database, wherein the imaging volume data comprises multi-modal imaging data acquired across multiple time points; determining, based on the text-based portion of the response output generated by the LLM, condition information and anatomy parcel information; identifying, based on the condition information, one or more relevant imaging protocols; identifying, based on the anatomy parcel information and the one or more relevant imaging protocols; a subset of image slices from the obtained imaging volume data.

14. The method of claim 13, wherein generating the non-text-based portion of the response output further comprises, when the first user query identifies a lesion of interest: automatically segmenting the subset of image slices; and size tracking the lesion of interest within the segmented subset of image slices over time.

15. The method of claim 13, wherein generating the non-text-based portion of the response output further comprises, when the first user query does not identify a lesion of interest; displaying the subset of image slices within the GUI of the healthcare management application; receiving user selection of a selected image slice of the subset of image slices, wherein the user selection identifies the lesion of interest within the selected image; in response to receiving the user selection, segmenting a subset of the subset of image slices, wherein the subset of the subset of image slices comprises the selected image slice and other image slices in the subset of image slices that include the lesion of interest ; and size tracking the lesion of interest within the subset of the subset of image slices over time.

16. The method of claim 12, wherein the non-text-based portion of the response output comprises one or more of a one or more unsegmented image slices; one or more segmented image slices, a trend graph, and a disease velocity label.

17. The method of claim 10, further comprising: receiving, from the healthcare management application, a second user query as a follow up to the first user query, wherein the response output to the first user query is a text-based response generated by an LLM of the one or more Al models; in response to determining that the second user query indicates a non-text-based response, leveraging, by the LLM, a foundation model of the one or more Al models to generate a second response output based on the text-based response, wherein the second response output is a nontext-based response; and displaying the response output and the second response output in the GUI of the healthcare management application, wherein the second response output comprises one or more ofunsegmented image slices; one or more segmented image slices, a trend graph, and a disease velocity label.

18. A method for representing disease velocity, comprising: receiving a first user query relating to a patient; obtaining patient data of the patient from one or more electronic medical records (EMRs); generating, by a large language model (LLM), a first response to the first user query based on the patient data in response to determining that the first user query indicates a text-based response; displaying the first response in a user interface (UI) on a display device, wherein the first response is a text-based response; receiving a second user query, wherein the second user query is a follow up to the first user query; generating, by a foundation model, a second response to the second user query based on the first response and imaging data obtained from an imaging database in response to determining that the second user query indicates a non-text based response; and displaying the second response in the UI on the display device, wherein the second response comprises one or more of unsegmented image slices; one or more segmented image slices, a trend graph, and a disease velocity label.

19. The method of claim 18, wherein generating the second user query comprises: determining, based on the first response generated by the LLM, condition information and anatomy parcel information; identifying, based on the condition information, one or more relevant imaging protocols; identifying, based on the anatomy parcel information and the one or more relevant imaging protocols; a subset of image slices from the imaging data; when the user query identifies a lesion of interest: automatically segmenting the subset of image slices; size tracking the lesion of interest within the segmented subset of image slices over time; anddisplaying the segmented subset of image slices and a trend graph generated based on the size tracking within the UI; and when the user query does not identify the lesion of interest; displaying the subset of image slices within the UI; receiving user selection of a selected image slice of the subset of image slices, wherein the user selection identifies the lesion of interest within the selected image; in response to receiving the user selection, segmenting a subset of the subset of image slices, wherein the subset of the subset of image slices comprises the selected image slice and other image slices in the subset of image slices that include the lesion of interest; size tracking the lesion of interest within the subset of the subset of image slices over time; and displaying the segmented subset of the subset of image slices and the trend graph generated based on the size tracking within the UI.

20. The method of claim 18, wherein the first and second user queries are received from a healthcare management application adapted to aggregate and display patient medical data.

Images