Disease velocity representation using conversational artificial intelligence

Abstract

Methods and systems for a disease velocity representation system are provided herein. In one example, a disease velocity representation system comprises: one or more processors; a memory storing instructions executable by the one or more processors that, when executed, cause the one or more processors to: receive a user query related to a patient; obtain patient data for the patient from one or more data repositories; generate, by one or more artificial intelligence (AI) models, a first response output to the user query based on the patient data, wherein the first response output represents a disease velocity; and output the first response output to a display device.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to Indian Patent Application No. 202341079521, filed on November 23, 2023. The entire contents of the above application are hereby incorporated by reference for all purposes. Technical Field

[0003] The implementation schemes of the topics disclosed herein relate to healthcare management, and more specifically, to the use of conversational artificial intelligence (AI) for disease velocity representation. Background Technology

[0004] Computerized healthcare management systems that capture departmental workflows and manage patient medical information can accumulate vast amounts of information, such as clinical records, patient records, inpatient and outpatient records, radiology and pathology reports, etc., generated throughout a patient's healthcare journey via the treatment cycle. Clinicians involved in the patient's healthcare journey at various stages of the treatment cycle may want to understand the longitudinal aspects of a patient's disease progression from multiple perspectives. For example, clinicians may want to know if a patient is responding to treatment, whether the disease is progressing or regressing, to visualize tumor lesions that have decreased in size, to review an overview of the patient's treatment regimen and any necessary changes, etc. However, with relevant information stored in multiple data repositories (including one or more electronic medical records (EMRs), image archiving and communication systems (PACS), laboratory systems, etc.), capturing relevant patient data on longitudinal disease progression can be fragmented, and therefore manual analysis by clinicians can be a daunting task that may potentially lead to the omission of relevant information. Summary of the Invention

[0005] In one example, a disease velocity representation system includes: one or more processors; a memory storing instructions executable by the one or more processors, which, when executed, cause the one or more processors to: receive a user query related to a patient; obtain patient data of the patient from one or more data repositories; generate a first response output to the user query based on the patient data by one or more artificial intelligence (AI) models, wherein the first response output represents the disease velocity; and output the first response output to a display device.

[0006] In another example, a method for a disease velocity representation system includes: receiving a first user query related to a patient from a healthcare management application; obtaining patient data of the patient from one or more data repositories; determining the query type of the first user query; in response to determining the query type, deploying one or more artificial intelligence (AI) models to generate a response output for the first user query; and displaying the response output in a graphical user interface (GUI) of the healthcare management application.

[0007] In another example, a method for representing disease velocity includes: receiving a first user query related to a patient; obtaining patient data for the patient from one or more electronic medical records (EMRs); in response to determining that the first user query indicates a text-based response, generating a first response to the first user query by a large language model (LLM) based on the patient data; displaying the first response in a user interface (UI) on a display device, wherein the first response is a text-based response; receiving a second user query, wherein the second user query is a follow-up to the first user query; in response to determining that the second user query indicates a non-text-based response, generating a second response to the second user query by a base model based on the first response and imaging data obtained from an imaging database; and displaying the second response in the UI on the display device, wherein the second response includes one or more of an unsegmented image slice, one or more segmented image slices, a trend graph, and a disease velocity label.

[0008] It should be understood that the above brief description is provided to introduce some concepts further described in the detailed embodiments in a simplified form. This is not intended to identify key or essential features of the claimed subject matter, the scope of which is uniquely defined by the claims following the detailed embodiments. Furthermore, the claimed subject matter is not limited to specific implementations that address any shortcomings pointed out above or in any part of this disclosure. Attached Figure Description

[0009] The invention will be better understood by referring to the following description of non-limiting embodiments, in which:

[0010] Figure 1 A block diagram of the disease velocity representation system is shown;

[0011] Figure 2 An example process flow is shown for employing a disease velocity representation system using one or more artificial intelligence (AI) models;

[0012] Figure 3 A block diagram of an example training system for AI models is shown.

[0013] Figure 4A high-level flowchart illustrating a method for using conversational AI to determine the speed of disease is shown;

[0014] Figure 5 A flowchart illustrating a method for generating and displaying a representation of disease velocity using one or more AI models in response to a user query is shown;

[0015] Figures 6A to 6B A flowchart illustrating a method for using one or more AI models to perform region localization and lesion tracking in response to a user query is shown;

[0016] Figure 7A The diagram illustrates the process of a first user query and the corresponding first response generated by one or more AI models.

[0017] Figure 7B The diagram illustrates the process of a second user query and the corresponding second response generated by one or more AI models.

[0018] Figure 8A A flowchart is shown to illustrate the process of identifying relevant medical images from an imaging database based on responses generated by an AI model in response to user queries.

[0019] Figure 8B Example images that can be stored in an imaging database are shown;

[0020] Figure 9 The process of using a second AI model to identify relevant medical images based on responses generated from a first AI model is illustrated.

[0021] Figure 10 The process of identifying relevant medical images is shown;

[0022] Figure 11 An example graphical user interface (GUI) is shown, from which a user query is entered and a disease rate representation is displayed; and

[0023] Figure 12 An example of a response output generated by one or more AI models and displayed within a GUI in response to a user query is shown. Detailed Implementation

[0024] The following description relates to various implementations of systems and methods for representing disease velocity. Specifically, systems and methods for representing disease velocity using one or more artificial intelligence (AI) models are provided. Hospitals and outpatient clinical medical facilities may provide computational systems with graphical user interfaces (GUIs) for displaying patient information to healthcare providers and other users. Such healthcare management systems can capture departmental workflows and manage patient information. In this way, healthcare providers can view up-to-date patient information and retrieve data from electronic medical records (EMRs), imaging results, laboratory results, etc. Furthermore, alerts or other notifications may be generated automatically and / or manually to indicate the patient's status within the healthcare environment. However, even when targeting a specific disease type (e.g., cancer, cardiology, etc.), such healthcare management systems may accumulate a large amount of information related to a patient's healthcare journey during the treatment cycle.

[0025] In some examples, patients within a healthcare setting may have ongoing medical treatments for one or more diagnoses, resulting in patient data for these treatments comprising various types of data over a period of time, including laboratory results, imaging results, provider consultation records, prescriptions, provider orders, etc. In the case of various data repositories storing such patient data, analyzing the progression of the disease state and / or the velocity of the disease can be challenging. Healthcare providers may have to manually monitor information and sift through multiple databases to obtain an accurate representation of disease progression to date and analyze the velocity of the disease to obtain care management recommendations. For example, a treating clinician who may be involved in a patient's healthcare journey at various stages of the treatment cycle may want to understand the longitudinal aspects of the patient's disease progression from multiple perspectives. As a non-limiting example, a patient recently diagnosed with cancer may first be referred to a surgeon for surgical resection before seeing an oncologist. By the time the oncologist first sees the patient, the patient may have already undergone multiple rounds of laboratory tests, multiple imaging scans, multiple consultations with other physicians (such as surgeons), and may even have already undergone surgery or other procedures. All patient data related to a patient's cancer treatment process may be stored in different data repositories, making it difficult for oncologists to efficiently understand the treatment progress to date, the disease progression to date, and the current rate of disease progression. Healthcare management systems can help providers visualize patient data from multiple sources, but they may not adequately present the relevant longitudinal data in a way that represents disease progression and rate of progression. For example, clinicians may want to know if a patient is responding to treatment, whether the disease is progressing or regressing, visualize tumor lesions that have decreased in size, and see an overview of the patient's treatment regimen and any necessary changes. With current methods, oncologists may have to manually sift through multiple EMRs to find information, which is very time-consuming and can lead to missed information.

[0026] This paper provides methods and systems for representing disease velocity representation systems for disease progression and velocity. The disease velocity representation system described herein utilizes one or more AI models to analyze patient data. For example, a first conversational AI model (such as a conversational large language model (LLM)) can be deployed to generate responses to user queries about a patient's treatment process. A second AI model (such as a base model) can be utilized by the first AI model to identify relevant patient data and process the identified relevant patient data to generate a representation of disease velocity. As a non-limiting example, the second AI model can be used to identify relevant medical images based on user queries and the output of the first AI model, and to segment those identified relevant medical images to track specified lesions across time.

[0027] In this way, the disease velocity representation system can collect patient information from multiple databases and, based on user queries, output disease velocity representations to users in the form of text, graphs, medical images, etc. By utilizing AI models, such as the base model, to identify relevant patient data, irrelevant patient data unrelated to user queries can be filtered out, thereby improving processing efficiency.

[0028] Now turn to the attached diagram. Figure 1 A block diagram of an exemplary computing system 100 including a disease velocity representation system 102 employing one or more AI models is shown. As will be described herein, the disease velocity representation system 102 can be configured to acquire data in various formats from different sources, process the data, and generate output for display on a display device.

[0029] The disease speed representation system 102 can be configured as part of a computing device, such as a server, personal computer, workstation, mobile device (e.g., cellular phone, smartphone, tablet computer, etc.), or any other type of computing device. For example, the disease speed representation system 102 can be configured as a standalone application or as part of an application that incorporates other functionalities. For instance, the disease speed representation system 102 can be configured as part of a healthcare management application that a user (e.g., a clinician, technician, etc.) can launch on the computing device.

[0030] The disease speed representation system 102 includes a processor 104 configured to execute machine-readable instructions stored in a non-transitory memory 106. The processor 104 may be single-core or multi-core, and the program executing on the processor may be configured for parallel or distributed processing. In some embodiments, the processor 104 may optionally include separate hardware components distributed across two or more devices, these separate hardware components being remotely located and / or configured for cooperative processing. In some embodiments, one or more aspects of the processor 104 may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration. The disease speed representation system 102 also includes the non-transitory memory 106. It should be understood that the disease speed representation system 102 may include additional memory devices, including volatile memory, mass storage devices, local memory, etc.

[0031] In some examples, nontransitory memory 106 may include components located at two or more devices, which may be remotely situated and / or configured for collaborative processing. In some embodiments, one or more aspects of nontransitory memory 106 may include remotely accessible networked storage devices configured in a cloud computing configuration. Processor 104 and nontransitory memory 106 may be coupled, for example, via a communication bus.

[0032] LLM module 108, training module 110, identification module 114, segmentation module 116, disease velocity module 118, and data presentation module 120 may be stored in non-transitory memory 106. LLM module 108 may include an LLM and instructions for implementing the LLM to generate responses to input user queries. For example, the LLM may be an LLM trained to acquire text queries, acquire obtained (e.g., from one or more data repositories) patient data, and generate a text-based response output to the text query based on the acquired patient data. As an example, a user query may be a request for a summary of a patient's treatment process. The LLM, based on its training, may process the acquired patient data and output a summary based on the patient data. It should be understood that while LLMs are described herein, other types of conversational AI are capable of acquiring user queries and patient data and outputting responses to user queries in the requested format. An LLM may be trained to extract clinically structured multimodal data in near real-time to generate responses to user queries. Insights generated by LLM and presented in the response can be based on reports stored in a patient database (e.g., lab results reports, patient record reports, etc.), imaging data (e.g., images on which tumor annotation or demarcation can be performed), or a combination thereof. Additionally, LLMs can be trained to identify the context used for envisioning and using other AI models (such as the base model) for selecting and visualizing anatomically specific and pathology-specific data.

[0033] The identification module 114 may store instructions for identifying subsets of relevant patient data from multimodal data. For example, the identification module 114 may store instructions for identifying subsets of medical images of a patient associated with a given user query and / or LLM output. The segmentation module 118 may store instructions for segmenting or otherwise processing medical images. For example, the segmentation module 118 may store instructions for segmenting lesions from a subset of relevant medical images according to a segmentation algorithm or model (such as an edge detection algorithm, a segmentation-all model, etc.) and measuring the size of the segmented lesions over time.

[0034] In some examples, the identification module 114 and the segmentation module 116 may be formed together as a second AI model. For example, the base model 112 may include the identification module 114 and the segmentation module 116. As an example, the base model 112 may be a self-distilled label-free (DINO) model that combines the capabilities of DINO and the Segmentation All Model (SAM). In an example scenario, based on an initial response provided by the LLM, a clinician may input a second user query into the disease velocity representation system 102, where the second user query identifies a lesion of interest. The LLM may then use the base model 112 to identify a subset of images related to the lesion of interest from the acquired patient data. The subset of images may be identified based on both images in which the lesion of interest is present and images that best represent the lesion. In this way, the base model can be trained to identify images related to the user query based on data from acquired image reports, user queries, etc. The LLM may additionally utilize the base model 112 to segment the lesion of interest from the identified subset of images and track (e.g., measure) the size of the lesion. Therefore, the size of lesions within the identified subset of the tracked image can indicate the progression of the disease over time relative to the lesion of interest, thereby quantifying the response to treatment and disease progression.

[0035] Training module 110 may include instructions for training and / or fine-tuning one or more AI models among the AI ​​models described herein. In some examples, a first set of instructions may be stored for training and / or fine-tuning the LLM, and a second set of instructions may be stored for training and / or fine-tuning the base model. The training module may include instructions that, when executed by processor 104, cause the disease velocity representation system 102 to perform a method for fine-tuning the corresponding AI model, wherein the method includes generating datasets of training and test pairs and fine-tuning the AI ​​model to perform a given function of the AI ​​model. For example, the LLM may be fine-tuned on a dataset including training inputs of patient data (including imaging reports, images, laboratory results, physician notes (e.g., text data), etc.) and pairs of text-based queries, and the output of the response to the query may be fine-tuned based on the patient data. The base model (where these pairs are natural images, medical images, or combinations thereof) may be fine-tuned on the generated pairs for outputs related to and / or segmented images.

[0036] The disease velocity module 118 may store instructions for determining disease velocity based on the output from the LLM and / or the base model. For example, disease velocity may be indicated by labels, where a given disease state is “stable,” “slowly regressing,” “rapidly regressing,” “slowly progressing,” or “rapidly progressing.” In some examples, the assignment of these labels may be based on quantitative measures, such as the progression of lesion size tracked or otherwise determined by the output of one or more AI models.

[0037] The data presentation module 120 may store instructions for presenting output information from one or more AI models on the display device 136. For example, the data presentation module 120 may store instructions for rendering graphs, text data, medical images, segmented lesions, etc., for display within the GUI of, for example, a healthcare management application 130.

[0038] The disease speed representation system 102 can be operatively coupled to a healthcare management application 130, one or more data repositories 132, one or more user input devices 134, and one or more display devices 136. Although in Figure 1 Not specifically shown, but the healthcare management application 130 may also be operatively coupled to one or more data repositories 132, one or more user input devices 134, and one or more display devices 136.

[0039] One or more user input devices 134 may include one or more of the following: a touchscreen, keyboard, mouse, touchpad, motion-sensing camera, or other devices configured to enable a user to interact with and manipulate data within the disease velocity representation system 102. One or more display devices 136 may utilize virtually any type of display technology, such as mobile device screens, computer monitors, tablet computer screens, etc. In some embodiments, each display device 136 may be combined with the processor 104, the non-transitory memory 106 having its modules, and / or a given user device 134 in a shared package, or may be a peripheral display device and may include monitors, touchscreens, projectors, or other display devices known in the art that enable a user to view medical data, such as images, patient notes, processed and aggregated patient data, in the GUI of the healthcare management application 130 or in the GUI of one or more of the one or more data repositories 132. Furthermore, user queries can be entered into the disease velocity representation system 102 via a GUI, such as that of the healthcare management application 130, displayed on the display device of the user input device.

[0040] The disease velocity representation system 102 can be configured to obtain patient data for one or more patients from one or more data repositories 132. The one or more data repositories 132 may include EMRs, imaging storage databases such as Picture Archiving and Communication Systems (PACS), laboratory systems, etc. The disease velocity representation system 102 can be further configured to receive user input, such as user queries, via a user input device 134 through a GUI of a healthcare management application 130, and can output a response representing the disease velocity to a display device 136.

[0041] The disease velocity representation system 102 disclosed herein can be implemented in a suitable computing environment that allows the various methods described herein to be performed. Although the implementations have been described above in the general context of computer-executable instructions that can run on one or more computers, those skilled in the art will recognize that these implementations can also be implemented in combination with other program modules or as a combination of hardware and software.

[0042] Typically, program modules include routines, programs, components, data structures, etc., that perform specific tasks or implement specific abstract data types. Furthermore, those skilled in the art will understand that the methods of this invention can be practiced with other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, Internet of Things (IoT) devices, distributed computing systems, and personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, each operatively coupled to one or more associated devices.

[0043] The embodiments illustrated in this paper can also be practiced in a distributed computing environment where a specific task is performed by a remote processing device linked via a communication network. In a distributed computing environment, program modules can reside on both local and remote memory storage devices.

[0044] Computing devices typically include a variety of media, which may include computer-readable storage media, machine-readable storage media, or communication media, wherein the two terms are used interchangeably herein, as described below. A computer-readable storage medium or a machine-readable storage medium can be any available storage medium accessible by a computer, and includes volatile and non-volatile media, removable and non-removable media. By way of example and not limitation, a computer-readable storage medium or a machine-readable storage medium can be implemented in conjunction with any method or technology used for storing information, such as computer-readable or machine-readable instructions, program modules, structured data, or unstructured data.

[0045] Computer-readable storage media may include, but is not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc read-only memory (CDROM), digital versatile disc (DVD), Blu-ray disc (BD) or other optical disc storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, solid-state drives or other solid-state storage devices, or other tangible or non-transitory media that can be used to store desired information. In this regard, the terms “tangible” or “non-transitory” used herein to describe storage devices, memories, or computer-readable media shall be understood to exclude only the propagation of transient signals themselves as a modifier, and shall not waive any rights enjoyed in respect of all standard storage devices, memories, or computer-readable media other than the propagation of transient signals themselves.

[0046] Computer-readable storage media can be accessed by one or more local or remote computing devices, for example, via access requests, queries or other data retrieval protocols, to perform various operations with respect to the information stored on the media.

[0047] Communication media typically contain computer-readable instructions, data structures, program modules, or other structured or unstructured data in a data signal, such as a modulated data signal, a carrier wave, or other transmission mechanism, and include any information transmission or delivery medium. The terms "modulated data signal" or "signal" refer to a signal whose one or more characteristics are set or altered to encode information in one or more signals. By way of example and not limitation, communication media include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media).

[0048] That is, the computing system 100 in which the disease speed representation system 102 is implemented may include a computer or other computing device, which includes a processing unit, system memory, and a bus. The bus couples system components, including but not limited to system memory, to the processing unit. The processing unit may be any of a variety of commercially available processors. Dual microprocessors and other multiprocessor architectures may also be used as processing units.

[0049] Therefore, the disease velocity representation system 102 implemented in the computing system 100 implements a dedicated protocol for efficient DICOM data processing, including a custom compression algorithm optimized for medical image sequences, parallel decompression and processing pipelines, efficient streaming of large image datasets to memory, integration with PACS systems using optimized network protocols, and real-time image quality assessment and enhancement to allow for efficient, high-quality image display within the GUI.

[0050] Turn now Figure 2The illustration shows an example process flow 200 for deploying the disease velocity representation system 102 described above, according to an embodiment of this disclosure. Various embodiments of this document can perform lesion analysis or region of interest (ROI) analysis based on medical images by utilizing multimodal data from text, non-imaging information (e.g., laboratory results, physician orders, etc.), and patient imaging information to identify meaningful time points in a patient's disease progression and combine these meaningful time points with illustrative information that can generate interpretable concepts. Much information related to a patient's disease progression can be provided to clinicians in clinical reports; however, such information can become extensive and thus lead to omissions by clinicians during review for time efficiency. Therefore, various embodiments of this document can retrieve a second set of information from imaging databases to display secondary information in an illustrative manner. Continue to refer to Figure 1 The process flow of 200 cases illustrates additional aspects of disease velocity representation based on multimodal patient data.

[0051] In process flow 200, EMR 202 can be accessed by both healthcare management application 204 and LLM 208. For example, to present longitudinal patient data according to its protocol, healthcare management application 204 can obtain patient data from EMR 202 (and other data repositories, if applicable), as shown by arrow 203. Healthcare management application 204 can display the patient data in GUI 206. GUI 206 may include a query search bar where users can enter queries.

[0052] LLM 208 can receive user queries from healthcare management application 204, as indicated by arrow 205. The query may be a patient-related prompt, which can be entered by a clinician (e.g., an oncologist, radiologist, surgeon, nurse practitioner, or other care provider). In response to receiving a user query, LLM 208 can access EMR 202 to obtain patient data specific to the patient indicated by the user query, as indicated by arrow 207. Based on the patient data, LLM 208 can extract clinically structured multimodal data from the patient data and can generate a response to the user query, which can be output to healthcare management application 204, for example, by displaying the response within GUI 206, as indicated by arrow 217.

[0053] When a user query indicates that a text-based response should be generated, such as a request for a summary of the progress of treatment for a disease, the LLM can generate a text-based response for output. This text-based response includes a text summary generated based on patient data, which may include care provider notes, imaging reports, medical images, laboratory results, current and previous medical orders, etc.

[0054] When a user query indicates that a non-text-based response should be generated, the LLM can generate a text-based portion of the response and can utilize or otherwise access the base model 212, as indicated by arrow 209. The base model 212 can access a PACS database 210 storing medical images and corresponding imaging reports, and can retrieve a set of images corresponding to a patient, as indicated by arrow 211. The base model 212 can identify an image set 214, as indicated by arrow 213, which is most relevant to the user query. Furthermore, the base model 212 can segment the images in the image set 214 (automatically or in response to user input) and can execute one or more image processing algorithms to measure the segmented lesions from the image set 214. Based on the segmentation of the image set 214, the base model 212 can generate a disease velocity output 216, as indicated by arrow 215. In some examples, the disease velocity output 216 may include one or more figures 218 and / or text data 220. Disease velocity output 216 and segmented images of image set 214 can be output for display within the GUI 206 of the healthcare management application 204, as indicated by arrow 219. While process flow 200 shows disease velocity output 216 directly output to the healthcare management application 204, it should be understood that in some examples, similar to the first response, the base model 212 may send disease velocity output 216 back to the LLM 208 and then to the healthcare management application 204.

[0055] As an example, in a first user query, a clinician may request a timeline of events during a patient's treatment. The LLM receives the first user query, accesses patient data from the EMR and extracts multimodal data from it, and generates a first text response displayed on the GUI of a healthcare management device. In a follow-up second user query, the clinician may request a quantitative analysis of disease progression for a specific lesion of interest (e.g., "How has the lesion in the patient's left temporal lobe changed?"). The LLM receives the second user query and can utilize the base model to identify a subset of images stored in the PACS database that are relevant to the lesion of interest in the context of the patient's treatment, and segment the lesion of interest from the subset of images, thereby measuring the size of the lesion of interest in the segmented images over time. In this way, volumetric data can be longitudinally tracked to generate multiple time points in the patient's treatment course. Then, based on the tracked longitudinal volumetric data, a disease velocity representation in the form of graphs, text data, etc., along with the segmented images themselves in some examples, can be output to the healthcare management system. Furthermore, the disease velocity representation system can generate disease velocity labels based on the quantitative information extracted by the base model and can display these disease velocity labels on the GUI of the healthcare management application.

[0056] Turn now Figure 3An example of an AI model training system 300 is shown. The AI ​​model training system 300 described herein is exemplary in nature and can be applied to a first AI model (e.g., LLM) or a second AI model (e.g., a base model) described herein. Furthermore, in some examples, the disease velocity representation system 102 described above may include, for example, a first training system for LLM and a second training system for the base model stored in training module 110. For example, the LLM may be a conversational model trained to collect patient medical data, extract multimodal data from the patient medical data, and generate textual responses to user queries based on the extracted multimodal data. The base model may be trained to collect various inputs, such as imaging data retrieved from an imaging database (e.g., PACS), user queries, and outputs from the LLM, to identify relevant images, segment lesions from images, and longitudinally track lesions within images across multiple time points. The AI ​​model training system 300 may be derived from a disease velocity representation system (such as a system for representing disease velocity). Figure 1 The disease velocity representation system 102 described herein is implemented to train one or both of the LLM and the base model to perform the functions described above.

[0057] AI model 302 can be stored within the AI ​​module 301 of the disease velocity representation system. For example, when AI model 302 is an LLM, AI model 302 can be stored in... Figure 1 Within the described LLM module 108, the AI ​​model training system 300 also includes a training module 304, which includes a training dataset comprising multiple training pairs divided into training pairs 306 and test pairs 308. The training module 304 can be... Figure 1 This is a non-limiting example of the training module 110 of the disease velocity representation system 102. As an example, when the training system 300 is configured to train an LLM, the training pair 306 and test pair 308 can be text-based data pairs in various formats, including text data, laboratory data, etc. The training data can be specially selected medical data designed to train the LLM to collect various types of medical data, including laboratory results, physician reports, medication orders (e.g., prescriptions), radiology reports, etc., and to generate outputs based on this medical data. When the training system 300 is configured to train a base model, the training pair 306 and test pair 308 can be image data pairs, such as specially selected medical image data and / or natural images designed to train the base model to collect imaging data, identify data related to inputs (e.g., outputs from the LLM and / or user queries), segmentation-related data, and measure segmented lesions.

[0058] Multiple training pairs 306 and test pairs 308 can be selected to ensure sufficient training data is available to prevent overfitting, thereby the AI ​​model 302 will learn features specific to samples in the training set that are not present in the test set.

[0059] Each training pair 306 and test pair 308 includes an input and a target. The AI ​​model training system 300 may include a training data generator 310, which can be used to generate the training pairs 306 and test pairs 308 for the training module 304.

[0060] Once each pair is generated, it can be assigned to either training pair 306 or test pair 308. In one implementation, the pair can be randomly assigned to either training pair 306 or test pair 308 in a pre-established proportion. For example, the pair can be randomly assigned to either training pair 306 or test pair 308 such that 90% of the generated pairs are assigned to training pair 306 and 10% are assigned to test pair 308. Alternatively, the pair can be randomly assigned to either training pair 306 or test pair 308 such that 85% of the generated pairs are assigned to training pair 306 and 15% are assigned to test pair 308. It should be understood that the examples provided herein are for illustrative purposes, and these pairs can be assigned to the training pair 306 dataset or the test pair 308 dataset via different processes and / or in different proportions without departing from the scope of this disclosure.

[0061] AI model training system 300 may include a validator 320 that verifies the performance of AI model 302 against test pairs 308. Validator 320 may take a partially trained AI model 302 and a dataset of test pairs 308 as input, and may output an evaluation of the performance of the partially trained AI model 302 against the dataset of test pairs 308.

[0062] Once AI model 302 has been validated, the trained AI model 322 (e.g., validated AI model 302) can be used based on data from one or more databases 330 (which may be...). Figure 1 The system uses data obtained from one or more data repositories 132 and user queries 331 to generate output, which can be entered into a healthcare management application via a GUI, as described above. In some examples, a trained AI model 322 may be stored within the inference module 321 of the disease velocity representation system.

[0063] Based on such as... Figure 3The described training process utilizes a novel multi-stage training procedure optimized for medical context understanding and disease progression analysis to specifically train the LLM. The training process includes: 1) pre-training on a sifted dataset of millions of medical documents to learn medical terminology and relationships; 2) fine-tuning on paired examples of patient records and expert-generated disease progression summaries; 3) further fine-tuning using reinforcement learning with expert feedback to optimize for accurate disease velocity assessment; and 4) implementing a specially weighted attention mechanism to identify temporal relationships and progression indicators in the medical data.

[0064] In a similar manner, based on, for example, regarding Figure 3 The training process described, in which the base model (e.g., DINO-SAM) undergoes specialized training, includes, as a non-limiting example, 1) transferring learning from natural images to the medical imaging domain using domain adaptation techniques; 2) training on a dataset of anatomically labeled medical images to learn anatomical partitions; 3) integrating a medical knowledge graph during training to establish relationships between imaging examination results and clinical symptoms; and 4) implementing custom loss functions optimized for accurate lesion boundary detection and measurement consistency across time points.

[0065] Turn now Figure 4 The diagram illustrates a high-level flowchart of a method 400 for determining a disease velocity representation via one or more AI models. Method 400 can be derived from a disease velocity representation system (such as...). Figure 1 The disease rate representation system 102) is executed by its processor. Some operations of method 400 may be stored in the non-transitory memory of the disease rate representation system (e.g., Figure 1 In the LLM module 108, the identification module 114, and the segmentation module 116), and by the processor of the disease speed representation system (e.g., Figure 1 The processor 104) executes.

[0066] At 402, method 400 includes receiving one or more user-input queries. As described above, clinicians or other users can input queries about a given patient into the GUI of a healthcare management application. The following will discuss... Figure 11 A more detailed description of the exemplary GUI follows. A first AI model (such as a conversational LLM) can be operatively coupled to a healthcare management application and can receive input user queries. Input user queries can be requests for summaries of data, timelines of data, impressions or insights about a patient's disease progression, requests for analysis of the progression of a specific lesion, insights about a patient's response to treatment, or other query types related to insights sought by clinicians regarding the rate of disease progression in a patient. One or more user input queries about a given patient can be entered.

[0067] At 404, method 400 includes obtaining patient data specific to the patient specified in a user query. For example, one or more data repositories may store patient data, including patient demographics, current medications, provider prescriptions, laboratory results, imaging reports, medical images, provider consultation records, etc. Each entry stored in one or more data repositories (e.g., EMR, laboratory systems, etc.) may be date-stamped. Patient data may be obtained by a first AI model (e.g., LLM) in one or more data formats, and the disease velocity representation system may, if needed, convert the data formats of the various obtained data to a common form that can be used by the disease velocity representation system. In some examples, the LLM may receive data from one or more data repositories for multiple patients and then, in response to a user query about a patient, view patient-specific patient data.

[0068] At 406, method 400 includes generating one or more responses representing disease progression to one or more user-input queries based on the obtained patient data. In some examples, the LLM may receive a user query and determine the type of response requested by the user query, as noted at 408. Specifically, the LLM may determine whether to utilize a base model based on the type of request in the user query. For example, when the user query includes a request for a text-based response (such as a summary, list, or timeline), the LLM may extract patient-specific multimodal data, as noted at 410, and may generate a text-based response without utilizing a second AI model (e.g., the base model), as noted at 412. Furthermore, certain types of disease processes (such as blood-based cancers or other hematological conditions) can be monitored using laboratory data. Laboratory data can be presented in text form, and therefore, for user queries about such disease processes, the LLM may generate a text-based response without utilizing a base model.

[0069] In other examples, when a user query includes a request to display non-text-based data (such as imaging data), the LLM may output a text-based portion of the response output, and additionally, utilize the base model to identify relevant patient data (e.g., relevant imaging slices) and process the patient data (e.g., segmenting lesions of interest, tracking the size of segmented lesions at multiple time points) to generate a non-text-based portion of the response output, as indicated at 414. The non-text-based portion of the response output generated by the base model may include relevant imaging slices (annotated with or without segmentation details) and / or size-tracked trend information (e.g., in the form of a graph).

[0070] At 416, method 400 includes determining disease velocity based on output. For example, based on quantitative longitudinal data output from the base model, disease velocity labels such as “stable,” “slow progression,” “slow regression,” “rapid progression,” or “rapid regression” can be assigned to patients and displayed within the GUI of a healthcare management application.

[0071] Turn now Figure 5 The diagram illustrates a flowchart of an exemplary method 500 for generating and displaying a representation of disease velocity using one or more AI models in response to a user query. Method 500 can be used for... Figure 1 The described use case scenario for the disease rate representation system is an example method and therefore should not be interpreted as the only specific implementation scenario of the system. Some operations of method 500 may be stored in the non-transitory memory of the disease rate representation system (e.g., Figure 1 In the LLM module 108, the identification module 114, and the segmentation module 116), and by the processor of the disease speed representation system (e.g., Figure 1 The processor 104) executes.

[0072] At 502, method 500 includes receiving a first user query of the first type. As described above, a clinician or other user can enter the query into the GUI of a healthcare management application. The following will discuss... Figure 11 An exemplary GUI is described in more detail. A first AI model (such as a conversational LLM) can be operatively coupled to a healthcare management application and can receive input user queries. Input user queries can be requests for summaries of data, timelines of data, impressions or insights about a patient's disease progression, requests for analysis of the progression of a specific lesion, insights about a patient's response to treatment, or other query types related to insights sought by clinicians regarding the rate of a patient's disease progression. In the scenario presented in method 500, the first user query can be a first type of query (where the response will be text-based (e.g., summary, timeline, etc.)), and therefore, upon receiving this first type of query, the LLM can determine that it does not need to utilize the base model. A first user query about a given patient can be input.

[0073] At 504, method 500 includes obtaining patient data from one or more data repositories. The patient data may correspond to a given patient against whom a first user query is input. In some examples, the patient data may be obtained by the LLM prior to receiving a query (e.g., while the LLM is continuously receiving data from one or more data repositories), and the data may be specifically parsed in response to the query to obtain data specific to a given patient. For example, one or more data repositories may store patient data, including patient demographics, current medications, provider prescriptions, laboratory results, imaging reports, medical images, provider consultation records, etc. Each entry stored in one or more data repositories (e.g., EMR, laboratory systems, etc.) may be date-stamped. The patient data may be obtained by the LLM in one or more data formats, and if necessary, the disease rate representation system may convert the data formats of the various obtained data to a common form that can be used by the disease rate representation system.

[0074] At 506, method 500 includes extracting clinically structured multimodal data from patient data based on a first user query. The clinically structured multimodal data may be a subset of the patient data obtained in relation to the user query. For example, the patient data may include all patient data from one or more data repositories for a given patient. However, the first user query may be specific to a particular disease process (e.g., diagnosis of cancer or neurodegenerative disease), and some patient data within the patient data (such as data obtained before the patient was diagnosed with a specific disease process) may be irrelevant to the user query. Multimodal data may refer to data that includes multiple types, such as imaging reports, laboratory results, provider annotations, different types of imaging data (e.g., MRI vs. CT vs. PET vs. X-ray, etc.). As noted above, extracting this multimodal data may include selective data reformatting, where the disease velocity representation system can detect the data format of the received data and, if necessary, convert the data into a format usable by the system.

[0075] At 508, method 500 includes generating a first output in response to a first user query. As described above, the LLM can be trained to collect patient data and user queries, and output a response to the user query based on the patient data. Therefore, the first output can be generated by the LLM as a response to the first user query. In some examples, the first user query can specify a particular text format to be presented in the output, such as XML format, and / or a form of response, such as a timeline, summary, list, etc. Therefore, the LLM can output its response in the requested text format and in the requested form.

[0076] In some examples, the first user query can be a query in which the LLM is able to output a text-based response, and thus patient data and queries can be processed without leveraging another AI model (e.g., a base model). For example, the first user query could be a request to the LLM to create a timeline of progression for a specified lesion of interest. As will be noted below at 518, the first output can be displayed as text output within a GUI.

[0077] At 510, method 500 determines whether a second user query of the second type has been received. In some examples, the second type of user query may indicate a request for non-text-based data. In some examples, the second user query may be entered by the user into the same GUI as the first user query. For example, in the scenario discussed herein, the LLM may output a first response as discussed above, and in response, the user may enter a second user query as a follow-up more specific to the disease process or lesion of interest included in the first output. If a second user query is received, method 500 proceeds to 512. If no second user query is received, method 500 proceeds to 518.

[0078] At 512, method 500 includes identifying a subset of data from the multimodal data that is relevant to the second user query. For example, as with regard to... Figures 6A to 6B Further, the second user query could be a request to display the disease process or the progression of a lesion of interest. In an example of displaying the disease process in imaging data as described herein, the LLM can therefore use its first output or generate a second text output, and can leverage the base model for the purpose of identifying relevant imaging slices and segmenting those slices to track the progression of lesions of interest within them. For example, the base model can take the second user query, the output from the LLM (either the first output or the second LLM output if the second query is significantly different), multimodal data, and imaging data obtained from an imaging database (e.g., PACS) as input. The base model can determine the relevant imaging protocol based on the symptom data presented in the first output and / or imaging report, and can determine the relevant image slices based on the identified anatomical partitioning information. In the case of combining these two sets of information, the base model can identify a subset of the data, such as a set of relevant imaging slices.

[0079] At 514, method 500 includes generating a second output in response to a second user query. In some examples, the second output may be a non-text-based response. Similarly, as with... Figures 6A to 6BFurther, the second output may be a set of relevant imaging slices. In some cases, as noted at 516, the set of relevant imaging slices may be segmented, and the size of lesions within them may be measured and tracked across multiple time points. In some examples, the segmentation and measurement of lesion size may be performed automatically by the base model, or in other examples in response to user input selecting displayed image slices, as will be further described below. In some examples, the second output may additionally include a disease progression trend presented in a graphical format.

[0080] At 518, method 500 includes displaying a first output and / or a second output within a GUI. In some examples, the first output may be automatically displayed within the GUI after its generation, prior to receiving a second user query. Then, upon receiving the second user query, the LLM may generate a second LLM output, and the base model may generate its output, both of which can be displayed in the GUI as described above. For example, the first output may be displayed in a designated area of ​​the GUI where the user query is entered when the output is generated (e.g., before the second query is entered). In the example where a second query is entered, the second output may also be displayed within the same GUI where the query is entered. For example, an image may be displayed in a pop-up UI within the GUI. Furthermore, the first and / or second outputs can be used, for example, to generate a representation of disease velocity using labels or colors as previously discussed.

[0081] It should be understood that, regarding Figure 5 The examples described are non-limiting. For instance, in some instances, the first user query may include a request that the LLM must make in response to its utilization of the base model. As an example, the first user query may be a request to show the progression of a lesion of interest in imaging data. In such an example, the LLM may output a text response similar to the first output described above, and the base model may simultaneously output a subset of imaging slices similar to the second output described above, based on the text response and the imaging data, as will be further described below.

[0082] Furthermore, it should be understood that disease velocity representation systems are not limited to imaging data. For example, longitudinal laboratory data can also be presented graphically. The data types identified and presented can be based on user queries and patient data for a given patient. For example, a patient with metastatic liver cancer that has metastasized to the brain may have imaging data most relevant to disease progression and response to treatment, while a patient with acute myeloid leukemia may have laboratory data most relevant to disease progression and response to treatment. Therefore, for patients with imaging data relevant to a given disease process, the LLM can utilize the base model as described herein. Conversely, for patients without imaging data relevant to disease progression (e.g., only relevant laboratory data), the LLM can generate a text-based graph or trend of the response and laboratory data without utilizing the base model. In this way, the LLM can act as a gatekeeper, thereby improving overall system efficiency by utilizing only a second AI model when necessary, based on the determination of the request type and relevance type.

[0083] In this way, the disease velocity representation system can receive user queries and output responses in a conversational manner via implementations of one or more AI models, including LLM and / or base models as described herein. Such responses reduce the time users spend manually searching various EMRs for fragmented medical data and improve overall accuracy. Furthermore, the system can be configured to respond to queries in a computationally efficient manner, where the base model is used for queries that require it, rather than for queries for which the LLM itself can generate a response. Additionally, as will be further described below, the base model reduces the overall processing power required for accessing, displaying, and segmenting medical images.

[0084] Figure 6A and Figure 6B A flowchart illustrating a method for identifying a subset of relevant medical images using a base model utilized by an LLM in response to a user query is shown. Method 600 can be used for... Figure 1 The example method for the described disease rate representation system is as follows. Therefore, some operations of method 600 can be stored in the non-transitory memory of the disease rate representation system (e.g., Figure 1 In the LLM module 108, the identification module 114, and the segmentation module 116), and by the processor of the disease speed representation system (e.g., Figure 1 The processor 104) executes.

[0085] from Figure 6ABeginning at 602, method 600 includes obtaining LLM output in response to a received user query. As an example, the LLM may generate output in response to a user query, as discussed above. The LLM output may be a text-based output including information about relevant patient data. As a non-limiting example, the LLM output may be a list detailing the progression of the lesion of interest based on obtained patient data (e.g., imaging reports, provider notes, medical orders, etc.) in response to such a user query request.

[0086] At 604, method 600 includes processing the LLM output. Processing the LLM output may include identifying region information expressed in the LLM output, as indicated at 606, and identifying symptom information expressed in the LLM output, as indicated at 608.

[0087] For example, the information presented in the LLM output may include information about the location of a specific lesion (e.g., mass, tumor, plaque, etc.). As a non-limiting example, liver lesions, brain lesions, etc., can be identified in the LLM output by leaf, segment, and / or other location identifiers (such as anterior / posterior, inferior / superior, etc.). These region identifiers can be extracted from the LLM output.

[0088] Furthermore, the information presented in the LLM output may include information about the patient's condition or features, such as those seen in the imaging scan. As a non-limiting example, conditions such as enhancement, mass effect, edema, etc., may be identified within the LLM output. The system can then use these identified features to identify relevant imaging protocols, as will be described further, based on a predefined lookup dictionary or other algorithms applied to the LLM output.

[0089] At 610, method 600 includes identifying relevant imaging protocols based on symptom information. For example, enhancing lesions may be best seen on T2-FLAIR imaging rather than other types of imaging, and therefore T2-FLAIR imaging may be identified as relevant. Specifically, the system may be equipped with an algorithm, lookup table, or model by which the identified symptom information is extracted from the LLM output.

[0090] At 612, method 600 includes matching region information extracted from the LLM output with template anatomical regions. The template anatomical regions may be stored in a dictionary similar to a lookup table. Each template anatomical region may be linked in the dictionary to relevant imaging slices from various medical imaging protocols. The system may process the region information from the LLM output to match each fragment of region information with one template anatomical region from the template anatomical regions.

[0091] At 614, method 600 includes locating regions in the protocol body data and identifying associated slices. As noted, each template anatomical region may be linked to an associated imaging slice. The template anatomical region may include a partitioned case that may include all partitions associated with a specific region of the body, such as the brain, liver, etc. The system can then use this information as a template to identify where the identified region extracted from the LLM output is located within a specific set of data.

[0092] At 616, method 600 includes determining a subset of imaging data relevant to a user query from the acquired imaging data. For example, a base model may be coupled to an imaging database, such as a PACS, and from which volumetric data can be obtained. Based on region information matching, the base model may identify image slices of volumetric data stored in the imaging database that are associated with the extracted region information and correspond to the identified relevant imaging protocol. In some examples, a single image slice from each volumetric image relevant to the user query may be identified, for example, a slice cut through the largest size of the lesion included in the volumetric image.

[0093] As an example, for a lesion of interest in the left temporal lobe of the brain, all imaging data that does not belong to the brain can be filtered out, brain imaging slices from the right half of the brain can be filtered out, and so on, continuing to identify the set of slices associated with the lesion of interest in the left temporal lobe within the relevant imaging protocol. This filtering can be based on where the identified anatomical partitions (e.g., extracted from LLM output) are located in the imaging data obtained from PACS, specifically, which image slices correspond to the identified anatomical partitions / regions. In some examples, such as when a patient has undergone multiple imaging scans of different types over time (e.g., CT, MRI, PET scans, X-rays, etc.), the slice set may include slices from multiple imaging bodies from multiple time points, and therefore may be multimodal imaging data.

[0094] In some examples, relevant imaging protocols can be identified in parallel (e.g., concurrently) with the identification of relevant imaging slices. In other examples, as presented herein, relevant imaging protocols can be identified before or after the identification of relevant imaging slices. For example, relevant imaging protocols can be identified first, and thus relevant imaging slices can be identified from a reduced library of relevant imaging protocols, thereby reducing overall image processing.

[0095] In this way, the identified subset of imaging data may include imaging data relevant to a user query. For example, when the user query is a request related to the disease process (rather than a specific lesion of interest), the subset of imaging data may include image slices of one or more lesions in one or more anatomical regions. For example, a patient with metastatic melanoma may have multiple brain metastases. The user query could be a request to show the progression of the patient's disease, and in response, the base model may identify a subset of image slices of relevant protocols, wherein this subset includes at least one slice for each of the multiple brain metastases. As another example, when the user query is a request related to a specific lesion of interest, the subset of imaging data may include image slices that include that specific lesion of interest. For example, for a patient with metastatic melanoma having multiple brain metastases, the user query could be a request to show the progression of a right parietal lobe metastasis, and therefore the subset of imaging data may include one or more image slices from relevant protocols for that right parietal lobe metastasis.

[0096] Continue to Figure 6B At 618, method 600 includes tracking the lesion of interest across time based on a subset of imaging data. As an example, as described above, the subset of imaging data may include a subset of image slices containing the lesion of interest. Tracking the lesion of interest across time may include segmenting the lesion in each image slice of the slice set, as indicated at 620. For example, a base model may be trained to segment the lesion of interest from each slice in the slices, for example, based on edge identification, thresholding, or a segmentation-all model. The segmented lesion of interest in each slice in the slices may be measured at its widest dimension, as indicated at 622. Since the slice set includes slices of imaging data from multiple time points, measurements of the segmented lesions can be performed across multiple time points, thereby tracking the lesion across time.

[0097] At 624, method 600 includes generating an output representing disease velocity based on a measurement. For example, a graph representing the trend of the size of the lesion of interest over time can be generated based on the measurement. Furthermore, a disease velocity label can be determined based on a trend. For example, an overall trend, recent changes in lesion size (e.g., between the two most recent time points), and other types of statistics can be used to determine disease velocity labels such as “stable,” “slow progression,” “slow regression,” “rapid progression,” and “rapid regression.”

[0098] At 626, method 600 includes displaying output and / or annotated slices (e.g., with segmentation annotations) on a display device. For example, output representing disease velocity and annotated slices may be displayed within the GUI of a healthcare management application. As an example, the output may be displayed in a first overlay or pop-up UI, and the annotated slices may be displayed in the same or separate overlays or pop-up UIs.

[0099] In some examples, the base model can automatically segment image slices and track lesions of interest across time to generate output. In other examples, the base model can first display these image slices, for example, in an overlay or pop-up UI in a GUI, before segmenting a subset of the image slices (unsegmented and unannotated), and then segment the image slices and track lesions of interest across time in response to a user selection of one of the displayed unsegmented image slices.

[0100] As an example, when a user query includes a request for information about overall disease progression (rather than specifying a particular lesion of interest), the base model can first output unsegmented, identified image slices for the user to view subsequently. The user can then select one slice from the slices, thereby (e.g., by selecting a specific region of interest within the selected slice) indicating a lesion of interest included within the selected slice. In response to receiving the user's selection of the selected slice, the LLM can then identify a subset of the image slices that includes the selected slice and a subset of other slices that include the same lesion of interest. The base model can then segment the subset of image slices and track the size of the lesion of interest over time. Conversely, when a user query includes a request indicating a specific lesion of interest, the base model can automatically proceed to segmenting slices containing the corresponding region of interest and track the lesion over time as indicated by user input. In such examples, the subset of image slices includes only slices that include the lesion of interest indicated by the user query.

[0101] Therefore, as presented above, the method in this paper reduces the processing demands on the system and the computing device running on it by first identifying relevant imaging protocols and relevant imaging slices to generate an image slice set. Thus, instead of processing (e.g., segmentation and / or measurement) irrelevant or repetitive imaging data, the amount of data that must be processed is reduced by narrowing down the data to include only relevant image slices (e.g., a single slice from each relevant image volume (e.g., the widest slice through which the lesion is cut)). Furthermore, the amount of data ultimately presented to the user in the GUI is also reduced. For example, in the current workflow, a clinician might open an imaging database, locate a patient, and manually filter through multiple image volumes, scroll through slices to find the most relevant ones, and then perform processes such as segmentation and measurement algorithms on each selected slice. This is not only time-consuming for the clinician, but accessing, displaying, and processing all the image volumes clicked by the user also increases the amount of processing power used by the user's device. The disease velocity representation system disclosed herein reduces this processing capacity by identifying relevant slices and then segmenting and measuring lesions only in those relevant slices or a subset of those relevant slices (e.g., in an example where segmentation and size tracking are performed in response to user selection).

[0102] To reiterate, the disease velocity representation system described in this paper implements a novel hierarchical filtering architecture that significantly reduces computational overhead compared to conventional methods. Specifically: 1) The LLM uses an attention-based pre-filtering mechanism to identify relevant segments of patient records before full processing; 2) The base model employs a cascaded inference approach, where initial low-resolution processing identifies regions of interest before applying computationally intensive high-resolution analysis (e.g., segmentation analysis); 3) Integration between the LLM and the base model uses an efficient API that minimizes data transfer overhead; 4) A parallel processing pipeline allows for simultaneous natural language understanding and image analysis; and 5) a caching mechanism stores commonly accessed anatomical regions and intermediate results of measurement computations. These technological improvements reduce processing time by, for example, 60%–80% compared to sequential processing of all available patient data, while maintaining accuracy above 95% compared to manual expert analysis.

[0103] Turn now Figure 7A The flowchart 700 illustrates a first example of a user query and its corresponding LLM output. Specifically, the user can input user query 702 into a healthcare management application (such as...) Figure 1 In the GUI of a healthcare management application (130), user query 702 may include a request for a specific format for the response to be given. For example, user query 702 may specify a request such as "list all lesions detected for the patient in XML structured format".

[0104] The 704 response can be generated by the LLM. In some examples, the 704 response can be generated by the LLM based on the obtained patient data and the specific request in the user query 702. For example, the 704 response can list the identified lesions in XML format for a specified patient as requested.

[0105] Figure 7B A second example of a user query and its corresponding LLM output is shown in flowchart 710. Similar to user query 702, user query 712 can be entered into the query search bar in the GUI of a healthcare management application. User query 712 may include a specific form of request for a response. For example, user query 712 may specify a request such as “Create a timeline of the progression of lesions in the left temporal lobe in list form”.

[0106] The corresponding response 714 can be generated by LLM based on user query 712 and multimodal data extracted from patient data obtained from EMR. Figure 7B In the specific example shown, response 714 includes a numbered list detailing the dates and imaging results associated with the specified lesion in the left temporal lobe. This type of LLM output can reduce the time providers spend sifting through different data repositories, including both EMR and PACS, to find relevant information.

[0107] Furthermore, because responses 704 and 714 can be generated based on text data extracted from patient data obtained from a self-owned data repository (e.g., EMR), LLM can generate responses 704 and 714 without utilizing the base model. In this way, by including more than one AI model, each configured to respond to different user queries, as an example, the overall processing requirements of the system can be reduced by selectively utilizing the base model only when the user query includes a request involving the processing of medical images.

[0108] Turn now Figure 8A This document illustrates a sample process flow 800 for identifying relevant medical images from a PACS database based on responses generated by the LLM to queries. This document references information from... Figure 1 The components described are used to describe the process flow 800.

[0109] Specifically, the LLM can generate a first response 804 based on the obtained patient data 806 in response to user query 802. Specifically, in the example shown, user query 802 could be a request for examination results of a left temporal lobe mass. The first response 804 from the LLM could be text data detailing a list of examination results, including the imaging date and the imaging details regarding the left temporal lobe mass. This is because the query did not specify a particular list or text format (as described above). Figure 7A and Figure 7B The request (as shown in the XML) allows LLM to also utilize the base model.

[0110] For example, a first response 804 can be fed into a base model. The base model can acquire imaging data from an imaging database 808 (e.g., a PACS 808). The base model can identify relevant images from the image set obtained from the PACS 808 based on the first response 804. For example, as per [reference to...] Figures 6A to 6B The area and type of lesion (including the presence of edema or enhancement) described can be used by the base model to identify relevant slices from imaging scans.

[0111] In the provided example, the lesion of interest is a left temporal lobe mass with identified surrounding edema and localized mass effect. Based on this information, the base model can identify a subset of the most relevant image slices, including slices cut through the left temporal lobe (e.g., but not slices cut through the cerebellum or other regions unrelated to the lesion of interest) with one or more relevant sequences (such as T2-Flair sequences and / or T1-contrast-enhanced sequences), but excluding other less relevant sequences (such as diffusion-weighted imaging sequences). In this way, a subset of image slices 810 relevant to a specific user query 802 can be identified for further processing, such as segmentation. Therefore, processing efficiency can be improved by reducing the amount of data processed to generate the final output. In this instance, the output may be a second response, which includes a set of segmented image slices with or without annotations, a list of examination results (e.g., as in the first response 804), and / or disease velocity labels.

[0112] Figure 8B An example of multiple images 812 that can be stored in a PACS 808 is shown. As illustrated, the multiple images 812 may include images of various body parts, including head images, torso images, etc. Therefore, only a subset of the multiple images 812 may be associated with the identified lesion of interest. Thus, a subset of image slices 810 may be a selected group of slices of the multiple images 812 determined by the underlying model to be associated with the lesion of interest as described herein.

[0113] Turn now Figure 9 This illustrates the process flow 900 for identifying relevant medical images via LLM and a base model, as presented herein. (Reference) Figure 9 One or more implementation schemes described above can be accessed via the above-mentioned... Figure 1 One or more components presented and about Figure 4 This is achieved through the method described in Figure 6. For the sake of brevity, repeated descriptions of similar components and / or processes used in the corresponding implementations have been omitted.

[0114] In the resulting first response 902, the LLM can identify features or conditions within the imaging based on the imaging report obtained as text data, such as, for example, mass effect, enhancement, dehydration, etc., for a patient. In some examples, the first response 902 may be only a portion of the response given by the LLM. In some examples, for each identified feature or condition, an imaging protocol dictionary 906 may exist, which may resemble a lookup table. The imaging protocol dictionary 906 can be used to determine the appropriate imaging protocol to display certain images, as illustrated at 904. For example, the imaging protocol dictionary 906 can be used to determine which images best display edema via T2-FLAIR weighted images, or which images associated with the words 'enhancement' or 'mass' appearing in the first response 902 generated by the LLM will be displayed as contrast-enhanced images, or which images will display solid components via PET data, etc. While this document presents a lookup-style approach for matching, it should be understood that other methods, such as regression models, AI models, etc., may also be used without departing from the scope of this disclosure. A base model such as DINO-v2 can select the appropriate imaging protocol for the appropriate image based on a priority order. For example, for solid components, PET data may be prioritized and images may be displayed via PET, diffusion-weighted imaging (DWI), or perfusion-weighted imaging (PWI) (if PET data is unavailable), or dynamic contrast enhancement (DCE) (if DWI and PWI are unavailable). By matching information from the first response 902 with an imaging protocol dictionary (or other algorithm), the underlying model can identify relevant protocol imaging series 910 from available imaging sequences obtained from PACS.

[0115] Before, after, or in parallel with matching a symptom to a protocol, the base model can utilize an anatomical partitioning template image library 912 to extract relevant slices based on the anatomical partitioning information in the first response 902. For example, the anatomical partitioning template image library 912 may include partitioned cases that contain all partitions associated with other anatomical regions of the brain, liver, or body. The base model can use this template to identify where the partitions are located on new data (e.g., imaging data obtained from PACS). The first response 902 includes anatomical partitioning information 908, such as the left inferior temporal lobe, left occipital lobe, occipital horn of the left ventricle, hepatic segment VI, and right posteromedial parietal lobe. The left inferior temporal lobe may indicate the temporal lobe of the brain, and the base model can automatically partition the left inferior temporal lobe on the image for the user based on the anatomical partitioning information 908. In this way, the base model can identify partitions 914 corresponding to specific slices that are associated with the anatomical partitioning information presented in the first response 902 from the LLM.

[0116] Therefore, the base model can have information about the relevant imaging protocol and the relevant imaging slices, which can be incorporated and applied to images obtained from the PACS to identify a subset of the relevant image slices 916. In this way, the base model can locate the ROI in the image slice based on the first response 902 for segmentation, and the ROI can be coupled to identify the requested region from the anatomical partition template.

[0117] Subsequently, the most relevant image slices can be presented to clinicians via a GUI, allowing them to scroll and observe the images, and in some instances, segment the images. In some examples, the base model can proceed to segmenting only the images selected by the user via the GUI. In other examples, the base model can automatically segment each identified image slice within the same partition in response to the identification of the image slice or in response to the selection of an image slice within a partition. For example, segmentation can be performed on all image slices of the left temporal lobe, either automatically or in response to a user selection of an image slice containing data on a left temporal lobe mass, when the base model identifies a subset of relevant image slices. Thus, the base model can segment image slices and measure the size of segmented lesions at multiple time points, generating segmented and measured image slices 918.

[0118] As a non-limiting example, a user query could be a request to display the progression of a mass in the left temporal lobe. The LLM can generate a first response similar to first response 902, which includes symptom information and anatomical partitioning information related to the lesion of interest (e.g., a mass in the left temporal lobe). The symptom information can be processed via the base model to determine relevant protocols, and the anatomical partitioning information can be processed via an anatomical partitioning template image library to identify relevant image slices. This information can be combined to identify subsets of image slices corresponding to relevant image slices and relevant image protocols. Given that the user query requests the display of the progression of a specific lesion of interest, the base model can automatically segment the corresponding lesion in the left temporal lobe from the subset of image slices and present the segmented lesion and its tracked measurements to the user. For example, the base model can segment the lesion at multiple time points, such as those within different image slices included in the subset of relevant image slices. The segmented lesion can be measured at multiple time points to determine quantitative disease progression.

[0119] As another non-limiting example, a user query could be a request to display the overall progression of a disease. The LLM can generate a first response similar to first response 902, which includes disease-related symptom information and anatomical partitioning information, potentially including information on multiple lesions in multiple regions of the body. The symptom information can be processed via the base model to determine relevant protocols, and the anatomical partitioning information can be processed via an anatomical partitioning template image library to identify relevant image slices. This information can be combined to identify subsets of image slices corresponding to relevant image slices and relevant image protocols. Because the user query does not specify a particular lesion of interest, a subset of image slices can be presented to the user as is via the GUI, and then, in response to user input receiving a selection of an image slice or a lesion within an image slice, the base model can be used to segment the selected image slice, and in some examples, segment other image slices including the same lesion at different time points. The segmented lesions can be measured at multiple time points to determine quantitative disease progression.

[0120] Figure 10 It shows Figure 9 The process flow 900 includes an example section 1000 for identifying relevant medical images by combining a base model with LLM responses. Specifically, Figure 10 An image corresponding to an anatomical partition template image library 912 is illustrated, which can be used by the base model to identify partitions 914. Based on the identified partitions 914 and the identified relevant imaging protocols, and based on the displayed symptoms / features identified in the LLM response, relevant image slices 916 can be identified. Based on the content corresponding to the user query, the base model can automatically or responsively (e.g., in response to user input) segment lesions from the relevant image slices 916 to output segmented and measured image slices 918.

[0121] As described above, measurements of image slices at multiple time points can be processed to determine disease velocity. For example, if measurements indicate a significant decrease in the size of a given lesion, a disease velocity label of "rapid regression" can be assigned to the lesion and / or disease process. Conversely, if measurements indicate no significant change in the size of a given lesion, a disease velocity label of "stable" or "dormant" can be assigned to the lesion and / or disease process. Disease velocity labels for lesions or disease processes can be determined based on lookup tables, regression models, AI models (e.g., deep neural networks or other machine learning algorithms), which can collect various measurements of segmented lesions and determine trends. Additionally, measurements can be processed to be presented in graphs or charts, which will be displayed in a GUI along with the segmented image slices.

[0122] Turn now Figure 11The example GUI 1100 is shown. GUI 1100 can be configured to present healthcare management applications (such as...) Figure 1 The information in the healthcare management application 130). The GUI 1100 may include a data presentation section 1102, which may include one or more panels, timelines, etc., for displaying aggregated acquired medical patient data. Elements within the panels, timelines, etc., may be selectable to launch pop-ups to be displayed on a portion of the data presentation section 1102, such as Figure 11 As shown.

[0123] The GUI 1100 may additionally include a query section 1104. The query section 1104 may include a user query line, which includes a query search bar 1106 where the user can enter (e.g., type) a query, and an initiating query element 1108 that, when selected, triggers the LLM to retrieve data from the data store, extract multimodal data from that data, and generate a response to the query.

[0124] Query segment 1104 may additionally include an LLM output line, which includes a display element 1110 in which the response to the query generated by the LLM can be displayed. Furthermore, query segment 1104 may include a run model element 1112, which, when selected, triggers the LLM to utilize the base model. In response to the base model identifying one or more image slices (segmented or unsegmented), those image slices, along with tracking information and disease velocity labels, can be displayed in a pop-up window on data presentation segment 1102.

[0125] The GUI described herein and the computing device on which the GUI is rendered and displayed can be configured to implement an interactive visualization framework capable of handling the real-time display of high-resolution medical images (e.g., as output by a base model), implementing procedures for displaying annotated images (e.g., after segmentation and measurement), and employing progressive loading techniques to maintain interface responsiveness when processing large datasets.

[0126] Figure 12Example outputs that can be displayed within GUI 1100 are shown. For example, text output 1200 generated by the LLM in response to a user query can be displayed within display element 1110. Segmented image slices 1202 can also be displayed within GUI 1100, for example, based on text output 1200 and the corresponding user query as described above. Based on measured segmented lesion data tracked at multiple time points, a graph 1204 showing the size trend can also be displayed within GUI 1100. In this way, users can visualize the presented disease velocity representation in various ways, including in text form, as in text output 1200; in image form, as in segmented image slices 1202; and / or in graphical form, as in graph 1204. Such information not only reduces the time spent by users searching for scattered information when analyzing patients but also aids in clinical decision-making. For example, based on presented textual data indicating the rate of disease progression and the treatments that have been administered so far (e.g., surgical resection, different drugs / chemotherapeutic agents, radiation, etc.), clinicians can more easily determine the next step in a patient's treatment process.

[0127] The technical benefit of the disease velocity representation system and method disclosed herein is that users can easily visualize the progression and velocity of a disease by inputting a query. The query is input into the LLM, and based on the content of the query, the LLM can generate text-based output based on identified relevant patient data. In some examples, the LLM can utilize a base model capable of identifying relevant patient data (e.g., relevant image slices) and processing that relevant patient data (e.g., segmenting and size tracking of lesions). Based on the output from the LLM and / or the base model, disease velocity labels can be assigned to the patient's disease process. Specifically, the LLM can obtain dispersed patient data from multiple data repositories (e.g., one or more EMRs, laboratory systems, medical devices, imaging databases, etc.) and aggregate this data in a single response output in the requested format. Furthermore, for certain user queries, the LLM can utilize a base model that identifies a subset of relevant image slices based on symptom information and anatomical partitioning information extracted from the LLM output. In this way, the amount of information accessed, displayed, and processed by the user is reduced because the base model filters out irrelevant data and processes and displays relevant information. Visualizing disease velocity, for example through disease velocity tags or trend graphs of lesion size over time, can reduce the time users spend searching patient medical data to understand the progression of a patient's disease.

[0128] This disclosure also provides support for a disease velocity characterization system, comprising: one or more processors; a memory storing instructions executable by the one or more processors, which, when executed, cause the one or more processors to: receive a user query related to a patient; obtain patient data from one or more data repositories; generate a first response output to the user query based on the patient data by one or more artificial intelligence (AI) models, wherein the first response output represents the disease velocity; and output the first response output to a display device. In a first example of the system, when the user query is related to text-based data, the first response output is generated by a first AI model among one or more AI models. In a second example of the system (optionally including the first example), the first AI model is a large language model (LLM). In a third example of the system (optionally including one or both of the first and second examples), the system further includes: instructions stored in memory that, when a user query relates to both text-based and imaging-based data, can be executed by one or more processors to: generate a text-based response output to the user query based on patient data by a first AI model among one or more AI models; determine a set of image slices related to the user query by a second AI model among one or more AI models based on imaging data obtained from an imaging database and the text-based response output; and output the set of image slices to a display device, wherein the set of image slices is displayed in a graphical user interface (GUI) on the display device. In a fourth example of the system (optionally including one or more or each of the first to third examples), the system further includes: instructions stored in memory that, when executed, cause one or more processors, upon a user query specifying a lesion of interest, to: automatically segment each image slice in the set of image slices; and perform size tracking of the lesion of interest across time using the set of image slices. In a fifth example of the system (optionally including one or more or each of the first to fourth examples), the system further includes: instructions stored in memory that, when executed, cause one or more processors, when a user query does not specify a lesion of interest, to: receive a user selection for a selected image slice in a set of image slices, wherein the user selection indicates a lesion of interest; in response to the user selection for the selected image, segment the selected image and other image slices in the set of image slices that include the lesion of interest; perform size tracking of the lesion of interest across the set of image slices over time; generate a trend graph based on the size tracking of the lesion of interest; and output the segmented set of image slices and the trend graph for display within a GUI.In a sixth example of the system (optionally including one or more, or each of, the first through fifth examples), to determine a set of image slices, the second AI model is configured to: extract symptom information and anatomical partitioning information from a text-based response output; determine one or more relevant imaging protocols based on the symptom information; and determine one or more relevant image slices based on the anatomical partitioning information. In a seventh example of the system (optionally including one or more, or each of the first through sixth examples), to determine a set of image slices, the second AI model is further configured to merge the determined one or more relevant imaging protocols and the determined one or more relevant image slices to identify the set of image slices. In an eighth example of the system (optionally including one or more, or each of the first through seventh examples), the second AI model is a base model.

[0129] This disclosure also provides support for a method for a disease velocity representation system, the method comprising: receiving a first user query related to a patient from a healthcare management application; obtaining patient data of the patient from one or more data repositories; determining the query type of the first user query; in response to determining the query type, deploying one or more artificial intelligence (AI) models to generate a response output to the first user query; and displaying the response output in a graphical user interface (GUI) of the healthcare management application. In a first example of the method, when the query type is a first query type indicating a text-based response, deploying one or more AI models to generate the response output includes deploying a large language model (LLM), wherein the response output is a text-based response output. In a second example of the method (optionally including the first example), when the query type is a second query type indicating a non-text-based response, deploying one or more AI models to generate the response output includes: deploying an LLM to generate a text-based portion of the response output; and utilizing a base model to generate a non-text-based portion of the response output. In a third example of the method (optionally including one or both of the first and second examples), the non-text-based portion of generating the response output includes: obtaining volumetric data from an imaging database, wherein the volumetric data includes multimodal imaging data acquired across multiple time points; determining lesion information and anatomical partitioning information based on the text-based portion of the response output generated by LLM; identifying one or more relevant imaging protocols based on the lesion information; and identifying subsets of image slices from the obtained volumetric data based on the anatomical partitioning information and one or more relevant imaging protocols. In a fourth example of the method (optionally including one or more, or each of, the first to third examples), the non-text-based portion of generating the response output further includes, when a first user queries to identify a lesion of interest: automatically segmenting a subset of image slices; and tracking the size of the lesion of interest within the segmented subset of the image slices over time. In a fifth example of the method (optionally including one or more or each of the first to fourth examples), the non-text-based portion of generating the response output further includes, when the first user query does not identify a lesion of interest: displaying a subset of image slices within the GUI of a healthcare management application; receiving a user selection of a selected image slice within the subset of image slices, wherein the user selection identifies a lesion of interest within the selected image; in response to receiving the user selection, segmenting a subset of the subset of image slices, wherein the subset of the subset of image slices includes the selected image slice and other image slices within the subset of image slices that include the lesion of interest; and tracking the size of the lesion of interest within the subset of the subset of image slices over time.In a sixth example of the method (optionally including one or more or each of the first to fifth examples), the non-text-based portion of the response output includes one or more of unsegmented image slices, one or more segmented image slices, a trend graph, and a disease velocity label. In a seventh example of the method (optionally including one or more or each of the first to sixth examples), the method further includes: receiving a second user query from a healthcare management application as a follow-up to a first user query, wherein the response output to the first user query is a text-based response generated by an LLM of one or more AI models; in response to determining that the second user query indicates a non-text-based response, generating a second response output by the LLM based on the text-based response using a base model from one or more AI models, wherein the second response output is a non-text-based response; and displaying the response output and the second response output in a GUI of the healthcare management application, wherein the second response output includes one or more of unsegmented image slices, one or more segmented image slices, a trend graph, and a disease velocity label.

[0130] This disclosure also provides support for a method for representing disease velocity, the method comprising: receiving a first user query related to a patient; obtaining patient data of the patient from one or more electronic medical records (EMRs); in response to determining that the first user query indicates a text-based response, generating a first response to the first user query based on the patient data by a large language model (LLM); displaying the first response in a user interface (UI) on a display device, wherein the first response is a text-based response; receiving a second user query, wherein the second user query is a follow-up to the first user query; in response to determining that the second user query indicates a non-text-based response, generating a second response to the second user query by a base model based on the first response and imaging data obtained from an imaging database; and displaying the second response in the UI on the display device, wherein the second response includes one or more of an unsegmented image slice, one or more segmented image slices, a trend graph, and a disease velocity label. In a first example of the method, generating a second user query includes: determining symptom information and anatomical partitioning information based on a first response generated by the LLM; identifying one or more relevant imaging protocols based on the symptom information; identifying subsets of image slices from the imaging data based on the anatomical partitioning information and one or more relevant imaging protocols; when a user query identifies a lesion of interest: automatically segmenting the subset of image slices; tracking the size of the lesion of interest within the segmented subset of the image slice over time; and displaying the segmented subset of the image slice and a trend graph generated based on the size tracking within the UI; and when using... When a user query does not identify a lesion of interest: a subset of image slices is displayed within the UI; a user selection of a selected image slice within the subset of image slices is received, wherein the user selection identifies a lesion of interest within the selected image; in response to receiving the user selection, a subset of the subset of image slices is segmented, wherein the subset of the subset of image slices includes the selected image slice and other image slices within the subset of image slices that include the lesion of interest; the size of the lesion of interest within the subset of the subset of image slices is tracked over time; and the segmented subset of the subset of image slices and a trend graph generated based on the size tracking are displayed within the UI. In a second example of the method (optionally including the first example), the first user query and the second user query are received from a healthcare management application adapted to aggregate and display patient medical data.

[0131] As used herein, elements or steps listed in the singular and beginning with the word "a" or "an" should be understood to not exclude multiple said elements or steps unless such exclusion is explicitly stated. Furthermore, references to "an embodiment" of the invention are not intended to be construed as excluding the existence of additional embodiments that also include the referenced features. Moreover, unless explicitly stated to the contrary, embodiments that "comprise," "include," or "have" elements or multiple elements having a particular characteristic may include additional such elements that do not have that characteristic. The terms "comprise" and "in" are used as concise linguistic equivalents to the corresponding terms "comprising" and "wherein." Furthermore, the terms "first," "second," and "third," etc., are used merely as notations and are not intended to impose numerical requirements or a particular order of placement on their objects.

[0132] This written description uses examples to disclose the invention, including the best mode, and also enables those skilled in the art to practice the invention, including making and using any device or system and performing any included methods. The scope of patentability of the invention is defined by the claims, but may include other examples that would occur to those skilled in the art. Such other examples are intended to fall within the scope of the claims if they have structural elements that are not indistinguishable from the literal language of the claims, or if they include equivalent structural elements that have minor differences from the literal language of the claims.

Claims

1. A disease velocity representation system, the disease velocity representation system comprising: One or more processors; A memory that stores instructions executable by the one or more processors, the instructions, when executed, causing the one or more processors to: Receive user queries related to patients; Obtain the patient's patient data from one or more data repositories; One or more artificial intelligence (AI) models generate a first response output to the user query based on the patient data, wherein the first response output represents the disease speed; and The first response is output to a display device.

2. The disease velocity representation system according to claim 1, wherein when the user query is related to text-based data, the first response output is generated by a first AI model among the one or more AI models.

3. The disease velocity representation system according to claim 2, wherein the first AI model is a large language model (LLM).

4. The disease velocity representation system according to claim 1, further comprising instructions stored in memory, which, when the user query relates to both text-based data and image-based data, can be executed by the one or more processors to: The first AI model among the one or more AI models generates a text-based response output to the user query based on the patient data; The second AI model among the one or more AI models determines a set of image slices relevant to the user query based on imaging data obtained from an imaging database and the text-based response output; and The image slice set is output to the display device, wherein the image slice set is displayed in a graphical user interface (GUI) on the display device.

5. The disease velocity representation system according to claim 4, further comprising instructions stored in a memory, the instructions, when executed, causing the one or more processors to: Automatically segment each image slice in the image slice set; and The size of the lesion of interest is tracked across time using the image slice set.

6. The disease velocity representation system according to claim 4, further comprising instructions stored in a memory, the instructions, when executed, causing the one or more processors to: when the user queries without specifying a lesion of interest: Receive user selection of a selected image slice from the set of image slices, wherein the user selection indicates the lesion of interest; In response to a user selection of a selected image, the selected image and other image slices in the set of image slices that include the lesion of interest are segmented; The size of the lesion of interest is tracked across time using the image slice set; A trend graph is generated based on the size tracking of the lesions of interest; and Output the set of segmented image slices and the trend graph for display within the GUI.

7. The disease velocity representation system of claim 4, wherein, in order to determine the set of image slices, the second AI model is configured as follows: Extract disease information and anatomical structure partitioning information from the text-based response output; Based on the disease information, one or more relevant imaging protocols are determined; and One or more relevant image slices are determined based on the anatomical structure partitioning information.

8. The disease velocity representation system of claim 7, wherein, in order to determine the image slice set, the second AI model is further configured to merge one or more determined relevant imaging protocols and one or more determined relevant image slices to identify the image slice set.

9. The disease velocity representation system according to claim 4, wherein the second AI model is a base model.

10. A method for a disease velocity representation system, the method comprising: Receive first-user queries related to patients from healthcare management applications; Obtain the patient's patient data from one or more data repositories; Determine the query type of the first user; In response to determining the query type, one or more artificial intelligence (AI) models are deployed to generate a response output to the first user query; as well as The response output is displayed in the graphical user interface (GUI) of the healthcare management application.

11. The method of claim 10, wherein when the query type is a first query type indicating a text-based response, deploying the one or more AI models to generate the response output includes deploying a large language model (LLM), wherein the response output is a text-based response output.

12. The method of claim 10, wherein when the query type is a second query type indicative of a non-text-based response, deploying the one or more AI models to generate the response output comprises: Deploy an LLM to generate the text-based portion of the response output; And the non-text-based portion of the response output is generated using the base model.

13. The method of claim 12, wherein the non-text-based portion of generating the response output comprises: Image volume data is obtained from an imaging database, wherein the image volume data includes multimodal imaging data acquired across multiple time points; Based on the text-based portion generated by the LLM in the response output, disease information and anatomical partition information are determined. Based on the disease information, identify one or more relevant imaging protocols; Based on the anatomical structure partitioning information and the one or more related imaging protocols, a subset of image slices is identified from the obtained imaging volume data.

14. The method of claim 13, wherein the non-text-based portion of generating the response output further includes, when the first user queries to identify a lesion of interest: Automatically segment the subset of image slices; and The size of the lesion of interest within the segmented subset of the image slice is tracked over time.

15. The method of claim 13, wherein the non-text-based portion of generating the response output further includes when the first user query does not identify the lesion of interest; The subset of image slices is displayed within the GUI of the healthcare management application; Receive user selection of a selected image slice from the subset of image slices, wherein the user selection identifies the lesion of interest within the selected image; In response to receiving the user selection, a subset of the subset of image slices is segmented, wherein the subset of the subset of image slices includes the selected image slice and other image slices in the subset of image slices that include the lesion of interest; and The size of the lesion of interest within the subset of the image slice is tracked over time.

16. The method of claim 12, wherein the non-text-based portion of the response output comprises one or more of one or more unsegmented image slices, one or more segmented image slices, a trend graph, and disease velocity labels.

17. The method according to claim 10, further comprising: The healthcare management application receives a second user query as a follow-up to the first user query, wherein the response output to the first user query is a text-based response generated by the LLM of the one or more AI models. In response to determining that the second user query indicates a non-text-based response, the LLM generates a second response output based on the text-based response using a base model from one or more AI models, wherein the second response output is a non-text-based response; as well as The response output and the second response output are displayed in the GUI of the healthcare management application, wherein the second response output includes one or more of the following: unsegmented image slices, one or more segmented image slices, trend graphs, and disease velocity labels.

18. A method for representing the speed of a disease, the method comprising: Receive first-user queries related to patients; The patient's patient data is obtained from one or more electronic medical records (EMRs); In response to determining that the first user query indicates a text-based response, a first response to the first user query is generated by a large language model (LLM) based on the patient data; The first response is displayed in a user interface (UI) on a display device, wherein the first response is a text-based response; Receive a second user query, wherein the second user query is a follow-up to the first user query; In response to determining that the second user query indicates a non-text-based response, a second response to the second user query is generated by the base model based on the first response and imaging data obtained from the imaging database; as well as The second response is displayed in the UI on the display device, wherein the second response includes one or more of the following: an unsegmented image slice, one or more segmented image slices, a trend graph, and a disease velocity label.

19. The method of claim 18, wherein generating the second user query comprises: Based on the first response generated by the LLM, disease information and anatomical partition information are determined; Based on the disease information, identify one or more relevant imaging protocols; Based on the anatomical structure partitioning information and the one or more related imaging protocols, a subset of image slices is identified from the imaging data; When the user queries for the lesion they are interested in: Automatically segment the subset of image slices; The size of the lesions of interest within the segmented subsets of the image slices is tracked over time; and The UI displays a segmented subset of the image slices and a trend graph generated based on the size tracking. and When the user query does not identify the lesion of interest; Display the subset of image slices within the UI; Receive user selection of a selected image slice from the subset of image slices, wherein the user selection identifies the lesion of interest within the selected image; In response to receiving the user selection, a subset of the subset of image slices is segmented, wherein the subset of the subset of image slices includes the selected image slice and other image slices in the subset of image slices that include the lesion of interest; The size of the lesion of interest within the subset of the image slice is tracked over time; and The UI displays a subset of the image slices, a segmented subset, and a trend graph generated based on the size tracking.

20. The method of claim 18, wherein the first user query and the second user query are received from a healthcare management application adapted to aggregate and display patient medical data.

Images