Chart and nearest neighbor patient mapping and LLM output

Abstract

A computer-implemented method comprising receiving, via one or more processors, an input query associated with a patient; obtaining, via one or more processors, a healthcare record associated with the patient; analyzing, via the one or more processors and using a trained machine learning model, the healthcare record associated with the patient to extract data elements relevant to the input query; and generating, via the one or more processors and using the trained machine learning model, a first mapping of the extracted data elements relevant to the query.

Description

[0001] CHART AND NEAREST NEIGHBOR PATIENT MAPPING AND LLM OUTPUT

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] [1] This application claims priority to U.S. Provisional Patent Application No. 63 / 616423, filed December 29, 2023, the entire disclosure of which is incorporated herein by reference.

[0004] TECHNICAL FIELD

[0005] [2] The present disclosure is generally directed to systems and methods for mapping a patient chart.

[0006] BACKGROUND

[0007] [3] A patient chart may contain multitudes of data regarding a patient. However, when a medical professional asses a patient’s record for a specific condition, the amount of information may be difficult and time-consuming to navigate. A medical professional may have difficulty identifying all relevant pieces of information within the medical record, and the information may not be organized neatly. Moreover, the medical professional’s information may be limited to only the patient’s healthcare record.

[0008] [4] Therefore, there is an opportunity for utilizing artificial intelligence and machine learning to assist in graphically mapping a patient chart.

[0009] BRIEF SUMMARY

[0010] [5] In an embodiment, a computer-implemented method including receiving, via one or more processors, an input query associated with a patient; obtaining, via the one or more processors, a healthcare record associated with the patient; analyzing, via the one or more processors and using a trained machine learning model, the healthcare record associated with the patient to extract data elements relevant to the input query; and generating, via the one or more processors and using the trained machine learning model, a first mapping of the extracted data elements relevant to the query.

[0011] [6] In another embodiment, a computing system including one or more processors, and one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing system to receive an input query associated with a patient; obtain a healthcare record associated with the patient; analyze the healthcare record associated with the patient to extract data elements relevant to the input query; and generate a first mapping of the extracted data elements relevant to the query.

[0012] [7] In another embodiment, a non-transitory computer-readable medium, having stored thereon instructions that when executed, cause a computer to receive an input query associated with a patient; obtain a healthcare record associated with the patient; analyze the healthcare record associated with the patient to extract data elements relevant to the input query; and generate a first mapping of the extracted data elements relevant to the query.

[0013] BRIEF DESCRIPTION OF THE FIGURES

[0014] [8] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0015] [9] The figures described below depict various aspects of the system and methods disclosed therein. It should be understood that each figure depicts one aspect of a particular aspect of the disclosed system and methods, and that each of the figures is intended to accord with a possible aspect thereof. Further, wherever possible, the following description refers to the reference numerals included in the following figures, in which features depicted in multiple figures are designated with consistent reference numerals.

[0016]

[0010] FIG. 1 depicts an exemplary computing environment in which the techniques disclosed herein may be implemented, according to some aspects, according to some aspects.

[0017]

[0011] FIG. 2 depicts a combined block and logic diagram in which exemplary computer- implemented methods and systems for training machine learning are implemented, according to some aspects.

[0018]

[0012] FIG. 3 depicts an exemplary mapping of a patient chart.

[0019]

[0013] FIG. 4 depicts a block diagram of an example architecture for mapping a patient chart, according to some aspects.

[0020]

[0014] FIG. 5 depicts a flow diagram of an exemplary computer-implemented method for mapping a patient chart using machine learning, according to one embodiment.

[0021]

[0015] The figures depict preferred aspects for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative aspects of the systems and methods illustrated herein may be employed without departing from the principles of the invention described herein.

[0022] DETAILED DESCRIPTION

[0023] Overview

[0024]

[0016] The present techniques provide methods and systems for, inter alia, generate a mapping of a patient chart, also referred to as a healthcare record herein, using machine learning. A machine learning model may receive an input query. The input query may be enhanced with additional context information pulled from notes regarding a medical specialty and / or other notes relevant to the query. The machine learning model may obtain a healthcare record. The healthcare record may be analyzed to extract data elements relevant to the input query. The machine learning model may generate a mapping of the extracted data elements. In some implementations, the machine learning model may find other patients similar to the patient being treated (e.g., a patient who is a nearest-match). The machine learning model may summarize treatment recommendations and rank the outcomes of such recommendations.

[0025]

[0017] The present disclosure includes specific features other than what is well-understood, routine, conventional activity in the field, or adding unconventional operations that demonstrate, in various embodiments, particular useful applications, e.g., receiving an input from a provider (e.g., via user query / prompt or other input, directly from the provider or from a provider computer terminal communicating the input), the input including input query associated with a patient, obtaining, via one or more processors, a healthcare record associated with the patient, analyzing, via the one or more processors and using a trained machine learning model, the healthcare record associated with the patient to extract data elements relevant to the input query, and generating, via the one or more processors and using the trained machine learning model, a first mapping of the extracted data elements relevant to the query, among others.

[0026] Exemplary Computing Environment

[0027]

[0018] FIG. 1 depicts an exemplary computing environment 100 in which the techniques disclosed herein may be implemented, according to some aspects. The environment 100 may include computing resources for training and / or operating machine learning models to generate a mapping of a patient chart.

[0028]

[0019] The computing environment 100 may include a client computing device 102, a server computing device 104, an electronic network 106, an electronic health record database 108, a model electronic database 112, and a notes database 1 16. The computing environment may further include one or more cloud application programming interfaces (APIs) 114. The components of the computing environment 100 may be communicatively connected to one another via the electronic network 106, in some aspects.

[0029]

[0020] The client computing device 102 may implement, inter alia, operation of one or more applications for generating a mapping of a patient chart. In some aspects, the client computing device 102 may be implemented as one or more computing devices (e.g., one or more servers, one or more laptops, one or more mobile computing devices, one or more tablets, one or more wearable devices, one or more cloud-computing virtual instances, etc.). In some aspects, a plurality of client computing devices may be part of the environment 100 - for example, a first user may access a client computing device 102 that is a laptop, while a second user accesses the client computing device 102 that is a smart phone, while yet a third user accesses a client computing device 102 that is a wearable device.

[0030]

[0021] The client computing device 102 may include one or more processors 120, one or more network interface controllers 122, one or more memories 124, an input device 126, an output device 128 and a client API 130. The one or more memories 124 may have stored thereon one or more modules 140 (e.g., one or more sets of instructions).

[0031]

[0022] In some aspects, the one or more processors 120 may include one or more central processing units, one or more graphics processing units, one or more field-programmable gate arrays, one or more application-specific integrated circuits, one or more tensor processing units, one or more digital signal processors, one or more neural processing units, one or more RISC-V processors, one or more coprocessors, one or more specialized processors / accelerators for artificial intelligence (Al) or machine learning (ML) -specific applications, one or more microcontrollers, etc.

[0032]

[0023] The client computing device 102 may include one or more network interface controllers 122, such as Ethernet network interface controllers, wireless network interface controllers, etc. The network interface controllers 122 may include advanced features, in some aspects, such as hardware acceleration, specialized networking protocols, etc.

[0033]

[0024] The memories 124 of the client computing device 102 may include volatile and / or nonvolatile storage media. For example, the memories 124 may include one or more random access memories, one or more read-only memories, one or more cache memories, one or more hard disk drives, one or more solid-state drives, one or more non-volatile memory express, one or more optical drives, one or more universal serial bus flash drives, one or more external hard drives, one or more network-attached storage devices, one or more cloud storage instances, one or more tape drives, etc.

[0034]

[0025] As noted, the memories 124 may have stored thereon one or more modules 140, for example, as one or more sets of computer-executable instructions. In some aspects, the modules 140 may include additional storage, such as one or more operating systems (e.g., Microsoft Windows, GNU / Linux, Mac OSX, etc.). The operating systems may be configured to run the modules 140 during operation of the client computing device 102 - for example, the modules 140 may include additional modules and / or services for receiving and processing data from one or more other components of the environment 100 such as the one or more cloud APIs 114 or the server computing device 104. The modules 140 may be implemented using any suitable computer programming language(s) (e.g., Python, JavaScript, C, C++, Rust, C#, Swift, Java, Go, LISP, Ruby, Fortran, etc.).

[0035]

[0026] The modules 140 may include an API module 144, an input processing module 146, an authentication / security module 148, and a context module 150 in some aspects. In some aspects, more or fewer modules 140 may be included. The modules 140 may be configured to communicate with one another (e.g., via inter-process communication, via a bus, via sockets, pipes, message queues, etc.).

[0036]

[0027] The API module 144 may include one or more sets of computer executable instructions for accessing one or more remote APIs, and / or for enabling one or more other components within the environment 100 to access functionality of the client computing device 102. In some aspects, the API module 144 may enable other client applications (i.e., not applications facilitated by the modules 140) to connect to the client computing device 102, for example, to send queries or prompts, and to receive responses from the client computing device 102. The API module 144 may include instructions for authentication, rate limiting and error handling.

[0037]

[0028] As noted, the client computing device 102 may enable one or more users to access one or more trained models by providing input prompts that are processed by one or more trained models. The input processing module 146 may perform pre-processing of user prompts prior to being input into one or more models, and / or post-processing of outputs output by one or more models. For example, the input processing module 146 may process data input into one or more input fields, voice inputs or other input methods (e.g., file attachments) depending upon the application. The input processing module 146 may receive inputs directly via the input device 126, in some aspects.

[0038]

[0029] In some aspects, the input processing module 146 may perform post-processing of input received from one or more trained models. In some aspects, post-processing (and / or preprocessing) may include implementing content moderation mechanisms, to prevent misuses of trained models or inappropriate content generation. The input processing module 146 may include instructions for handling errors and for displaying errors to users (e.g., via the output device 128). The input processing module 146 may cause one or more graphical user interfaces to be displayed, for example to enable the user to enter information directly via a text field.

[0039]

[0030] The authentication / security module 148 may include one or more sets of computerexecutable instructions for implementing access control mechanisms for one or more trained models, ensuring that the model and the underlying healthcare records can only be accessed by those who are authorized to do so, and that the access of those users is private and secure. For example, information included in electronic healthcare records should only be accessed by authorized users.

[0040]

[0031] Generally, trained models, especially trained models, require state information in order to meaningfully carry on a dialogue with a user or with another trained model. For example, if a user prompts a trained model with a question such as “What is the weather in Chicago today?” followed by a second prompt “And how about tomorrow?” the model should understand that, in context, the second query relates to the first query, insofar as the user is asking about the weather tomorrow in the same location (Chicago).

[0041]

[0032] However, language models (e.g., large language models (LLMs)) are generally stateless, meaning that after they process a prompt, they have no internal record or memory of the information that was input, or the information that was generated as part of the language model’s processing. Thus, many systems add statefulness to models using context information. This may be implemented using sliding context windows, wherein a predetermined number of tokens (e.g., 4096 maximum tokens in the case of GPT 3.5, equivalent to about 3000 words) may be “remembered” by the LLM and can be used to enrich multiple sequential prompts input into the LLM (for example, when the LLM is used in a chat mode). GPT is a trademark of OpenAI Corp, of San Francisco, CA, and is currently available at https: / / openai.com / .

[0042]

[0033] The context module 150 may include one or more sets of computer-executable instructions for maintaining state of the type found in this example, and other types of state information. The context module 150 may implement sliding window context, in some aspects. In other aspects, the context module 150 may perform other types of state maintaining strategies. For example, the context module 150 may implement a strategy in which information from the immediately preceding prompt is part of the window, regardless of the size of that prior prompt.

[0043]

[0034] In some aspects, the context module 150 may implement a strategy in which one or more prior prompts are included in each current prompt. This prompt stuffing technique, or prompt concatenation, may be limited by prompt size constraints — once the total size of the prompt exceeds the prompt limit, the model immediately loses state information related to parts of the prompt truncated from the prompt.

[0044]

[0035] The server computing device 104 may include one or more processors 160, one or more network interface controllers 162, one or more memories 164, an input device (not depicted), an output device (not depicted) and a server API 166. The one or more memories 164 may have stored thereon one or more modules 170 (e.g., one or more sets of instructions).

[0036] In some aspects, the one or more processors 160 may include one or more central processing units, one or more graphics processing units, one or more field-programmable gate arrays, one or more application-specific integrated circuits, one or more tensor processing units, one or more digital signal processors, one or more neural processing units, one or more RISC-V processors, one or more coprocessors, one or more specialized processors / accelerators for artificial intelligence or machine learning-specific applications, one or more microcontrollers, etc.

[0045]

[0037] The server computing device 104 may include one or more network interface controllers 162, such as Ethernet network interface controllers, wireless network interface controllers, etc. The network interface controllers 162 may include advanced features, in some aspects, such as hardware acceleration, specialized networking protocols, etc.

[0046]

[0038] The memories 164 of the server computing device 104 may include volatile and / or nonvolatile storage media. For example, the memories 164 may include one or more random access memories, one or more read-only memories, one or more cache memories, one or more hard disk drives, one or more solid-state drives, one or more non-volatile memory express, one or more optical drives, one or more universal serial bus flash drives, one or more external hard drives, one or more network-attached storage devices, one or more cloud storage instances, one or more tape drives, etc.

[0047]

[0039] As noted, the memories 164 may have stored thereon one or more modules 170, for example, as one or more sets of computer-executable instructions. In some aspects, the modules 170 may include additional storage, such as one or more operating systems (e.g., Microsoft Windows, GNU / Linux, Mac OSX, etc.). The operating systems may be configured to run the modules 170 during operation of the server computing device 104 - for example, the modules 170 may include additional modules and / or services for receiving and processing data from one or more other components of the environment 100 such as the one or more cloud APIs 114 or the client computing device 102. The modules 170 may be implemented using any suitable computer programming language(s) (e.g., Python, JavaScript, C, C++, Rust, C#, Swift, Java, Go, LISP, Ruby, Fortran, etc.).

[0048]

[0040] In some aspects, the modules 170 may include a data collection module 172, a data preprocessing module 174, a model pretraining module 176, a fine-tuning module 178, a model training module 180, a checkpointing module 182, a hyperparameter tuning module 184, a validation and testing module 186, an auto-prompting module 188, and an operating module

[0049] 190. In some aspects, more or fewer modules 170 may be included. The modules 170 may be configured to communicate with one another (e.g., via inter-process communication, via a bus, via sockets, pipes, message queues, etc.). The modules 170 may respond to network requests (e.g., via the API 166) or other requests received via the network 106 (e.g., via the client computing device 102 or other components of the environment 100).

[0050]

[0041] The data collection module 172 may be configured to collect information used to train one or more modules. In general, the information collected may be any suitable information used for training a language model. The data collection module 172 may collect data via web scraping, via API calls / access, via database extract-transform-load (ETL) processes, etc. Sources accessed by the data collection module 172 include social media websites, books, websites, academic publications, web forums / interest sites (e.g., Reddit, Facebook, bulletin boards, etc.), etc. The data collection module 172 may access data sources by active techniques (e.g., scraping or other retrieval) or may access existing corpuses. The data collection module 172 may include sets of instructions for performing data collection in parallel, in some aspects. The data collection module 172 may store collected data in one or more electronic databases, such as a database accessible via the cloud APIs 114 or via a local electronic database (not depicted). The data may be stored in a structured and / or unstructured format. In some aspects, the data collection module 172 may store large data volumes used for training one or more models (i.e., training data). For example the data collection module 172 may store terabytes, petabytes, exabytes or more of training data.

[0051]

[0042] In some aspects, the data collection module 172 may retrieve data from the electronic health record database 108 or from the notes database 1 16. For example, the data collection module 172 may process the retrieved / received data and sort the data into multiple subsets based on information included within the electronic health record database 108. For example, the data collection module 172 may receive one or more sets of unstructured text (e.g., unstructured clinical notes). The data collection module 172 may chunk the data according to time (e.g., hourly, daily, quarterly, etc.).

[0052]

[0043] The model preprocessing module 174 may include instructions for pre-processing data collected by the data collection module 172. In particular, the model preprocessing module 174 may perform text extraction and / or cleaning operations on data collected by the data collection module 172. The data pre-processing module 174 may perform preprocessing operations, such as lexical parsing, tokenizing, case conversions and other string splitting / munging. In some aspects, the data collection module 172 may perform data deduplication, filtering, annotation, compliance, version control, validation, quality control, etc. In some aspects, one or more human reviewers may be looped into the process of pre-processing data collected by the data pre-processing module 174. For example, a distributed work queue may be used to transmit batch jobs and receive human-computed responses from one or more human workers. Once pre-processed, the data pre-processing module 174 may store copied and / or modified copies of the training date in an electronic database.

[0053]

[0044] In some aspects the data pre-processing module 174 may include instructions for parsing the unstructured text received by the data collection module 172 to structure the text. For example, when the text relates to meeting minutes (e.g., text transcripts) of meetings of a clinical review board, the data pre-processing module 174 may generate a time series data structure in which each meeting is represented by one or more timestamps, and at each timestamp, text of one or more speakers is labeled. The data pre-processing module 174 may also label the data according to the identity of one or more speaker and / or one or more topic. For example, the time series data may be labeled according to one or more speakers associated with textual speech, in a language transcript form. The time series may include one or more keywords associated with the transcript. In some aspects, the present techniques may use a separate trained text summarization module to generate keywords used for this purpose. In this way, the data pre-processing module 174 may generate structured data corresponding to unstructured meeting minutes, such that the structured data is enriched with information about the meeting that is suitable for training. This structured data may be processed by downstream processes / modules.

[0054]

[0045] Generally, the present techniques may train one or more models to perform language generation tasks that include token generation. Both training inputs and model outputs may be tokenized. Herein, tokenization refers to the process by which text used for training is divided into units such as words, subwords or characters. Tokenization may break a single word into multiple subwords (e.g., “LLM” may be tokenized as “L” and “LM”). The present techniques may train one or more models using a set of tokens (e.g., a vocabulary) that includes many (e.g., thousands or more) of tokens. These tokens may be embedded into a vector. This vector of token or “embeddings” may include numerical representations of the individual tokens in the vocabulary in high-dimensional vector space. The modules 170 may access and modify the embeddings during training to learn relationships between tokens. These relationships effectively represent semantic language meaning.

[0055]

[0046] In some aspects, a specialized database (e.g., a vector store, a graph database, etc.) may be used to store and query the embeddings. Embedding databases may include specialized features, such as efficient retrieval, similarity search and scalability. For example, the server computing device 104 may include a local electronic embedding database (not depicted). In some aspects, a remote embedding database service may be used (e.g., via the cloud APIs 1 14). Such a remote embedding database service may be based on an open source or proprietary model (e.g., Milvus, Pinecone, Redis, Postgres, MongoDB, Facebook Al Similarity Search (FAISS), etc.). The server computing device 104 may include instructions (e.g., in the data collection module 172) for adding training data to one or more specialized databases, and for accessing it to train models.

[0056]

[0047] The present techniques may include language modeling, wherein one or more deep learning models are trained by processing token sequences using a large language model architecture. For example, in some aspects, a transformer architecture may be used to process a sequence of tokens. Such a transformer model may include a plurality of layers including selfattention and feedforward neural networks. This architecture may enable the model to learn contextual relationships between the tokens, and to predict the next token in a sequence, based upon the preceding tokens. During training, the model is provided with the sequence of tokens and it learns to predict a probability distribution over the next token in the sequence. This training process may include updating one or more model parameters (e.g., weights or biases) using an objective function that minimizes the difference between the predicted distribution and a true next token in the training data. Particular techniques for training are discussed in further detail below in Fig. 2.

[0057]

[0048] Alternatives to the transformer architecture may include recurrent neural networks, long short-term memory networks, gated recurrent networks, convolutional neural networks, recursive neural networks, and other modeling architectures.

[0058]

[0049] In some aspects, the modules 170 may include instructions for performing pretraining of a language model (e.g., an LLM), for example, in a pretraining module 176. The pretraining module 176 may include one or more sets of instructions for performing pretraining, which as used herein, generally refers to a process that may span pre-processing of training data via the data pre-processing module 174 and initialization of an as-yet untrained language model. In general, a pre-trained model is one that has no prior training of specific tasks. For example, the model pretraining module 176 may include instructions that initialize one more model weights.

[0059] In some aspects, model pretraining module 176 may initialize the weights to have random values. The model pretraining module 176 may train one or more models using unsupervised learning, wherein the one or more models process one or more tokens (e.g., preprocessed data output by the data pre-processing module 174) to learn to predict one or more elements (e.g., tokens). The model pretraining module 176 may include one or more optimizing objective functions that the model pretraining module 176 applies to the one or more models, to cause the one or more models to predict one or more most-likely next tokens, based on the likelihood of tokens in the training data. In general, the model pretraining module 176 causes the one or more models to learn linguistic features such as grammar and syntax. The pretraining module 176 may include additional steps, including training, data batching, hyperparameter tuning and / or model checkpointing.

[0060]

[0050] The model pretraining module 176 may include instructions for generating a model that is pretrained for a general purpose, such as general text processing / understanding. This model may be known as a “base model” in some aspects. The base model may be further trained by downstream training process(es), for example, those training processes described with respect to the fine-tuning module 178. The model pretraining module 176 generally trains foundational models that have general understanding of language and / or knowledge. Pretraining may be a distinct stage of model training in which training data of a general and diverse nature (i.e., not specific to any particular task or subset of knowledge) is used to train the one or more models.

[0061] In some aspects, a single model may be trained and copied. Copies of this model may serve as respective base models for a plurality of fine-tuned models.

[0062]

[0051] In some aspects, base models may be trained to have specific levels of knowledge common to more advanced agents. For example, the model pretraining module 176 may train a medical student base model that may be subsequently used to fine tune an internist model, a surgeon model, a resident model, etc. In this way, the base model can start from a relatively advanced stage, without requiring pretraining of each more advanced model individually. This strategy represents an advantageous improvement, because pretraining can take a long time (many days), and pretraining the common base model only requires that pretraining process to be performed once.

[0063]

[0052] The modules 170 may include a fine-tuning module 178. The fine-tuning module 178 may include instructions that train the one or models further to perform specific tasks.

[0064] Specifically, the fine-tuning module 178 may include instructions that train one or more models to generate respective language outputs (e.g., text generation), summarization, question answering or translation activities based on healthcare data.

[0065]

[0053] Continuing the example, the fine-tuning module 178 may include sets of instructions for retrieving one or more structured data sets, such as time series generated by the data preprocessing module 174. These structured data sets may be sorted by time and / or speaker to train one or more machine learning models (e.g., one or more language models) that may be used within the environment 100 to generate mappings of a patient chart. For example, the fine-tuning module 178 may include instructions for configuring an objective function for performing a specific task, such as generating text that is similar to text found within the corpus of training data associated with a particular individual by role.

[0054] In some aspects, to manage complexity of fine-tuning and other machine learning operations of the server computing device 104, one or more open source frameworks may be used. Example frameworks include TensorFlow, Keras, MXNet, Caffe, SciKit learn, PyTorch. Specifically for training and operating language models, frameworks such as OpenLLM and LangChain may be used, in some aspects. The fine-tuning module 178 may use an algorithm such as stochastic gradient descent or another optimization technique to adjust weights of the pretrained model.

[0066]

[0055] Fine-tuning may be an optional operation, in some aspects. In some aspects, training may be performed by the training module 180 after pretraining by the model pretraining module 176. In some aspects, the model training module 180 may perform task-specific training like the fine-tuning module 178, on a smaller scale or with a more tailored objective.

[0067]

[0056] The training module 180 may include one or more submodules, including the checkpointing module 182, the hyperparameter tuning module 184, the validation and testing module 186 and the auto-prompting module 188. The checkpointing module 182 may perform checkpointing, which is saving of a model’s parameters. The checkpointing module 182 may store checkpoints during training and at the conclusion of training, for example, in the model electronic database 1 12. In this way, the model may be run (e.g., for testing and validation) at multiple stages and its training parameters loaded, and also retrained from a checkpoint. In this way, the model can be run and trained forward without being re-trained from the beginning, which may save significant time (e.g., days of computation). The hyperparameter tuning module 184 may include hyperparameters such as batch size, model size, learning rate, etc. These hyperparameters may be adjusted to influence model training. The hyperparameter tuning module 184 may include instructions for tuning hyperparameters by successive evaluation. The validation and testing module 186 may include sets of instructions for validating and testing one or more machine learning models, including those generated by the model pretraining module 176, the fine-tuning module 178 and the model training module 180. The auto-prompting module 188 may include sets of instructions for performing auto-prompting of one or more models. Specifically, the auto-prompting module 188 may enrich a prompt with additional information. The auto-prompting module 188 may include additional information in a prompt, so that the model receiving the prompt has additional context or directions that it can use. This may allow the auto-prompting module 188 to fine-tune a base model using one-shot or few-shot learning, in some aspects. The auto-prompting module 188 may also be used to focus the output of the one or more models.

[0057] In some aspects, the training module 180 may include instructions for training one or more additional machine learning models, such as supervised or unsupervised machine learning models. For example, as discussed below, in some aspects, the present techniques may include processing imaging data to identify related images and determine changes in the data over time, such as tumor progression data, etc. In that case, the patient’s imaging data may be processed by a model (e.g., a convolutional neural network) and the results processed further (e.g., by a language model) and / or provided to the client computing device 102. The training module 180 may train such a supervised model separately from training one or more language models. Further, the server computing device 104 may select one or more trained models at runtime based on data about a specific patient, based upon data contained in a prompt or based on other conditions that may be preprogrammed into the server computing device 104.

[0068]

[0058] In some aspects, the training module 180 may train multi-modal models. For example, the training module 180 may train a plurality of models each capable of drawing from multimodal data types such as written text, imaging data, laboratory data, real-time monitoring data, pathology images, etc. In some cases, the training module 180 may train a single model capable of processing the multimodal data types. In some aspects, a trained multi-modal model may be used in conjunction with another model (e.g., a large language model) to provide nontext data interactions with users. Non-text data may be analyzed and integrated into the debate functions discussed herein. In some embodiments, the training and / or fine-tuning operations described herein may be periodically repeated (e.g., daily, weekly, monthly) to incorporate new input data and / or feedback from users / providers. This retraining may produce updated versions of the respective models. In this way, some embodiments may evolve over time to produce outputs that match the particularized uses and preferences of its users / providers.

[0069]

[0059] The operating module 190 may operate one or more trained models. Specifically, the model operation module 190 may initialize one or more trained models, load parameters into the model(s), and provide the model(s) with inference data (e.g., prompt inputs). In some aspects, the model operation module 190 may deploy one or more trained model (e.g., a pretrained model, a fine tuned model and / or a trained model) onto a cloud computing device (e.g., via the API 166). The model operation module 190 may receive one or more inputs, for example from the client computing device 102, and provide those inputs (e.g., one or more prompts) to the trained model. In some aspects, the API 166 may include elements for receiving requests to the model, and for generating outputs based on model outputs. For example, the API 166 may include a RESTful API that receives a GET or POST request including a prompt parameter. The model operation module 190 may receive the request from the API 166, and pass the prompt parameter into the trained model and receive a corresponding input. For example, the prompt parameter may be “What are treatments for atrial fibrillation?”. The prompt output may be “Treatments for atrial fibrillation include medication, cardioversion, ablation, or surgery.”

[0070]

[0060] The model operation module 190 may operate models in different modes. A medical team may want to use the present techniques to assess a patient chart or assess treatment options. The user may access the client computing device 102. The input processing module 146 may identify the patient via the name or other identifying information provided, and retrieve patient information (e.g., electronic health records) from an electronic health record database 108.

[0071]

[0061] As discussed, in some aspects, multi-modal modeling may be used. The data preprocessing module may, for example, process and understand image data, audio data, video data, etc. The server computing device 104 may interpret and respond to queries that involve understanding content from these different modalities. For example, the server computing device 104 may include an image processing module (not depicted) including instructions for performing image analysis on images provided by users, or images retrieved from patient EHR data. The server computing device 104 may process the images to determine a similarity between the images and group the images based on similarity. In some aspects, the server computing device 104 may generate outputs in modalities other than text. For example, the server computing device 104 may generate an audio response, an image, etc. Combining multi-modal data may enable the present models to perform more comprehensive analysis of patient conditions, based on information processed in multiple different modes simultaneously.

[0072]

[0062] The operating module 190 may include a set of computer-executable instructions that when executed by one or more processors (e.g., the processors 160) cause a computer (e.g., the server computing device 104) to perform retrieval-augmented generation. Specifically, the operating module 190 may perform retrieval-augmented generation based upon inputs or queries received from the user. This allows the operating module 190 to tailor responses of a model based on the specific input and context, such as the medical issue under discussion. For example, one or more models may be pre-trained, fine-tuned and / or trained as discussed above. During that training, the model may learn to generate tokens based on general language understanding as well as application-specific training. Such a model at that point may be static, insofar as it cannot access further information when presented with an input query.

[0073]

[0063] When the model is used at runtime, such as when deployed in the environment 100, the operating module 190 may perform retrieval operations from the notes database 1 16. The operating module 190 may perform retrieval operations such as searching or selecting information from additional documents, a databases, or other sources. The operating module 190 may include instructions for processing user input and for performing a keyword search, a regular expression search, a similarity search, etc. based upon that user input. The operating module 190 may input the results of that search, along with the user input, into the trained model. Thus, the trained model may process this additional retrieved information to augment, or contextualize, the generation of tokens that represent responses to the user’s query. In sum, retrieval augmented generation applied in this manner allows the model to dynamically generate outputs that are more relevant to the user’s input query at runtime. Information that may be retrieved may include data corresponding to a patient (e.g., patient demographic information, medical history, clinical notes, diagnoses, medications, allergies, immunizations, laboratory results, oncology information, radiation and imaging information, vitals, etc.) and additional training information, such as medical journals, notes or speech transcripts from symposia or other meetings / conferences, etc.

[0074]

[0064] The present techniques may trigger retrieval augmented generation by processing a prompt, in some aspects. For example, a prompt may be processed by the input processing module 146 of the client computing device 102, prior to processing the prompt by the one or more generative models. The input processing module 146 may trigger retrieval augmented generation based on the presence of certain inputs, such as patient information, or a request for specific information, in the form of keywords. For example, the input processing module 146 may identify a medical specialty from the prompt. The input processing module 146 may perform entity recognition or other natural language processing functions to determine whether the prompt should be processed using retrieval augmented generation prior to being provided to the trained model. The input processing module may retrieve information from the notes database 116 to contextualize the prompt.

[0075]

[0065] As discussed above, prompts may be received via the input processing module 146 of the client computing device 102 and transmitted to the server computing device 104 via the electronic network 106. In some aspects, the output of the model may be modulated prior to being transmitted, output or otherwise displayed to a user.

[0076]

[0066] The client computing device 102 and the server computing device 104 may communicate with one another via the network 106. In some aspects, the client computing device 102 and / or the server computing device 104 may offload some or all of their respective functionality to the one or more cloud APIs 114. In aspects, the one or more cloud APIs 114 may include one or more public clouds, one or more private clouds and / or one or more hybrid clouds. The one or more cloud APIs 114 may include one or resources provided under one or more service models, such as Infrastructure as a Service (laaS), Platform as a Service (PaaS), Software as a Service (SaaS), and Function as a Service (FaaS). For example, the one or more cloud APIs 114 may include one or more cloud computing resources, such as computing instances, electronic databases, operating systems, email resources, etc. The one or more cloud APIs 114 may include distributed computing resources that enable, for example, the model pretraining module 176 and / or other of the modules 170 to distribute parallel model training jobs across many processors.

[0077]

[0067] In some aspects, the one or more cloud APIs 1 14 may include one or more language operation APIs, such as OpenAI, Bing, Claude. ai, etc. In other aspects, the one or more cloud APIs 114 may include an API configured to operate one or more open source models, such as Llama 2.

[0078]

[0068] The electronic network 106 may be a collection of interconnected devices, and may include one or more local area networks, wide area networks, subnets, and / or the Internet. The network 106 may include one or more networking devices such as routers, switches, etc. Each device within the network 106 may be assigned a unique identifier, such as an IP address, to facilitate communication. The network 106 may include wired (e.g., Ethernet cables) and wireless (e.g., Wi-Fi) connections. The network 106 may include a topology such as a star topology (devices connected to a central hub), a bus topology (devices connected along a single cable), a ring topology (devices connected in a circular fashion), and / or a mesh topology (devices connected to multiple other devices). The electronic network 106 may facilitate communication via one or more networking protocols, such as packet protocols (e.g., Internet Protocol (IP)) and / or application-layer protocols (e.g., HTTP, SMTP, SSH, etc.). The network 106 may perform routing and / or switching operations using routers and switches. The network 106 may include one or more firewalls, file servers and / or storage devices. The network 106 may include one or more subnetworks such as a virtual LAN (VLAN).

[0079]

[0069] The environment 100 may include one or more electronic databases, such as a relational database that uses structured query language (SQL) and / or a NoSQL database or other schema-less database suited for the storage of unstructured or semi-structured data.

[0080]

[0070] The present techniques may store training data, training parameters and / or trained models in an electronic database such as the database 1 12. Specifically, one or more trained machine learning models may be serialized and stored in a database (e.g., as a binary, a JSON object, etc.). Such a model can later be retrieved, deserialized and loaded into memory and then used for predictive purposes. The one or more trained models and their respective training parameters (e.g., weights) may also be stored as blob objects. Cloud computing APIs may also be used to stored trained models, via the cloud APIs 114. Examples of these services include AWS SageMaker, Google Al Platform and Azure Machine Learning.

[0081] Exemplary Training of a Machine Learning Model

[0082]

[0071] Figure 2 depicts a combined block and logic diagram 200 for training a machine learning model, in which the techniques described herein may be implemented, according to some embodiments. Some of the blocks in Figure 2 may represent hardware and / or software components, other blocks may represent data structures or memory storing these data structures, registers, or state variables (e.g., 212), and other blocks may represent output data (e.g., 225). Input and / or output signals may be represented by arrows labeled with corresponding signal names and / or other identifiers. The methods and systems may include one or more servers 202, 204, 206, such as the server 04104 or an external computing device.

[0083]

[0072] In one aspect, the server 202 may fine-tune a pretrained language model 210. The pretrained language model 210 may be obtained by the server 202 and be stored in a memory, such as memory 4124. The pretrained language model 210 may be loaded into an ML training module, such as the MLTM 142, by the server 202 for retraining / fine-tuning. A supervised training dataset 212 may be used to fine-tune the pretrained language model 210 wherein each data input prompt to the pretrained language model 210 may have a known output response for the pretrained language model 210 to learn from. The supervised training dataset 212 may be stored in a memory of the server 202, e.g., the memory 4124 (for example, in the data collection module 172)). In one aspect, the data labelers may create the supervised training dataset 212 prompts and appropriate responses. The pretrained language model 210 may be fine-tuned using the supervised training dataset 212 resulting in the SFT ML model 215 which may provide appropriate responses to user prompts once trained. The trained SFT ML model 215 may be stored in a memory of the server 202, e.g., memory 4124.

[0084]

[0073] In some embodiments, the server 202 may fine-tune the pretrained language model 210 using a set of vectors associated with a set of training data. In some instances, the set of training data may include prompts associated with questions and documents, and responses associated with the prompts. Creating the set of vectors may include (1 ) splitting the text of the prompts, associated questions and / or associated documents into semantic clusters, and (2) encoding the semantic clusters as the set of vectors. The semantic clusters may be one or more words, a portion of a word, or a character. A distance between the vectors (e.g., a cosine distance, a Euclidean distance) may depend on a relevance between the semantic clusters corresponding to the vectors.

[0074] In one aspect, training the machine learning model 250 may include the server 204 training a reward model 220 to provide as an output a scaler value / reward 225. It may be desirable for the reward model 220 to leverage Reinforcement Learning with Human Feedback (RLHF) in which a model (e.g., machine learning model 250) learns to produce outputs which maximize its reward 225, and in doing so may provide responses which are better aligned to user prompts.

[0085]

[0075] Training the reward model 220 may include the server 204 providing a single prompt 222 to the SFT ML model 215 as an input. The input prompt 222 may be provided via an input device (e.g., a keyboard) via the I / O module of the server, such as I / O module 146. The prompt 222 may be previously unknown to the SFT ML model 215, e.g., the labelers may generate new prompt data, the prompt 222 may include testing data stored in the data collection module 172, and / or any other suitable prompt data. The SFT ML model 215 may generate multiple, different output responses 224A, 224B, 224C, 224D to the single prompt 222. The server 204 may output the responses 224A, 224B, 224C, 224D via an I / O module (e.g., I / O module 146) to a user interface device, such as a display (e.g., as text responses), a speaker (e.g., as audio / voice responses), and / or any other suitable manner of output of the responses 224A, 224B, 224C, 224D for review by the data labelers.

[0086]

[0076] The data labelers may provide feedback via the server 204 on the responses 224A, 224B, 224C, 224D when ranking 226 them from best to worst based upon the prompt-response pairs. The data labelers may rank 226 the responses 224A, 224B, 224C, 224D by labeling the associated data. The ranked prompt-response pairs 228 may be used to train the reward model 220. In one aspect, the server 204 may load the reward model 220 via the ML module (e.g., the ML module 140) and train the reward model 220 using the ranked response pairs 228 as input. The reward model 220 may provide as an output the scalar reward 225.

[0087]

[0077] In one aspect, the scalar reward 225 may include a value numerically representing a human preference for the best and / or most expected response to a prompt, i.e., a higher scaler reward value may indicate the user is more likely to prefer that response, and a lower scalar reward may indicate that the user is less likely to prefer that response. For example, inputting the “winning” prompt-response (i.e., input-output) pair data to the reward model 220 may generate a winning reward. Inputting a “losing” prompt-response pair data to the same reward model 220 may generate a losing reward. The reward model 220 and / or scalar reward 225 may be updated based upon labelers ranking 226 additional prompt-response pairs generated in response to additional prompts 222.

[0078] In one example, a data labeler may provide to the SFT ML model 215 as an input prompt 222, “What are potential treatments for atrial fibrillation?.” The input may be provided by the labeler via the user device 102 over network 110 to the server 204 running an application utilizing the SFT ML model 215. The SFT ML model 215 may provide as output responses to the labeler via the user device 102: (i) “medication” 224A; (ii) “medications such as a beta blocker” 224B; and (iii) “there are no treatments” 224C. The data labeler may rank 226, via labeling the prompt-response pairs, prompt-response pair 222 / 224B as the most preferred answer; prompt-response pair 222 / 224A as a less preferred answer; and prompt-response 222 / 224C as the least preferred answer. The labeler may rank 226 the prompt-response pair data in any suitable manner. The ranked prompt-response pairs 228 may be provided to the reward model 220 to generate the scalar reward 225.

[0088]

[0079] While the reward model 220 may provide the scalar reward 225 as an output, the reward model 220 may not generate a response {e.g., text). Rather, the scalar reward 225 may be used by a version of the SFT ML model 215 to generate more accurate responses to prompts, i.e., the SFT model 215 may generate the response such as text to the prompt, and the reward model 220 may receive the response to generate a scalar reward 225 of how well humans perceive it. Reinforcement learning may optimize the SFT model 215 with respect to the reward model 220 which may realize the configured machine learning model 250.

[0089]

[0080] In one aspect, the server 206 may train the machine learning model 250 e.g., via the ML module 140) to generate a response 234 to a random, new and / or previously unknown user prompt 232. To generate the response 234, the machine learning model 250 may use a policy 235 e.g., algorithm) which it learns during training of the reward model 220, and in doing so may advance from the SFT model 215 to the machine learning model 250. The policy 235 may represent a strategy that the machine learning model 250 learns to maximize the reward 225. As discussed herein, based upon prompt-response pairs, a human labeler may continuously provide feedback to assist in determining how well the machine learning model’s 250 responses match expected responses to determine the rewards 225. The rewards 225 may feed back into the machine learning model 250 to evolve the policy 235. Thus, the policy 235 may adjust the parameters of the machine learning model 250 based upon the rewards 225 it receives for generating good responses. The policy 235 may update as the machine learning model 250 provides responses 234 to additional prompts 232.

[0090]

[0081] In one aspect, the response 234 of the machine learning model 250 using the policy 235 based upon the reward 225 may be compared using a cost function 238 to the SFT ML model 215 (which may not use a policy) response 236 of the same prompt 232. The cost function 238 may be trained in a similar manner and / or contemporaneous with the reward model 220. The server 206 may compute a cost 240 based upon the cost function 238 of the responses 234, 236. The cost 240 may reduce the distance between the responses 234, 236, i.e., a statistical distance measuring how one probability distribution is different from a second, in one aspect the response 234 of the machine learning model 250 versus the response 236 of the SFT model 215. Using the cost 240 to reduce the distance between the responses 234, 236 may avoid a server over-optimizing the reward model 220 and deviating too drastically from the human- intended / preferred response. Without the cost 240, the machine learning model 250 optimizations may result in generating responses 234 which are unreasonable but may still result in the reward model 220 outputting a high reward 225.

[0091]

[0082] In one aspect, the responses 234 of the machine learning model 250 using the current policy 235 may be passed by the server 206 to the rewards model 220, which may return the scalar reward 225. The machine learning model 250 response 234 may be compared via the cost function 238 to the SFT ML model 215 response 236 by the server 206 to compute the cost 240. The server 206 may generate a final reward 242 which may include the scalar reward 225 offset and / or restricted by the cost 240. The final reward 242 may be provided by the server 206 to the machine learning model 250 and may update the policy 235, which in turn may improve the functionality of the machine learning model 250.

[0092]

[0083] To optimize the machine learning model 250 over time, RLHF via the human labeler feedback may continue ranking 226 responses of the machine learning model 250 versus outputs of earlier / other versions of the SFT ML model 215, i.e., providing positive or negative rewards 225. The RLHF may allow the servers (e.g., servers 204, 206) to continue iteratively updating the reward model 220 and / or the policy 235. As a result, the machine learning model 250 may be retrained and / or fine-tuned based upon the human feedback via the RLHF process, and throughout continuing conversations may become increasingly efficient.

[0093]

[0084] Although multiple servers 202, 204, 206 are depicted in the exemplary block and logic diagram 200, each providing one of the three steps of the overall machine learning model 250 training, fewer and / or additional servers may be utilized and / or may provide the one or more steps of the machine learning model 250 training. In one aspect, one server may provide the entire machine learning model 250 training.

[0094] Exemplary Mapping of a Patient Chart

[0095]

[0085] Fig. 3 depicts an exemplary mapping of a patient healthcare record as related to an input query. The mapping 300 is made up of cluster graphs. The cluster graphs may be differently colored to indicate different clustering groups. In some aspects, the clustering graphs indicate the strength of connections between clustered dated based the depicted size of the connections between nodes (which may be bubbles herein). In some aspects, clusters may be connected to one or more other connectors, as shown through connections between nodes (bubbles). The nodes of the mapping may be queries or pieces of information that are related to the query. That is, in some examples, the nodes in the mapping overall or in particular clusters of the mapping may themselves be LLM created summaries of data, termed bubbles, connected to other bubbles of LLM created summaries of data based on vector distance determined by the LLM. In some implementations, pieces of information that are more closely related may be graphically displayed as being closer to each other.

[0096]

[0086] In some implementations, each cluster may represent one or morepatients with similar medical histories and conditions (e.g., a patient who is a nearest-match). A trained machine learning model may retrieve such other patients based on the query so that a medical professional may compare the patients. For example, if a medical professional is seeing a patient with atrial fibrillation, the large language model could automatically retrieve healthcare records of other patients with atrial fibrillation and who are medically similar to the patient being treated. The information from the other patients may be graphically displayed, creating a cloud of patients who are nearest-matches to the patient being treated. On the mapping, patients who are similar to the patient being treated may be displayed as closer to the patient being treated.

[0097]

[0087] Selection on any given bubble in a cluster provides an LLM created summary, and a list of similar notes / patients based on vector distance. It is recognized that hyperparameters can be adjusted to define the number of clusters. For example, selecting the bubble 302 causes the summary 304 to be displayed.

[0098]

[0088] As described above, in some aspects, the LLM’s herein are multimodal LLM’s trained on multiple data types and therefore configured to generate mappings that include multiple image types, such as mappings that include medical images or medical image derived data that is then graphically clustered and their vector distances depicted through the mapping. In some aspects, these mappings then can be used to identify changes over time in medical data, such as changes in tumor progression, etc. Examples of medical images includes radiology images (X-ray, chest radiology, etc.), histopathology images, magnetic resonance images, ultrasound images, nuclear medicine imaging, etc.

[0099] Exemplary Architecture

[0100]

[0089] FIG. 4 depicts a block diagram of exemplary architecture for generating a chart and nearest neighbor mapping.

[0090] An architecture 400 may include an electronic healthcare record database 402. The electronic healthcare record database 402 may store the healthcare records of patients (e.g., patient medical history). Electronic healthcare records may include information about patients such as demographics, notes, medications, vital signs, immunizations, laboratory data, reports, etc. The electronic healthcare records may include text and / or image data. The architecture 400 may also include a notes database 404, which may include notes and other information about particular medical specialties (e.g., oncology, cardiology, etc.).

[0101]

[0091] The electronic healthcare records in the electronic healthcare record database 402 and notes database 404 may be converted into vector representations of the text (i.e. , embeddings) via the machine learning engine 412. The machine learning engine 412 may include an LLM that may be used to generate the vector representations of text. The vector representations of the health records may be stored in a vector database 406. In some embodiments, the electronic healthcare records may include images (e.g., x-rays, ECGs). A multimodal model may be used to generate the vector representations of the images. The vector embeddings may be used to retrieve data from the electronic healthcare records database 402 and notes database 404.

[0102]

[0092] A user may interact (e.g., via an input device 126) with a user computing device 406 (e.g., user computing device 102 of FIG. 1) to provide a prompt 410 to the machine learning engine 412. The prompt 410 may be converted into a vector representation by the machine learning engine 412. The prompt vectors may be compared to the electronic healthcare record vectors of a particular patient’s electronic healthcare record stored in the vector database 406. The vectorized distance between the prompt vectors and electronic healthcare records may be used to retrieve relevant (i.e., the vector distance between the vectors of the prompt and the vectors of the parts of an electronic healthcare record are within a threshold) information (e.g., medical chart data, healthcare record notes) from the electronic healthcare record database 402. In some embodiments, the electronic healthcare records of other patients may be retrieved and provided to the machine learning engine 412. In some embodiments, the prompt vectors may also be compared to vectors representing notes from the notes database 404. The vectorized distance between the prompt vectors and notes may be used to retrieve relevant information from the notes database 404.

[0103]

[0093] The relevant portions of the electronic healthcare record are included with the prompt 410 (i.e., augment the prompt) and provided to the machine learning engine 412. The machine learning engine 410 may include one or more machine learning models (e.g., machine learning models 215, 250), and is trained (e.g., via a training process described in FIG. 2) to generate an output 414 corresponding to the prompt 410. The machine learning engine 412 may include a large language model (LLM), a multi-modal machine learning model, and / or generative models The machine learning engine 412 may be trained to generate a response to a query for a specific patient. In some embodiments, the machine learning models may be trained to identify other, similar patients and / or patients with similar diagnoses as the specific patient. For example, for a specific patient with atrial fibrillation, the machine learning models may be trained to identify other patients with atrial fibrillation who have similar attributes and medical histories as the specific patient. In some embodiments, the machine learning models may also be trained to generate a score for treatments based on the outcomes rank the outcomes (e.g., hospitalization, mortality, arrythmia-free survival, etc.) of a particular treatment.

[0104]

[0094] In some embodiments, a machine learning model included in the machine learning engine 412 may use the embeddings to determine nearest neighbor data elements relevant to the prompt 410 (e.g., an input query). As described above, patient electronic healthcare records may be converted into vector representations. The nearest neighbor search locates a set of nearest neighbors or all neighbors within a specified distance to the prompt 410, based on a specified distance metric (e.g., distance between the vectors representing the electronic healthcare records). The machine learning model may be an LLM or another machine learning model. In some embodiments, an LLM may access a separate nearest neighbor model. In some embodiments, the nearest neighbor model may be trained, retrained, and / or fine-tuned to a healthcare provider’s feedback. For example, the parameters of the nearest neighbor search may be adjusted to reduce the number of nearest neighbors identified (e.g., generate more relevant data elements) or increase the number of nearest neighbors (e.g., broaden a search for relevant data elements). In some embodiments, the score for treatments based on the outcomes may be used to adjust the nearest neighbor model. For example, the model may be adjusted such that only nearest neighbor treatments within a set distance from the score for the treatments may be included in the mapping. The nearest neighbor search may utilize techniques and algorithms such as a linear search, space partitioning, local sensitivity hashing, approximate nearest neighbor, fixed-radius nearest neighbors, etc. In some embodiments, a search for nearest neighbor data elements may include retrieving a specified number of data elements that are closest to the input query. For example, the model may be trained to retrieve the ten closest (i.e., most similar) data elements. In some embodiments, a search for nearest data elements may be a function of the distance between the data elements. For example, data elements whose distance is at most 2 times the distance of an exact nearest neighbor may be retrieved. Such techniques may retrieve additional data elements not contemplated by a user inputting the prompt 410. For example, a prompt 410 asking for outcomes of potential treatments for atrial fibrillation may retrieve healthcare information of patients who are of the same race as the subject patient as well as healthcare information of patients who are not of the same race but are similar to the patient in other ways (e.g., vital readings, height, weight, gender, other medical conditions, etc.) or vice versa.

[0105]

[0095] The output 414 may include a mapping 414a and one or more summaries 414b. The relevant information from the electronic healthcare record database 402 may be used to generate a map (e.g., a thought cloud) including bubbles of information that is responsive to the query. The mapping 414a may graphically display information that is most relevant to the query as closer to a graphical representation of the query. For example, a distance between a vector representing the input query and vectors representing extracted data elements may be determined, and the positioning of the extracted data elements displayed in the mapping may be based on the determined distance. The distance between the input query and the extracted data elements may indicate how closely related, relevant, and / or similar (e.g., in terms of subject matter, demographics, genotype, phenotype, medical history, diagnoses, treatment, treatment success, etc.) the input query and the extracted data elements are. A bubble representing a first extracted data element that is depicted as closer to a bubble representing an input query may indicate that the data element is more closely related (i.e. , has a smaller distance) to the prompt 410 (e.g., an input query) than a bubble representing a second extracted datapoint that is depicted further away. For example, for a prompt 410 asking about a specific diagnosis, information that is more relevant to the diagnosis may be depicted more closely to the prompt 410. In another example, a patient that is more demographically similar to the patient indicated in the prompt 410 may be depicted more closely to the patient of the prompt 410. In some embodiments, the mapping 414a may be color-coded such that each color corresponds to a grouping of a feature (e.g., extracted data elements). For example, bubbles related to one diagnosis may be in one color, and bubbles related to another diagnosis may be in another color. In some embodiments, the mapping 414a may include the notes of similar patients. For example, for a patient with atrial fibrillation, a mapping 414a may include a graphical representation (e.g., a circle or bubble) of the patient with atrial fibrillation surrounded by graphical representations of other similar patients. The mapping of similar patients may be color-coded. For example, bubbles representing patients with a similar condition as a subject patient mentioned in the input query may be color-coded with the same color in a mapping that includes the subject patient and similar patients. In some embodiments, a machine learning model (e.g., an LLM) of the machine learning engine 412 may determine the number of bubbles (e.g., connections to data elements related to the input query), the grouping of bubbles representing the extracted data elements, and the distance on the graphical display of the mapping between the groupings of the bubbles and / or between the bubbles and the input query. In some embodiments, groupings may be generated by a machine learning model (e.g., an LLM) of the machine learning engine 412. In some embodiments, the groupings may be determined based on the electronic healthcare record rather than generated by the machine learning model.

[0106]

[0096] The output 414 may include one or more summaries 414b. The summaries 414b may be responsive to the query. For example, a summary 414b may include a ranked list of treatments for atrial fibrillation in response to a query for treatments for atrial fibrillation. In some embodiments, the summaries 414b may include a list of patients and / or patient notes that are similar to a particular patient and / or a particular patient’s notes. The summaries 414b may be viewed by interacting (e.g., via the user computing device 406) with the bubbles in the mapping 414a. For example, the mapping 414a may initially only display bubbles, and selecting a bubble may cause a summary 414b of the information of a bubble to be displayed.

[0107] Exemplary Computer-Implemented Methods

[0108]

[0097] FIG. 5 depicts an example of a computer-implemented method 500 for generating a mapping of a patient chart using machine learning, in accordance with various embodiments herein, and as may be implemented by the exemplary computing environment 100.

[0109]

[0098] At block 502, the method 500 includes receiving an input query associated with a patient. The input query may be received via one or more processors (e.g., the processor 160 of the server 104 in FIG. 1 ) from a computing device (e.g., computing device 102 in FIG. 1 ). In some implementations, the method 500 may include identifying, by the one or more processors, a medical specialty from the input query. The method 500 may include retrieving a first set of context data corresponding to the medical specialty, and retrieving a second set of context data corresponding to the input query. The context data corresponding to the medical specialty may be stored in and retrieved from a database such as the notes database 116 of FIG. 1 . The method 500 may include injecting the first and second set of context data into the input query.

[0110]

[0099] At block 504, the method 500 includes obtaining, by the one or more processors (e.g., processor 160), a healthcare record associated with the patient. The context data corresponding to the input query may correspond to patient data from an electronic healthcare record and be stored in and retrieved from a database such as the electronic healthcare record database 108 of FIG. 1 .

[0111]

[0100] At block 506, the method 500 includes analyzing, by the one or more processors (e.g., processor 160), the healthcare record associated with the patient to extract data elements relevant to the input query by using a trained machine learning model (e.g., a model of the machine learning engine 412). In some implementations, the machine learning model may be a large language model that has been pre-trained, fine-tuned, and / or trained using historical healthcare record data. In some implementations, the trained machine learning model may be a multi-modal machine learning model.

[0112]

[0101] At an optional block 508, the method 500 may include searching, using a trained machine learning model (e.g., a model of the machine learning engine 412), for nearest neighbors using embeddings to extract additional data elements (e.g., information) relevant to the input query.

[0113]

[0102] At block 510, the method 500 includes generating, via the one or more processors and using the trained machine learning model, a first mapping the extracted data elements relevant to the query. In some implementations, the positioning of the extracted data elements as displayed on the mapping may be based on a distance to the input query. In some embodiments, the distance may be a vector distance. Data elements that are more closely related (i.e., have a smaller distance) to the input query may be depicted closer together, while data elements that are less related may be depicted as farther apart.

[0114]

[0103] In some implementations, the method 500 may further include obtaining healthcare records associated with one or more additional patients. The method 500 may include obtaining, one or more additional healthcare records associated with one or more additional patients from an electronic healthcare record database 108. The trained machine learning model (e.g., a model of the machine learning engine 412) may determine a distance indicating a level of similarity between the first patient associated with the healthcare record and one or more additional patients based on the one or more additional healthcare records. For example, the first and second patients’ healthcare records may be broken down into smaller portions representing different features of the patient, and each portion may be represented by a vector. The distance between the vectors representing features of the first patient and the vectors representing features of the second patient may be calculated to determine the level of similarity between the two patients. The method 500 may include using the machine learning model to identify a set of patients of the one or more additional patients, wherein for each patient included in the set the distance between the first patient and the one or more additional patients is less than a threshold distance (e.g., a patient who is a nearest-match). The method 500 may include using the trained machine learning model to generate a second mapping of the first patient to each of the one or more additional patients included in the set of patients.

[0104] In some implementations, the method may include generating a summary of one or more recommendations for treatment based on one or more additional healthcare records associated with the one or more additional patients included in the set of patients. In some implementations, the method may include retrieving (e.g., from an electronic healthcare record database 108) an outcome of a treatment used on each of the one or more additional patients included in the set of patients from the one or more additional healthcare records associated with the one or more additional patients included in the set of patients. The method may include using the trained machine learning model to generate a score evaluating the outcome of the treatment used on each of the one or more additional patients included in the set of patients. The method may include using the trained machine learning mode to rank the one or more outcomes based on the scores.

[0115]

[0105] Thus, as detailed herein, the present application provides systems and methods for the use of a graphical representation for rapid review of a patient chart. In this context, large language model utilizing retrievable augmented generation would take a query, assess the vectorized distance between the query regarding a specific patient, and pertinent notes in the chart and create a thought cloud, which is color coded. Thus, those pieces of information in the chart that are closest to the query being requested for a specific patient would be graphically displayed as being closest. This could also be done with text where the pertinent text notes could be rapidly reviewed and displayed. This would allow a clinician to immediately review the germane parts of the note based on their clinical background and clinical questions.

[0116]

[0106] Additionally or alternatively, some embodiments may use the disclosed methods to compare similar patients. Thus, by means of a specific example, if a clinician is seeing a patient with atrial fibrillation, the large language model could automatically pull patients who have similar medical histories and summarize the recommendations created by experts to see how other experts would have treated the patients. The large language model can thus generate a mapping of such data, including clusters of different summarized recommendations, in some examples. The mappings may represent nearest neighbor determinations across any of a variety of data upon which the large language model has been trained. An extension of this approach would be to also rank the outcomes, whether it is in terms of hospitalization, mortality, arrhythmia-free survival, or other germane outcomes for similarly treated patients, thus creating a cloud of patients who are nearest-matches, with vector distance between the indexed patient and other similar patients. The large language model can thus generate a mapping of such data. In some aspects, the generated maps use the length and / or thickness of nodal connections to indicate the vector closeness of various nodes to the patient, to a diagnosis of the patient, to a treatment of the patient, and / or to other data herein.

[0117]

[0107] Furthermore, the generation may be improved via a combination of pulling specialtyspecific notes that are recent as well as query-derived notes with the closest vector distance to the index query to generate a cohort of data input that is germane, so the generative data becomes clinically useful and actionable.

[0118]

[0108] While this disclosure refers to chart review and patient nearest-matches, the concept of contextual adjancy and text vector distance with graphical display can be apply in any field, for example review of competitive applications for an RFA. Furthermore, while these are specific embodiments as related to utilizing examples of cardiology and heart rhythm patients, it is recognized that this would be generalizable and widely useable by means of large language models that are fine tuned to for medical utilization, deploy retrievable augmented generation, and would have access to HPI.

[0119]

[0109] The following list of examples reflects a variety of embodiments contemplated by the present disclosure:

[0120]

[0110] Example 1 . A computer-implemented method comprising: receiving, via one or more processors, an input query associated with a patient; obtaining, via the one or more processors, a healthcare record associated with the patient; analyzing, via the one or more processors and using a trained machine learning model, the healthcare record associated with the patient to extract data elements relevant to the input query; and generating, via the one or more processors and using the trained machine learning model, a first mapping of the extracted data elements relevant to the query.

[0121]

[0111] Example 2. The method of example 1 , wherein the trained machine learning model is (a) a large language model (LLM) pre-trained, fine-tuned, and / or trained using historical healthcare record data, or (b) a multi-modal machine learning model.

[0122]

[0112] Example 3. The method of example 1 , further comprising: obtaining, via one or more processors, one or more additional healthcare records associated with one or more additional patients; determining, via the one or more processors and using the machine learning model, a distance between the first patient associated with the healthcare record and one or more additional patients based on the one or more additional healthcare records; identifying, via the one or more processors and using the machine learning model, a set of patients of the one or more additional patients, wherein for each patient included in the set the distance between the first patient and the one or more additional patients is less than a threshold distance; and generating, via the one or more processors and using the trained machine learning model, a second mapping of the first patient to each of the one or more additional patients included in the set of patients.

[0123]

[0113] Example 4. The method of example 3, further comprising: generating, via the one or more processors and using the machine learning model, a summary of one or more recommendations for treatment based on one or more additional healthcare records associated with the one or more additional patients included in the set of patients.

[0124]

[0114] Example 5. The method of example 4, further comprising: retrieving, via the one or more processors, an outcome of a treatment used on each of the one or more additional patients included in the set of patients from the one or more additional healthcare records associated with the one or more additional patients included in the set of patients; generating, via the one or more processors and using the trained machine learning model, a score evaluating the outcome of the treatment used on each of the one or more additional patients included in the set of patients; and ranking, via the one or more processors and using the trained machine learning model, the one or more outcomes based on the scores.

[0125]

[0115] Example 6. The method of example 1 , further comprising: identifying, via the one or more processors, a medical specialty from the input query; retrieving, via the one or more processors, a first set of context data corresponding to the medical specialty; retrieving, via the one or more processors, a second set of context data corresponding to the input query; and injecting, via the one or more processors, the first set of context data and second set of context data into the input query.

[0126]

[0116] Example 7. The method of example 1 , wherein a positioning of the extracted data elements displayed in the first mapping is based on a distance indicating a level of relation to the input query.

[0127]

[0117] Example 8. A computing system comprising: one or more processors, and one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing system to: receive an input query associated with a patient; obtain a healthcare record associated with the patient; analyze the healthcare record associated with the patient to extract data elements relevant to the input query; and generate a first mapping of the extracted data elements relevant to the query.

[0128]

[0118] Example 9. The computing system of example 8, wherein the trained machine learning model is (a) a large language model (LLM) pre-trained, fine-tuned and / or trained using historical healthcare data or (b) a multi-modal machine learning model.

[0119] Example 10. The computing system of example 8, the one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, further cause the computing system to: obtain one or more additional healthcare records associated with one or more additional patients; determine a distance between the first patient associated with the healthcare record and one or more additional patients based on the one or more additional healthcare records; identify a set of patients of the one or more additional patients, wherein for each patient included in the set the distance between the first patient and the one or more additional patients is less than a threshold distance; and generate a second mapping of the first patient to each of the one or more additional patients included in the set of patients.

[0129]

[0120] Example 11 . The computing system of example 10, the one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, further cause the computing system to: generate a summary of one or more recommendations for treatment based on one or more additional healthcare records associated with the one or more additional patients included in the set of patients.

[0130]

[0121] Example 12. The computing system of example 11 , the one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, further cause the computing system to: retrieve an outcome of a treatment used on each of the one or more additional patients included in the set of patients from the one or more additional healthcare records associated with the one or more additional patients included in the set of patients; generate a score evaluating the outcome of the treatment used on each of the one or more additional patients included in the set of patients; and rank the one or more outcomes based on the scores.

[0131]

[0122] Example 13. The computing system of example 8, the one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, further cause the computing system to: identify a medical specialty from the input query; retrieve a first set of context data corresponding to the medical specialty; retrieve a second set of context data corresponding to the input query; and inject the first set of context data and second set of context data into the input query.

[0132]

[0123] Example 14. The computing system of example 8, wherein a positioning of the extracted data elements displayed in the first mapping is based on a distance to the input query.

[0133]

[0124] Example 15. A non-transitory computer-readable medium, having stored thereon instructions that when executed, cause a computer to: receive an input query associated with a patient; obtain a healthcare record associated with the patient; analyze the healthcare record associated with the patient to extract data elements relevant to the input query; and generate a first mapping of the extracted data elements relevant to the query.

[0134]

[0125] Example 16. The non-transitory computer-readable medium of example 15, wherein the trained machine learning model is (a) a large language model (LLM) pre-trained, fine-tuned and / or trained using historical healthcare data or (b) a multi-modal machine learning model.

[0135]

[0126] Example 17. The non-transitory computer-readable medium of example 15, having stored thereon instructions that when executed, further cause a computer to: obtain one or more additional healthcare records associated with one or more additional patients; determine a distance between the first patient associated with the healthcare record and one or more additional patients based on the one or more additional healthcare records; identify a set of patients of the one or more additional patients, wherein for each patient included in the set the distance between the first patient and the one or more additional patients is less than a threshold distance; and generate a second mapping of the first patient to each of the one or more additional patients included in the set of patients.

[0136]

[0127] Example 18. The non-transitory computer-readable medium of example 17, having stored thereon instructions that when executed, further cause a computer to: generate a summary of one or more recommendations for treatment based on one or more additional healthcare records associated with the one or more additional patients included in the set of patients.

[0137]

[0128] Example 19. The non-transitory computer-readable medium of example 17, having stored thereon instructions that when executed, further cause a computer to: retrieve an outcome of a treatment used on each of the one or more additional patients included in the set of patients from the one or more additional healthcare records associated with the one or more additional patients included in the set of patients; generate a score evaluating the outcome of the treatment used on each of the one or more additional patients included in the set of patients; and rank the one or more outcomes based on the scores.

[0138]

[0129] Example 20. The non-transitory computer-readable medium of example 15, having stored thereon instructions that when executed, further cause a computer to: identify a medical specialty from the input query; retrieve a first set of context data corresponding to the medical specialty; retrieve a second set of context data corresponding to the input query; and inject the first set of context data and second set of context data into the input query.

[0130] Example 21 . The non-transitory computer-readable medium of example 15, wherein a positioning of the extracted data elements displayed in the first mapping is based on a distance to the input query.

[0139] Additional Considerations

[0140]

[0131] The following considerations also apply to the foregoing discussion. Throughout this specification, plural instances may implement operations or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may be performed concurrently, and nothing requires that the operations be performed in the order illustrated. These and other variations, modifications, additions, and improvements fall within the scope of the subject matter herein.

[0141]

[0132] It should also be understood that, unless a term is expressly defined in this patent using the sentence "As used herein, the term " " is hereby defined to mean . . . " or a similar sentence, there is no intent to limit the meaning of that term, either expressly or by implication, beyond its plain or ordinary meaning, and such term should not be interpreted to be limited in scope based on any statement made in any section of this patent (other than the language of the claims). To the extent that any term recited in the claims at the end of this patent is referred to in this patent in a manner consistent with a single meaning, that is done for sake of clarity only so as to not confuse the reader, and it is not intended that such claim term be limited, by implication or otherwise, to that single meaning. Finally, unless a claim element is defined by reciting the word "means" and a function without the recital of any structure, it is not intended that the scope of any claim element be interpreted based on the application of 35 U.S.C. § 112(f).

[0142]

[0133] Unless specifically stated otherwise, discussions herein using words such as "processing," "computing," "calculating," "determining," "presenting," "displaying," or the like may refer to actions or processes of a machine (e.g., a computer) that manipulates or transforms data represented as physical (e.g., electronic, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or a combination thereof), registers, or other machine components that receive, store, transmit, or display information.

[0143]

[0134] As used herein any reference to "one aspect" or "an aspect" means that a particular element, feature, structure, or characteristic described in connection with the aspect is included in at least one aspect. The appearances of the phrase "in one aspect" in various places in the specification are not necessarily all referring to the same aspect.

[0135] As used herein, the terms "comprises," "comprising," "includes," "including," "has," "having" or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, "or" refers to an inclusive or and not to an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).

[0144]

[0136] In addition, use of "a" or "an" is employed to describe elements and components of the aspects herein. This is done merely for convenience and to give a general sense of the invention. This description should be read to include one or at least one and the singular also includes the plural unless it is obvious that it is meant otherwise.

[0145]

[0137] Upon reading this disclosure, those of skill in the art will appreciate still additional alternative structural and functional designs for implementing the concepts disclosed herein, through the principles disclosed herein. Thus, while particular aspects and applications have been illustrated and described, it is to be understood that the disclosed aspects are not limited to the precise construction and components disclosed herein. Various modifications, changes and variations, which will be apparent to those skilled in the art, may be made in the arrangement, operation and details of the method and apparatus disclosed herein without departing from the spirit and scope defined in the appended claims.

Claims

What is claimed is:1 . A computer-implemented method comprising: receiving, via one or more processors, an input query associated with a patient; obtaining, via the one or more processors, a healthcare record associated with the patient; analyzing, via the one or more processors and using a trained machine learning model, the healthcare record associated with the patient to extract data elements relevant to the input query; and generating, via the one or more processors and using the trained machine learning model, a first mapping of the extracted data elements relevant to the query.

2. The method of claim 1 , wherein the trained machine learning model is (a) a large language model (LLM) pre-trained, fine-tuned, and / or trained using historical healthcare record data, or (b) a multi-modal machine learning model.

3. The method of claim 1 , further comprising: obtaining, via one or more processors, one or more additional healthcare records associated with one or more additional patients; determining, via the one or more processors and using the machine learning model, a distance between the first patient associated with the healthcare record and one or more additional patients based on the one or more additional healthcare records; identifying, via the one or more processors and using the machine learning model, a set of patients of the one or more additional patients, wherein for each patient included in the set the distance between the first patient and the one or more additional patients is less than a threshold distance; and generating, via the one or more processors and using the trained machine learning model, a second mapping of the first patient to each of the one or more additional patients included in the set of patients.

4. The method of claim 3, further comprising: generating, via the one or more processors and using the machine learning model, a summary of one or more recommendations for treatment based on one or more additional healthcare records associated with the one or more additional patients included in the set of patients.

5. The method of claim 4, further comprising: retrieving, via the one or more processors, an outcome of a treatment used on each of the one or more additional patients included in the set of patients from the one or more additional healthcare records associated with the one or more additional patients included in the set of patients; generating, via the one or more processors and using the trained machine learning model, a score evaluating the outcome of the treatment used on each of the one or more additional patients included in the set of patients; and ranking, via the one or more processors and using the trained machine learning model, the one or more outcomes based on the scores.

6. The method of claim 1 , further comprising: identifying, via the one or more processors, a medical specialty from the input query; retrieving, via the one or more processors, a first set of context data corresponding to the medical specialty; retrieving, via the one or more processors, a second set of context data corresponding to the input query; and injecting, via the one or more processors, the first set of context data and second set of context data into the input query.

7. The method of claim 1 , wherein a positioning of the extracted data elements displayed in the first mapping is based on a distance to the input query.

8. A computing system comprising: one or more processors, and one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing system to: receive an input query associated with a patient; obtain a healthcare record associated with the patient; analyze, using a trained machine learning model, the healthcare record associated with the patient to extract data elements relevant to the input query; and generate a first mapping of the extracted data elements relevant to the query.

9. The computing system of claim 8, wherein the trained machine learning model is (a) a large language model (LLM) pre-trained, fine-tuned and / or trained using historical healthcare data or (b) a multi-modal machine learning model.

10. The computing system of claim 8, the one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, further cause the computing system to: obtain one or more additional healthcare records associated with one or more additional patients; determine a distance between the first patient associated with the healthcare record and one or more additional patients based on the one or more additional healthcare records; identify a set of patients of the one or more additional patients, wherein for each patient included in the set the distance between the first patient and the one or more additional patients is less than a threshold distance; and generate a second mapping of the first patient to each of the one or more additional patients included in the set of patients.1 1. The computing system of claim 10, the one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, further cause the computing system to: generate a summary of one or more recommendations for treatment based on one or more additional healthcare records associated with the one or more additional patients included in the set of patients.

12. The computing system of claim 11 , the one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, further cause the computing system to: retrieve an outcome of a treatment used on each of the one or more additional patients included in the set of patients from the one or more additional healthcare records associated with the one or more additional patients included in the set of patients; generate a score evaluating the outcome of the treatment used on each of the one or more additional patients included in the set of patients; and rank the one or more outcomes based on the scores.

13. The computing system of claim 8, the one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, further cause the computing system to: identify a medical specialty from the input query; retrieve a first set of context data corresponding to the medical specialty; retrieve a second set of context data corresponding to the input query; and inject the first set of context data and second set of context data into the input query.

14. The computing system of claim 8, wherein a positioning of the extracted data elements displayed in the first mapping is based on a distance to the input query.

15. A non-transitory computer-readable medium, having stored thereon instructions that when executed, cause a computer to: receive an input query associated with a patient; obtain a healthcare record associated with the patient; analyze, using a trained machine learning model, the healthcare record associated with the patient to extract data elements relevant to the input query; and generate a first mapping of the extracted data elements relevant to the query.

16. The non-transitory computer-readable medium of claim 15, wherein the trained machine learning model is (a) a large language model (LLM) pre-trained, fine-tuned and / or trained using historical healthcare data or (b) a multi-modal machine learning model.

17. The non-transitory computer-readable medium of claim 15, having stored thereon instructions that when executed, further cause a computer to: obtain one or more additional healthcare records associated with one or more additional patients; determine a distance between the first patient associated with the healthcare record and one or more additional patients based on the one or more additional healthcare records; identify a set of patients of the one or more additional patients, wherein for each patient included in the set the distance between the first patient and the one or more additional patients is less than a threshold distance; and generate a second mapping of the first patient to each of the one or more additional patients included in the set of patients.

18. The non-transitory computer-readable medium of claim 17, having stored thereon instructions that when executed, further cause a computer to: generate a summary of one or more recommendations for treatment based on one or more additional healthcare records associated with the one or more additional patients included in the set of patients.

19. The non-transitory computer-readable medium of claim 17, having stored thereon instructions that when executed, further cause a computer to: retrieve an outcome of a treatment used on each of the one or more additional patients included in the set of patients from the one or more additional healthcare records associated with the one or more additional patients included in the set of patients; generate a score evaluating the outcome of the treatment used on each of the one or more additional patients included in the set of patients; and rank the one or more outcomes based on the scores.

20. The non-transitory computer-readable medium of claim 15, having stored thereon instructions that when executed, further cause a computer to: identify a medical specialty from the input query; retrieve a first set of context data corresponding to the medical specialty; retrieve a second set of context data corresponding to the input query; and inject the first set of context data and second set of context data into the input query.21 . The non-transitory computer-readable medium of claim 15, wherein a positioning of the extracted data elements displayed in the first mapping is based on a distance to the input query.

Images