Method for determining completed actions and next steps in a flow model

Abstract

A method may include receiving, from a computing device, a prompt including a portion of the flow model and a node on the portion of the flow model; retrieving contextual data associated with the portion of the flow model; providing the prompt and the contextual data to a trained machine learning (ML) model to determine completion of one or more actions in the flow model; generating, by processing the prompt and contextual data with the trained ML model, one or more next steps based on the completion of one or more actions in the pathway; generating, using the ML model, a summary including a justification for the one or more next steps supported by one or more datapoints from the contextual data; storing the summary, the one or more completed actions, and a node associated with the completed actions; and displaying the one or more next steps on the computing device.

Description

METHOD FOR DETERMINING COMPLETED ACTIONS AND NEXT STEPS IN A FLOW MODELCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 602,522, filed November 24, 2023, which is incorporated herein by reference in its entirety.TECHNICAL FIELD

[0002] The present disclosure is generally directed to systems and methods for determining completed actions in a flow model or steps in a decision tree and generating next steps.BACKGROUND

[0003] Many processes that include a sequence of steps and decisions may be represented by a flow model, such as various healthcare processes. However, such flow models may be large and complex, involving many steps and decisions. For example, a discharge process may include decisions that branch off into multiple alternate paths.

[0004] Tracking progress through a flow model may be time-consuming and error prone. For example, efficient discharge processes are essential for improved patient outcomes and the operational efficiency of the hospital, but current discharge processes have proven to be inefficient due to delays in communication and information gathering. These delays result in higher costs for patients and reduced ability to optimize patient census and throughput. Identification of patient discharge needs is labor intensive, involving manual chart review, interpersonal information sharing at preset times (multidisciplinary conference), and is often limited to defined settings (e.g., intensive care unit (ICU) vs floor). Furthermore, manual formulation of discharge summaries is time consuming and addressing patient concerns following discharge is often sub-optimally coordinated. Similar challenges exist for other healthcare processes, such as a pre-operation / surgery process that determines whether a patient is ready for surgery.

[0005] Therefore, there is an opportunity for utilizing artificial intelligence and machine learning to assist in determining completed actions in a flow model and generating one or more next steps, such as discharging a patient and providing post-discharge care for a flow model representing a patient discharge process.BRIEF SUMMARY

[0006] In an embodiment, a method includes receiving, from a computing device, a prompt including a portion of the flow model and a node on the portion of the flow model; retrievingcontextual data associated with the portion of the flow model; providing, via the one or more processors, the prompt and the contextual data to a trained machine learning (ML) model to determine completion of one or more actions in the flow model; generating, by processing the prompt and contextual data with the trained machine learning model, one or more next steps based on the completion of one or more actions in the pathway; generating, via the one or more processors and using the ML model, a summary including a justification for the one or more next steps supported by one or more datapoints from the contextual data; storing, via the one or more processors, the summary, the one or more completed actions, and a node associated with the completed actions; and displaying, via the one or more processors, the one or more next steps on the computing device.BRIEF DESCRIPTION OF THE FIGURES

[0007] The figures described below depict various aspects of the system and methods disclosed therein. It should be understood that each figure depicts one aspect of a particular aspect of the disclosed system and methods, and that each of the figures is intended to accord with a possible aspect thereof. Further, wherever possible, the following description refers to the reference numerals included in the following figures, in which features depicted in multiple figures are designated with consistent reference numerals.

[0008] FIG. 1 depicts an exemplary computing environment in which the techniques disclosed herein may be implemented, according to some aspects, according to some aspects.

[0009] FIG. 2 depicts a combined block and logic diagram in which exemplary computer- implemented methods and systems for training machine learning are implemented, according to some aspects.

[0010] FIG. 3 depicts an exemplary flow model for a patient discharge process, according to some aspects.

[0011] FIG. 4 depicts a flow diagram of an exemplary computer-implemented method for determining completed actions in a flow model and generating next steps, according to some aspects.

[0012] FIG. 5 depicts a block diagram of exemplary architecture for determining completed actions and generating next steps, according to some aspects.

[0013] FIG. 6 depicts a combined block and flow diagram for determining completed actions and generating next steps in an exemplary care pathway flow model.

[0014] FIGS. 7A-7G depicts an exemplary flow model for a pre-operation / surgery process, according to some aspects.

[0015] The figures depict preferred aspects for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative aspects of the systems and methods illustrated herein may be employed without departing from the principles of the invention described herein.DETAILED DESCRIPTIONOverview

[0016] The present techniques provide methods and systems for, inter alia, using machine learning to determine completed actions in a flow model and identify one or more next steps. A prompt including a portion of the flow model and a node (e.g., an action or task) on the portion of the flow model may be received from a computing device. Contextual data associated with a portion of the flow model may be retrieved (e.g., from a database). The prompt and contextual data may be provided to a trained machine learning (ML) model to determine completion of one or more actions in the flow model. The trained machine learning model may use the prompt and contextual data to generate one or more steps based on the completion of one or more actions in the flow model. The trained machine learning model may generate a summary including a justification for the completed steps that are identified and the one or more next steps supported by one or more datapoints from the contextual data and the flow model. The summary, the one or more completed actions, and a node associated with the completed actions may be stored, and the one or more next steps may be displayed on the computing device.

[0017] As noted above, it may be difficult to track progress through a complex flow model, including flow models representing healthcare processes such as a discharge process. For example, manually tracking progress may be time-consuming and error-prone due to the large volume of tasks, decisions, and personnel involved in completing a process. Such a problem may be solved by using machine learning. However, using a machine learning model is not without risks of inaccuracies. For example, a machine learning model may generate an output that is irrelevant to the input or otherwise incorrect. Such inaccurate responses may be disastrous in a healthcare setting, as a patient’s life and well-being may be at risk. Additionally, training a model requires large volumes of data, and training a model on historical data may suffer from drawbacks such as lack of data or lack of sufficiently-labeled data. For example, for a healthcare process such as a patient discharge process, there may be a lack of patient data approved for training use that adequately covers different discharge situations.

[0018] To overcome these technical hurdles, the present application describes systems and methods that provide the determination of completed tasks in a flow model and generation of next steps via a machine learning model. The machine learning model can efficiently determine progress along a pathway. For example, the machine learning model can quickly and efficiently determine which tasks of a patient discharge process have been completed, and one or more next steps in a patient discharge process. Additionally, the machine learning model can provide justifications for determining a task is complete and recommending a next step by citing datapoints (e.g., numerical data, text data, image data, audio data, video data) from the contextual data and the flow model. For example, the machine learning model may generate a summary of the completed actions and next steps, including a justification citing portions of an electronic healthcare record. Furthermore, another machine learning model may be trained to generate labelled synthetic data, which may be used to train the machine learning model to determine progress in a flow model, and / or to measure accuracy in which the machine learning model is able to identify completed tasks in a flow model. For example, synthetic patient healthcare data may be generated to train a machine learning model that determines progress in a patient discharge process.

[0019] Thus, the techniques of the present disclosure provide a technical improvement over conventional techniques at least by improving the functionality of a computing device (e.g., server executing machine learning model). In particular, the computing device analyzes data using a machine learning model and determines completed actions in a flow model and generates one or more next steps. Performing these actions enables the tracking of progress through the flow model with an efficiency not achieved using conventional techniques. Moreover, the machine learning model generates a summary including a justification, citing contextual data, for determining that tasks are complete and one or more next steps, which enables accuracy that is not achieved using conventional techniques. That is, the present disclosure describes improvements in the functioning of the computer itself because the computing device more efficiently and accurately determines completed tasks and generates next steps as a direct result of the machine learning model. This improves over the prior art at least because existing systems are unable to track progress in a flow model with the accuracy and efficiency resulting from the disclosed machine learning model. Additionally, using another machine learning model to generate labelled synthetic testing data, such as synthetic patient healthcare data, improves the training process of other machine learning models. This improves over the prior art at least because existing systems must be trained on historical data which, due to external factors, may be limited, inadequately address enough scenarios to train a machine learning model, or inadequately labeled to train a machine learning model.Exemplary Computing Environment

[0020] FIG. 1 depicts an exemplary computing environment 100 in which the techniques disclosed herein may be implemented, according to some aspects. The environment 100 may include computing resources for training and / or operating machine learning models to handle flow models.

[0021] The computing environment 100 may include a client computing device 102, a server computing device 104, an electronic network 106, a context database 108, and a model electronic database 112. The computing environment may further include one or more cloud application programming interfaces (APIs) 114. The components of the computing environment 100 may be communicatively connected to one another via the electronic network 106, in some aspects.

[0022] The client computing device 102 may implement, inter alia, operation of one or more applications for performing steps in a flow model, i.e., a representation of a sequence of steps and decisions to perform a process. In some embodiments, the flow model may include a medical or healthcare process. For example, the flow model may include a process for facilitating patient discharge, a pre-operation or pre-surgery process managing cardiology issues such as admission for atrial fibrillation and heart failure, determining hospital admissions, diagnosing a disease, etc. In some aspects, the client computing device 102 may be implemented as one or more computing devices (e.g., one or more servers, one or more laptops, one or more mobile computing devices, one or more tablets, one or more wearable devices, one or more cloud-computing virtual instances, etc.). In some aspects, a plurality of client computing devices may be part of the environment 100 - for example, a first user may access a client computing device 102 that is a laptop, while a second user accesses the client computing device 102 that is a smart phone, while yet a third user accesses a client computing device 102 that is a wearable device.

[0023] The client computing device 102 may include one or more processors 120, one or more network interface controllers 122, one or more memories 124, an input device 126, an output device 128 and a client API 130. The one or more memories 124 may have stored thereon one or more modules 140 (e.g., one or more sets of instructions).

[0024] In some aspects, the one or more processors 120 may include one or more central processing units, one or more graphics processing units, one or more field-programmable gate arrays, one or more application-specific integrated circuits, one or more tensor processing units, one or more digital signal processors, one or more neural processing units, one or more RISC-V processors, one or more coprocessors, one or more specialized processors / accelerators forartificial intelligence (Al) or machine learning (ML) -specific applications, one or more microcontrollers, etc.

[0025] The client computing device 102 may include one or more network interface controllers 122, such as Ethernet network interface controllers, wireless network interface controllers, etc. The network interface controllers 122 may include advanced features, in some aspects, such as hardware acceleration, specialized networking protocols, etc.

[0026] The memories 124 of the client computing device 102 may include volatile and / or nonvolatile storage media. For example, the memories 124 may include one or more random access memories, one or more read-only memories, one or more cache memories, one or more hard disk drives, one or more solid-state drives, one or more non-volatile memory express, one or more optical drives, one or more universal serial bus flash drives, one or more external hard drives, one or more network-attached storage devices, one or more cloud storage instances, one or more tape drives, etc.

[0027] As noted, the memories 124 may have stored thereon one or more modules 140, for example, as one or more sets of computer-executable instructions. In some aspects, the modules 140 may include additional storage, such as one or more operating systems (e.g., Microsoft Windows, GNU / Linux, Mac OSX, etc.). The operating systems may be configured to run the modules 140 during operation of the client computing device 102 - for example, the modules 140 may include additional modules and / or services for receiving and processing data from one or more other components of the environment 100 such as the one or more cloud APIs 114 or the server computing device 104. The modules 140 may be implemented using any suitable computer programming language(s) (e.g., Python, JavaScript, C, C++, Rust, C#, Swift, Java, Go, LISP, Ruby, Fortran, etc.).

[0028] The modules 140 may include an API module 144, an input processing module 146, an authentication / security module 148, and a context module 150 in some aspects. In some aspects, more or fewer modules 140 may be included. The modules 140 may be configured to communicate with one another (e.g., via inter-process communication, via a bus, via sockets, pipes, message queues, etc.).

[0029] The API module 144 may include one or more sets of computer executable instructions for accessing one or more remote APIs, and / or for enabling one or more other components within the environment 100 to access functionality of the client computing device 102. In some aspects, the API module 144 may enable other client applications (i.e. , not applications facilitated by the modules 140) to connect to the client computing device 102, for example, to send queries or prompts, and to receive responses from the client computingdevice 102. The API module 144 may include instructions for authentication, rate limiting and error handling.

[0030] As noted, the client computing device 102 may enable one or more users to access one or more trained models by providing input prompts that are processed by one or more trained models. The input processing module 146 may perform pre-processing of user prompts prior to being input into one or more models, and / or post-processing of outputs output by one or more models. For example, the input processing module 146 may process data input into one or more input fields, voice inputs or other input methods (e.g., file attachments) depending upon the application. The input processing module 146 may receive inputs directly via the input device 126, in some aspects.

[0031] In some aspects, the input processing module 146 may perform post-processing of input received from one or more trained models. In some aspects, post-processing (and / or preprocessing) may include implementing content moderation mechanisms, to prevent misuses of trained models or inappropriate content generation. The input processing module 146 may include instructions for handling errors and for displaying errors to users (e.g., via the output device 128). The input processing module 146 may cause one or more graphical user interfaces to be displayed, for example to enable the user to enter information directly via a text field.

[0032] The authentication / security module 148 may include one or more sets of computerexecutable instructions for implementing access control mechanisms for one or more trained models, ensuring that the model can only be accessed by those who are authorized to do so, and that the access of those users is private and secure. It should be appreciated that the security module 148 may permission users based upon tasks in a flow model.

[0033] Generally, trained models, especially trained models, require state information in order to meaningfully carry on a dialogue with a user or with another trained model. For example, if a user prompts a trained model with a question such as “What is the weather in Chicago today?” followed by a second prompt “And how about tomorrow?” the model should understand that, in context, the second query relates to the first query, insofar as the user is asking about the weather tomorrow in the same location (Chicago).

[0034] However, language models (e.g., large language models (LLMs)) are generally stateless, meaning that after they process a prompt, they have no internal record or memory of the information that was input, or the information that was generated as part of the language model’s processing. Thus, many systems add statefulness to models using contextual data. This may be implemented using sliding context windows, wherein a predetermined number oftokens (e.g., 4096 maximum tokens in the case of GPT 3.5, equivalent to about 3000 words) may be “remembered” by the LLM and can be used to enrich multiple sequential prompts input into the LLM (for example, when the LLM is used in a chat mode).

[0035] The context module 150 may include one or more sets of computer-executable instructions for maintaining state of the type found in this example, and other types of state information. The context module 150 may implement sliding window context, in some aspects. In other aspects, the context module 150 may perform other types of state maintaining strategies. For example, the context module 150 may implement a strategy in which information from the immediately preceding prompt is part of the window, regardless of the size of that prior prompt.

[0036] In some aspects, the context module 150 may implement a strategy in which one or more prior prompts are included in each current prompt. This prompt stuffing technique, or prompt concatenation, may be limited by prompt size constraints — once the total size of the prompt exceeds the prompt limit, the model immediately loses state information related to parts of the prompt truncated from the prompt.

[0037] The server computing device 104 may include one or more processors 160, one or more network interface controllers 162, one or more memories 164, an input device (not depicted), an output device (not depicted) and a server API 166. The one or more memories 164 may have stored thereon one or more modules 170 (e.g., one or more sets of instructions).

[0038] In some aspects, the one or more processors 160 may include one or more central processing units, one or more graphics processing units, one or more field-programmable gate arrays, one or more application-specific integrated circuits, one or more tensor processing units, one or more digital signal processors, one or more neural processing units, one or more RISC-V processors, one or more coprocessors, one or more specialized processors / accelerators for artificial intelligence or machine learning-specific applications, one or more microcontrollers, etc.

[0039] The server computing device 104 may include one or more network interface controllers 162, such as Ethernet network interface controllers, wireless network interface controllers, etc. The network interface controllers 162 may include advanced features, in some aspects, such as hardware acceleration, specialized networking protocols, etc.

[0040] The memories 164 of the server computing device 104 may include volatile and / or non-volatile storage media. For example, the memories 164 may include one or more random access memories, one or more read-only memories, one or more cache memories, one or more hard disk drives, one or more solid-state drives, one or more non-volatile memory express, oneor more optical drives, one or more universal serial bus flash drives, one or more external hard drives, one or more network-attached storage devices, one or more cloud storage instances, one or more tape drives, etc.

[0041] As noted, the memories 164 may have stored thereon one or more modules 170, for example, as one or more sets of computer-executable instructions. In some aspects, the modules 170 may include additional storage, such as one or more operating systems (e.g., Microsoft Windows, GNU / Linux, Mac OSX, etc.). The operating systems may be configured to run the modules 170 during operation of the server computing device 104 - for example, the modules 170 may include additional modules and / or services for receiving and processing data from one or more other components of the environment 100 such as the one or more cloud APIs 114 or the client computing device 102. The modules 170 may be implemented using any suitable computer programming language(s) (e.g., Python, JavaScript, C, C++, Rust, C#, Swift, Java, Go, LISP, Ruby, Fortran, etc.).

[0042] In some aspects, the modules 170 may include a data collection module 172, a data pre-processing module 174, a model pretraining module 176, a fine-tuning module 178, a model training module 180, a checkpointing module 182, a hyperparameter tuning module 184, a validation and testing module 186, an auto-prompting module 188, an operating module 190, and a chatbot module 192. In some aspects, more or fewer modules 170 may be included. The modules 170 may be configured to communicate with one another (e.g., via inter-process communication, via a bus, via sockets, pipes, message queues, etc.). The modules 170 may respond to network requests (e.g., via the API 166) or other requests received via the network 106 (e.g., via the client computing device 102 or other components of the environment 100).

[0043] The data collection module 172 may be configured to collect information used to train one or more modules. In general, the information collected may be any suitable information used for training a language model. The data collection module 172 may collect data via web scraping, via API calls / access, via database extract-transform-load (ETL) processes, etc. Sources accessed by the data collection module 172 include social media websites, books, websites, academic publications, web forums / interest sites (e.g., Reddit, Facebook, bulletin boards, etc.), etc. The data collection module 172 may access data sources by active means (e.g., scraping or other retrieval) or may access existing corpuses. The data collection module 172 may include sets of instructions for performing data collection in parallel, in some aspects. The data collection module 172 may store collected data in one or more electronic databases, such as a database accessible via the cloud APIs 114 or via a local electronic database (not depicted). The data may be stored in a structured and / or unstructured format. In someaspects, the data collection module 172 may store large data volumes used for training one or more models (i.e., training data). For example, the data collection module 172 may store terabytes, petabytes, exabytes or more of training data.

[0044] In some aspects, the data collection module 172 may retrieve data from the context database 108. For example, the data collection module 172 may process the retrieved / received data and sort the data into multiple subsets based on information included within the context database 108. For example, the data collection module 172 may receive one or more sets of unstructured text (e.g., unstructured clinical notes, an electronic healthcare record, etc.). The data collection module 172 may chunk the data according to time (e.g., hourly, daily, quarterly, etc.). In some embodiments, data may be stored and / or updated in the context database 108. For example, an electronic healthcare record may be updated with additional patient data such as new measurements for a patient, new insurance information, a node (e.g., task or decision) indicating a patient’s progress in a healthcare flow model (e.g., a patient discharge flow model, a pre-operation / surgery flow model, etc.), completed actions in a flow model, next steps in a flow model, a summary including a justification for the next steps, etc. In some embodiments, the context database 108 may be updated in real-time. For example, an electronic healthcare record stored in the context database 108 may be updated in real-time. In some embodiments, the context database 108 may include one or more flow models that may be retrieved based on the prompt.

[0045] The model preprocessing module 174 may include instructions for pre-processing data collected by the data collection module 172. In particular, the model preprocessing module 174 may perform text extraction and / or cleaning operations on data collected by the data collection module 172. The data pre-processing module 174 may perform preprocessing operations, such as lexical parsing, tokenizing, case conversions and other string splitting / munging. In some aspects, the data collection module 172 may perform data deduplication, filtering, annotation, compliance, version control, validation, quality control, etc. In some aspects, one or more human reviewers may be looped into the process of preprocessing data collected by the data pre-processing module 174. For example, a distributed work queue may be used to transmit batch jobs and receive human-computed responses from one or more human workers. Once pre-processed, the data pre-processing module 174 may store copied and / or modified copies of the training data in an electronic database.

[0046] In some aspects the data pre-processing module 174 may include instructions for parsing the unstructured text received by the data collection module 172 to structure the text. For example, when the text relates to unstructured healthcare provider notes, text may belabeled according to the identity of one or more authors and / or one or more topics. For example, the data may be labeled with one or more keywords associated with the transcript. In some aspects, the present techniques may use a separate trained text summarization module to generate keywords used for this purpose. In this way, the data pre-processing module 174 may generate structured data corresponding to unstructured notes, such that the structured data is enriched with information about the meeting that is suitable for training. This structured data may be processed by downstream processes / modules.

[0047] Generally, the present techniques may train one or more models to perform language generation tasks that include token generation. Both training inputs and model outputs may be tokenized. Herein, tokenization refers to the process by which text used for training is divided into units such as words, subwords or characters. Tokenization may break a single word into multiple subwords (e.g., “LLM” may be tokenized as “L” and “LM”). The present techniques may train one or more models using a set of tokens (e.g., a vocabulary) that includes many (e.g., thousands or more) of tokens. These tokens may be embedded into a vector. This vector of token or “embeddings” may include numerical representations of the individual tokens in the vocabulary in high-dimensional vector space. The modules 170 may access and modify the embeddings during training to learn relationships between tokens. These relationships effectively represent semantic language meaning.

[0048] In some aspects, a specialized database (e.g., a vector store, a graph database, etc.) may be used to store and query the embeddings. Embedding databases may include specialized features, such as efficient retrieval, similarity search and scalability. For example, the server computing device 104 may include a local electronic embedding database (not depicted). In some aspects, a remote embedding database service may be used (e.g., via the cloud APIs 114). Such a remote embedding database service may be based on an open source or proprietary model (e.g., Milvus, Pinecone, Redis, Postgres, MongoDB, Facebook Al Similarity Search (FAISS), etc.). The server computing device 104 may include instructions (e.g., in the data collection module 172) for adding training data to one or more specialized databases, and for accessing it to train models.

[0049] The present techniques may include language modeling, wherein one or more deep learning models are trained by processing token sequences using a large language model architecture. For example, in some aspects, a transformer architecture may be used to process a sequence of tokens. Such a transformer model may include a plurality of layers including selfattention and feedforward neural networks. This architecture may enable the model to learn contextual relationships between the tokens, and to predict the next token in a sequence, basedupon the preceding tokens. During training, the model is provided with the sequence of tokens and it learns to predict a probability distribution over the next token in the sequence. This training process may include updating one or more model parameters (e.g., weights or biases) using an objective function that minimizes the difference between the predicted distribution and a true next token in the training data. Particular techniques for training are discussed in further detail below in Fig. 2.

[0050] Alternatives to the transformer architecture may include recurrent neural networks, long short-term memory networks, gated recurrent networks, convolutional neural networks, recursive neural networks, and other modeling architectures.

[0051] In some aspects, the modules 170 may include instructions for performing pretraining of a language model (e.g., an LLM), for example, in a pretraining module 176. The pretraining module 176 may include one or more sets of instructions for performing pretraining, which as used herein, generally refers to a process that may span pre-processing of training data via the data pre-processing module 174 and initialization of an as-yet untrained language model. In general, a pre-trained model is one that has no prior training of specific tasks. For example, the model pretraining module 176 may include instructions that initialize one more model weights. In some aspects, model pretraining module 176 may initialize the weights to have random values. The model pretraining module 176 may train one or more models using unsupervised learning, wherein the one or more models process one or more tokens (e.g., preprocessed data output by the data pre-processing module 174) to learn to predict one or more elements (e.g., tokens). The model pretraining module 176 may include one or more optimizing objective functions that the model pretraining module 176 applies to the one or more models, to cause the one or more models to predict one or more most-likely next tokens, based on the likelihood of tokens in the training data. In general, the model pretraining module 176 causes the one or more models to learn linguistic features such as grammar and syntax. The pretraining module 176 may include additional steps, including training, data batching, hyperparameter tuning and / or model checkpointing.

[0052] The model pretraining module 176 may include instructions for generating a model that is pretrained for a general purpose, such as general text processing / understanding. This model may be known as a “base model” in some aspects. The base model may be further trained by downstream training process(es), for example, those training processes described with respect to the fine-tuning module 178. The model pretraining module 176 generally trains foundational models that have general understanding of language and / or knowledge. Pretraining may be a distinct stage of model training in which training data of a general anddiverse nature (i.e., not specific to any particular task or subset of knowledge) is used to train the one or more models. In some aspects, a single model may be trained and copied. Copies of this model may serve as respective base models for a plurality of fine-tuned models.

[0053] In some aspects, base models may be trained to have specific levels of knowledge common to more advanced agents. In this way, the base model can start from a relatively advanced stage, without requiring pretraining of each more advanced model individually. This strategy represents an advantageous improvement, because pretraining can take a long time (many days), and pretraining the common base model only requires that pretraining process to be performed once.

[0054] The modules 170 may include a fine-tuning module 178. The fine-tuning module 178 may include instructions that train the one or models further to perform specific tasks. Specifically, the fine-tuning module 178 may include instructions that train one or more models to generate respective language outputs (e.g., text generation), summarization, question answering or translation activities based on healthcare data.

[0055] Continuing the example, the fine-tuning module 178 may include sets of instructions for retrieving one or more structured data sets. The fine-tuning module 178 may include instructions for configuring an objective function for performing a specific task, such as generating text that is similar to text found within the corpus of training data. For example, the fine-tuning module 178 may include instructions for fine-tuning a base model to identify completed actions in a flow model, such as a flow model for a discharge process, and generate one or more next steps based on the completed actions and flow model. In some aspects, the fine-tuning module 178 may include user-selectable parameters that affect the fine-tuning of the one or more models.

[0056] In some aspects, to manage complexity of fine-tuning and other machine learning operations of the server computing device 104, one or more open source frameworks may be used. Example frameworks include TensorFlow, Keras, MXNet, Caffe, SciKit learn, PyTorch. Specifically for training and operating language models, frameworks such as OpenLLM and LangChain may be used, in some aspects. The fine-tuning module 178 may use an algorithm such as stochastic gradient descent or another optimization technique to adjust weights of the pretrained model.

[0057] Fine-tuning may be an optional operation, in some aspects. In some aspects, training may be performed by the training module 180 after pretraining by the model pretraining module 176. In some aspects, the model training module 180 may perform task-specific training like the fine-tuning module 178, on a smaller scale or with a more tailored objective.

[0058] The training module 180 may include one or more submodules, including the checkpointing module 182, the hyperparameter tuning module 184, the validation and testing module 186 and the auto-prompting module 188. The checkpointing module 182 may perform checkpointing, which is saving of a model’s parameters. The checkpointing module 182 may store checkpoints during training and at the conclusion of training, for example, in the model electronic database 112. In this way, the model may be run (e.g., for testing and validation) at multiple stages and its training parameters loaded, and also retrained from a checkpoint. In this way, the model can be run and trained forward without being re-trained from the beginning, which may save significant time (e.g., days of computation). The hyperparameter tuning module 184 may include hyperparameters such as batch size, model size, learning rate, etc. These hyperparameters may be adjusted to influence model training. The hyperparameter tuning module 184 may include instructions for tuning hyperparameters by successive evaluation. The validation and testing module 186 may include sets of instructions for validating and testing one or more machine learning models, including those generated by the model pretraining module 176, the fine-tuning module 178 and the model training module 180. The auto-prompting module 188 may include sets of instructions for performing auto-prompting of one or more models. Specifically, the auto-prompting module 188 may enrich a prompt with additional information. The auto-prompting module 188 may include additional information in a prompt, so that the model receiving the prompt has additional contextual data or directions that it can use. This may allow the auto-prompting module 188 to fine-tune a base model using one- shot of few-shot learning, in some aspects. The auto-prompting module 188 may also be used to focus the output of the one or more models.

[0059] In some aspects, the training module 180 may include instructions for training one or more additional machine learning models, such as supervised or unsupervised machine learning models. For example, as discussed below, in some aspects, the present techniques may include processing data to determine next steps in a process, such as recommendations for discharging a patient. In that case, the patient’s data, such as notes and / or images included in an electronic healthcare record, may be processed by a model (e.g., a convolutional neural network) and the results processed further (e.g., by a language model) and / or provided to the client computing device 102. In other aspects, a machine learning model may be trained to generate synthetic contextual data, such as synthetic patient healthcare data, which may be used to train another machine learning model to perform another task. The training module 180 may train such a supervised model separately from training one or more language models. Further, the server computing device 104 may select one or more trained models at runtimebased on data about a specific patient, based upon data contained in a prompt or based on other conditions that may be preprogrammed into the server computing device 104.

[0060] In some aspects, the training module 180 may train multi-modal models. For example, the training module 180 may train a plurality of models each capable of drawing from multimodal data types such as written text, imaging data, laboratory data, real-time monitoring data, pathology images, etc. In some cases, the training module 180 may train a single model capable of processing the multimodal data types. In some aspects, a trained multi-modal model may be used in conjunction with another model (e.g., a large language model) to provide nontext data interactions with users. Non-text data may be analyzed and integrated into generating next steps in a process and / or generating a summary for the next steps further discussed herein.

[0061] The operating module 190 may operate one or more trained models. Specifically, the model operation module 190 may initialize one or more trained models, load parameters into the model(s), and provide the model(s) with inference data (e.g., prompt inputs). In some aspects, the model operation module 190 may deploy one or more trained model (e.g., a pretrained model, a fine tuned model and / or a trained model) onto a cloud computing device (e.g., via the API 166). The model operation module 190 may receive one or more inputs, for example from the client computing device 102, and provide those inputs (e.g., one or more prompts) to the trained model. In some aspects, the API 166 may include elements for receiving requests to the model, and for generating outputs based on model outputs. For example, the API 166 may include a RESTful API that receives a GET or POST request including a prompt parameter.The model operation module 190 may receive the request from the API 166, and pass the prompt parameter into the trained model and receive a corresponding input. For example, the prompt parameter may be “What is the smallest bone in the human body?”. The prompt output may be “The stapes bone of the inner ear is the smallest bone in the human body.”

[0062] The model operation module 190 may receive a prompt input via the client computing device 102, and provide that input to a machine learning model for processing. The output of the machine learning model may be collected and transmitted back to the client computing device 102 for display. In one aspect, the model operation module 190 may receive additional inputs from the client computing device 102 that enable the user to interact with the one or more trained models in a question-answer format.

[0063] For example, a flow model may represent a patient discharge process. A medical team may want to use the present techniques to investigate whether the patient is ready for discharge and post-discharge treatment options according to the patient discharge flow model.The user may access the client computing device 102. The input processing module 146 may identify the patient via the name or other identifying information provided, and retrieve patient information (e.g., electronic health records) from a context database 108.

[0064] In a question-answer aspect, the user may ask follow up questions. For example, the user may enter a prompt such as “What are post-discharge options for treatment?” As noted, the server computing device 104 may have access to historical patient data, and the server 104 (e.g., the data preprocessing module 174) may include instructions for retrieving additional data from knowledge databases regarding patients to provide additional contextual data to the one or more language models. These knowledge databases may include the electronic healthcare records database discussed above, as well as external sources such as academic papers, case studies, transcripts, etc. In some aspects, the language models may be trained using this additional data ahead of time, and may not retrieve the data at runtime.

[0065] As discussed, in some aspects, multi-modal modeling may be used. The data preprocessing module may, for example, process and understand image data, audio data, video data, etc. The server computing device 104 may interpret and respond to queries that involve understanding content from these different modalities. For example, the server computing device 104 may include an image processing module (not depicted) including instructions for performing image analysis on images provided by users, or images retrieved from patient EHR data. In some aspects, the server computing device 104 may generate outputs in modalities other than text. For example, the server computing device 104 may generate an audio response, an image, etc. Combining multi-modal data may enable the present models to perform more comprehensive analysis of patient conditions, based on information processed in multiple different modes simultaneously.

[0066] The operating module 190 may include a set of computer-executable instructions that when executed by one or more processors (e.g., the processors 160) cause a computer (e.g., the server computing device 104) to perform retrieval-augmented generation. Specifically, the operating module 190 may perform retrieval-augmented generation based upon inputs or queries received from the user. This allows the operating module 190 to tailor responses of a model based on the specific input and context, such as patient data (e.g., an electronic healthcare record). For example, one or more models may be pre-trained, fine-tuned and / or trained as discussed above. During that training, the model may learn to generate tokens based on general language understanding as well as application-specific training. Such a model at that point may be static, insofar as it cannot access further information when presented with an input query.

[0067] When the model is used at runtime, however, such as when deployed in the environment 100, the operating module 190 may perform retrieval operations, such as searching or selecting information from a document, a database, or another source. The operating module 190 may include instructions for processing user input and for performing a keyword search, a regular expression search, a similarity search, etc. based upon that user input. The operating module 190 may input the results of that search, along with the user input, into the trained model. Thus, the trained model may process this additional retrieved information to augment, or contextualize, the generation of tokens that represent responses to the user’s query. In sum, retrieval augmented generation applied in this manner allows the model to dynamically generate outputs that are more relevant to the user’s input query at runtime. Information that may be retrieved may include data corresponding to a patient (e.g., patient demographic information, medical history, clinical notes, diagnoses, medications, allergies, immunizations, laboratory results, oncology information, radiation and imaging information, vitals, etc.) and additional training information, such as medical journals, notes or speech transcripts from symposia or other meetings / conferences, etc.

[0068] The present techniques may trigger retrieval augmented generation by processing a prompt, in some aspects. For example, a prompt may be processed by the input processing module 146 of the client computing device 102, prior to processing the prompt by the one or more generative models. The input processing module 146 may trigger retrieval augmented generation based on the presence of certain inputs, such as patient information, or a request for specific information, in the form of keywords. The input processing module 146 may perform entity recognition or other natural language processing functions to determine whether the prompt should be processed using retrieval augmented generation prior to being provided to the trained model.

[0069] As discussed above, prompts may be received via the input processing module 146 of the client computing device 102 and transmitted to the server computing device 104 via the electronic network 106. In some aspects, the output of the model may be modulated prior to being transmitted, output or otherwise displayed to a user.

[0070] The modules 170 may include a chatbot 192 which may be programmed to simulate human conversation, interact with users, understand their needs, and recommend an appropriate line of action with minimal and / or no human intervention, among other things. This may include providing the best response of any query that it receives and / or asking follow-up questions. For example, a discharged patient may input a question to the chatbot such as“What medication was prescribed in my treatment plan?" and the chatbot may respond “Tylenol.”

[0071] In some embodiments, the voice bots or chatbots 150 discussed herein may be configured to utilize Al and / or ML techniques. For instance, the voice bot or chatbot 150 may be a ChatGPT chatbot, an InstructGPT bot, a Codex bot, or a Google Bard bot. The voice bot or chatbot 150 may employ supervised or unsupervised ML techniques, which may be followed by and / or used in conjunction with reinforced or reinforcement learning techniques. The voice bot or chatbot 150 may employ the techniques utilized for ChatGPT, InstructGPT bot, Codex bot, or Google Bard bot.

[0072] The client computing device 102 and the server computing device 104 may communicate with one another via the network 106. In some aspects, the client computing device 102 and / or the server computing device 104 may offload some or all of their respective functionality to the one or more cloud APIs 114. In aspects, the one or more cloud APIs 114 may include one or more public clouds, one or more private clouds and / or one or more hybrid clouds. The one or more cloud APIs 114 may include one or resources provided under one or more service models, such as Infrastructure as a Service (laaS), Platform as a Service (PaaS), Software as a Service (SaaS), and Function as a Service (FaaS). For example, the one or more cloud APIs 114 may include one or more cloud computing resources, such as computing instances, electronic databases, operating systems, email resources, etc. The one or more cloud APIs 114 may include distributed computing resources that enable, for example, the model pretraining module 176 and / or other of the modules 170 to distribute parallel model training jobs across many processors.

[0073] In some aspects, the one or more cloud APIs 114 may include one or more language operation APIs, such as OpenAI, Bing, Claude. ai, etc. In other aspects, the one or more cloud APIs 114 may include an API configured to operate one or more open source models, such as Llama 2.

[0074] The electronic network 106 may be a collection of interconnected devices, and may include one or more local area networks, wide area networks, subnets, and / or the Internet. The network 106 may include one or more networking devices such as routers, switches, etc. Each device within the network 106 may be assigned a unique identifier, such as an IP address, to facilitate communication. The network 106 may include wired (e.g., Ethernet cables) and wireless (e.g., Wi-Fi) connections. The network 106 may include a topology such as a star topology (devices connected to a central hub), a bus topology (devices connected along a single cable), a ring topology (devices connected in a circular fashion), and / or a mesh topology(devices connected to multiple other devices). The electronic network 106 may facilitate communication via one or more networking protocols, such as packet protocols (e.g., Internet Protocol (IP)) and / or application-layer protocols (e.g., HTTP, SMTP, SSH, etc.). The network 106 may perform routing and / or switching operations using routers and switches. The network 106 may include one or more firewalls, file servers and / or storage devices. The network 106 may include one or more subnetworks such as a virtual LAN (VLAN).

[0075] The environment 100 may include one or more electronic databases, such as a relational database that uses structured query language (SQL) and / or a NoSQL database or other schema-less database suited for the storage of unstructured or semi-structured data.

[0076] The present techniques may store training data, training parameters and / or trained F FFor example, for a flow model representing a patient discharge process, a machine learning model 215 or 250 may recommend a next step including additional testing for the patient based on the patient’s most recently measured vital signs in an electronic healthcare record, and generate a summary justifying the next step by citing the patient’s most recently measured vital signs listed in the electronic healthcare record.

[0077] In some embodiments, a machine learning model may be trained to generate labelled synthetic contextual data for training a machine learning model. For example, a machine learning model 215 or 250 may be trained to generate labelled synthetic patient healthcare data (e.g., synthetic electronic healthcare records), which may be used to train another machine learning model to identify completed actions and generate next steps in a patient discharge process flow model.

[0078] In some embodiments, a machine learning model may be a chatbot (e.g., a chatbot 192) trained to answer healthcare questions from a patient. For example, the trained chatbot (e.g., chatbot 192) on a healthcare server (e.g., server computing device 104) may be accessible via the cloud to a recently discharged patient. The chatbot may access an electronic healthcare record associated with the patient to retrieve next steps in a discharge process. For example, for patient discharge process, a chatbot may generate responses including information about a patient's post-discharge treatment plan, medications prescribed to the patient, or information related to the patient’s diagnosis. In some embodiments, the chatbot may be trained to determine that the healthcare question indicates a situation that requires urgent action and may generate and transmit a notification of the situation requiring urgent action to an output device associated with a healthcare provider, and / or connect the patient to the healthcare provider directly. In some embodiments, the chatbot may be trained to generate reminders for the recently discharged patient, such as a notification of a scheduled appointment.

[0079] In one aspect, the server 202 may fine-tune a pretrained language model 210. The pretrained language model 210 may be obtained by the server 202 and be stored in a memory, such as memory 122. The pretrained language model 210 may be loaded into an ML training module, such as the MLTM 142, by the server 202 for retraining / fine-tuning. A supervised training dataset 212 may be used to fine-tune the pretrained language model 210 wherein each data input prompt to the pretrained language model 210 may have a known output response for the pretrained language model 210 to learn from. The supervised training dataset 212 may be stored in a memory of the server 202, e.g., the memory 122 or the training database 126. In one aspect, the data labelers may create the supervised training dataset 212 prompts and appropriate responses. The pretrained language model 210 may be fine-tuned using the supervised training dataset 212 resulting in the SFT ML model 215 which may provide appropriate responses to user prompts once trained. The trained SFT ML model 215 may be stored in a memory of the server 202, e.g., memory 122.

[0080] In some embodiments, the training data 212 may include prompts associated with a flow model, responses associated with the prompts, and other data relevant to the flow model. In some embodiments, a flow model may be directed to a medical or healthcare process, and the training data may include historical patient healthcare data, prompts directed to the medical / healthcare process, and responses associated with the patient healthcare data and prompts. For example, for a patient discharge process, the training data 212 may include historical electronic healthcare records, a prompt asking for the next steps in a patient discharge process, and a response recommending one or more next steps in a patient discharge process. In another example, the training data 212 may include a historical electronic healthcare records, a prompt to assess a patient’s readiness for surgery, and a response recommending or not recommending the patient for surgery. In some embodiments, the training data 212 may include examples of summaries including justifications for identifying completed actions and / or recommending next steps.

[0081] In some embodiments, the training data 212 may be used to train a machine learning model to generate labelled synthetic contextual data, which may be used to train another machine learning model to handle a flow model. The training data 212 may include labelled historical patient healthcare data (e.g., historical electronic healthcare records).

[0082] In some embodiments, training data 212 for a chatbot answering questions about a discharge process may include example prompts including questions from a recently discharged patient and responses answering the questions. In some embodiments, the training data 212 may include example prompts including questions indicating situations requiring urgent actionand example responses that include generating and transmitting a notification of situation requiring urgent action to a healthcare provider. In some embodiments, training data 212 for a chatbot may include example notifications of scheduled appointments to train the chatbot to generate notifications of scheduled appointments.

[0083] In some embodiments, the server 202 may fine-tune the pretrained language model 210 using a set of vectors associated with a set of training data. In some instances, the set of training data may include prompts associated with a flow model, and responses associated with the prompts. Creating the set of vectors may include (1) splitting the text of the prompts, associated questions and / or associated documents into semantic clusters, and (2) encoding the semantic clusters as the set of vectors. The semantic clusters may be one or more words, a portion of a word, or a character. A distance between the vectors (e.g., a cosine distance, a Euclidean distance) may depend on a relevance between the semantic clusters corresponding to the vectors.

[0084] In one aspect, training the machine learning model 250 may include the server 204 training a reward model 220 to provide as an output a scaler value / reward 225. The reward model 220 may be required to leverage Reinforcement Learning with Human Feedback (RLHF) in which a model (e.g., machine learning model 250) learns to produce outputs which maximize its reward 225, and in doing so may provide responses which are better aligned to user prompts.

[0085] Training the reward model 220 may include the server 204 providing a single prompt 222 to the SFT ML model 215 as an input. The input prompt 222 may be provided via an input device (e.g., a keyboard) via the I / O module of the server, such as I / O module 146. The prompt 222 may be previously unknown to the SFT ML model 215, e.g., the labelers may generate new prompt data, the prompt 222 may include testing data stored on training database 126, and / or any other suitable prompt data. The SFT ML model 215 may generate multiple, different output responses 224A, 224B, 224C, 224D to the single prompt 222. The server 204 may output the responses 224A, 224B, 224C, 224D via an I / O module (e.g., I / O module 146) to a user interface device, such as a display (e.g., as text responses), a speaker (e.g., as audio / voice responses), and / or any other suitable manner of output of the responses 224A, 224B, 224C, 224D for review by the data labelers.

[0086] The data labelers may provide feedback via the server 204 on the responses 224A, 224B, 224C, 224D when ranking 226 them from best to worst based upon the prompt-response pairs. The data labelers may rank 226 the responses 224A, 224B, 224C, 224D by labeling the associated data. The ranked prompt-response pairs 228 may be used to train the reward model220. In one aspect, the server 204 may load the reward model 220 via the ML module (e.g., the ML module 140) and train the reward model 220 using the ranked response pairs 228 as input. The reward model 220 may provide as an output the scalar reward 225. In some embodiments, the prompt of a prompt-response pair may include labelled synthetic data, and the response of the prompt-response pair may include an expected output including an identification of completed tasks, next steps, and data points supporting the completed tasks and next steps.

[0087] In one aspect, the scalar reward 225 may include a value numerically representing a human preference for the best and / or most expected response to a prompt, i.e., a higher scaler reward value may indicate the user is more likely to prefer that response, and a lower scalar reward may indicate that the user is less likely to prefer that response. For example, inputting the “winning” prompt-response (i.e., input-output) pair data to the reward model 220 may generate a winning reward. Inputting a “losing” prompt-response pair data to the same reward model 220 may generate a losing reward. The reward model 220 and / or scalar reward 225 may be updated based upon labelers ranking 226 additional prompt-response pairs generated in response to additional prompts 222.

[0088] In one example, a flow model may include a patient discharge process. A data labeler may provide to the SFT ML model 215 as an input prompt 222, a request for next steps in a patient discharge process, healthcare data for a patient (e.g., a historical electronic healthcare record or synthetic healthcare contextual data), a node, and a portion of the flow model. The node may indicate that the last stopping point in a flow model was a task to check a patient’s vital signs. The portion of the flow model may include tasks (e.g., additional nodes) that follow the task to check the patient’s vital signs. The input may be provided by the labeler via the user device 102 over network 110 to the server 204 running a chatbot application utilizing the SFT ML model 215. The SFT ML model 215 may provide as output responses to the labeler via the user device 102: (i) “the next step is to order discharge testing and setting a discharge date” 224A; (ii) “the next step is to identify a patient care giver” 224B; and (iii) “the next step is to complete discharge testing” 224C. The data labeler may rank 226, via labeling the promptresponse pairs, prompt-response pair 222 / 224A as the most preferred answer; promptresponse pair 222 / 224C as a less preferred answer; and prompt-response 222 / 224B as the least preferred answer. The labeler may rank 226 the prompt-response pair data in any suitable manner. The ranked prompt-response pairs 228 may be provided to the reward model 220 to generate the scalar reward 225.

[0089] In another example, an SFT ML model 215 may be trained to generate synthetic patient healthcare data. A data labeler may provide to the SFT ML model 215 as an inputprompt 222, a request to generate synthetic patient healthcare data. The input may be provided by the labeler via the user device 102 over network 110 to the server 204 running a chatbot application utilizing the SFT ML model 215. The SFT ML model 215 may provide as output responses to the labeler via the user device 102: (i) “the patient’s heart rate is 80 bpm” 224A; (ii) “the patient’s heart rate is 120 bpm” 224B; and (iii) “the patient’s heart rate is 1000 bpm” 224C. The data labeler may rank 226, via labeling the prompt-response pairs, prompt-response pair 222 / 224A and prompt-response pair 222 / 224B as preferred answers; and prompt-response 222 / 224c as an answer that is not preferred. The SFT ML model 215 may be trained to generate synthetic healthcare data that is possible for a human by labeling prompt-response pairs that include responses with healthcare values that are possible for a human as preferred. The labeler may rank 226 the prompt-response pair data in any suitable manner. The ranked prompt-response pairs 228 may be provided to the reward model 220 to generate the scalar reward 225.

[0090] In another example, an SFT ML model 215 may be a chatbot (e.g., chatbot 192) that may be trained to answer patient questions and identify situations requiring urgent action. A data labeler may provide to the SFT ML model 215 as an input prompt 222, an example question from a recently discharged patient. The input may be provided by the labeler via the user device 102 over network 110 to the server 204 running a chatbot application utilizing the SFT ML model 215. The SFT ML model 215 may provide as output responses to the labeler via the user device 102: (i) “contact your doctor immediately” 224A; (ii) “wait to see if your condition improves or worsens” 224B; and (iii) “do nothing” 224C. The data labeler may rank 226, via labeling the prompt-response pairs, prompt-response pair 222 / 224A as the most preferred answer as the response is appropriate for a situation in which a recently discharged patient’s chest hurts, i.e. , a situation requiring urgent action; prompt-response pair 222 / 224B as a less preferred answer; and prompt-response 222 / 224C as the least preferred answer. The labeler may rank 226 the prompt-response pair data in any suitable manner. The ranked promptresponse pairs 228 may be provided to the reward model 220 to generate the scalar reward 225.

[0091] While the reward model 220 may provide the scalar reward 225 as an output, the reward model 220 may not generate a response (e.g., text). Rather, the scalar reward 225 may be used by a version of the SFT ML model 215 to generate more accurate responses to prompts, i.e., the SFT model 215 may generate the response such as text to the prompt, and the reward model 220 may receive the response to generate a scalar reward 225 of how wellhumans perceive it. Reinforcement learning may optimize the SFT model 215 with respect to the reward model 220 which may realize the configured machine learning model 250.

[0092] In one aspect, the server 206 may train the machine learning model 250 (e.g., via the ML module 140) to generate a response 234 to a random, new and / or previously unknown user prompt 232. To generate the response 234, the machine learning model 250 may use a policy 235 (e.g., algorithm) which it learns during training of the reward model 220, and in doing so may advance from the SFT model 215 to the machine learning model 250. The policy 235 may represent a strategy that the machine learning model 250 learns to maximize the reward 225. As discussed herein, based upon prompt-response pairs, a human labeler may continuously provide feedback to assist in determining how well the machine learning model’s 250 responses match expected responses to determine the rewards 225. The rewards 225 may feed back into the machine learning model 250 to evolve the policy 235. Thus, the policy 235 may adjust the parameters of the machine learning model 250 based upon the rewards 225 it receives for generating good responses. The policy 235 may update as the machine learning model 250 provides responses 234 to additional prompts 232.

[0093] In one aspect, the response 234 of the machine learning model 250 using the policy 235 based upon the reward 225 may be compared using a cost function 238 to the SFT ML model 215 (which may not use a policy) response 236 of the same prompt 232. The cost function 238 may be trained in a similar manner and / or contemporaneous with the reward model 220. The server 206 may compute a cost 240 based upon the cost function 238 of the responses 234, 236. The cost 240 may reduce the distance between the responses 234, 236, i.e., a statistical distance measuring how one probability distribution is different from a second, in one aspect the response 234 of the machine learning model 250 versus the response 236 of the SFT model 215. Using the cost 240 to reduce the distance between the responses 234, 236 may avoid a server over-optimizing the reward model 220 and deviating too drastically from the human-intended / preferred response. Without the cost 240, the machine learning model 250 optimizations may result in generating responses 234 which are unreasonable but may still result in the reward model 220 outputting a high reward 225.

[0094] In one aspect, the responses 234 of the machine learning model 250 using the current policy 235 may be passed by the server 206 to the rewards model 220, which may return the scalar reward 225. The machine learning model 250 response 234 may be compared via the cost function 238 to the SFT ML model 215 response 236 by the server 206 to compute the cost 240. The server 206 may generate a final reward 242 which may include the scalar reward 225 offset and / or restricted by the cost 240. The final reward 242 may be provided by the server206 to the machine learning model 250 and may update the policy 235, which in turn may improve the functionality of the machine learning model 250.

[0095] To optimize the machine learning 250 over time, RLHF via the human labeler feedback may continue ranking 226 responses of the machine learning model 250 versus outputs of earlier / other versions of the SFT ML model 215, i.e., providing positive or negative rewards 225. The RLHF may allow the servers (e.g., servers 204, 206) to continue iteratively updating the reward model 220 and / or the policy 235. As a result, the machine learning model 250 may be retrained and / or fine-tuned based upon the human feedback via the RLHF process, and throughout continuing conversations may become increasingly efficient.

[0096] In some embodiments, updated data may be obtained and used to retrain a machine learning model 215, 250. For example, updated healthcare data (e.g., healthcare data for additional patients, additional healthcare data for a particular patient, etc.) may be used to retrain a machine learning model 215, 250 for a pre-operation / pre-surgery or a patient discharge process. In some embodiments, labelled synthetic data may be used in training and / or retraining the machine learning model 215, 250 evaluate the accuracy of the machine learning model 215, 250 at identifying completed actions. The labelled synthetic data may include evidence of completed actions, and may provided in a prompt to the machine learning model 215, 250. The machine learning model 215, 250 may be evaluated based on a similarity between the expected output and the output generated between a machine learning model 215, 250 in response to the prompt that includes the labelled synthetic data. For example, if the machine learning model 215, 250 generates an output that is very similar to the expected output, then the machine learning model 215, 250 is more accurate. If the machine learning model 215, 250 generates an output that is dissimilar from the expected output, then the machine learning model 215, 250 may be less accurate.

[0097] Although multiple servers 202, 204, 206 are depicted in the exemplary block and logic diagram 200, each providing one of the three steps of the overall machine learning model 250 training, fewer and / or additional servers may be utilized and / or may provide the one or more steps of the machine learning model 250 training. In one aspect, one server may provide the entire machine learning model 250 training.Exemplary Flow Model

[0098] FIG. 3 depicts an exemplary flow model that may be handled in accordance with various embodiments herein.

[0099] A flow model 300 such as a patient discharge process may include a complex series of tasks and decisions. A flow model 300 may include many nodes (that form branches through the flow model. For example, a branch of the flow model may include the nodes 302, 304, 306, and 308. Nodes may represent actions or steps in a flow model, and may include task nodes, decision nodes, and / or output nodes. A task node represents activities and / or actions, and a machine learning model (e.g., machine learning model 215, 250) may determine whether tasks are complete. For example, node 302a may be a task node representing an action of removing chest tubes from a patient (“Chest tubes out”). A decision node represents a decision to be made which has one or more outcomes. The machine learning model may determine whether a decision has been made and which outcome was chosen. For example, node 320 be a decision node for determining a discharge destination (“Discharge destination?”), which has three outcome discharge destinations. An output node generates an output resulting from the execution of a flow model, i.e. , after a flow model has been evaluated, the output is the aggregation of all output nodes that have been reached or traversed when evaluating the flow model. The output may include text, transmitting information, submitting an order, etc. For example, node 314 may be an output node that generates an after-visit summary. In some embodiments, the output may be generated automatically upon evaluation of an output node. In some embodiments, the output may be generated upon user approval.

[0100] A machine learning model (e.g., machine learning model 215, 250) may be trained to identify completed tasks in the flow model and generate one or more next steps. A prompt which may include a node and a portion of the flow model, may be provided to a machine learning model. For example, a prompt may be used to evaluate the completion of a task and / or determine if a decision has been made and the outcome of the decision in a discharge process for a particular patient. The prompt and a branch 310 may be provided to the machine learning model. Context data (e.g., an electronic healthcare record) for the particular patient may be retrieved from a database (e.g., a context database 108). In some embodiments, contextual data such as healthcare data may be included directly in the prompt. The machine learning model may extract a node 304 from the branch 310. The node 304 may indicate a stopping point along the flow model from when the method generating next steps was last previously performed, and the portion of the flow model may include one or more nodes (e.g., tasks or decisions) that occur sequentially follow the node included in the prompt. For example, the node 304 included in the prompt may indicate that all tasks or actions in a patient discharge process up to the node 304 had been identified as completed the last time the method was run for a particular patient, and the portion of the flow model may include the branch 310. In some embodiments, a singular node may be the portion of the flow model included in the prompt.

[0101] The machine learning model may identify completed tasks and generate one or more next steps. For example, for a prompt including the node 312 and a portion of the flow model including nodes 312 and 314, and healthcare data for a particular patient, the machine learning model may determine that a healthcare provider has reviewed and approved prescriptions (node 312), and that next steps may include filling pharmacy scripts, completing an after-visit summary, printing the after-visit summary, and reviewing the after-visit summary with the patient (node 314). The machine learning model may generate a summary that includes completed actions (e.g., completed tasks, decisions), next steps (e.g., next tasks to be performed, next decisions to be made, next output to be generated), and a justification for both completed steps and next steps. For example, for the prompt that includes the node 312 and the portion of the flow model including nodes 312 and 314, the summary may include a citation to an electronic healthcare record showing a prescription that has been signed by a doctor to show that a healthcare provider has reviewed and approved the prescriptions and that a next step should include filling the prescription. The one or more next steps may be displayed on a computing device. In some embodiments, the machine learning model may identify a user associated with a next step and transmit next steps to the user associated with the next step. For example, the machine learning model may identify the next step of filling a prescription to a computing device associated with a pharmacy. In some embodiments, after the flow model has been evaluated, an output node of the flow model may generate an output. For example, after evaluating the node 314, the machine learning model may generate an after-visit summary as an output.

[0102] The node included in the prompt, the identified completed actions, and the summary may be stored in a database (e.g., context database 108). For example, an indication that a healthcare provider has reviewed and approved prescriptions, the node 314, and the machine learning model-generated summary may be stored in an electronic healthcare record associated with a patient., which may be stored in the context database 108. In some embodiments, the flow model 300 may include metadata to indicate (e.g., via tagging a node) whether completion of a node is considered permanent or transient. Nodes tagged with permanent completion are considered completed for the remainder of the progress through a flow model once they are found to be complete. Nodes tagged with transient completion are only determined to be completed for a particular performance of the method determining progress through a flow model, i.e., every time the method is performed, the machine learning model must determine whether the task / decision / output of the node has been completed. For example, node 302c may be tagged with transient completion and node 302d may be tagged with permanent completion.

[0103] Continuing the example, a method (e.g., method 400) may be performed on Day 1 to determine a patient’s progress through a patient discharge process flow model 300, which determines that a patient is hemodynamically stable (node 302c), but that a patient care giver has not yet been identified (node 302d). On Day 2, the method may be performed again, and determine that a patient caregiver has been identified (node 302d). Because node 302c for confirming that a patient is hemodynamically stable is tagged with transient completion, the machine learning model (e.g., model 215, 250) must evaluate again whether node 302c is complete, and may determine that node 302c is not complete because the patient is not hemodynamically stable. On Day 3, the method may be performed again. Because node 302d is tagged with permanent completion, node 302d is considered complete from the evaluation of the flow model on Day 2, and the machine learning model does not evaluate node 302d again. Node 302c, which is tagged with transient completion, node 302c must be evaluated again to determine whether the patient is hemodynamically stable.

[0104] In some embodiments, portions of a flow model may be evaluated sequentially. For example, for some patient care processes, particular patient care tasks must be performed and completed in a particular order. In another example, a next task may be dependent on the outcome. For example, a decision node 320 must be evaluated before any of the task nodes 322a, 322b, or 322c because the outcome of decision node 320 determines which of the nodes 322a, 322b, or 322c is evaluated next. In another example, node 312 must be evaluated before node 314, because the task of node 312, which includes a provider reviewing and approving prescriptions, must be performed before the task of node 314, which includes filling a pharmacy script.

[0105] In some embodiments, more than one portion of the flow model may be evaluated in parallel. For example, a first prompt including node 302a and a portion of the flow model 302a, 304, 306, and 308 may be evaluated in parallel against a second prompt including node 302b and a portion of the flow model 304, 306, and 308. In some embodiments, activation criteria may determine the evaluation of portions of the flow model in parallel. For example, for a patient with atrial fibrillation (AFib) in an intensive care unit (ICU) setting, a flow model for AFib patients may be evaluated in parallel with a separate ICU flow model. The AFib flow model may include a set of activation criteria that is used to activate the AFib flow model, and the ICU flow model may have a different set of activation criteria that is used to activate the ICU flow model.

[0106] The flow model in FIG. 3 depicts a patient discharge process. A patient discharge process may include several branches beginning with nodes 302a-302f. A patient discharge process may include identifying an available patient care giver (node 302d). After identifying acare giver, the next node in the flow may include educating the patient and patient care giver about the patient’s post-discharge care. After patient and care giver education has occurred, the flow model proceeds to the completion of pharmacy medication reconciliation.

[0107] The patient discharge process may also include removing chest tubes from the patient (node 302a), removing pacing wires from a patient (node 302b), determining the patient has hemodynamic stability (i.e. , heart rate 60-100, systolic blood pressure 100-140, respiratory rate 14-20) (node 302c), and that the patient has acceptable weight (i.e., the patient’s weight has changed 2kg or less since before the patient’s operation) (node 302e). The nodes 302a,b,c,e may be evaluated in parallel, and determined as complete before ordering discharge testing and setting a specific date for discharge (node 304). After discharge testing is ordered and a specific date for discharge is set, the discharge testing may be completed (node 306) and then reviewed (node 308). After discharge testing has been reviewed, the flow model proceeds to the completion of pharmacy medication reconciliation.

[0108] The patient discharge process may also include assessing discharge needs for a patient, including patient mobility and patient expectations (302f). If needed, a physical therapy and / or occupational therapy consultation and recommendations are completed. Care management and social work activities are assessed next, if needed. A discharge destination may be set next. The flow model may include a determination node 320 that determines the patient’s discharge destination. A patient may be discharged to a home unassisted, home with home healthcare, family home, or hotel; or a patient may be discharged to a skilled nursing facility (SNF), long-term acute care (LTAC), or acute rehab. If the patient is discharged to a home / hotel, oxygen may be set up if needed, or a notation of patient dismissal without oxygen may be provided (node 322a). If the patient requires infusion therapy, wound vacuum-assisted closure (VAC), and / or dialysis, the location and frequency of such treatments may be set (node 322b). After oxygen has been set (if needed) and treatments have been set, the flow model proceeds to the completion of pharmacy medication reconciliation.

[0109] After pharmacy medication reconciliation is complete, a healthcare provider may review and approve prescriptions (node 312). The flow model then proceeds to filling pharmacy scripts, and completion, printing, and reviewing of the patient’s after-visit summary (AVS) (node 314).

[0110] If a patient is to be discharged to an SNF, LTAC, or acute rehab, the patient facility may be chosen based on patient needs (node 322c). The chosen facility may confirm the acceptance of the patient. The patient may be informed and / or counseled on the facility. Transport may be set up to transport the patient to the chosen facility. The flow model thenproceeds to filling pharmacy scripts, and completion, printing, and reviewing of the patient’s AVS.

[0111] While FIG. 3 depicts a flow model directed to a patient discharge process, other flow models may be tracked in accordance with the method described herein. For example, a flow model may represent a pre-operation / surgery process that determines patient readiness for surgery, as depicted in FIG. 7. Nodes in a pre-operation / surgery process may include ensuring that a date for surgery has been scheduled, a patient has provided information about their health risks (e.g., medical conditions, allergies, etc.), a pre-operation checkup and any additional tests have been performed, the surgery and risks have been reviewed with the patient, etc.Exemplary Computer-Implemented Methods

[0112] FIG. 4 depicts an exemplary computer-implemented method 400 in accordance with various embodiments herein, and as may be implemented by the exemplary computing environment 100.

[0113] At block 402, the method 400 may include receiving, from a computing device, a prompt including a portion of the flow model and a node on the portion of the flow model. In some embodiments, the flow model may be associated with a healthcare process. For example, in some embodiments, the flow model may be for a pre-operation / surgery process or for a patient discharge process.

[0114] At block 404, the method 400 may include retrieving contextual data associated with the portion of the flow model. In some embodiments, the contextual data may include patient data. For example, in some embodiments, the contextual data may include an electronic healthcare record updated in real-time.

[0115] At block 406, the method 400 may include providing the prompt and the contextual data to a trained machine learning (ML) model to determine completion of one or more actions in the flow model. In some embodiments, the machine learning model is a large language model (LLM) or a multi-modal machine learning model.

[0116] At block 408, the method may 400 include generating, by processing the prompt and contextual data with the trained machine learning model, one or more next steps based on the completion of one or more actions in the pathway.

[0117] At block 410, the method 400 may include generating, using the trained machine learning model, a summary including a justification for the one or more next steps supported byone or more datapoints from the contextual data. In some embodiments, the justification may be supported by citing to the flow model.

[0118] At block 412, the method 400 may include storing the summary, the one or more completed actions, and the node.

[0119] At block 414, the method 400 may include displaying the one or more completed steps and the one or more next steps on the computing device. In some embodiments, the method 400 may include determining, based on the one or more next steps and the contextual data and using the machine learning model, a user associated with the at least one of the one or more next steps. At least one of the one or more next steps may be transmitted to a computing device associated with the user.

[0120] In some embodiments, the method 400 may include providing an additional portion of the flow model to the trained machine learning model to determine completion of one or more actions in the additional portion of the flow model, wherein the trained machine learning model is configured with an agent to process the portion of the flow model and an additional agent to process the additional portion of the flow model. The method 400 may may further include generating, by processing the additional portion of the flow model, the prompt, and the contextual data with the trained machine learning model, one or more additional next steps based on the completion of one or more actions in the additional portion of the flow model, the one or more additional next steps being generated in parallel with the one or more next steps.

[0121] In some embodiments, the method 400 may include generating, by using a second machine learning model, labelled synthetic contextual data. In some embodiments, the method 400 may include receiving a test prompt including the portion of the flow model and the node from the computing device. The method 400 may include training the machine learning model on the synthetic contextual data and the test prompt.

[0122] In some embodiments, the flow model may be for a virtual discharge process. The method 400 may include obtaining one or more portions of a flow model describing a patient discharge process and historical electronic healthcare record and training a machine learning model, using the one or more portions of the flow model describing a patient discharge process and the historical electronic health care to generate one or more recommendations for discharging a patient.

[0123] In some embodiments, the method may be used for a virtual discharge process flow model to determine whether the patient can be graduated to a next phase of care, such as discharge, as described in various examples herein. For example, where the healthcare dataincludes real-time updated healthcare data, the method 400 may determine if the patient is to be graduated to discharge status in real-time. In an example, wherein patient meets progressive care status (PCU) for transfer out of an ICU, the trained machine learning mode, receiving healthcare data, may determine the discharge status and execute an alert within EHR and / or a prompt through a chatbot to prompt nursing and / or a healthcare provider of the status determination.

[0124] The method 400 used for a virtual discharge process flow model may include generating, by processing the healthcare data using a trained machine learning model, an output that includes one or more recommendations for discharging a patient. The recommendations may include a step for continuing treatment (e.g., physical therapy), a medication, a continuing treatment location (a hemodialysis facility for existing End-Stage Renal Disease (ESRD)), a medical device (e.g., a LifeVest® for ejection fraction < 35%), a predicted time of discharge, or a follow-up appointment time. The machine learning model may identify whether the patient can be graduated to the next phase of care in real-time. For example, the machine learning model may determine that a patient meets progressive care status (PCU) for transfer out of the ICU. The machine learning model may also generate a discharge summary or other paperwork for discharge.

[0125] The one or more recommendations to be displayed in an output device. This may include an alert to providers. In some examples, the one or more recommendations of discharge summary is communicated through a chatbot prompt or other electronic summary to an attending nursing computing station or other healthcare provided computing station.

[0126] In some embodiments, the trained machine learning model may receive real-time updated healthcare data of a patient (e.g., health data from electronic monitors, healthcare data updates entered by a healthcare professional, etc.). The machine learning model may identify factors that may delay discharge of a patient and generate an output including a notification of the factors that may delay discharge.

[0127] In some implementations, the method 400 may include receiving a healthcare question from a patient. For example, the method 400 may integrate one or more recommendations with a trained LLM chatbot accessible to a patient via cloud-based access to the healthcare provider server computing device.

[0128] A chatbot may generate a response to the question, which may include a patient's post-discharge treatment plan, medications prescribed to the patient, or information related to the patient’s diagnosis. In some implementations, the chatbot may determine that the healthcare question indicates that the patient is experiencing a situation that requires urgentaction. The chatbot may generate a notification of the situation that requires urgent action and transmit the notification to a device associated with a healthcare provider. In some implementations, the chatbot may connect the patient and healthcare provided directly.

[0129] The one or more recommendations generated by the method 400 may be include recommendations tailored to different users. For example, in some implementations the method 400 can provide recommendations to multidisciplinary team interdependencies, including nursing, primary team, physical therapy, occupational therapy, social work, case management, respiratory therapy, and consulting services. With the method 400, the trained LLM model may align care plans across all discharge planning teams in real-time by providing notices of changes in care needs.

[0130] Thus, as detailed herein, the present application provides systems and methods deploying novel Artificial Intelligence (Al) (i.e. , Machine Learning (ML)) interfaces capable of handling complex flow models to identify completed actions in the flow model and generate one or more next steps based on contextual data and the flow model. In some embodiments, the method is capable of optimizing anticipated care dimensions of patient discharge. Identification of patient discharge needs is labor intensive involving manual chart review, interpersonal information sharing at preset times (multidisciplinary conference) and often limited to defined settings (e.g., intensive care unit (ICU) vs floor). Furthermore, manual formulation of discharge summaries is time consuming and addressing patient concerns following discharge is often sub optimally coordinated.

[0131] Overcoming these limitations, the present application, in some implementations, provides real-time LLM model determination of discharge needs that can occur at initial presentation but that can also be continually updated based on clinically relevant events as described in hospital records. Using an iterative learning process, the LLM model (and other trained machine learning models) can be trained to deliver highly accurate discharge recommendations. Furthermore, LLM can autonomously excerpt hospital events in a preliminary discharge summary / Hospital Course that can be edited by the care provider as needed thus saving effort that can be applied to other tasks. Post-discharge, a chatbot can triage patients' concerns that can be routed to care providers in order of urgency. This ensures expedited evaluation of more serious patient concerns such as shortness of breath, syncope, occurrence of falls and / or wound complications.

[0132] Thus, LLM models (and / or any other trained machine learning models) can be fully integrated throughout a discharge planning pipeline, which thus can provide healthcare providers anticipatory discharge planning and production of discharge documentation to triagedpost-discharge communication. Further, the LLM models (and / or any other trained machine learning models) can be continuously evolved with human imputed corrections which allows adaptation to changing discharge paradigms. While allowing the timely discharge of the index patient, this approach has upstream effects of streamlining patient throughput and reducing health worker load and burnout, in addition to improving patient satisfaction and liberating clinical resources.Exemplary Architecture

[0133] FIG. 5 depicts a block diagram of exemplary architecture for determining completed actions and generating next steps.

[0134] An architecture 500 may include a flow model 502. The flow model 502 may represent a medical or healthcare process (also called a patient care pathway or a care pathway). For example, flow model 502 may represent a patient discharge process such as the patient discharge process 300 in FIG. 3. The flow model 502 may comprise a series of nodes (e.g., actions 504a, 504b,...504n) which represent tasks or actions in the flow model. For example, in a patient discharge process, a node may include determining that a provider has reviewed and approved prescriptions (e.g., node 312). The flow model 502 may be provided to machine learning engine 510 so that the machine learning engine 510 may determine what actions in the flow model 502 have been completed and generate next steps to complete from the flow model 502. The flow model 502 may be updated by a dynamic flow model updater 506.

[0135] The dynamic flow model updater 506 may be used to update the flow model. For example, external events (e.g., hospital policy changes) and / or events within the subject matter of the flow model (e.g., a change in a patient condition or status) may require that the flow model 502 may be updated. In some embodiments, activation criteria, which may indicate whether a particular flow model is applicable, may require that the flow model 502 be updated. For example, for a medical or healthcare flow model, a patient’s condition and / or status may include activation criteria that activate other flow models that should be evaluated by the machine learning engine 510. For example, one flow model may be a flow model representing intensive care unit (ICU) processes, and a second flow model may be a flow model representing progressive care unit (PCU) processes, and an activation criterion may include patient discharge from an ICU to a PCU. Upon discharging the patient from the ICU to the PCU (i.e. , satisfaction of the activation criterion), the ICU flow model may be deactivated and the PCU flow model may be activated, such that a machine learning engine 510 will evaluate the PCU flow model and not evaluate the ICU flow model.

[0136] The architecture 500 may include a contextual data layer 508. The contextual data layer 508 provides the machine learning engine 510 with contextual data, which the machine learning engine 510 may use to determine completed actions in a flow model 502 and generate next steps. The contextual data layer 508 may include data from electrically monitored sources 508a, an electronic healthcare record 508b, a user computing devices 508c and 508d, and a synthetic data ML model 508e. Electrically monitored sources 508a may include electronic medical devices such as monitoring equipment (e.g., heart rate monitors, glucometers, etc.), diagnostic equipment (e.g., ultrasound imaging, CT scanners, x-ray machines, etc.), implantable devices (e.g., pacemakers, defibrillators, etc.) and / or other devices. An electronic healthcare record (EHR) 508b may be retrieved from an electronic healthcare record database, which may be a context database 108 as in FIG. 1. User computing devices 508c and 508d may be a computing device 102 as in FIG. 1. A user may interact (e.g., via entering a prompt using an input device 126) with a user computing device 508c and / or 508d to provide contextual data to the machine learning engine 510.

[0137] In some embodiments, a synthetic data machine learning model 508e may provide labelled synthetic contextual data (e.g., labelled synthetic patient data) to the machine learning engine 510 when training the machine learning engine 510. The synthetic data machine learning model 508e may be trained, using labelled contextual data, to generate labelled synthetic contextual data when provided with a prompt including labels. The labelled synthetic contextual data is generated with attributes of the data already labeled, without the need for manual labelling. The controller 512 determines whether labelled synthetic contextual data is provided to the machine learning engine 510. The controller 512 may retrieve the flow model 502 and contextual data (e.g., patient data from electrically monitored sources 508a, electronic healthcare record 508b, user computing devices 508c and 508d) and use the data to generate labelled synthetic contextual data (e.g., labelled synthetic patient data). The labelled synthetic contextual data may be provided to the machine learning engine 510 if the controller 512 determines that the machine learning engine 510 requires further training or re-training (e.g., due to the machine learning engine 510 generating outputs with low confidence levels). In some embodiments, the controller 512 may provide the labelled synthetic contextual data after receiving an instruction from a user computing device (e.g., user computing device 508c or 508d, client computing device 102).

[0138] The machine learning engine 510 may include one or more machine learning models (e.g., machine learning models 215, 250, 630), and is trained (e.g., via a training process described in FIG. 2) to generate determine completed actions (e.g., completed actions 514a) ina flow model (e.g., flow model 502) and generate one or more next steps (e.g., next steps 514b). The completed actions and next steps may be part of an output 514, as described below. The machine learning engine 510 may include a large language model (LLM) (e.g., LLM 630) or a multi-modal machine learning model.

[0139] In some embodiments, the completion of tasks may be continuously recorded and stored (e.g., in a database 108). The machine learning engine 510 may be trained to assess the completed tasks and determine whether the completed tasks have deviated from the flow model 502. The machine learning engine 510 may be trained to analyze the changes in task completion and determine whether the changes have led to better results or outcomes based on user feedback and / or user behavior in response to the changes. The machine learning engine 510 may assess the flow model 502 and make recommendations to improve efficiency of the flow model 502, e.g., removing redundant nodes in the flow model 502, changing the order of nodes, etc.

[0140] The output 514 may include completed actions 514a, next steps 514b, a summary 514c, and an associated user / associated user / user computing device 514d. The completed actions 514a may include which of the actions 504a, 504b, ...504n have been completed. The next steps 514b may include which of the actions 504a, 504b,...504n need to be completed next. The summary may include information about the completed actions 514a and the next steps 514b, including a justification explaining why or how the machine learning engine 510 determined completed actions 514a and next steps 514b.

[0141] The summary 514c may include data from the contextual data from the contextual data layer 508. For example, the action 504a may include a healthcare provider reviewing and approving a prescription for a patient, and the action 504b may include filling the prescription. The machine learning engine 510 may determine that action 504a is a completed action (i.e., a healthcare provider has reviewed and approved a prescription) and that the action 504b is a next step (i.e., the next action to complete is filling the prescription). The summary 514c may include a justification explaining why the machine learning engine 510 determined that the action 504b is a next step. The justification may include a quotation, such as a prescription with a doctor’s electronic signature, from an electronic healthcare record to show that a healthcare provider has reviewed and approved the prescription, and that a next step should include filling the prescription. In some embodiments, the justification may include citations to one or more portions of the flow model. In some embodiments, the justification for a next step may include an absence of evidence indicating that the next step has been completed.

[0142] In some embodiments, the output 514 may include an identification of an associated user / user computing device 514d (e.g., a name, a device identifier, etc ). The machine learning engine 510 may determine which user is responsible for completing a next step 514b, a user who completed a completed action 514a, and / or any other user who can view the output 514. The completed actions 514a, next steps 514b, and / or summary 514c may be transmitted to a user computing device (e.g., user computing devices 508c and / or 508d) based on the identification of the associated user / user computing device 514d.Exemplary Evaluation of Progress in a Care Pathway Flow Model

[0143] FIG. 6 depicts a combined block and flow diagram for determining completed actions (tasks) and generating next steps (tasks) in an exemplary care pathway flow model. In some embodiments, the care pathway flow model may represent a medical or healthcare process.

[0144] A system may include a periodic patient care pathway assessment processor 620, care pathway flow model database 622, a patient care pathway status database 624, a patient electronic healthcare record data store 626, a patient data vector store 628, and a large language model (LLM) 630.

[0145] A periodic patient care pathway assessment processor (e.g., processor 160 of server 104 in FIG. 1) may receive a request (e.g., a prompt) to assess a patient’s progress in a care pathway (e.g., a flow model) such as a patient discharge process, a pre-operation / surgery process, etc.

[0146] At block 602, the patient’s current care pathway status may be retrieved from a patient care pathway status database 624. In some embodiments, the patient care pathway status may indicate a node of the patient care pathway. The node may include one or more tasks. The patient care pathway status database 624 may be a context database 108. In some embodiments, the patient care pathway status database 624 may be the same database as the patient electronic healthcare record data store 626 and / or the patient data vector store 628. In some embodiments in which the patient care pathway status database 624 and the patient electronic healthcare record data store 626 are the same, the patient the care pathway status for a patient may be stored in an electronic healthcare record associated with the patient. In some embodiments, the patient care pathway status database 624 may be a separate database. Indications of completed tasks (i.e. , “state” information that indicates a patient’s progress through a flow model, including the last task that was determined to be complete) may be stored in the patient care pathway status database. In some embodiments, patient care pathway status database may store information as to whether completion of a task or decision represented by a node is considered permanent or transient (i.e., complete for one particularevaluation of a patient’s progress through a flow model, but requires re-evaluation every time the patient’s progress through a flow model is evaluated).

[0147] At block 604, one or more next tasks may be retrieved from a care pathway flow model database 622. The care pathway flow model database 622 may store a plurality of flow models representing different medical or healthcare processes. For example, the care pathway flow model may store a patient discharge flow model (e.g., the patient flow model 300 of FIG. 3), a pre-operation / surgery flow model, etc. The care pathway flow model may be a context database 108 as depicted in FIG. 1. In some embodiments, tasks may be retrieved based on the current patient care pathway status. In some embodiments, the one or more next tasks may be retrieved based on a determination output by the LLM 630, as described below.

[0148] In some embodiments, tasks from more than one care pathway flow models may be retrieved. Activation criteria may indicate which care pathway flow models are applicable for a patient, and more than one pathway may be applicable, such that flow models may be combined to provide guidance for various situations. For example, activation criteria for a PCU flow model that applies to all PCU patients may include a patient being admitted to a PCU but not yet discharged from the PCU. Activation criteria for a heart failure flow model that applies to patients with heart failure may include a patient with heart failure. A patient in the PCU with heart failure would activate the PCU flow model and the heart failure flow model, such that the one or more next tasks retrieved from a care pathway flow model database 622 may include tasks from the PCU flow model and / or the care pathway flow model. Rather than requiring a flow model that specifically addresses patients with heart failure in a PCU, a flow model for a patient in the PCU and a separate flow model for a patient with heart failure may be evaluated to determine next tasks.

[0149] At block 606, the periodic patient care pathway assessment processor 620 may determine whether there are more next tasks for the patient to be added to a list of tasks that need to be completed. If there are no more next tasks to be added to the list of tasks, the list of tasks is provided to user, e.g., via a computing device 102. If there is a next task, at block 608 a prompt may be passed to an LLM 630 querying whether the task is complete. The prompt may include user-provided information to be used for retrieval augmented generation (RAG). The information in the prompt may be used to search for relevant contextual information from a patient electronic healthcare record using a patient data vector store 628 and provide the relevant contextual information to the LLM 630.

[0150] A patient electronic healthcare record may be stored in the patient electronic healthcare record data store 626. The patient electronic healthcare record data store 626 maybe a context database 108. In some embodiments, the patient care pathway status database 624 may be the same database as the patient electronic healthcare record data store 626 and / or the patient data vector store 628. In some embodiments, the patient care pathway status database 624 may be a separate database.

[0151] The patient data vector store 628 may be a context database 108. In some embodiments, the patient care pathway status database 624 may be the same database as the patient electronic healthcare record data store 626 and / or the patient care pathway status database 624. In some embodiments, the patient data vector store 628 may be a separate database. The patient data vector store 628 may store a set of vectors associated with data from patient electronic healthcare records. For example, text of a patient electronic healthcare record may be split into semantic clusters, and which may be encoded as the set of vectors. The semantic clusters may be one or more words, a portion of a word, or a character. To provide a machine learning model (e.g., LLM 630) with contextual data (e.g., healthcare data), a similarity search between a prompt that has also been split into semantic clusters and encoded as a set of vectors and data from the patient data vector store 628 may be conducted. The similarity between prompt vectors and the patient data vectors may be determined by calculating a distance between the prompt vectors and patient data vectors.

[0152] The prompt may be augmented with the contextual healthcare data resulting from the similarity search. The prompt may be provided to an LLM 630, which determines if the task is complete (block 610). If the task is incomplete, at block 612 a task list may be updated with the incomplete task and the method may return to block 606 to determine if there are any further incomplete tasks to be added to the task list.

[0153] If the LLM 630 determines the task is complete (block 610), the patient care pathway status may be updated at block 614. The updated patient care pathway status may be updated in the patient care pathway status database 624. The method may then return to the block 604 to determine next tasks based on the completed tasks and updated patient care pathway status.Exemplary Pre-Operation / Surgery Flow Model

[0154] FIGS. 7A-7G depict an exemplary flow model for a pre-operation / surgery process.

[0155] The flow model for the pre-operation / surgery process may include nodes that may be evaluated in parallel, and nodes that may be evaluated sequentially. The nodes may include tasks, decisions, and output generation for a pre-operation / surgery process.

[0156] As shown in FIG. 7A, a node may be a decision node for determining whether the patient has had prior heart surgery. If the patient has not had prior heart surgery, no furtheraction needs to be taken. If the patient has had prior heart surgery, the flow may proceed to a node for ordering a chest CT scan. The flow may also proceed from the decision node for the prior heart surgery to another decision node for determining whether a sternotomy is listed as part of the patient’s upcoming operation, which may be evaluated in parallel with the node ordering a chest CT scan. If not, no further action needs to be taken. If a sternotomy is listed, the flow sequentially proceeds to another decision node determining whether the patient has had a prior sternotomy. If not, no further action needs to be taken. If the patient had a prior sternotomy, the flow sequentially proceeds to a node to update the surgery listing to “redo sternotomy.”

[0157] Another node, which may be evaluated in parallel with the prior heart surgery node, may be a decision node for determining whether the patient has a pacemaker or an implantable cardioverter-defibrillator (ICD). If not, no further action needs to be taken. If the patient has a pacemaker or ICD, the flow may sequentially proceed to a node ordering investigation of the pacemaker or ICD device.

[0158] Another node, which may be evaluated in parallel with the prior heart surgery and pacemaker / ICD nodes, may be a decision node for determining whether the patient has a penicillin allergy. If not, no further action needs to be taken. If the patient has a penicillin allergy, the flow may proceed to a node for ordering a penicillin skin test. The flow may also proceed to a node for determining the type of allergic reaction to penicillin, which may be evaluated in parallel with the skin test. If the patient has a Cefazolin drug allergy or drug fever, Cephalosporin anaphylaxis, Stevens-Johnson Syndrome / Toxic Epidermal Necrolysis (SJS / TEN), Drug Reaction with Eosinophilia and Systemic Systems (DRESS), Acute Generalized Exanthematous Pustulosis (AGEP), and / or Generalized Bullous Fixed Drug Eruption (gbFDE, FDE), then the flow may proceed to a node for changing the interoperative medicine (e.g., Ancef to Vanco).

[0159] Continuing the flow model in FIG. 7B, a decision node, which may be evaluated in parallel with the nodes depicted in FIG. 7A, may be for determining whether the patient has a contrast dye allergy. If not, no further action needs to be taken. If the patient has a contrast dye allergy, the flow may proceed to a node ordering pre-medication for any contrast dye procedures.

[0160] Another node, which may be evaluated in parallel with the nodes depicted in FIG. 7A and the contrast dye allergy node, may be for determining if a dental form is available for the patient. If there is a dental form on file for the patient, no further action needs to be taken. Ifthere is no dental form, the flow may proceed to a node for sending a dental form to the patient to be completed by the patient.

[0161] Another node, which may be evaluated in parallel with the nodes depicted in FIG. 7A, the contrast dye allergy node, and the dental form node, may be for determining whether the patient had a transfusion in the last 90 days. If the patient has had a transfusion in the last 90 days, the flow may proceed to a node for ordering a blood test that determines a patient's ABO blood type and Rh factor. If a patient has not had a blood transfusion in the last 90 days, the flow model may proceed to a node for ordering a blood test that determines the patients ABO blood type, Rh factor, and screens for potential red blood cell antibodies.

[0162] Continuing the flow model in FIG. 7C, a decision node, which may be evaluated in parallel with the nodes in FIGS. 7A-B, may be for determining whether the patient is anemic. If not, no further action needs to be taken. If the patient is anemic, the flow may proceed to anode for ordering a preoperative anemia consultation.

[0163] Another node, which may be evaluated in parallel with the nodes in FIGS. 7A-B and the anemia node, may be for determining whether the patient is a current smoker. If not, no further action needs to be taken. If the patient is a current smoker, the flow may proceed to a node for offering a nicotine consultation.

[0164] Another node, which may be evaluated in parallel with the nodes in FIGS. 7A-B, the anemia node, and the current smoker node, may be for determining whether the patient takes a blood thinner. If not, no further action needs to be taken. If the patient takes a blood thinner, the flow model may proceed to a decision node for determining if the blood thinner is Warfarin. If the patient does not take Warfarin, the flow may proceed to a node for instructing the patient to take the last dose of direct oral anticoagulant (DOAC) three days before angiography and 6 days before surgery. If the patient does take Warfarin, the flow may proceed to a node instructing a healthcare provider to determine when the patient should take the last dose of Warfarin. The flow may also evaluate, in parallel with the node instructing a healthcare provider to determine when the patient should take the last does of Warfarin, a decision node for determining whether the patient has a thromboembolic risk. If not, no further action needs to be taken. If the patient has a thromboembolic risk, the flow may proceed to a node for providing the patient with Lovenox.

[0165] Continuing the flow model in FIG. 7D, a decision node, which may be evaluated in parallel with the nodes in FIGS. 7A-C, may be for determining whether the patient takes antiplatelet medication. If not, no further action needs to be taken. If the patient takes antiplatelet medication, the flow may proceed to a decision node for determining whether thepatient started the antiplatelet medication within the last 12 months. If not, the flow may proceed to a node stating that the last dose of the antiplatelet medication eight days before surgery. If the patient started the antiplatelet medication within the last 12 months, the flow may proceed to a node for asking the surgeon for medication instructions.

[0166] Another node, which may be evaluated in parallel with the nodes in FIGS. 7A-C and the antiplatelet node, may be for determining whether the patient takes aspirin. If not, no further action needs to be taken. If the patient takes aspirin, the flow may proceed to a node for determining whether the patient has coronary artery bypass grafting (CABG), a percutaneous coronary intervention (PCI) stent, or carotid stenosis. If not, the flow may proceed to a node advising that the last dose of aspirin should be taken eight days before surgery. If the patient has CABG, a PCI stent, or carotid stenosis, the flow may proceed to a node advising that the last dose of aspirin should be taken one day before surgery.

[0167] Another node, which may be evaluated in parallel with the nodes in FIGS. 7A-C, the antiplatelet node, and the aspirin node, may be a decision node for determining whether the patient takes krill or fish oil. If not, no further action needs to be taken. If the patient takes krill or fish oil, the flow may proceed to a node advising that the last dose of krill or fish oil should be taken eight days before surgery.

[0168] Another node, which may be evaluated in parallel with the nodes in FIGS. 7A-C, the antiplatelet node, the aspirin node, and the krill / fish oil node, may be a decision node for determining whether the patient takes an angiotensin-converting enzyme (ACE) inhibitor or an angiotensin II receptor blocker (ARB). If not, no further action needs to be taken. If the patient takes an ACE inhibitor or an ARB, the flow may proceed to a node advising that the last dose should be taken three days before surgery.

[0169] Another node, which may be evaluated in parallel with the nodes in FIGS. 7A-C, the antiplatelet node, the aspirin node, the krill / fish oil node, and the ACE inhibitor / ARB node, may be a decision node for determining whether the patient takes a biologic or immunosuppressant. If not, no further action needs to be taken. If the patient takes a biologic or immunosuppressant, the flow may proceed to a node advising that the last dose should be taken based on advice from an expert or tool for determining the last dose of biologic or immunosuppressant.

[0170] Another node, which may be evaluated in parallel with the nodes in FIGS. 7A-C, the antiplatelet node, the aspirin node, the krill / fish oil node, the ACE inhibitor / ARB node, and the biologic / immunosuppressant node, may be a decision node for determining whether the patient takes vitamins, supplements, or over-the-counter (OTC) drugs. If not, no further action needs tobe taken. If the patient takes vitamins, supplements or OTC drugs, the flow may proceed to a node advising that the last dose should be taken eight days before surgery.

[0171] Continuing the flow model in FIG. 7E, a decision node, which may be evaluated in parallel with the nodes in FIGS. 7A-D, may be for determining whether the patient takes diabetes medication. If not, no further action needs to be taken. If the patient takes diabetes medication, the flow may proceed to a decision node for determining the administration method of the diabetes medication. If the diabetes medication is taken orally, the flow may proceed to a route for determining whether the medication is Steglatro. If not, no further action needs to be taken. If the patient takes Steglatro, the flow may proceed to a node advising that the last dose should be taken five days before surgery. The Steglatro node may be evaluated in parallel with a decision node for determining whether the patient takes Invokana, Faxiga, or Jardiance. If not, no further action needs to be taken. If the patient takes Invokana, Faxiga, or Jardiance, the flow may proceed to a node advising that the last dose should be taken four days before surgery. Another decision node, which may be evaluated in parallel with the Steglatro and Invokana / Faxiga / Jardiance nodes, may be for determining whether the patient takes Metformin, Glucotrol, Januvia, Onglyza, Tradjenta, Rybelsus, Glynase, Glucovance, Actos, Prandin, glimepiride, glipizide, or glyburide. If not, no further action needs to be taken. If the patient takes Metformin, Glucotrol, Januvia, Onglyza, Tradjenta, Rybelsus, Glynase, Glucovance, Actos, Prandin, glimepiride, glipizide, or glyburide, the flow may proceed to a node advising that the last dose should be taken one day before surgery, in the morning. The Steglatro, Invokana / Faxiga / Jardiance node, and the Metformin etc. node may be evaluated in parallel with a node for determining whether the patient takes other diabetes medication. If not, no further action needs to be taken. If the patient takes other diabetes medication, the node proceeds to a flow advising that the last dose should be determined by a healthcare provider.

[0172] If the diabetes medication is taken via injection, the flow may proceed to a decision node for determining whether the patient takes Byetta, Victoza, Bydureon, Ozempic, Mounjaro, or Trulicity. If not, no further action needs to be taken. If the patient takes Byetta, Victoza, Bydureon, Ozempic, Mounjaro, or Trulicity, the flow may proceed to a decision node for determining the frequency of use of the medication. If the medication is taken daily, the last dose should be taken one day before the surgery. If the medication is taken weekly, the last dose should be taken eight days before the surgery. Another node, which may be evaluated in parallel with the Byetta etc. node, may be for determining whether the patient is taking insulin. If not, no further action needs to be taken. If the patient is taking insulin, the flow may proceed to a node for advising that the last dose should be taken one day before the surgery. The Byettaetc. node and the insulin node may be evaluated in parallel with a node for determining whether the patient takes other injected diabetes medications. If not, no further action needs to be taken. If the patient takes other injected diabetes medications, the flow may proceed to a node advising that a healthcare provider should determine the last dose.

[0173] Continuing the flow model in FIG. 7F, a decision node, which may be evaluated in parallel with the nodes in FIGS. 7A-E, may be for determining whether the patient’s A1C level is 9 or higher. If not, no further action needs to be taken. If the patient’s A1C level is 9 or higher, the flow may proceed to a node advising that the surgeon should be contacted.

[0174] Another node, which may be evaluated in parallel with the nodes in FIGS. 7A-E and the A1C node, may be a decision node for determining whether the patient uses an oral steroid or steroid cream. If not, no further action needs to be taken. If the patient uses an oral steroid or steroid cream, the flow may proceed to a decision node for determining whether the patient takes an oral steroid. If not, no further action needs to be taken. If the patient takes an oral steroid, the flow may proceed to a node instructing a healthcare provider to note the dose and determine oral steroid management.

[0175] Another node, which may be evaluated in parallel with the nodes in FIGS. 7A-E, the A1C node, and the steroid node, may be a decision node for determining whether the patient takes a non-steroidal anti-inflammatory drug (NSAID). If not, no further action needs to be taken. If the patient takes an NSAID, the flow proceeds a node advising that the last dose should be taken three days before surgery.

[0176] Another node, which may be evaluated in parallel with the nodes in FIGS. 7A-E, the A1C node, the steroid node, and the NSAID node, may be a decision node for determining whether the patient takes prescription pain medication. If not, no further action needs to be taken. If the patient takes prescription pain medication, the flow proceeds a node advising that the healthcare provider to determine when the last dose should be taken.

[0177] Another node, which may be evaluated in parallel with the nodes in FIGS. 7A-E, the A1C node, the steroid node, the NSAID node, and the prescription pain medication node, may be a decision node for determining whether the patient has taken a thyroid stimulating hormone (TSH) test within one year of the surgery. If the patient has taken a TSH test within one year of surgery, no further action needs to be taken. If the patient has not taken a TSH test within one year of surgery, the flow may proceed to a node ordering a TSH test.

[0178] Another node, which may be evaluated in parallel with the nodes in FIGS. 7A-E, the A1C node, the steroid node, the NSAID node, the prescription pain medication node, and theTSH node, may be a decision node for determining whether the patient has taken an A1C test within one year of the surgery. If the patient has taken an A1C test within three months of surgery, no further action needs to be taken. If the patient has not taken a A1C test within one year of surgery, the flow may proceed to a node ordering an A1C test.

[0179] Another node, which may be evaluated in parallel with the nodes in FIGS. 7A-E, the A1C node, the steroid node, the NSAID node, the prescription pain medication node, the TSH node, and the A1C node, may be a decision node for determining whether the patient has had a chest x-ray (CXR) within six weeks of the surgery. If the patient has taken a CXR within six weeks of surgery, no further action needs to be taken. If the patient has not taken a CXR within six weeks of surgery, the flow may proceed to a node ordering a CXR.

[0180] Continuing the flow model in FIG. 7G, a decision node, which may be evaluated in parallel with the nodes in FIGS. 7A-F, may be for determining whether the patient has had an electrocardiogram (EKG) within six weeks of the surgery. If the patient has had an EKG within six weeks of surgery, no further action needs to be taken. If the patient has not had an EKG within six weeks of surgery, the flow may proceed to a node ordering the EKG.

[0181] Another decision node, which may be evaluated in parallel with the nodes in FIGS. 7A-F and the EKG node, may be for determining whether the patient has had a complete blood count (CBC) test within six weeks of the surgery. If the patient has had a CBC test within six weeks of surgery, no further action needs to be taken. If the patient has not had a CBC test within six weeks of surgery, the flow may proceed to a node ordering the CBC test.

[0182] Another decision node, which may be evaluated in parallel with the nodes in FIGS. 7A-F, the EKG node, and the CBC node, may be for determining whether the patient has had a human chorionic gonadotropin (hCG) test within six weeks of the surgery. If the patient has had an hCG test within six weeks of surgery, no further action needs to be taken. If the patient has not had an hCG test within six weeks of surgery, the flow may proceed to a node ordering the hcG test.

[0183] Another decision node, which may be evaluated in parallel with the nodes in FIGS. 7A-F, the EKG node, and the CBC node, may be for determining whether the patient has had a human chorionic gonadotropin (hCG) test within six weeks of the surgery. If the patient has had an hCG test within six weeks of surgery, no further action needs to be taken. If the patient has not had an hCG test within six weeks of surgery, the flow may proceed to a node ordering the hcG test.

[0184] Another decision node, which may be evaluated in parallel with the nodes in FIGS. 7A-F, the EKG node, and the CBC node, and the hCG node may be for determining whether the patient has had a basic metabolic panel (BMP) test within six weeks of the surgery. If the patient has had a BMP test within six weeks of surgery, no further action needs to be taken. If the patient has not had an BMP test within six weeks of surgery, the flow may proceed to a node ordering the BMP test.

[0185] Another decision node, which may be evaluated in parallel with the nodes in FIGS. 7A-F, the EKG node, and the CBC node, the hCG node, and the BMP node may be for determining whether the patient has had a prothrombin time international normalized ratio (PT / INR) test within six weeks of the surgery. If the patient has had a PT / INR test within six weeks of surgery, no further action needs to be taken. If the patient has not had a PT / INR test within six weeks of surgery, the flow may proceed to a node ordering the PT / INR test.

[0186] Another decision node, which may be evaluated in parallel with the nodes in FIGS. 7A-F, the EKG node, and the CBC node, the hCG node, the BMP node, and the PT / INR node, may be for determining whether the patient has had a type and screen (T&S) test withinsix weeks of the surgery. If the patient has had a T&S test within six weeks of surgery, no further action needs to be taken. If the patient has not had a T&S test within six weeks of surgery, the flow may proceed to a node ordering the T&S test.

[0187] While FIG. 7 depicts a flow model for a pre-operation / surgery process, the systems and methods described herein may be used for other types of flow models. For example, the techniques of the present disclosure may be used to evaluate a patient discharge flow model (as depicted in FIG. 3) and / or other medical and / or healthcare flow models, ticket sales flow models, insurance coverage flow models, travel itinerary flow models, flight booking flow models, flow models for autonomous analysis of complex data sets, etc.Additional Considerations

[0188] The following considerations also apply to the foregoing discussion. Throughout this specification, plural instances may implement operations or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may be performed concurrently, and nothing requires that the operations be performed in the order illustrated. These and other variations, modifications, additions, and improvements fall within the scope of the subject matter herein.

[0189] It should also be understood that, unless a term is expressly defined in this patent using the sentence "As used herein, the term " " is hereby defined to mean . . . " or a similar sentence, there is no intent to limit the meaning of that term, either expressly or by implication, beyond its plain or ordinary meaning, and such term should not be interpreted to be limited in scope based on any statement made in any section of this patent (other than the language of the claims). To the extent that any term recited in the claims at the end of this patent is referred to in this patent in a manner consistent with a single meaning, that is done for sake of clarity only so as to not confuse the reader, and it is not intended that such claim term be limited, by implication or otherwise, to that single meaning. Finally, unless a claim element is defined by reciting the word "means" and a function without the recital of any structure, it is not intended that the scope of any claim element be interpreted based on the application of 35 U.S.C. § 112(f).

[0190] Unless specifically stated otherwise, discussions herein using words such as "processing," "computing," "calculating," "determining," "presenting," "displaying," or the like may refer to actions or processes of a machine (e.g., a computer) that manipulates or transforms data represented as physical (e.g., electronic, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or a combination thereof), registers, or other machine components that receive, store, transmit, or display information.

[0191] As used herein any reference to "one aspect" or "an aspect" means that a particular element, feature, structure, or characteristic described in connection with the aspect is included in at least one aspect. The appearances of the phrase "in one aspect" in various places in the specification are not necessarily all referring to the same aspect.

[0192] As used herein, the terms "comprises," "comprising," "includes," "including," "has," "having" or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, "or" refers to an inclusive or and not to an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).

[0193] In addition, use of "a" or "an" is employed to describe elements and components of the aspects herein. This is done merely for convenience and to give a general sense of the invention. This description should be read to include one or at least one and the singular also includes the plural unless it is obvious that it is meant otherwise.

[0194] Upon reading this disclosure, those of skill in the art will appreciate still additional alternative structural and functional designs for implementing the concepts disclosed herein, through the principles disclosed herein. Thus, while particular aspects and applications have been illustrated and described, it is to be understood that the disclosed aspects are not limited to the precise construction and components disclosed herein. Various modifications, changes and variations, which will be apparent to those skilled in the art, may be made in the arrangement, operation and details of the method and apparatus disclosed herein without departing from the spirit and scope defined in the appended claims.

[0195] Aspects of the techniques described in the present disclosure may include any of the following aspects, either alone or in combination:

[0196] 1 . A method comprising: receiving, via one or more processors and from a computing device, a prompt including a portion of a flow model and a node on the portion of the flow model; retrieving, via the one or more processors, contextual data associated with the portion of the flow model; providing, via the one or more processors, the prompt and the contextual data to a trained machine learning (ML) model to determine completion of one or more actions in the portion of the flow model; generating, by processing the prompt and contextual data with the trained machine learning model, one or more next steps based on the completion of one or more actions in the pathway; generating, via the one or more processors and using the ML model, a summary including a justification for the one or more completed actions and next steps supported by one or more datapoints from the contextual data; storing, via the one or more processors, the summary, the one or more completed actions, and a node associated with the completed actions; andgenerating, via the one or more processors, a display, a notification, or an alert of the one or more next steps on the computing device.

[0197] 2. The method of claim 1 , further comprising: providing, via the one or more processors, an additional portion of the flow model to the trained machine learning model to determine completion of one or more actions in the additional portion of the flow model, wherein the trained machine learning model is configured with an agent to process the portion of the flow model and an additional agent to process the additional portion of the flow model; and generating, via the one or more processors by processing the additional portion of the flow model, the prompt, and the contextual data with the trained machine learning model, one or more additional next steps based on the completion of one or more actions in the additional portion of the flow model, the one or more additional next steps being generated in parallel with the one or more next steps.

[0198] 3. The method of claim 1 , further comprising: generating, via the one or more processors and using a second machine learning model, labelled synthetic contextual data; receiving, via the one or more processors, a test prompt from the computing device including the portion of the flow model and the node; and training, via the one or more processors, the ML model by providing the synthetic contextual data and the test prompt to the output and determining an accuracy of the machine learning model.

[0199] 4. The method of claim 1 , further comprising: determining, based on the one or more next steps and the contextual data and using the ML model, a user associated with at least one of the one or more next steps; and transmitting, via the one or more processors, the at least one of the one or more next steps to a computing device associated with the user.

[0200] 5. The method of claim 1 , wherein the flow model is associated with a medical or healthcare process, and wherein the contextual data includes an electronic healthcare record updated in real-time.

[0201] 6. The method of claim 1 , wherein the flow model is for a pre-operation / surgery process, and wherein the contextual data includes an electronic healthcare record.

[0202] 7. The method of claim 1 , wherein the flow model is associated with a patient discharge process, and wherein the contextual data includes an electronic healthcare record (EHR) updated in real-time.

[0203] 8. The method of claim 7, further comprising: obtaining, via the one or more processors, one or more portions of a flow model describing a patient discharge process and historical electronic healthcare records; and training, via the one or more processors, a machine learning model, using the one or more portions of the flow model describing a patient discharge process and the historical electronic health care to generate one or more recommendations for discharging a patient.

[0204] 9. The method of claim 7, further comprising: receiving, by the one or more processors, an update to the healthcare data including real-time data collected for the patient; and retraining, by the one or more processors, the trained machine learning model using the update to the healthcare data.

[0205] 10. The method of claim 7, wherein the recommendations for discharging a patient include one or more of (i) a step for continuing treatment, (ii) a medication, (iii) a continuing treatment location, (iv) a medical device, (v) a predicted time of discharge, or (vi) a follow-up appointment time.

[0206] 11. The method of claim 6, further comprising: receiving, via the one or more processors, an input including a healthcare question; and generating, via a chatbot, an output including healthcare information.

[0207] 12. The method of claim 10, further comprising: determining, via the chatbot, that the healthcare question indicates a situation that requires urgent action; generating, via the chatbot, a notification of the situation that requires urgent action; and transmitting, via the one or more processors, the notification to an output device associated with a healthcare provider.

[0208] 13. The method of claim 7, further comprising: receiving, via one or more processors, an update to the healthcare data; identifying, by processing the update to the healthcare data using the trained machine learning model, one or more factors causing a delay in discharge; and generating, by using the trained machine learning model, an output including a notification of the one or more factors causing a delay in discharge.

[0209] 14. The computer-implemented method of claim 6, further comprising: generating, via a chatbot, an output including a notification of a scheduled appointment.

[0210] 15. The method of claim 1 , wherein the machine learning model is a large language model (LLM) or a multi-modal machine learning model.

Claims

What is claimed is:1 . A method comprising: receiving, via one or more processors and from a computing device, a prompt including a portion of a flow model and a node on the portion of the flow model; retrieving, via the one or more processors, contextual data associated with the portion of the flow model; providing, via the one or more processors, the prompt and the contextual data to a trained machine learning (ML) model to determine completion of one or more actions in the portion of the flow model; generating, by processing the prompt and contextual data with the trained machine learning model, one or more next steps based on the completion of one or more actions in the pathway; generating, via the one or more processors and using the ML model, a summary including a justification for the one or more completed actions and next steps supported by one or more datapoints from the contextual data; storing, via the one or more processors, the summary, the one or more completed actions, and a node associated with the completed actions; and generating, via the one or more processors, a display, a notification, or an alert of the one or more next steps on the computing device.

2. The method of claim 1 , further comprising: providing, via the one or more processors, an additional portion of the flow model to the trained machine learning model to determine completion of one or more actions in the additional portion of the flow model, wherein the trained machine learning model is configured with an agent to process the portion of the flow model and an additional agent to process the additional portion of the flow model; and generating, via the one or more processors by processing the additional portion of the flow model, the prompt, and the contextual data with the trained machine learning model, one or more additional next steps based on the completion of one or more actions in the additional portion of the flow model, the one or more additional next steps being generated in parallel with the one or more next steps.

3. The method of claim 1 , further comprising: generating, via the one or more processors and using a second machine learning model, labelled synthetic contextual data;receiving, via the one or more processors, a test prompt from the computing device including the portion of the flow model and the node; and training, via the one or more processors, the ML model by providing the synthetic contextual data and the test prompt to the output and determining an accuracy of the machine learning model.

4. The method of claim 1 , further comprising: determining, based on the one or more next steps and the contextual data and using the ML model, a user associated with at least one of the one or more next steps; and transmitting, via the one or more processors, the at least one of the one or more next steps to a computing device associated with the user.

5. The method of claim 1 , wherein the flow model is associated with a medical or healthcare process, and wherein the contextual data includes an electronic healthcare record updated in real-time.

6. The method of claim 1 , wherein the flow model is for a pre-operation / surgery process, and wherein the contextual data includes an electronic healthcare record.

7. The method of claim 1 , wherein the flow model is associated with a patient discharge process, and wherein the contextual data includes an electronic healthcare record (EHR) updated in real-time.

8. The method of claim 7, further comprising: obtaining, via the one or more processors, one or more portions of a flow model describing a patient discharge process and historical electronic healthcare records; and training, via the one or more processors, a machine learning model, using the one or more portions of the flow model describing a patient discharge process and the historical electronic health care to generate one or more recommendations for discharging a patient.

9. The method of claim 7, further comprising: receiving, by the one or more processors, an update to the healthcare data including real-time data collected for the patient; and retraining, by the one or more processors, the trained machine learning model using the update to the healthcare data.

10. The method of claim 7, wherein the recommendations for discharging a patient include one or more of (i) a step for continuing treatment, (ii) a medication, (iii) a continuing treatment location, (iv) a medical device, (v) a predicted time of discharge, or (vi) a follow-up appointment time.

11. The method of claim 6, further comprising: receiving, via the one or more processors, an input including a healthcare question; and generating, via a chatbot, an output including healthcare information.

12. The method of claim 10, further comprising: determining, via the chatbot, that the healthcare question indicates a situation that requires urgent action; generating, via the chatbot, a notification of the situation that requires urgent action; and transmitting, via the one or more processors, the notification to an output device associated with a healthcare provider.

13. The method of claim 7, further comprising: receiving, via one or more processors, an update to the healthcare data; identifying, by processing the update to the healthcare data using the trained machine learning model, one or more factors causing a delay in discharge; and generating, by using the trained machine learning model, an output including a notification of the one or more factors causing a delay in discharge.

14. The computer-implemented method of claim 6, further comprising: generating, via a chatbot, an output including a notification of a scheduled appointment.

15. The method of claim 1 , wherein the machine learning model is a large language model (LLM) or a multi-modal machine learning model.

Images