Methods and systems for optimizing healthcare data management using generative artificial intelligence agents

Abstract

Systems and methods for optimizing healthcare data management using generative artificial intelligence (Al) agents are disclosed herein. A computing system includes a processor and a memory having stored thereon computer-executable instructions that, when executed by the processor, cause the computing system to: receive a prompt input referencing healthcare data included in an electronic health record (EHR); determine, by executing a trained machine learning (ML) model of an agent object, a user intent corresponding to the prompt input indicating retrieval of the healthcare data from the EHR; transmit, via the agent object and through an associated application programming interface (API), a control instruction for a digital tool to retrieve the healthcare data from the EHR; generate, by executing the trained ML model, a prompt output that includes the healthcare data; and cause the prompt output to be displayed in an output device.

Description

METHODS AND SYSTEMS FOR OPTIMIZING HEALTHCARE DATA MANAGEMENT USING GENERATIVE ARTIFICIAL INTELLIGENCE AGENTSCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 601 ,956, entitled “Methods and Systems for Optimizing Healthcare Data Management Using Generative Artificial Intelligence Agents,” filed on November 22, 2023, the disclosure of which is hereby incorporated herein by reference.TECHNICAL FIELD[2] The present disclosure is generally directed to methods and systems for optimizing healthcare data management, and more particularly, to techniques for training and operating one or more generative artificial intelligence (Al) agents to optimize healthcare data management.BACKGROUND[3] The evolution of Electronic Health Records (EHRs) has brought a plethora of information into the fingertips of healthcare providers. Today's EHRs are a far cry from their rudimentary predecessors, encompassing a vast range and volume of patient data. This encompasses structured and unstructured notes, lab data, pathology records, imaging findings, outside records, and more. Every patient's journey is meticulously documented, creating a digital mosaic of their health narrative. While this data-rich environment is a treasure trove for personalized patient care, it also introduces challenges in terms of data accessibility and usability. Providers often find themselves navigating through multiple layers of the EHR, clicking repetitively to extract snippets of vital information. This not only hampers their efficiency but also poses risks in terms of potential errors due to oversight. The fragmented, disparate, and non-intuitive presentation of information in today's EHRs further exacerbates the challenge, contributing to provider fatigue and detracting from the time that could be better spent on direct patient care.[4] In an effort to combat these inefficiencies, the healthcare sector is witnessing a surge in the development of Al technologies, including the promising realms of generative Al and large language models (LLMs). These innovations hold the potential to address significant clinical and operational challenges. However, the translation of these solutions into practical, everyday clinical use remains a hurdle. One such obstacle is the difficulty of seamless integration into existing EHR systems and clinical and administrative workflows. Even if an Al tool / solution isdeemed viable, its deployment is often stymied by the need for EHR engineers to configure it, leading to delays and potential mismatches with specific institutional needs.[5] Moreover, introducing these conventional tools and solutions into clinical settings is fraught with challenges, primarily due to concerns around Protected Health Information (PHI). Furthermore, while certain tasks in the clinical environment might appear trivial, such as paging a provider or collating clinical documents, their cumulative impact on time and efficiency is significant. Overall, these tasks remain painstakingly cumbersome in healthcare settings.[6] Thus, the modern healthcare landscape, while enriched with data, faces significant challenges in terms of data accessibility, Al integration, and productivity optimization. Collaboration between clinicians is often difficult with the design of modern day EHRs. Such EHRs are largely designed to function for one clinician at a time, and even as communication tools have rapidly evolved to facilitate instant multi-modal communication, EHR centric communication remains highly asynchronous and suboptimal. Addressing these, and other, issues is crucial to ensure that healthcare institutions continue to provide care that is not only comprehensive and accurate, but also efficient and tailored to the unique needs of both providers and patients.[7] Therefore, there is an opportunity for improved platforms for healthcare data management using generative Al agents, capable of delivering more timely, accurate, and comprehensive outcomes in complex clinical cases.BRIEF SUMMARY[8] In an embodiment, a computing system for optimizing healthcare data management using generative artificial intelligence (Al) agents comprises: one or more processors; and one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing system to: receive, at the one or more processors, a prompt input referencing healthcare data included in an electronic health record (EHR), determine, by the one or more processors executing a trained machine learning (ML) model of an agent object, a user intent corresponding to the prompt input indicating retrieval of the healthcare data from the EHR, transmit, by the one or more processors via the agent object and through an associated application programming interface (API), a control instruction for a digital tool to retrieve the healthcare data from the EHR, generate, by the one or more processors executing the trained ML model, a prompt output that includes the healthcare data, and cause, by the one or more processors, the prompt output to be displayed in an output device.[9] In a variation of this embodiment, the one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, further cause the computing system to: generate, by the one or more processors, a preprocessed prompt input corresponding to the prompt input; and generate, by the one or more processors, post-processed healthcare data that is formatted based on the user intent, and wherein the prompt output includes the post-processed healthcare data. Further in this variation, the postprocessed healthcare data includes at least one of: (i) a graphical representation of the healthcare data, (ii) a collated document that includes the healthcare data, (iii) an image included as part of the healthcare data, (iv) a selectable link that directs users to the healthcare data in a source location, or (v) a chat option configured to connect a user with a second user that is associated with the healthcare data and / or indicated in the prompt input or the user intent.

[0010] In another variation of this embodiment, the trained ML model of the agent object is (a) a large language model (LLM) that is configured to receive input healthcare data and generate a prompt output based on the input healthcare data or (b) a multi-modal machine learning model configured to output multi-modal data associated with the input healthcare data.

[0011] In yet another variation of this embodiment, the digital tool is one of a plurality of digital tools, and the one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, further cause the computing system to: analyze, by the one or more processors, each of the plurality of digital tools to determine whether any of the plurality of digital tools is configured to retrieve the healthcare data from the EHR; identify, by the one or more processors, that the digital tool is configured to retrieve the healthcare data from the EHR; and generate, by the one or more processors, the control instruction for the digital tool to retrieve the healthcare data.

[0012] In still another variation of this embodiment, the one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, further cause the computing system to: receive, at the one or more processors, a subsequent prompt input related to the prompt output; determine, by the one or more processors executing the trained ML model of the agent object, a subsequent user intent corresponding to the subsequent prompt input; transmit, by the one or more processors via the agent object and through a second API, a subsequent control instruction for a second digital tool to perform an action related to the subsequent user intent, wherein the second digital tool is different from the digital tool; generate, by the one or more processors executing the trained ML model, a subsequent prompt output based on the action performed by the second digital tool; and cause,by the one or more processors, the subsequent prompt output to be displayed in the output device.

[0013] In yet another variation of this embodiment, a first user submits the prompt input, the output device is a first device of the first user, the prompt output includes a selectable option to connect with a second device of a second user in real-time, and the one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, further cause the computing system to: receive, at the one or more processors, a subsequent prompt input indicating a selection of the selectable option to connect with the second device of the second user; transmit, by the one or more processors via the agent object and through a second API, a subsequent control instruction for a second digital tool to connect the first device of the first user with the second device of the second user across a live data stream, wherein the second digital tool is different from the digital tool; and cause, by the one or more processors, the prompt output to continue to be displayed in the output device during the live data stream between the first device of the first user and the second device of the second user.

[0014] In still another variation of this embodiment, the prompt input is a verbal input from a user, and the one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, further cause the computing system to: transcribe, by the one or more processors executing the trained ML model of the agent object, the verbal input into a textual transcription; and determine, by the one or more processors executing the trained ML model of the agent object, the user intent based on the textual transcription.

[0015] In another embodiment, a non-transitory computer-readable medium includes instructions that when executed, cause a computer to: non-transitory computer-readable medium, having stored thereon instructions that when executed, cause a computer to: receive a prompt input referencing healthcare data included in an electronic health record (EHR); determine, by executing a trained machine learning (ML) model of an agent object, a user intent corresponding to the prompt input indicating retrieval of the healthcare data from the EHR; transmit, via the agent object and through an associated application programming interface (API), a control instruction for a digital tool to retrieve the healthcare data from the EHR; generate, by executing the trained ML model, a prompt output that includes the healthcare data; and cause the prompt output to be displayed in an output device.

[0016] In a variation of this embodiment, the instructions, when executed, further cause the computer to: generate a preprocessed prompt input corresponding to the prompt input; andgenerate post-processed healthcare data that is formatted based on the user intent, and wherein the prompt output includes the post-processed healthcare data. Further in this variation, the post-processed healthcare data includes at least one of: (i) a graphical representation of the healthcare data, (ii) a collated document that includes the healthcare data, (iii) an image included as part of the healthcare data, (iv) a selectable link that directs users to the healthcare data in a source location, or (v) a chat option configured to connect a user with a second user that is associated with the healthcare data and / or indicated in the prompt input or the user intent.

[0017] In another variation of this embodiment, the trained ML model of the agent object is (a) a large language model (LLM) that is configured to receive input healthcare data and generate a prompt output based on the input healthcare data or (b) a multi-modal machine learning model configured to output multi-modal data associated with the input healthcare data.

[0018] In yet another variation of this embodiment, the digital tool is one of a plurality of digital tools, and the instructions, when executed, cause a computer to: analyze each of the plurality of digital tools to determine whether any of the plurality of digital tools is configured to retrieve the healthcare data from the EHR; identify that the digital tool is configured to retrieve the healthcare data from the EHR; and generate the control instruction for the digital tool to retrieve the healthcare data.

[0019] In still another variation of this embodiment, the instructions, when executed, further cause the computer to: receive a subsequent prompt input related to the prompt output; determine, by executing the trained ML model of the agent object, a subsequent user intent corresponding to the subsequent prompt input; transmit, via the agent object and through a second API, a subsequent control instruction for a second digital tool to perform an action related to the subsequent user intent, wherein the second digital tool is different from the digital tool; generate, by executing the trained ML model, a subsequent prompt output based on the action performed by the second digital tool; and cause the subsequent prompt output to be displayed in the output device.

[0020] In yet another variation of this embodiment, a first user submits the prompt input, the output device is a first device of the first user, the prompt output includes a selectable option to connect with a second device of a second user in real-time, and the instructions, when executed, further cause the computer to: receive a subsequent prompt input indicating a selection of the selectable option to connect with the second device of the second user; transmit, via the agent object and through a second API, a subsequent control instruction for a second digital tool to connect the first device of the first user with the second device of thesecond user across a live data stream, wherein the second digital tool is different from the digital tool; and cause the prompt output to continue to be displayed in the output device during the live data stream between the first device of the first user and the second device of the second user.

[0021] In yet another embodiment, a computer-implemented method for optimizing healthcare data management using generative artificial intelligence (Al) agents comprises: receiving, at one or more processors, a prompt input referencing healthcare data included in an electronic health record (EHR); determining, by the one or more processors executing a trained machine learning (ML) model of an agent object, a user intent corresponding to the prompt input indicating retrieval of the healthcare data from the EHR; transmitting, by the one or more processors via the agent object and through an associated application programming interface (API), a control instruction for a digital tool to retrieve the healthcare data from the EHR; generating, by the one or more processors executing the trained ML model, a prompt output that includes the healthcare data; and causing, by the one or more processors, the prompt output to be displayed in an output device.

[0022] In a variation of this embodiment, the method further comprises: generating, by the one or more processors, a preprocessed prompt input corresponding to the prompt input; and generating, by the one or more processors, post-processed healthcare data that is formatted based on the user intent, wherein the prompt output includes the post-processed healthcare data, and wherein the post-processed healthcare data includes at least one of: (i) a graphical representation of the healthcare data, (ii) a collated document that includes the healthcare data, (iii) an image included as part of the healthcare data, (iv) a selectable link that directs users to the healthcare data in a source location, or (v) a chat option configured to connect a user with a second user that is associated with the healthcare data and / or indicated in the prompt input or the user intent.

[0023] In another variation of this embodiment, the trained ML model of the agent object is (a) a large language model (LLM) that is configured to receive input healthcare data and generate a prompt output based on the input healthcare data or (b) a multi-modal machine learning model configured to output multi-modal data associated with the input healthcare data.

[0024] In yet another variation of this embodiment, the digital tool is one of a plurality of digital tools, and the method further comprises: analyzing, by the one or more processors, each of the plurality of digital tools to determine whether any of the plurality of digital tools is configured to retrieve the healthcare data from the EHR; identifying, by the one or more processors, that thedigital tool is configured to retrieve the healthcare data from the EHR; and generating, by the one or more processors, the control instruction for the digital tool to retrieve the healthcare data.

[0025] In still another variation of this embodiment, the method further comprises: receiving, at the one or more processors, a subsequent prompt input related to the prompt output; determining, by the one or more processors executing the trained ML model of the agent object, a subsequent user intent corresponding to the subsequent prompt input; transmitting, by the one or more processors via the agent object and through a second API, a subsequent control instruction for a second digital tool to perform an action related to the subsequent user intent, wherein the second digital tool is different from the digital tool; generating, by the one or more processors executing the trained ML model, a subsequent prompt output based on the action performed by the second digital tool; and causing, by the one or more processors, the subsequent prompt output to be displayed in the output device.BRIEF DESCRIPTION OF THE FIGURES

[0026] The figures described below depict various aspects of the system and methods disclosed therein. It should be understood that each figure depicts one aspect of a particular aspect of the disclosed system and methods, and that each of the figures is intended to accord with a possible aspect thereof. Further, wherever possible, the following description refers to the reference numerals included in the following figures, in which features depicted in multiple figures are designated with consistent reference numerals.

[0027] FIG. 1 depicts an exemplary computing environment in which the techniques disclosed herein may be implemented, in accordance with various embodiments herein.

[0028] FIG. 2 depicts an exemplary agent-controller configuration, in accordance with various embodiments herein.

[0029] FIG. 3 depicts an exemplary computer-implemented method for optimizing healthcare data management using generative Al agents, in accordance with various embodiments herein.

[0030] FIG. 4 depicts a first example graphical user interface (GUI) associated with optimizing healthcare data management using generative Al agents, in accordance with various embodiments herein.

[0031] FIG. 5 depicts a second example graphical user interface (GUI) associated with optimizing healthcare data management using generative Al agents, in accordance with various embodiments herein.

[0032] FIG. 6 depicts a third example graphical user interface (GUI) associated with optimizing healthcare data management using generative Al agents, in accordance with various embodiments herein.

[0033] FIG. 7 depicts a fourth example graphical user interface (GUI) associated with optimizing healthcare data management using generative Al agents, in accordance with various embodiments herein.

[0034] FIG. 8 depicts a fifth example graphical user interface (GUI) associated with optimizing healthcare data management using generative Al agents, in accordance with various embodiments herein.

[0035] FIG. 9 depicts a sixth example graphical user interface (GUI) associated with optimizing healthcare data management using generative Al agents, in accordance with various embodiments herein.

[0036] FIG. 10 depicts a seventh example graphical user interface (GUI) associated with optimizing healthcare data management using generative Al agents, in accordance with various embodiments herein.

[0037] FIG. 1 1 depicts an eighth example graphical user interface (GUI) associated with optimizing healthcare data management using generative Al agents, in accordance with various embodiments herein.

[0038] FIG. 12 depicts a ninth example graphical user interface (GUI) associated with optimizing healthcare data management using generative Al agents, in accordance with various embodiments herein.

[0039] FIG. 13 depicts a tenth example graphical user interface (GUI) associated with optimizing healthcare data management using generative Al agents, in accordance with various embodiments herein.

[0040] FIG. 14 depicts an eleventh example graphical user interface (GUI) associated with optimizing healthcare data management using generative Al agents, in accordance with various embodiments herein.

[0041] The figures depict preferred aspects for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative aspects of the systems and methods illustrated herein may be employed without departing from the principles of the invention described herein.DETAILED DESCRIPTIONOverview

[0042] The present techniques provide methods and systems for, inter alia, optimizing healthcare data management, and more particularly, to techniques for training and operating one or more Al agents to optimize healthcare data management using generative Al agents (e.g., in a clinical setting). Specifically, the present techniques may include technologies for training one or more ML models (e.g., one or more language models, one or more artificial neural networks, etc.) to approximate human decision makers, for example, using a corpus of EHR data, correspondence, and / or other metadata. Once trained, these agents may generate outputs in response to user prompts related to healthcare data contained in, for example, an EHR corresponding to a particular patient.

[0043] The present techniques may further include one or more client computing devices, or applications that may be installed on client computing devices, that enable users to access one or more trained models by providing input prompts that are processed by one or more trained models. These prompts may be propagated to the computerized agents (e.g., trained models) for processing. The client computing device may receive responses from these agents, and convey those responses to the user, optionally after performing pre-processing and / or postprocessing steps.

[0044] As discussed herein, the systems and methods of the present disclosure may seamlessly integrate with an electronic medical record system, and thereby reduce / eliminate manual tasks by pulling up essential patient details such as documents, medication lists, and past medical records. More specifically, the natural language interface of the assistance chatbot described herein, powered by LLMs, offers a unique orchestration layer capability to, for example, summarize notes, answer specific questions about a patient’s history based on information from the patient's EHR in an interactive dialog format, and / or other similar functions. As a result, the assistance chatbot described herein may obviate certain clinical and administrative tasks, which are often burdensome resulting in operational inefficiencies, user burnout and increased fatigue, and potential human errors through manual, time-consuming tasks

[0045] Broadly, the assistance chatbot described herein may utilize an intelligent orchestration layer which may execute and / or otherwise leverage LLMs (e.g., agent objects with LLMs) to decide what / which "plugin” or other tool may best answer any question(s) included as part of an input prompt based on a user’s intent, as determined from the input prompt. The assistance chatbot may receive data from these tools, format the data as necessary (e.g., graphical representation, formatted document, high-level summary, etc.), and may cause a user device to display the information to the user as part of an output / response prompt.

[0046] The assistance chatbot described herein may further assist busy clinicians with a checkout feature, allowing such clinicians to extract text snippets from the EHR, incorporate the snippets into generated documents (e.g., PDFs), and share the generated documents with colleagues. Leveraging these functions, the assistance chatbot may quickly and accurately create teaching files, tumor board notes, and / or shareable summaries for simple and efficient “curbside” consults between and / or with medical practitioners. Thus, the assistance chatbot described herein may be deeply integrated into communication platforms and utilize the multimodal communication capabilities of such platforms while also seamlessly integrating into more legacy communication tools like paging systems.

[0047] Further, the assistance chatbot described herein allows integration of additional tools through the intelligent orchestration layer previously mentioned. These tools may include Al output, distilled LLM models, and / or other analytics (e.g., calculators and dashboards) that may enable the assistance chatbot to generate data related to a patient’s EHR. For example, the assistance chatbot described herein may utilize a distilled LLM as a task specific LLM that removes uncertainty in responses associated with traditional LLMs.

[0048] In accordance with the above, and with the disclosure herein, the present disclosure includes improvements in computer functionality or in improvements to other technologies at least because the disclosure describes that, e.g., a hosting server (e.g., server computing device), or otherwise computing device (e.g., a client computing device), is improved where the intelligence or predictive ability of the hosting server or computing device is enhanced by a trained machine learning chatbot / voice bot. This model, executing on the hosting server or user computing device, is able to accurately and efficiently determine relevant responses to user queries / prompts related to and / or otherwise associated with a patient’s EHR. That is, the present disclosure describes improvements in the functioning of the computer itself or “any other technology or technical field” because a hosting server or user computing device, is enhanced with a trained machine learning chatbot / voice bot to accurately detect, evaluate, predict, and generate query responses configured to improve a user / operator’s interactive / diagnostic efforts related to EHRs and associated devices. This improves over the prior art at least because existing systems lack such evaluative and / or predictive functionality, and are generally unable to accurately analyze such query data / prompts on a real-time basis to output predictive and / or otherwise recommended responses designed to improve a user / operator’s overall interactive / diagnostic efforts related to EHRs and associated devices.

[0049] As mentioned, the model(s) may be trained using machine learning and may utilize machine learning during operation. Therefore, in these instances, the techniques of the presentdisclosure may further include improvements in computer functionality or in improvements to other technologies at least because the disclosure describes such models being trained with a plurality of training data (e.g., 10,000s of training data corresponding to EHRs, user queries / prompts, output / response prompts, etc.) to output the relevant prompt responses configured to improve the user / operator’s interactive / diagnostic efforts related to EHRs and associated devices.

[0050] Moreover, the present disclosure includes effecting a transformation or reduction of a particular article to a different state or thing, e.g., transforming or reducing the interaction processing demand of an EHR system (and associated subsystems / components / devices) and clinician occupation times evaluating such EHRs from a non-optimal or error state to an optimal state by automatically retrieving / determining relevant data from EHRs while eliminating untimely / erroneous clinician searching through such EHR systems.

[0051] Still further, the present disclosure includes specific features other than what is well- understood, routine, conventional activity in the field, or adding unconventional steps that demonstrate, in various embodiments, particular useful applications, e.g., receiving a prompt input referencing healthcare data included in an electronic health record (EHR), determining, by executing a trained machine learning (ML) model of an agent object, a user intent corresponding to the prompt input indicating retrieval of the healthcare data from the EHR, transmitting, via the agent object and through an associated application programming interface (API), a control instruction for a digital tool to retrieve the healthcare data from the EHR, and / or generating, by executing the trained ML model, a prompt output that includes the healthcare data, among others.Exemplary Computing Environment

[0052] FIG. 1 depicts an exemplary computing environment 100 in which the techniques disclosed herein may be implemented, according to some aspects. The environment 100 may include computing resources for training and / or operating machine learning models to perform optimized healthcare data management, in some aspects.

[0053] The computing environment 100 may include a client computing device 102, a server computing device 104, an electronic network 106, and a model electronic database 112. The computing environment may further include one or more cloud application programming interfaces (APIs) 114. The components of the computing environment 100 may be communicatively connected to one another via the electronic network 106, in some aspects.

[0054] The client computing device 102 may be implemented as one or more computing devices e.g., one or more servers, one or more laptops, one or more mobile computing devices, one or more tablets, one or more wearable devices, one or more cloud-computing virtual instances, etc.). In some aspects, a plurality of client computing devices may be part of the environment 100 - for example, a first user may access a client computing device 102 that is a laptop, while a second user accesses the client computing device 102 that is a smart phone, while yet a third user accesses a client computing device 102 that is a wearable device. Each of these respective users may access the chatbot-based services via their use of their respective client computing device 102.

[0055] The client computing device 102 may include one or more processors 120, one or more network interface controllers 122, one or more memories 124, an input device 126, an output device 128 and a client API 130. The one or more memories 124 may have stored thereon one or more modules 140 (e.g., one or more sets of instructions).

[0056] In some aspects, the one or more processors 120 may include one or more central processing units, one or more graphics processing units, one or more field-programmable gate arrays, one or more application-specific integrated circuits, one or more tensor processing units, one or more digital signal processors, one or more neural processing units, one or more RISC-V processors, one or more coprocessors, one or more specialized processors / accelerators for artificial intelligence or machine learning-specific applications, one or more microcontrollers, etc.

[0057] The client computing device 102 may include one or more network interface controllers 122, such as Ethernet network interface controllers, wireless network interface controllers, etc. The network interface controllers 122 may include advanced features, in some aspects, such as hardware acceleration, specialized networking protocols, etc.

[0058] The memories 124 of the client computing device 102 may include volatile and / or nonvolatile storage media. For example, the memories 124 may include one or more random access memories, one or more read-only memories, one or more cache memories, one or more hard disk drives, one or more solid-state drives, one or more non-volatile memory express, one or more optical drives, one or more universal serial bus flash drives, one or more external hard drives, one or more network- attached storage devices, one or more cloud storage instances, one or more tape drives, etc.

[0059] As noted, the memories 124 may have stored thereon one or more modules 140, for example, as one or more sets of computer-executable instructions. In some aspects, the modules 140 may include additional storage, such as one or more operating systems (e.g., Microsoft Windows, GNU / Linux, Mac OSX, etc.). The operating systems may be configured torun the modules 140 during operation of the client computing device 102 - for example, the modules 140 may include additional modules and / or services for receiving and processing data from one or more other components of the environment 100 such as the one or more cloud APIs 114 or the server computing device 104. The modules 140 may be implemented using any suitable computer programming language(s) (e.g., Python, JavaScript, C, C++, Rust, C#, Swift, Java, Go, LISP, Ruby, Fortran, etc.).

[0060] The modules 140 may include an API module 144, an input processing module 146, an authentication / security module 148, a context module 150 and an assistance module 152, in some aspects. In some aspects, more or fewer modules 140 may be included. The modules 140 may be configured to communicate with one another (e.g., via inter-process communication, via a bus, via sockets, pipes, message queues, etc.).

[0061] The API module 144 may include one or more sets of computer executable instructions for accessing one or more remote APIs, and / or for enabling one or more other components within the environment 100 to access functionality of the client computing device 102. For example, the API module 144 may enable a user to query the electronic objects corresponding to generative Al agents that are configured and stored in the model database 112. In some aspects, the API module 144 may enable other client applications (i.e., not applications facilitated by the modules 140) to connect to the client computing device 102, for example, to send queries or prompts, and to receive responses from the client computing device 102.

[0062] As noted, the client computing device 102 may enable one or more users to access one or more trained models by providing input prompts that are processed by one or more trained models. The input processing module 146 may perform pre-processing of user prompts prior to being input into one or more models, and / or post-processing of outputs output by one or more models. For example, the input processing module 146 may process data input into one or more input fields, voice inputs or other input methods (e.g., file attachments) depending upon the application. The input processing module 146 may receive inputs directly via the input device 126, in some aspects.

[0063] In some aspects, the input processing module 146 may perform post-processing of input received from one or more trained models. In some aspects, post-processing (and / or preprocessing) may include formatting outputs from the trained ML models in a manner consistent with the intended display that may be interpreted from the user intent. The input processing module 146 may include instructions for handling the display of such post-processed content to users (e.g., via the output device 128). The input processing module 146 may cause one ormore graphical user interfaces to be displayed, for example to enable the user to enter information directly via a text field.

[0064] The authentication / security module 148 may include one or more sets of computerexecutable instructions for implementing access control mechanisms for one or more trained models, ensuring that the model can only be accessed by those who are authorized to do so, and that the access of those users is private and secure. It should be appreciated that the security module 148 may permission users / agents based upon their respective membership and / or otherwise authorization to access information / records stored in a patient EHR. For example, a user that has not been added and / or otherwise authorized to access certain sensitive records / information may not be able to access the contents of that information contained in a particular EHR via the client computing device 102.

[0065] Generally, trained models, especially trained models, require state information in order to meaningfully carry on a dialogue with a user or with another trained model. For example, if a user prompts a trained model with a question such as “What is the weather in Chicago today?” followed by a second prompt “And how about tomorrow?” the model should understand that, in context, the second query relates to the first query, insofar as the user is asking about the weather tomorrow in the same location (Chicago).

[0066] However, language models (e.g., large language models (LLMs)) are generally stateless, meaning that after they process a prompt, they have no internal record or memory of the information that was input, or the information that was generated as part of the language model’s processing. Thus, many systems add statefulness to models using context information. This may be implemented using sliding context windows, wherein a predetermined number of tokens (e.g., 4096 maximum tokens in the case of GPT 3.5, equivalent to about 3000 words) may be “remembered” by the LLM and can be used to enrich multiple sequential prompts input into the LLM (for example, when the LLM is used in a chat mode).

[0067] The context module 150 may include one or more sets of computer-executable instructions for maintaining state of the type found in this example, and other types of state information. The context module 150 may implement sliding window context, in some aspects. In other aspects, the context module 150 may perform other types of state maintaining strategies. For example, the context module 150 may implement a strategy in which information from the immediately preceding prompt is part of the window, regardless of the size of that prior prompt.

[0068] In some aspects, the context module 150 may implement a strategy in which one or more prior prompts are included in each current prompt. This prompt stuffing technique, orprompt concatenation, may be limited by prompt size constraints — once the total size of the prompt exceeds the prompt limit, the model immediately loses state information related to parts of the prompt truncated from the prompt.

[0069] The assistance controller module 152 may include one or more sets of computerexecutable instructions for accessing one or more trained models (e.g., via the model database 112 and / or the EHR database 108). The assistance controller module 152 may include instructions for creating / adding electronic agent objects, for adding assistance controllers to those respective agent objects and / or for configuring one or more links between the electronic agent objects and / or the one or more assistance controllers. The assistance controller module 152 may access the one or more trained models via the network 106 and the server computing device 104, in some aspects.

[0070] The server computing device 104 may include one or more processors 160, one or more network interface controllers 162, one or more memories 164, an input device (not depicted), an output device (not depicted) and a server AP1 166. The one or more memories 164 may have stored thereon one or more modules 170 (e.g., one or more sets of instructions).

[0071] In some aspects, the one or more processors 160 may include one or more central processing units, one or more graphics processing units, one or more field-programmable gate arrays, one or more application-specific integrated circuits, one or more tensor processing units, one or more digital signal processors, one or more neural processing units, one or more RISC-V processors, one or more coprocessors, one or more specialized processors / accelerators for artificial intelligence or machine learning-specific applications, one or more microcontrollers, etc.

[0072] The server computing device 104 may include one or more network interface controllers 162, such as Ethernet network interface controllers, wireless network interface controllers, etc. The network interface controllers 162 may include advanced features, in some aspects, such as hardware acceleration, specialized networking protocols, etc.

[0073] The memories 164 of the server computing device 104 may include volatile and / or nonvolatile storage media. For example, the memories 164 may include one or more random access memories, one or more read-only memories, one or more cache memories, one or more hard disk drives, one or more solid-state drives, one or more non-volatile memory express, one or more optical drives, one or more universal serial bus flash drives, one or more external hard drives, one or more network-attached storage devices, one or more cloud storage instances, one or more tape drives, etc.

[0074] As noted, the memories 164 may have stored thereon one or more modules 170, for example, as one or more sets of computer-executable instructions. In some aspects, the modules 170 may include additional storage, such as one or more operating systems (e.g., Microsoft Windows, GNU / Linux, Mac OSX, etc.). The operating systems may be configured to run the modules 170 during operation of the server computing device 104 - for example, the modules 170 may include additional modules and / or services for receiving and processing data from one or more other components of the environment 100 such as the one or more cloud APIs 114 or the client computing device 102. The modules 170 may be implemented using any suitable computer programming language(s) e.g., Python, JavaScript, C, C++, Rust, C#, Swift, Java, Go, LISP, Ruby, Fortran, etc.).

[0075] In some aspects, the modules 170 may include a data collection module 172, a data preprocessing module 174, a model pretraining module 176, a fine-tuning module 178, a model training module 180, a checkpointing module 182, a hyperparameter tuning module 184, a validation and testing module 186, an auto-prompting module 188, and an assistance chatbot 190. In some aspects, more or fewer modules 170 may be included. The modules 170 may be configured to communicate with one another (e.g., via inter-process communication, via a bus, via sockets, pipes, message queues, etc.). The modules 170 may respond to network requests (e.g., via the AP1 166) or other requests received via the network 106 (e.g., via the client computing device 102 or other components of the environment 100).

[0076] The data collection module 172 may be configured to collect information used to train one or more modules. In general, the information collected may be any suitable information used for training a language model. The data collection module 172 may collect data via web scraping, via API calls / access, via database extract-transform-load (ETL) processes, etc. Sources accessed by the data collection module 172 include social media websites, books, websites, academic publications, web forums / interest sites (e.g., Reddit, Facebook, bulletin boards, etc.), etc. The data collection module 172 may access data sources by active means (e.g., scraping or other retrieval) or may access existing corpuses. The data collection module 172 may include sets of instructions for performing data collection in parallel, in some aspects. The data collection module 172 may store collected data in one or more electronic databases, such as a database accessible via the cloud APIs 1 14 or via a local electronic database (not depicted). The data may be stored in a structured and / or unstructured format. In some aspects, the data collection module 172 may store large data volumes used for training one or more models (i.e., training data). For example, the data collection module 172 may store terabytes, petabytes, exabytes or more of training data.

[0077] In some aspects, the data collection module 172 may retrieve data from the EHR database 108. For example, the data collection module 172 may process the retrieved / received data and sort the data into multiple subsets based on information included within the EHR database 108. For example, the data collection module 172 may receive one or more sets of unstructured text (e.g., transcripts of one or more historical meeting minutes of a clinical review board). The data collection module 172 may chunk the data according to time (e.g., hourly, daily, quarterly, etc.).

[0078] The model preprocessing module 174 may include instructions for pre-processing data collected by the data collection module 172. In particular, the model preprocessing module 174 may perform text extraction and / or cleaning operations on data collected by the data collection module 172. The data pre-processing module 174 may perform preprocessing operations, such as lexical parsing, tokenizing, case conversions and other string splitting / munging. In some aspects, the data collection module 172 may perform data deduplication, filtering, annotation, compliance, version control, validation, quality control, etc. In some aspects, one or more human reviewers may be looped into the process of pre-processing data collected by the data pre-processing module 174. For example, a distributed work queue may be used to transmit batch jobs and receive human-computed responses from one or more human workers. Once pre-processed, the data pre-processing module 174 may store copied and / or modified copies of the training date in an electronic database.

[0079] In some aspects the data pre-processing module 174 may include instructions for parsing the unstructured text received by the data collection module 172 to structure the text. For example, when the text relates to meeting minutes (e.g., text transcripts) of meetings of a clinical review board, the data pre-processing module 174 may generate a time series data structure in which each meeting is represented by one or more timestamps, and at each timestamp, text of one or more speakers is labeled. The data pre-processing module 174 may also label the data according to the identity of one or more speaker and / or one or more topic. For example, the time series data may be labeled according to one or more speakers associated with textual speech, in a language transcript form. The time series may include one or more keywords associated with the transcript. In some aspects, the present techniques may use a separate trained text summarization module to generate keywords used for this purpose. In this way, the data pre-processing module 174 may generate structured data corresponding to unstructured meeting minutes, such that the structured data is enriched with information about the meeting that is suitable for training. This structured data may be processed by downstream processes / modules.

[0080] Generally, the present techniques may train one or more models to perform language generation tasks that include token generation. Both training inputs and model outputs may be tokenized. Herein, tokenization refers to the process by which text used for training is divided into units such as words, subwords or characters. Tokenization may break a single word into multiple subwords (e.g., “LLM” may be tokenized as “L” and “LM”). The present techniques may train one or more models using a set of tokens (e.g., a vocabulary) that includes many (e.g., thousands or more) of tokens. These tokens may be embedded into a vector. This vector of token or “embeddings” may include numerical representations of the individual tokens in the vocabulary in high-dimensional vector space. The modules 170 may access and modify the embeddings during training to learn relationships between tokens. These relationships effectively represent semantic language meaning.

[0081] In some aspects, a specialized database (e.g., a vector store, a graph database, etc.) may be used to store and query the embeddings. Embedding databases may include specialized features, such as efficient retrieval, similarity search and scalability. For example, the server computing device 104 may include a local electronic embedding database (not depicted). In some aspects, a remote embedding database service may be used (e.g., via the cloud APIs 1 14). Such a remote embedding database service may be based on an open source or proprietary model (e.g., Milvus, Pinecone, Redis, Postgres, MongoDB, Facebook Al Similarity Search (FAISS), etc.). The server computing device 104 may include instructions (e.g., in the data collection module 172) for adding training data to one or more specialized databases, and for accessing it to train models.

[0082] The present techniques may include language modeling, wherein one or more deep learning models are trained by processing token sequences using an LLM architecture. For example, in some aspects, a transformer architecture may be used to process a sequence of tokens. Such a transformer model may include a plurality of layers including self-attention and feedforward neural networks. This architecture may enable the model to learn contextual relationships between the tokens, and to predict the next token in a sequence, based upon the preceding tokens. During training, the model is provided with the sequence of tokens and it learns to predict a probability distribution over the next token in the sequence. This training process may include updating one or more model parameters (e.g., weights or biases) using an objective function that minimizes the difference between the predicted distribution and a true next token in the training data.

[0083] Alternatives to the transformer architecture may include recurrent neural networks, long short-term memory networks, gated recurrent networks, convolutional neural networks, recursive neural networks, and other modeling architectures.

[0084] More generally speaking, programmable chatbots, such as the assistance chatbot 190, may provide tailored, conversational-like abilities when interacting with a user. The chatbot may be capable of understanding user requests / responses, providing relevant information, etc. Additionally, the chatbot may generate data from user interactions which the enterprise may use to personalize future support and / or improve the chatbot’s functionality, e.g., when retraining and / or fine-tuning the chatbot.

[0085] A ML chatbot e.g., assistance chatbot 190) may provide advance features as compared to a non-ML chatbot, which may include, and / or derive functionality from, a large language model (LLM). The ML chatbot may be trained on a server, such as the server computing device 104, using large training datasets of text which may provide sophisticated capability for naturallanguage tasks, such as answering questions and / or holding conversations. The ML chatbot may include a general-purpose pretrained LLM which, when provided with a starting set of words (prompt) as an input, may attempt to provide an output (response) of the most likely set of words that follow from the input. In one aspect, the prompt may be provided to, and / or the response received from, the ML chatbot and / or any other ML model, via a user interface of the server. This may include a user interface device operably connected to the server via an I / O module, such as the input device 126 and / or the output device 128. Exemplary user interface devices may include a touchscreen, a keyboard, a mouse, a microphone, a speaker, a display, and / or any other suitable user interface devices.

[0086] Multi-turn (i.e., back-and-forth) conversations may require LLMs to maintain context and coherence across multiple user utterances and / or prompts, which may require the ML chatbot to keep track of an entire conversation history as well as the current state of the conversation. The ML chatbot may rely on various techniques to engage in conversations with users, which may include the use of short-term and long-term memory. Short-term memory may temporarily store information (e.g., in the memory 164 of the server 104) that may be required for immediate use and may keep track of the current state of the conversation and / or to understand the user’s latest input in order to generate an appropriate response. Long-term memory may include persistent storage of information (e.g., on the model database 112 and / or the EHR database 108 connected to the server computing device 104) which may be accessed over an extended period of time. The long-term memory may be used by the ML chatbot to store information about the user (e.g., preferences, chat history, etc.) and may be useful for improving an overalluser experience by enabling the ML chatbot to personalize and / or provide more informed responses.

[0087] The system and methods to generate and / or train an ML chatbot model (e.g., via the training module 180 of the server computing device 104) which may be used by the ML chatbot, may consist of three steps: (1 ) a supervised fine-tuning (SFT) step where a pretrained language model (e.g., an LLM) may be fine-tuned on a relatively small amount of demonstration data curated by human labelers to learn a supervised policy (SFT ML model) which may generate responses / outputs from a selected list of prompts / inputs. The SFT ML model may represent a cursory model for what may be later developed and / or configured as the ML chatbot model; (2) a reward model step where human labelers may rank numerous SFT ML model responses to evaluate the responses which best mimic preferred human responses, thereby generating comparison data. The reward model may be trained on the comparison data; and / or (3) a policy optimization step in which the reward model may further fine-tune and improve the SFT ML model. The outcome of this step may be the ML chatbot model using an optimized policy. In one aspect, step one may take place only once, while steps two and three may be iterated continuously, e.g., more comparison data is collected on the current ML chatbot model, which may be used to optimize / update the reward model and / or further optimize / update the policy.

[0088] More specifically, the modules 170 may include instructions for performing pretraining of a language model (e.g., an LLM), for example, in a pretraining module 176. The pretraining module 176 may include one or more sets of instructions for performing pretraining, which as used herein, generally refers to a process that may span pre-processing of training data via the data pre-processing module 174 and initialization of an as-yet untrained language model. In general, a pre-trained model is one that has no prior training of specific tasks. For example, the model pretraining module 176 may include instructions that initialize one more model weights.In some aspects, model pretraining module 176 may initialize the weights to have random values. The model pretraining module 176 may train one or more models using unsupervised learning, wherein the one or more models process one or more tokens (e.g., preprocessed data output by the data pre-processing module 174) to learn to predict one or more elements (e.g., tokens). The model pretraining module 176 may include one or more optimizing objective functions that the model pretraining module 176 applies to the one or more models, to cause the one or more models to predict one or more most-likely next tokens, based on the likelihood of tokens in the training data. In general, the model pretraining module 176 causes the one or more models to learn linguistic features such as grammar and syntax. The pretraining module176 may include additional steps, including training, data batching, hyperparameter tuning and / or model checkpointing.

[0089] The model pretraining module 176 may include instructions for generating a model that is pretrained for a general purpose, such as general text processing / understanding. This model may be known as a “base model” in some aspects. The base model may be further trained by downstream training process(es), for example, those training processes described with respect to the fine-tuning module 178. The model pretraining module 176 generally trains foundational models that have general understanding of language and / or knowledge. Pretraining may be a distinct stage of model training in which training data of a general and diverse nature (i.e., not specific to any particular task or subset of knowledge) is used to train the one or more models.In some aspects, a single model may be trained and copied. Copies of this model may serve as respective base models for a plurality of fine-tuned models.

[0090] The modules 170 may include a fine-tuning module 178. The fine-tuning module 178 may include instructions that train the one or models further to perform specific tasks. For example, the fine-tuning module 178 may train each of a plurality of agents to generate one or more outputs that are based on each respective agents’ personality, characteristics, and / or sets of tasks and / or tools with which the agent is expected to interact / perform. Specifically, the fine- tuning module 178 may include instructions that train one or more models to generate respective language outputs (e.g., text generation), summarization, question answering or translation activities based on the characteristics of each respective agent.

[0091] Continuing the example, the fine-tuning module 178 may include sets of instructions for retrieving one or more structured data sets, such as time series generated by the data preprocessing module 174. These structured data sets may be sorted by time and / or speaker to train one or more machine learning models (e.g., one or more language models) that may be used within the environment 100 to provide accurate, timely healthcare data management. For example, the fine-tuning module 178 may include instructions for configuring an objective function for performing a specific task, such as generating text that is similar to text found within the corpus of training data associated with a particular individual by role.

[0092] In some aspects, the fine-tuning module 178 may include user-selectable parameters that affect the fine-tuning of the one or more models. For example, a “caution” bias parameter may be included that represents medical conservativeness. This bias parameter may be adjusted to affect the cautiousness with which the resulting trained model (i.e., agent) approaches medical decision-making. Additional models may be trained, for additional personas / tasks, as discussed below.

[0093] In some aspects, to manage complexity of fine-tuning and other machine learning operations of the server computing device 104, one or more open source frameworks may be used. Example frameworks include TensorFlow, Keras, MXNet, Caffe, SciKit learn, PyTorch. Specifically for training and operating language models, frameworks such as OpenLLM and LangChain may be used, in some aspects. The fine-tuning module 178 may use an algorithm such as stochastic gradient descent or another optimization technique to adjust weights of the pretrained model.

[0094] Fine-tuning may be an optional operation, in some aspects. In some aspects, training may be performed by the training module 180 after pretraining by the model pretraining module 176. In some aspects, the model training module 180 may perform task-specific training like the fine-tuning module 178, on a smaller scale or with a more tailored objective.

[0095] The training module 180 may include one or more submodules, including the checkpointing module 182, the hyperparameter tuning module 184, the validation and testing module 186 and the auto-prompting module 188. The checkpointing module 182 may perform checkpointing, which is saving of a model’s parameters. The checkpointing module 182 may store checkpoints during training and at the conclusion of training, for example, in the model electronic database 112. In this way, the model may be run (e.g., for testing and validation) at multiple stages and its training parameters loaded, and also retrained from a checkpoint. In this way, the model can be run and trained forward without being re-trained from the beginning, which may save significant time (e.g., days of computation). The hyperparameter tuning module 184 may include hyperparameters such as batch size, model size, learning rate, etc. These hyperparameters may be adjusted to influence model training. The hyperparameter tuning module 184 may include instructions for tuning hyperparameters by successive evaluation. The validation and testing module 186 may include sets of instructions for validating and testing one or more machine learning models, including those generated by the model pretraining module 176, the fine-tuning module 178 and the model training module 180. The auto-prompting module 188 may include sets of instructions for performing auto-prompting of one or more models. Specifically, the auto-prompting module 188 may enrich a prompt with additional information. The auto-prompting module 188 may include additional information in a prompt, so that the model receiving the prompt has additional context or directions that it can use. This may allow the auto-prompting module 188 to fine-tune a base model using one-shot of few-shot learning, in some aspects. The auto-prompting module 188 may also be used to focus the output of the one or more models.

[0096] In some aspects, the training module 180 may include instructions for training one or more additional machine learning models, such as supervised or unsupervised machine learning models. For example, as discussed below, in some aspects, the present techniques may include processing imaging data (e.g., radiology scans) retrieved from the EHR database 108. In that case, the patient’s imaging data may be processed by a model (e.g., a convolutional neural network) and the results processed further (e.g., by a language model) and / or provided to the client computing device 102. The training module 180 may train such a supervised model separately from training one or more language models. Further, the server computing device 104 may select one or more trained models at runtime based on data about a specific patient, based upon data contained in a prompt or based on other conditions that may be preprogrammed into the server computing device 104.

[0097] In some aspects, the training module 180 may train multi-modal models. For example, the training module 180 may train a plurality of models each capable of drawing from multimodal data types such as written text, imaging data, laboratory data, real-time monitoring data, pathology images, etc. In some cases, the training module 180 may train a single model capable of processing the multimodal data types. In some aspects, a trained multi-modal model may be used in conjunction with another model (e.g., a large language model) to provide nontext data interactions with users. Non-text data may be analyzed and integrated into the chat functions discussed herein.

[0098] The assistance chatbot 190 may operate one or more trained models. Specifically, the assistance chatbot 190 may initialize one or more trained models, load parameters into the model(s), and provide the model(s) with inference data (e.g., prompt inputs). In some aspects, the assistance chatbot 190 may deploy one or more trained model (e.g., a pretrained model, a fine tuned model and / or a trained model) onto a cloud computing device (e.g., via the AP1 166). The assistance chatbot 190 may receive one or more inputs, for example from the client computing device 102, and provide those inputs (e.g., one or more prompts) to the trained model. In some aspects, the API 166 may include elements for receiving requests to the model, and for generating outputs based on model outputs. For example, the API 166 may include a RESTful API that receives a GET or POST request including a prompt parameter. The assistance chatbot 190 may receive the request from the API 166 and pass the prompt parameter into the trained model and receive a corresponding input. For example, the prompt parameter may be “What is patient X’s most recent blood pressure measurements?”. The prompt output may be “Patient X’s most recent blood pressure measurements were 122 / 84 mm Hg.” In certain aspects, outputs may also include numeric confidence levels.

[0099] Broadly, the assistance chatbot 190 may serve as a conversational agent. A user may provide an input prompt to the assistance chatbot 190, and the assistance chatbot 190 may determine a user intent based on the input prompt. For example, a user intent may indicate that the user intends to retrieve the data from patient X’s EHR, and more specifically, that the user intends to retrieve the most recent heart rate data from patient X’s EHR. The assistance chatbot 190 may then determine and / or otherwise designate a tool configured to retrieve pertinent information (e.g., most recent heart rate) related to the input prompt.

[0100] When the tool retrieves the relevant data, the assistance chatbot 190 may format the data into a response / output prompt, and the user may engage in an LLM-based Q&A session with the assistance chatbot 190. For example, such a Q&A session may include the assistance chatbot 190 visualizing the data graphically for the user or collating the data for comprehensive insights. In this manner, the assistance chatbot 190 may succinctly summarize intricate notes, thereby making user healthcare data consumption more efficient than conventional techniques requiring users to filter through extensively complex EHRs. For instance, complex tasks such as outside record summarization or retrieval augmented generation, may be streamlined through the distilled LLMs of the assistance chatbot 190.

[0101] In particular, the assistance chatbot 190 integration layer may be equipped with LLM agents configured to decipher user prompts and uses a series of tools / plugins to fulfil the users’ needs, as determined from the user’s input prompt. These agents may determine and / or otherwise analyze the intent behind an input prompt and decide what type of database (e.g., EHR database 108) to search or what kind of application to call to fulfill the user’s intent. This intent analysis ensures accurate data retrieval and suggests further inquiries, thereby enhancing the depth of data interaction. Each piece of information generated by the assistance chatbot 190 may also be backed by links and / or references that enable users to delve deeper or verify the data's accuracy.

[0102] As part of this data retrieval / analysis, the assistance chatbot 190 is configured to connect with both external and internally stored digital tools via APIs (e.g., APIs 166, 114). For example a specific specialty tool may be configured to predict hospital-acquired infections, and the assistance chatbot 190 may seamlessly link the user's input prompt to this tool to receive relevant feedback. This seamless integration with Al tools through suitable APIs 166, 114 enables the assistance chatbot 190 to be a continuously evolving system, such that insights and data results retrieved and / or otherwise generated by the Al tools may be used to re-train, update, and / or otherwise influence the outputs of the assistance chatbot 190 agents.

[0103] Further, the assistance chatbot 190 may easily and automatically identify and connect users with other relevant users (e.g., via page, email, communication platform chat, text message, phone call, etc.) without users needing to toggle between multiple platforms. The assistance chatbot 190 may be integrated with any existing services, such as a paging system, communications platform, and / or any other communications network to enable the assistance chatbot 190 to offer and automatically connect a user with another user in response to an input prompt. Additionally, the assistance chatbot 190 may provide a "checkout" feature, which allows users to collate retrieved items into a formatted document (e.g., PDF), thereby facilitating the creation of teaching files, document submissions, and / or aiding in curbside consultations. This functionality may also enable the assistance chatbot 190 to integrate radiology image links and / or other relevant image data into output / response prompts, which enables users to view imaging exams with a single click.

[0104] As part of these functionalities, the assistance chatbot 190 may include any suitable configuration of agents and / or corresponding ML models. For example, the assistance chatbot 190 may include an existing LLM that is fine-tuned for two tasks: pre-authorization and outside record summarization. These pre-existing models may be distilled down for these specific tasks and integrated as part of an agent and / or otherwise incorporated into the assistance chatbot190. However, it should be understood that the assistance chatbot 190 may include any suitable number of LLMs, agents, and that such LLMs and / or agents may be configured to perform (e.g., fine-tuned) any suitable number and / or types of tasks.

[0105] In any event, in certain aspects, the assistance chatbot 190 and / or other modules described herein may utilize multi-modal modeling. The data preprocessing module may, for example, process and understand image data, audio data, video data, etc. The server computing device 104 may interpret and respond to queries that involve understanding content from these different modalities. For example, the server computing device 104 may include an image processing module (not depicted) including instructions for performing image analysis on images provided by users, or images retrieved from patient EHR data. In some aspects, the server computing device 104 may generate outputs in modalities other than text. For example, the server computing device 104 may generate an audio response, an image, etc. Still further, the agent objects may process both text data and other data (e.g., an image uploaded by a user, an image from patient EHR, etc.) during a data stream with a user. Combining multimodal data may enable the present agents to perform more comprehensive analysis of patient conditions, based on information processed in multiple different modes simultaneously.

[0106] The assistance chatbot 190 may include a set of computer-executable instructions that when executed by one or more processors (e.g., the processors 160) cause a computer (e.g., the server computing device 104) to perform retrieval-augmented generation. Specifically, the assistance chatbot 190 may perform retrieval-augmented generation based upon inputs or queries received from the user. This allows the assistance chatbot 190 to tailor responses of a model based on the specific input and context, such as the medical issue under discussion. For example, one or more models may be pre-trained, fine-tuned and / or trained as discussed above. During that training, the model may learn to generate tokens based on general language understanding as well as application-specific training. Such a model at that point may be static, insofar as it cannot access further information when presented with an input query.

[0107] When the model is used at runtime, however, such as when deployed in the environment 100, the assistance chatbot 190 may perform retrieval operations, such as searching or selecting information from a document, a database, or another source. The assistance chatbot 190 may include instructions for processing user input and for performing a keyword search, a regular expression search, a similarity search, etc. based upon that user input. The assistance chatbot 190 may input the results of that search, along with the user input, into the trained model. Thus, the trained model may process this additional retrieved information to augment, or contextualize, the generation of tokens that represent responses to the user’s query. In sum, retrieval augmented generation applied in this manner allows the model to dynamically generate outputs that are more relevant to the user’s input query at runtime. Information that may be retrieved may include data corresponding to a patient (e.g., patient demographic information, medical history, clinical notes, diagnoses, medications, allergies, immunizations, laboratory results, oncology information, radiation and imaging information, vitals, etc.) and additional training information, such as medical journals, notes or speech transcripts from symposia or other meetings / conferences, etc.

[0108] The assistance chatbot 190 may include generating a patient summary or abstract. Essential information may be generated at this stage. For example, a patient’s KRAS mutation status may be essential for determining how aggressive to be in treating a patient. Thus, the assistance chatbot 190 may access an electronic health records summary module (not depicted) that generates summary statistics for a patient. This patient abstract may be provided to the agents by the assistance chatbot 190, to provide additional context. Additional example summary information may include chemotherapy types and duration that the patient has had, if any. In this case, an agent may be asked “which radiation therapy is best for the patient?” andin response, may generate “a suggested dosage is 50 gy in 6 daily fractions.” The output of the agent may include links to medical literature supporting this generated response.

[0109] The present techniques may trigger retrieval augmented generation by processing a prompt, in some aspects. For example, a prompt may be processed by the input processing module 146 of the client computing device 102, prior to processing the prompt by the one or more generative models. The input processing module 146 may trigger retrieval augmented generation based on the presence of certain inputs, such as patient information, or a request for specific information, in the form of keywords. The input processing module 146 may perform entity recognition or other natural language processing functions to determine whether the prompt should be processed using retrieval augmented generation prior to being provided to the trained model.

[0110] As discussed above, prompts may be received via the input processing module 146 of the client computing device 102 and transmitted to the server computing device 104 via the electronic network 106. In some aspects, the output of the model may be modulated prior to being transmitted, output or otherwise displayed to a user.

[0111] The client computing device 102 and the server computing device 104 may communicate with one another via the network 106. In some aspects, the client computing device 102 and / or the server computing device 104 may offload some or all of their respective functionality to the one or more cloud APIs 114. In aspects, the one or more cloud APIs 114 may include one or more public clouds, one or more private clouds and / or one or more hybrid clouds. The one or more cloud APIs 1 14 may include one or resources provided under one or more service models, such as Infrastructure as a Service (laaS), Platform as a Service (PaaS), Software as a Service (SaaS), and Function as a Service (FaaS). For example, the one or more cloud APIs 1 14 may include one or more cloud computing resources, such as computing instances, electronic databases, operating systems, email resources, etc. The one or more cloud APIs 1 14 may include distributed computing resources that enable, for example, the model pretraining module 176 and / or other of the modules 170 to distribute parallel model training jobs across many processors.

[0112] In some aspects, the one or more cloud APIs 114 may include one or more language operation APIs, such as OpenAI, Bing, Claude.ai, etc. In other aspects, the one or more cloud APIs 114 may include an API configured to operate one or more open-source models, such as Llama 2.

[0113] The electronic network 106 may be a collection of interconnected devices, and may include one or more local area networks, wide area networks, subnets, and / or the Internet. Thenetwork 106 may include one or more networking devices such as routers, switches, etc. Each device within the network 106 may be assigned a unique identifier, such as an IP address, to facilitate communication. The network 106 may include wired (e.g., Ethernet cables) and wireless (e.g., Wi-Fi) connections. The network 106 may include a topology such as a star topology (devices connected to a central hub), a bus topology (devices connected along a single cable), a ring topology (devices connected in a circular fashion), and / or a mesh topology (devices connected to multiple other devices). The electronic network 106 may facilitate communication via one or more networking protocols, such as packet protocols (e.g., Internet Protocol (IP)) and / or application-layer protocols (e.g., HTTP, SMTP, SSH, etc.). The network 106 may perform routing and / or switching operations using routers and switches. The network 106 may include one or more firewalls, file servers and / or storage devices. The network 106 may include one or more subnetworks such as a virtual LAN (VLAN).

[0114] The environment 100 may include one or more electronic databases, such as a relational database that uses structured query language (SQL) and / or a NoSQL database or other schema-less database suited for the storage of unstructured or semi-structured data.

[0115] For example, the EHR database 108 may be an electronic database that stores medical records of a patient. This data may describe past treatment of patients, past measurements associated with the patient (e.g., blood pressure, heart rate, lab results, radiology images, etc.) and / or other information related to patient care. The EHR database 108 may also include correspondence between physicians (e.g., a plurality of sub-specialists). The present techniques may include populating the EHR database 108 with data relating to a specific patient during a discussion with a healthcare provider during a discussion with the assistance chatbot 190 regarding the specific patient. This patient-specific information may include imaging data related to the patient, treatment data, biometric data, and / or other suitable data.

[0116] The present techniques may store training data, training parameters and / or trained models in an electronic database such as the model database 112. Specifically, one or more trained machine learning models may be serialized and stored in a database (e.g., as a binary, a JSON object, etc.). Such a model can later be retrieved, deserialized and loaded into memory and then used for predictive purposes. The one or more trained models and their respective training parameters (e.g., weights) may also be stored as blob objects. Cloud computing APIs may also be used to stored trained models, via the cloud APIs 1 14. Examples of these services include AWS SageMaker, Google Al Platform and Azure Machine Learning.

[0117] In operation, a user may access a prompt graphical user interface via the client computing device 102. The prompt graphical user interface may be configured, generated,and / or displayed by the input processing module 146 via the output device 128. Specifically, the input processing module 146 may configure the graphical user interface to accept prompts and display corresponding prompt outputs generated by one or more models processing the accepted outputs. The input processing module 146 may also be configured to transmit the prompts input by the user via the network electronic network 106 to the AP1 166 of the server computing device 104. The AP1 166 may process the user inputs via one or more trained models, or agents.

[0118] At the time the user accesses the prompt graphical user interface, one or more models may already be trained, including pretraining and fine-tuning. These trained models may be selectively loaded into the one or more agent objects based on configuration parameters, and / or based upon the content of the user’s input prompts. Further, in some aspects, the user may engage in a question-answer session with the client computing device 102.Exemplary Agent-Controller Configuration

[0119] FIG. 2 depicts an exemplary agent-controller configuration 200, according to one or more aspects. The exemplary agent-controller configuration 200 may include an assistance controller 202 that receives one or more prompts 204 and provides the prompts 204 to one or more agent objects 206. The assistance controller 202 may correspond to the assistance controller module 152 of FIG. 1 , in some aspects. The exemplary agent-controller configuration 200 may comprise the one or more agents 206 and additional information, as discussed above.

[0120] Generally, the one or more agents 206 correspond, respectively, to one or more trained models. For example, each one or more agents 206 may include a trained model and additional metadata, such as a profile name, training date, checkpoint information, etc. The assistance controller 202 may receive the respective responses of the one or more agents 206 and generate one or more controller responses 208 based on the agent responses. In some aspects, the one or more controller responses 208 may include each agent’s response (e.g., a batched response). In some aspects, the one or more controller responses 208 may include one or more individual agent responses.

[0121] The agent-controller configuration 200 may include any number of agent objects. For example, the agent-controller configuration 200 depicts an agent object 206-A, which is the first agent object, an agent object 206-B, which is the second agent object, an agent object 206-C, which is the third agent object, an agent object 206-D, which is the fourth agent object and an agent object 206-E, which is the nth agent object, wherein n is a positive natural number.

[0122] Moreover, as illustrated in FIG. 2, agent objects 206-A and 206-B may access, call, and / or otherwise utilize one or more tools 207a, 207b, 207c, which may be configured to perform various actions related to a patient’s EHR. For example, the first tool 207a may be configured to keyword search through a patient’s entire EHR file history to identify and / or retrieve relevant medical information / data in response to the input prompt 204 and / or for inclusion in the response 208. As another example, the second tool 207b may be a calculator configured to intake data from the first tool 207a, data retrieved by the second tool 207b, and / or data from any other suitable source / location to calculate a mortality percentage for a patient in the event that the patient acquires a particular infection or disease. Regardless, these tools 207a-c (also referenced herein as “plugins”) may facilitate the agent objects 206-A-N acquiring the data necessary to format and / or otherwise generate all and / or portions of the response 208 output by the assistance controller.

[0123] It should be appreciated that the agent objects 206-A-N may be connected to and / or otherwise have access to any suitable number of tools 207a-c, and these tools may perform any suitable action(s) or combinations thereof.Exemplary Computer-Implemented Methods

[0124] FIG. 3 depicts an exemplary computer-implemented method 300 for optimizing healthcare data management using generative Al agents, in accordance with various embodiments herein.

[0125] At block 302, the method 300 includes receiving, at the one or more processors, a prompt input referencing healthcare data included in an electronic health record (EHR). The method 300 further include determining, by one or more processors executing a trained machine learning (ML) model of an agent object, a user intent corresponding to the prompt input indicating retrieval of the healthcare data from the EHR (block 304). The method 300 further includes transmitting, by one or more processors via an agent object and through an associated application programming interface (API), a control instruction for a digital tool to retrieve the healthcare data from the EHR (block 306). The method 300 further includes generating, by one or more processors executing a trained ML model, a prompt output that includes the healthcare data (block 308). The method 300 further includes causing, by one or more processors, the prompt output to be displayed in an output device (block 310).

[0126] In certain embodiments, the method 300 may further include: generating, by the one or more processors, a preprocessed prompt input corresponding to the prompt input; and generating, by the one or more processors, post-processed healthcare data that is formatted based on the user intent, and wherein the prompt output includes the post-processed healthcaredata. Further in this variation, the post-processed healthcare data includes at least one of: (i) a graphical representation of the healthcare data, (ii) a collated document that includes the healthcare data, (iii) an image included as part of the healthcare data, (iv) a selectable link that directs users to the healthcare data in a source location, or (v) a chat option configured to connect a user with a second user that is associated with the healthcare data and / or indicated in the prompt input or the user intent.

[0127] In some embodiments, the trained ML model of the agent object is (a) a large language model (LLM) that is configured to receive input healthcare data and generate a prompt output based on the input healthcare data or (b) a multi-modal machine learning model configured to output multi-modal data associated with the input healthcare data.

[0128] In certain embodiments, the digital tool is one of a plurality of digital tools, and the method 300 may further include: analyzing, by the one or more processors, each of the plurality of digital tools to determine whether any of the plurality of digital tools is configured to retrieve the healthcare data from the EHR; identifying, by the one or more processors, that the digital tool is configured to retrieve the healthcare data from the EHR; and generating, by the one or more processors, the control instruction for the digital tool to retrieve the healthcare data.

[0129] In some embodiments, the method 300 may further include: receiving, at the one or more processors, a subsequent prompt input related to the prompt output; determining, by the one or more processors executing the trained ML model of the agent object, a subsequent user intent corresponding to the subsequent prompt input; transmitting, by the one or more processors via the agent object and through a second API, a subsequent control instruction for a second digital tool to perform an action related to the subsequent user intent, wherein the second digital tool is different from the digital tool; generating, by the one or more processors executing the trained ML model, a subsequent prompt output based on the action performed by the second digital tool; and causing, by the one or more processors, the subsequent prompt output to be displayed in the output device.

[0130] In certain embodiments, a first user submits the prompt input, the output device is a first device of the first user, the prompt output includes a selectable option to connect with a second device of a second user in real-time, and the method 300 may further include: receiving, at the one or more processors, a subsequent prompt input indicating a selection of the selectable option to connect with the second device of the second user; transmitting, by the one or more processors via the agent object and through a second API, a subsequent control instruction for a second digital tool to connect the first device of the first user with the second device of the second user across a live data stream, wherein the second digital tool is different from thedigital tool; and causing, by the one or more processors, the prompt output to continue to be displayed in the output device during the live data stream between the first device of the first user and the second device of the second user.

[0131] In some embodiments, the prompt input is a verbal input from a user, and the method 300 may further include: transcribing, by the one or more processors executing the trained ML model of the agent object, the verbal input into a textual transcription; and determining, by the one or more processors executing the trained ML model of the agent object, the user intent based on the textual transcription.Example Graphical User Interfaces (GUIs)

[0132] FIG. 4 depicts a first example graphical user interface (GUI) 400 associated with optimizing healthcare data management using generative Al agents, in accordance with various embodiments herein. Generally, the first GUI 400 illustrates sets of patient data that the systems described herein may extract, retrieve, and / or otherwise access from an EHR. A chatbot (e.g., assistance chatbot 190), as described herein, may be embedded within the EHR, allowing the chatbot to directly access and interact with the patient data stored within the EHR. As a result, when a practitioner uses the chatbot, the chatbot may immediately retrieve and process the necessary information from the patient's records. The chatbot may employ a retrieval-augmented generation (RAG) process, meaning the chatbot may pull specific data from the EHR, analyze it, and then generate summaries of information included therein, lists of medications, treatment recommendations, and / or other suitable information or combinations thereof based on the information it has accessed.

[0133] This process not only saves time for the practitioners by providing quick access to a patient's medical history, medication list, and / or other relevant information, but may also ensure that the information is presented in a concise and easily understandable format. In certain embodiments, the chatbot may utilize de-identified patient data to ensure privacy and adherence to regulatory standards, such that identifiable information that may be used to trace back to the patient may be removed and / or obscured before the chatbot processes the data. The chatbot's ability to work with de-identified data ensures that this functionality may not compromise patient privacy.

[0134] Furthermore, integrating the chatbot with the EHR facilitates the display of patient data in the first GUI 400 to allow practitioners to visually interact with the information, making it easier to navigate through the patient's medical history. Practitioners many interact (e.g., click, tap, swipe, voice commands, gesture) with specific elements within the first GUI 400, such as an appointment listing for the practitioner, associated with a patient (e.g., Patient X), and thechatbot may immediately access / retrieve detailed information from the EHR and / or automatically transition from the first GUI 400 to a second GUI that includes the information from the EHR.

[0135] For example, FIG. 5 depicts a second example GUI 500 associated with optimizing healthcare data management using generative Al agents, in accordance with various embodiments herein. In the second GUI 500 of FIG. 5, the practitioner may have selected an appointment listed in the first GUI 400 associated with “Patient X”. In response, the systems described herein may automatically retrieve and / or cause previously retrieved medical history information to render in the second GUI 500 associated with Patient X.

[0136] In certain embodiments, the practitioner selecting the appointment associated with Patient X may automatically invoke a macro configured to prompt the chatbots described herein to determine, retrieve, access, and / or otherwise present relevant medical information from the EHR associated with Patient X for display to the practitioner in the second GUI 500. These prompts may be designed to provide the most relevant portions of the patient's medical history in preparation for the practitioner's upcoming appointment with the patient. This EHR information retrieval process may be facilitated through RAG, as described herein. For example, the chatbot may retrieves search results from the EHR based on the initial prompts provided by the macro "startup", which may be configured to elicit the most pertinent information from the patient's medical history that would be relevant for the practitioner's review prior to or during a patient appointment. The systems described herein may search the EHR for information that matches these prompts, which could include specific conditions, previous diagnoses, medication history, and / or other relevant medical data.

[0137] With the relevant information, the chatbot may then proceed to sort and filter these search results to ensure that only the most relevant and accurate information is selected for generating answers. The sorting and filtering algorithms may prioritize data based on its relevance to the prompts, the recency of the information, its significance to the patient's current health status, and / or other relevant criteria that may be included or indicated in one or more of the prompts. The chatbot may also use this relevant information to generate answers, such as a summary of the patient’s medical history. This summary and / or other answers may be constructed using natural language processing (NLP) techniques (e.g., as part of the chatbots) to ensure that they are easily understandable and directly relevant to the practitioner's needs. The generated answers may thereby provide a concise and accurate summary of the patient's medical history, highlighting key aspects that are pertinent to the upcoming appointment.

[0138] As illustrated in the second GUI 500, the prompts included as part of the “startup” macro may include “summarize the patient medical history” and “list all the medications”. As a result, the chatbots described herein may retrieve / analyze data from the EHR to generate the patient’s medical history summary and the medications prescribed to the patient in the past. Moreover, each prompt may include instructions constraining the chatbots to provide answers that are supported by the EHR. The chatbots may also include citations to the sources within the EHR, enabling practitioners to see the sources of information that the chatbot has used to generate its answers. By constraining the chatbots in this manner and providing visibility into the data sources, the system enables practitioners to verify the accuracy of the information for themselves and reduces the likelihood of the chatbot presenting incorrect or irrelevant information (e.g., hallucinations), thereby enhancing the trustworthiness and reliability of the chatbot's responses.

[0139] Practitioners may input additional prompts to the chatbots described herein, e.g., after receiving the summary and / or list of medications to receive more specific information about the patient the practitioner may want prior to their appointment. For example, FIG. 6 depicts a third example GUI 600 associated with optimizing healthcare data management using generative Al agents, in accordance with various embodiments herein. As illustrated in the third GUI 600, the practitioner may be in the process of inputting a new query into the chat window for the chatbot, and the systems described herein may automatically provide the practitioner with a list of suggested questions and / or macros that may quickly provide the practitioner with the desired information.

[0140] These suggestions / recommendations may be tailored based on several factors, such as the content of the answers previously provided to the practitioner, the practitioner's specialization or group affiliation, and / or the historical questions asked by the practitioner. This approach significantly enhances the interaction between the practitioner and the system, making it more dynamic, user-friendly, and efficient. For example, the third GUI 600 includes a list of suggested questions, such as “Show the active list of medication including dosages and frequency of administration,” “Is there a possibility of drug interactions for the active medications?,” and “What are the side effects associated with the medications the patient is currently taking?”, among others.

[0141] The systems described herein may analyze the practitioner's behavior, including the types of questions asked and the information sought, as well as the practitioner 's specialization. The systems described herein may thereby determine and adapt to the specific needs and preferences of individual practitioners over time. As a particular practitioner interacts more withthe systems described herein, the system tailors its functionalities to match the practitioner's unique inquiry patterns and information needs. For instance, if a practitioner frequently inquires about cardiovascular-related patient histories, the system may learn to prioritize and suggest questions related to cardiology. This personalization ensures that the system becomes more effective and efficient for each practitioner, enhancing the overall user experience.

[0142] In addition to individual learning, the systems described herein may also incorporate learning from groups of specialists. For example, the systems described herein may aggregate common questions and practices from specific specialist groups, such as cardiologists or endocrinologists, and may use this information to provide more relevant question suggestions to practitioners within those specialties. This group learning functionality ensures that the systems described herein may offer suggestions that are personalized and informed by the collective expertise and inquiry patterns of specialist groups.

[0143] These concepts further extend to macros, where the systems described herein may tailor macro recommendations to individual practitioners and / or groups of practitioners based on historical interaction / preferences data. For example, the third GUI 600 includes a list of macros, such as "Default,” “Liver Macro,” and “Pre-auth CT”, among others. In certain embodiments, the systems described herein may recommend and / or set standard questions or macros for any particular group / department, ensuring that all team members have access to a consistent set of queries that are most relevant to their collective needs. This feature ensures that practitioners can quickly and efficiently access the most pertinent information, reducing variability in the information retrieval process and promoting a more standardized approach to patient care across the group / department.

[0144] As mentioned, each output of the chatbots described herein may be constrained to the EHR and / or other information sources described herein, and each output from the chatbots described herein may include a citation to the information sources. For example, FIG. 7 depicts a fourth example GUI 700 associated with optimizing healthcare data management using generative Al agents, in accordance with various embodiments herein. The fourth GUI 700 includes a statement associated with a patient’s medical history that states “The patient has a history of exertional chest pain, diabetes, and asthma. She also has a family history of coronary artery disease.” After both of these sentences, the fourth GUI 700 may include a citation indication, with which a practitioner may interact to receive additional information indicating the source (e.g., from the patient’s entry in the EHR) corresponding to the information provided in the answer / output.

[0145] As illustrated in the fourth GUI 700, the practitioner may interact with the citation indication following the second sentence of the chatbot output, which may open the citation text box illustrated below the chatbot output. The citation text box may include a date / time associated with the entry of the cited information in the EHR, a type indication corresponding to the type of data from the EHR, a snippet quoting directly from the EHR data entry, and an interactive report link. The interactive report link may cause the systems described herein to transition from the fourth GUI 700 to a fifth GUI 800, as illustrated in FIG. 8.

[0146] The fifth GUI 800 includes a full representation / reproduction of the document / file from which the chatbots / systems described herein may retrieve information to use within the output / answer. In certain embodiments, the chatbots described herein may examine both structured and unstructured data within the EHR. Structured data may generally refer to information that is organized in a defined manner, such as database entries, while unstructured data may include free-form text or scanned documents that do not follow a specific format. The chatbots described herein may employ algorithms to analyze and interpret this data, extracting relevant information to answer queries posed by practitioners. In doing so, the chatbots described herein may cite specific portions of the data to support its generated outputs, providing practitioners with direct references to the information sources within the EHR.

[0147] Regardless, when the chatbots / systems described herein analyze data / f Iles incorporated / stored as part of the EHR, the chatbots may associate these documents / files with their outputs to be able to provide the citation and the full text review after the practitioner interacts with the interactive report link. This fifth GUI 800 further includes a "generate summary" option, which may allow practitioners to obtain a concise summary of the document based on the information they are interested in, as indicated in their initial query and / or the output of the chatbot. In response to the practitioner interacting with the generate summary option / button, the systems described herein may input prompts into one or more of the chatbots described herein indicating the desired summary of the corresponding document. The prompts input to the chatbots described herein may include constraints and / or other indications that cause the chatbot to summarize the document based on, e.g., the context of the practitioner's initial query, ensuring that the summary is tailored to the specific information needs of the practitioner.

[0148] In certain embodiments, the systems described herein may utilize one or more Al agents to facilitate the various query response functionalities described herein. For example, FIG. 9 depicts a sixth example GUI 900 associated with optimizing healthcare data management using generative Al agents, in accordance with various embodiments herein. Thissixth GUI 900 illustrates a practitioner and / or other user inputting a query to invoke a particular Al agent (e.g., “AskMayoExpert”) that is configured to query particular information sources, perform certain functions, and / or otherwise respond to the user’s query in a manner that constrains the outputs.

[0149] As illustrated in FIG. 9, the sixth GUI 900 includes an output indicating that Patient X may have a recent HbA1c level of 8.5% that was documented on October 3, 2024. In response, the user may input a query specifically to invoke the AskMayoExpert Al agent that states “what is the recommended first line of treatment?” Once invoked, the Al agent may utilize one or more knowledge sources to provide accurate and relevant information. These sources may include internal databases that contain medical information, guidelines, local knowledge, and / or patient data, as well as external knowledge bases that may offer additional insights or corroborate the information found internally. Local knowledge may refer to hospital-specific guidelines, protocols, and / or practices that may influence the choice of treatment for a patient. By incorporating this local context, the systems described herein may ensure that the recommendations are not only based on general medical knowledge but also aligned with the specific practices and standards of the healthcare facility in question.

[0150] For instance, when retrieving information on the first line of treatment for a patient with an HbA1c level of 8.5%, the "AskMayoExpert" Al agent may analyze the query in the context of the latest diabetes management guidelines, taking into account any hospital-specific protocols that might affect treatment selection. The Al agent may then provide the practitioner with a context-aware answer that includes the recommended first line of treatment, considering both the general guidelines and any relevant local knowledge.

[0151] To illustrate such a context-aware answer, FIG. 10 depicts a seventh example GUI 1000 associated with optimizing healthcare data management using generative Al agents, in accordance with various embodiments herein. This seventh GUI 1000 includes an example output of the “AskMayoExpert” Al agent that received the practitioner input illustrated in the sixth GUI 900. This output includes a summary of the condition(s) faced by the patient, as well as the first-line treatment for the patient’s condition(s) (e.g., HbA1c of 8.5%), such as lifestyle modifications, glycemic control, blood pressure management. As part of the output, the Al agent may further include alternative treatments, potential risks, side effects, and contraindications, further investigations, and / or a conclusion section that may summarize the recommended treatments for the patient’s condition(s).

[0152] Additionally, the Al agent may provide citations to one or more of the outputs provided as part of the overall response. For example, as illustrated in the seventh GU1 1000, the Alagent may provide citation indications as part of the conclusion to indicate one or more sources to support the information provided in the conclusion, such that the practitioner evaluating the conclusion may also evaluate the validity of the information included therein. These citations may indicate both the source of the information (e.g., “CBC (Epic:LAB294) Lipid panel”) as well as the location of the source (e.g., “Intranet”), indicating to the practitioner what the information source is and where the Al agent retrieved the source from.

[0153] Of course, it should be appreciated that the Al agents described herein may be automatically invoked by the chatbots and / or other systems herein to perform one or more tasks (e.g., information retrieval / analysis) based on the user’s input. Additionally, or alternatively, the practitioner may invoke one or more other Al agents to perform an identical task. For example, FIG. 1 1 depicts an eighth example GU1 1100 associated with optimizing healthcare data management using generative Al agents, in accordance with various embodiments herein. This eighth GUI 1 100 includes a response from an Al agent (e.g., “Ask Al” agent) to a similar / identical query to the query prompted to the “AskMayoExpert” Al agent in FIG. 10. This Al agent may utilize different internal and / or external knowledge sources to develop a relevant answer for the user.

[0154] This Al agent represented in FIG. 1 1 may also provide an assessment of the first-line treatment of the patient’s condition (e.g., HbA1c of 8.5%), such as lifestyle modifications (e.g., dietary changes, physical activity, weight management) and metformin. The Al agent may also provide alternative treatments and potential risks, side effects, and contraindications. Thus, similar to the Al agent in FIG. 10, this Al agent may provide first-line treatment options, but may include different information from the information that was output by the “AskMayoExpert” Al agent of FIG. 10.

[0155] Additionally, or alternatively, the systems described herein may generate recommendations that assist a practitioner in treating patients. For example, FIG. 12 depicts a ninth example GU1 1200 associated with optimizing healthcare data management using generative Al agents, in accordance with various embodiments herein. This ninth example GUI 1200 includes a recommended treatment plan for a patient (e.g., Patient X) that is based on the information the systems described herein may retrieve from the patient’s EHR entries, as well as information retrieved / extracted from internal / external knowledge bases.

[0156] Upon receiving a request for a treatment plan for the patient, the systems described herein may activate an appropriate Al agent based on the query's specificity and complexity. The "AskMayoExpert" Al agent, for example, may be tasked with gathering and synthesizing information relevant to the patient's conditions from a variety of knowledge sources. This mayinclude internal databases such as the EHR in which the systems described herein may be embedded that house detailed medical records, treatment guidelines, and / or patient outcomes, as well as external knowledge bases that may offer additional insights and / or recent research findings related to the conditions mentioned in the patient’s medical record.

[0157] The Al agent(s) may also cross-reference information from various databases and knowledge agents to ensure that the recommendations are grounded in the most current and comprehensive medical knowledge available. In particular, the Al agent(s) may analyze treatment guidelines for each of the patient's conditions, considering the interactions between the patient's multiple health issues, and / or evaluate the potential impact of the family history of coronary artery disease on treatment options. The Al agent may also apply local knowledge to significantly enhance the relevance and accuracy of the treatment plan. For example, if a healthcare facility has specific protocols for managing patients with multiple comorbidities or a family history of coronary artery disease, the Al agent may account for such information when formulating the treatment plan. This ensures that the recommendations provided by the system are not only medically sound but also aligned with the practices and standards of the specific healthcare setting. As illustrated in FIG. 12, the treatment plan may include, among other recommendations, that the initial investigations to treat Patient X’s conditions include one or more of an ECG, an echocardiogram, a chest X-ray, a pulmonary function test, and / or utilizing cardiac biomarkers. Thus, the treatment recommendation output by the systems described herein may provide a practitioner with one or more next steps or actions that may be relevant to treating a particular patient based on their specific medical history.

[0158] Additionally, or alternatively, the systems described herein may receive inputs from practitioners / users that represent the outcomes of such treatment plans and / or otherwise updates to the EHR. The systems described herein may receive these outcomes or other updates and may automatically update the EHR to reflect these outcomes without the practitioner / user having to update the EHR manually. For example, the practitioner / user may input a prompt to the systems described herein indicating that a patient has received an X-ray associated with a particular condition. The practitioner / user may also upload a file / image of the X-ray and may instruct the systems described herein to update the EHR for the patient with this X-ray and / or implications associated with the X-ray (e.g., indicating that one or more bones in the patient’s foot are broken).

[0159] In certain embodiments, the systems described herein enable users / practitioners to develop macros that may automatically prompt and / or otherwise instruct one or more of the chatbots / AI agents described herein to perform tasks. For example, FIG. 13 depicts a tenthexample GU1 1300 associated with optimizing healthcare data management using generative Al agents, in accordance with various embodiments herein. To create a macro, practitioners / users may interact with the tenth GU1 1300 to select from a list of recent questions they have posed or by adding new questions to which they frequently need answers. This initial step allows for the creation of a macro that is tailored to the specific informational needs of the practitioner, ensuring that they can quickly access the most relevant patient data without navigating through the entire EHR system.

[0160] Practitioners / users may further refine their macros by applying advanced filters that, e.g., restrict the answers provided by the macro to certain data types and / or to information from specific date ranges. For example, a practitioner could create a macro that only retrieves patient information related to blood test results from the last six months. The use of advanced filters may significantly enhance the precision and relevance of the outputs generated by the chatbots / AI agents executing the macro, ensuring that practitioners receive the most pertinent information for their clinical decision-making.

[0161] In addition to specifying questions and applying filters, practitioners may also use preambles to customize the format and presentation of the answers generated by the macro. Preambles may allow for the structuring of responses in a specific way, such as presenting the information in a bulleted list format. This customization feature ensures that the information retrieved is presented in a manner that is most useful and accessible to the practitioner, further improving the efficiency of data retrieval.

[0162] Moreover, practitioners may have the option to specify which chatbot and / or Al agents to use when executing the macro. This allows for the selection of the most appropriate chatbot / AI agent based on the nature of the questions included in the macro and the specific knowledge sources that are most likely to provide accurate and relevant answers. By specifying the chatbot and / or Al agents, practitioners can leverage the specialized expertise of different agents, enhancing the overall effectiveness of the macro.

[0163] In certain embodiments, the systems described herein may further provide internal resources and / or answers by leveraging the chatbots / AI agents described herein. For example, FIG. 14 depicts an eleventh example GUI 1400 associated with optimizing healthcare data management using generative Al agents, in accordance with various embodiments herein. As described herein, the chatbot / AI agents may be equipped with access to internal knowledge resources (e.g., an internal IT knowledge base). This knowledge base may contain a comprehensive collection of information and documentation related to common queries, procedures, and / or troubleshooting steps. Thus, when a practitioner inquires about, forexample, an IT-related question such as changing their password on an internal system the chatbot / AI agents described herein may process this request by searching the internal IT knowledge base for relevant information.

[0164] The chatbot / AI agents may employ NLP techniques to understand the context and intent behind the user's query and may then retrieve the most relevant information from the knowledge base, which may include step-by-step instructions for changing a password, links to internal IT policies, and / or automated forms that the user can fill out to request a password reset. The chatbot may present this relevant information to the user in an easily understandable format, such as providing a concise summary of the steps to change a password, offering a direct link to the password change page, and / or initiating an automated process to guide the user through the password change procedure.

[0165] Further, the chatbot / AI agent functionality may be extended to handle a wide range of questions associated with any suitable internal process / systems. For example, the chatbot / AI agent may be configured to provide information on software updates, accessing internal systems, troubleshooting common technical issues, guiding users through the use of various IT services, and / or otherwise assisting users with information associated with any suitable internal processes / systems. By leveraging the internal knowledge bases, the chatbots / AI agents may offer quick and accurate answers to these queries, functioning effectively as a virtual service desk. The chatbot / AI agents may continuously improve these outputs through machine learning algorithms that may learn from each interaction, enabling the chatbot / AI agents to become more efficient over time, and thereby enhancing its ability to provide relevant and accurate information based on the specific needs of the users.Aspects of the Disclosure

[0166] Aspects of the techniques described in the present disclosure may include any of the following aspects, either alone or in combination:

[0167] 1 . A computing system for optimizing healthcare data management using generative artificial intelligence (Al) agents, comprising: one or more processors; and one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing system to: receive, at the one or more processors, a prompt input referencing healthcare data included in an electronic health record (EHR), determine, by the one or more processors executing a trained machine learning (ML) model of an agent object, a user intent corresponding to the prompt input indicating retrieval of the healthcare data from the EHR, transmit, by the one or more processors via the agent object and through an associated application programming interface (API), a control instruction for adigital tool to retrieve the healthcare data from the EHR, generate, by the one or more processors executing the trained ML model, a prompt output that includes the healthcare data, and cause, by the one or more processors, the prompt output to be displayed in an output device.

[0168] 2. The computing system of aspect 1 , the one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, further cause the computing system to: generate, by the one or more processors, a preprocessed prompt input corresponding to the prompt input; and generate, by the one or more processors, post-processed healthcare data that is formatted based on the user intent, and wherein the prompt output includes the post-processed healthcare data.

[0169] 3. The computing system of aspect 2, wherein the post-processed healthcare data includes at least one of: (i) a graphical representation of the healthcare data, (ii) a collated document that includes the healthcare data, (iii) an image included as part of the healthcare data, (iv) a selectable link that directs users to the healthcare data in a source location, or (v) a chat option configured to connect a user with a second user that is associated with the healthcare data and / or indicated in the prompt input or the user intent.

[0170] 4. The computing system of any one of aspects 1 through 3, wherein the trained ML model of the agent object is (a) a large language model (LLM) that is configured to receive input healthcare data and generate a prompt output based on the input healthcare data or (b) a multimodal machine learning model configured to output multi-modal data associated with the input healthcare data.

[0171] 5. The computing system of any one of aspects 1 through 4, wherein the digital tool is one of a plurality of digital tools, and the one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, further cause the computing system to: analyze, by the one or more processors, each of the plurality of digital tools to determine whether any of the plurality of digital tools is configured to retrieve the healthcare data from the EHR; identify, by the one or more processors, that the digital tool is configured to retrieve the healthcare data from the EHR; and generate, by the one or more processors, the control instruction for the digital tool to retrieve the healthcare data.

[0172] 6. The computing system of any one of aspects 1 through 5, the one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, further cause the computing system to: receive, at the one or more processors, a subsequent prompt input related to the prompt output; determine, by the one or more processors executing the trained ML model of the agent object, a subsequent user intentcorresponding to the subsequent prompt input; transmit, by the one or more processors via the agent object and through a second API, a subsequent control instruction for a second digital tool to perform an action related to the subsequent user intent, wherein the second digital tool is different from the digital tool; generate, by the one or more processors executing the trained ML model, a subsequent prompt output based on the action performed by the second digital tool; and cause, by the one or more processors, the subsequent prompt output to be displayed in the output device.

[0173] 7. The computing system of any one of aspects 1 through 6, wherein a first user submits the prompt input, the output device is a first device of the first user, the prompt output includes a selectable option to connect with a second device of a second user in real-time, and the one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, further cause the computing system to: receive, at the one or more processors, a subsequent prompt input indicating a selection of the selectable option to connect with the second device of the second user; transmit, by the one or more processors via the agent object and through a second API, a subsequent control instruction for a second digital tool to connect the first device of the first user with the second device of the second user across a live data stream, wherein the second digital tool is different from the digital tool; and cause, by the one or more processors, the prompt output to continue to be displayed in the output device during the live data stream between the first device of the first user and the second device of the second user.

[0174] 8. The computing system of any one of aspects 1 through 7, wherein the prompt input is a verbal input from a user, and the one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, further cause the computing system to: transcribe, by the one or more processors executing the trained ML model of the agent object, the verbal input into a textual transcription; and determine, by the one or more processors executing the trained ML model of the agent object, the user intent based on the textual transcription.

[0175] 9. A non-transitory computer-readable medium, having stored thereon instructions that when executed, cause a computer to at least: receive a prompt input referencing healthcare data included in an electronic health record (EHR); determine, by executing a trained machine learning (ML) model of an agent object, a user intent corresponding to the prompt input indicating retrieval of the healthcare data from the EHR; transmit, via the agent object and through an associated application programming interface (API), a control instruction for a digital tool to retrieve the healthcare data from the EHR; generate, by executing the trained ML model,a prompt output that includes the healthcare data; and cause the prompt output to be displayed in an output device.

[0176] 10. The non-transitory computer-readable medium of aspect 9, having stored thereon instructions that when executed, further cause the computer to at least: generate a preprocessed prompt input corresponding to the prompt input; and generate post-processed healthcare data that is formatted based on the user intent, and wherein the prompt output includes the post-processed healthcare data.

[0177] 1 1. The non-transitory computer-readable medium of aspect 10, wherein the postprocessed healthcare data includes at least one of: (i) a graphical representation of the healthcare data, (ii) a collated document that includes the healthcare data, (iii) an image included as part of the healthcare data, (iv) a selectable link that directs users to the healthcare data in a source location, or (v) a chat option configured to connect a user with a second user that is associated with the healthcare data and / or indicated in the prompt input or the user intent.

[0178] 12. The non-transitory computer-readable medium of any one of aspects 9 through11 , wherein the trained ML model of the agent object is (a) a large language model (LLM) that is configured to receive input healthcare data and generate a prompt output based on the input healthcare data or (b) a multi-modal machine learning model configured to output multi-modal data associated with the input healthcare data.

[0179] 13. The non-transitory computer-readable medium of any one of aspects 9 through12, wherein the digital tool is one of a plurality of digital tools, and the instructions, when executed, cause a computer to: analyze each of the plurality of digital tools to determine whether any of the plurality of digital tools is configured to retrieve the healthcare data from the EHR; identify that the digital tool is configured to retrieve the healthcare data from the EHR; and generate the control instruction for the digital tool to retrieve the healthcare data.

[0180] 14. The non-transitory computer-readable medium of any one of aspects 9 through13, having stored thereon instructions that when executed, further cause the computer to: receive a subsequent prompt input related to the prompt output; determine, by executing the trained ML model of the agent object, a subsequent user intent corresponding to the subsequent prompt input; transmit, via the agent object and through a second API, a subsequent control instruction for a second digital tool to perform an action related to the subsequent user intent, wherein the second digital tool is different from the digital tool; generate, by executing the trained ML model, a subsequent prompt output based on the action performedby the second digital tool; and cause the subsequent prompt output to be displayed in the output device.

[0181] 15. The non-transitory computer-readable medium of any one of aspects 9 through 14, wherein a first user submits the prompt input, the output device is a first device of the first user, the prompt output includes a selectable option to connect with a second device of a second user in real-time, and the instructions, when executed, further cause the computer to: receive a subsequent prompt input indicating a selection of the selectable option to connect with the second device of the second user; transmit, via the agent object and through a second API, a subsequent control instruction for a second digital tool to connect the first device of the first user with the second device of the second user across a live data stream, wherein the second digital tool is different from the digital tool; and cause the prompt output to continue to be displayed in the output device during the live data stream between the first device of the first user and the second device of the second user.

[0182] 16. A computer-implemented method for optimizing healthcare data management using generative artificial intelligence (Al) agents, comprising: receiving, at one or more processors, a prompt input referencing healthcare data included in an electronic health record (EHR); determining, by the one or more processors executing a trained machine learning (ML) model of an agent object, a user intent corresponding to the prompt input indicating retrieval of the healthcare data from the EHR; transmitting, by the one or more processors via the agent object and through an associated application programming interface (API), a control instruction for a digital tool to retrieve the healthcare data from the EHR; generating, by the one or more processors executing the trained ML model, a prompt output that includes the healthcare data; and causing, by the one or more processors, the prompt output to be displayed in an output device.

[0183] 17. The computer-implemented method of aspect 16, further comprising: generating, by the one or more processors, a preprocessed prompt input corresponding to the prompt input; and generating, by the one or more processors, post-processed healthcare data that is formatted based on the user intent, wherein the prompt output includes the post-processed healthcare data, and wherein the post-processed healthcare data includes at least one of: (i) a graphical representation of the healthcare data, (ii) a collated document that includes the healthcare data, (iii) an image included as part of the healthcare data, (iv) a selectable link that directs users to the healthcare data in a source location, or (v) a chat option configured to connect a user with a second user that is associated with the healthcare data and / or indicated in the prompt input or the user intent.

[0184] 18. The computer-implemented method of any one of aspects 16 through 17, wherein the trained ML model of the agent object is (a) a large language model (LLM) that is configured to receive input healthcare data and generate a prompt output based on the input healthcare data or (b) a multi-modal machine learning model configured to output multi-modal data associated with the input healthcare data.

[0185] 19. The computer-implemented method of any one of aspects 16 through 18, wherein the digital tool is one of a plurality of digital tools, and the method further comprises: analyzing, by the one or more processors, each of the plurality of digital tools to determine whether any of the plurality of digital tools is configured to retrieve the healthcare data from the EHR; identifying, by the one or more processors, that the digital tool is configured to retrieve the healthcare data from the EHR; and generating, by the one or more processors, the control instruction for the digital tool to retrieve the healthcare data.

[0186] 20. The computer-implemented method of any one of aspects 16 through 19, further comprising: receiving, at the one or more processors, a subsequent prompt input related to the prompt output; determining, by the one or more processors executing the trained ML model of the agent object, a subsequent user intent corresponding to the subsequent prompt input; transmitting, by the one or more processors via the agent object and through a second API, a subsequent control instruction for a second digital tool to perform an action related to the subsequent user intent, wherein the second digital tool is different from the digital tool; generating, by the one or more processors executing the trained ML model, a subsequent prompt output based on the action performed by the second digital tool; and causing, by the one or more processors, the subsequent prompt output to be displayed in the output device.Additional Considerations

[0187] The following considerations also apply to the foregoing discussion. Throughout this specification, plural instances may implement operations or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may be performed concurrently, and nothing requires that the operations be performed in the order illustrated. These and other variations, modifications, additions, and improvements fall within the scope of the subject matter herein.

[0188] It should also be understood that, unless a term is expressly defined in this patent using the sentence "As used herein, the term " " is hereby defined to mean . . . " or a similar sentence, there is no intent to limit the meaning of that term, either expressly or by implication, beyond its plain or ordinary meaning, and such term should not be interpreted to be limited in scope basedon any statement made in any section of this patent (other than the language of the claims). To the extent that any term recited in the claims at the end of this patent is referred to in this patent in a manner consistent with a single meaning, that is done for sake of clarity only so as to not confuse the reader, and it is not intended that such claim term be limited, by implication or otherwise, to that single meaning. Finally, unless a claim element is defined by reciting the word "means'' and a function without the recital of any structure, it is not intended that the scope of any claim element be interpreted based on the application of 35 U.S.C. § 112(f).

[0189] Unless specifically stated otherwise, discussions herein using words such as "processing," "computing," "calculating," "determining," "presenting," "displaying," or the like may refer to actions or processes of a machine (e.g., a computer) that manipulates or transforms data represented as physical (e.g., electronic, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or a combination thereof), registers, or other machine components that receive, store, transmit, or display information.

[0190] As used herein any reference to "one aspect" or "an aspect" means that a particular element, feature, structure, or characteristic described in connection with the aspect is included in at least one aspect. The appearances of the phrase "in one aspect" in various places in the specification are not necessarily all referring to the same aspect.

[0191] As used herein, the terms "comprises," "comprising," "includes," "including," "has," "having" or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, "or" refers to an inclusive or and not to an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).

[0192] In addition, use of "a" or "an" is employed to describe elements and components of the aspects herein. This is done merely for convenience and to give a general sense of the invention. This description should be read to include one or at least one and the singular also includes the plural unless it is obvious that it is meant otherwise.

[0193] The various embodiments described above can be combined to provide further embodiments. All U.S. patents, U.S. patent application publications, U.S. patent application, foreign patents, foreign patent application and non-patent publications referred to in this specification and / or listed in the Application Data Sheet are incorporated herein by reference, in their respective entireties, for all purposes. Aspects of the embodiments can be modified ifnecessary to employ concepts of the various patents, applications, and publications to provide yet further embodiments.

[0194] These and other changes can be made to the embodiments in light of the abovedetailed description. In general, in the following claims, the terms used should not be construed to limit the claims to the specific embodiments disclosed in the specification and the claims but should be construed to include all possible embodiments along with the full scope of equivalents to which such claims are entitled. Accordingly, the claims are not limited by the disclosure.

[0195] Upon reading this disclosure, those of skill in the art will appreciate still additional alternative structural and functional designs for implementing the concepts disclosed herein, through the principles disclosed herein. Thus, while particular aspects and applications have been illustrated and described, it is to be understood that the disclosed aspects are not limited to the precise construction and components disclosed herein. Various modifications, changes and variations, which will be apparent to those skilled in the art, may be made in the arrangement, operation and details of the method and apparatus disclosed herein without departing from the spirit and scope defined in the appended claims.

Claims

What is claimed is:1 . A computing system for optimizing healthcare data management using generative artificial intelligence (Al) agents, comprising: one or more processors; and one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing system to: receive, at the one or more processors, a prompt input referencing healthcare data included in an electronic health record (EHR), determine, by the one or more processors executing a trained machine learning (ML) model of an agent object, a user intent corresponding to the prompt input indicating retrieval of the healthcare data from the EHR, transmit, by the one or more processors via the agent object and through an associated application programming interface (API), a control instruction for a digital tool to retrieve the healthcare data from the EHR, generate, by the one or more processors executing the trained ML model, a prompt output that includes the healthcare data, and cause, by the one or more processors, the prompt output to be displayed in an output device.

2. The computing system of claim 1 , the one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, further cause the computing system to: generate, by the one or more processors, a preprocessed prompt input corresponding to the prompt input; and generate, by the one or more processors, post-processed healthcare data that is formatted based on the user intent, and wherein the prompt output includes the post-processed healthcare data.

3. The computing system of claim 2, wherein the post-processed healthcare data includes at least one of: (i) a graphical representation of the healthcare data, (ii) a collated document that includes the healthcare data, (iii) an image included as part of the healthcare data, (iv) a selectable link that directs users to the healthcare data in a source location, or (v) a chat option configured to connect a user with a second user that is associated with the healthcare data and / or indicated in the prompt input or the user intent.

4. The computing system of claim 1 , wherein the trained ML model of the agent object is (a) a large language model (LLM) that is configured to receive input healthcare data and generate a prompt output based on the input healthcare data or (b) a multi-modal machine learning model configured to output multi-modal data associated with the input healthcare data.

5. The computing system of claim 1 , wherein the digital tool is one of a plurality of digital tools, and the one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, further cause the computing system to: analyze, by the one or more processors, each of the plurality of digital tools to determine whether any of the plurality of digital tools is configured to retrieve the healthcare data from the EHR; identify, by the one or more processors, that the digital tool is configured to retrieve the healthcare data from the EHR; and generate, by the one or more processors, the control instruction for the digital tool to retrieve the healthcare data.

6. The computing system of claim 1 , the one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, further cause the computing system to: receive, at the one or more processors, a subsequent prompt input related to the prompt output; determine, by the one or more processors executing the trained ML model of the agent object, a subsequent user intent corresponding to the subsequent prompt input; transmit, by the one or more processors via the agent object and through a second API, a subsequent control instruction for a second digital tool to perform an action related to the subsequent user intent, wherein the second digital tool is different from the digital tool; generate, by the one or more processors executing the trained ML model, a subsequent prompt output based on the action performed by the second digital tool; and cause, by the one or more processors, the subsequent prompt output to be displayed in the output device.

7. The computing system of claim 1 , wherein a first user submits the prompt input, the output device is a first device of the first user, the prompt output includes a selectable optionto connect with a second device of a second user in real-time, and the one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, further cause the computing system to: receive, at the one or more processors, a subsequent prompt input indicating a selection of the selectable option to connect with the second device of the second user; transmit, by the one or more processors via the agent object and through a second API, a subsequent control instruction for a second digital tool to connect the first device of the first user with the second device of the second user across a live data stream, wherein the second digital tool is different from the digital tool; and cause, by the one or more processors, the prompt output to continue to be displayed in the output device during the live data stream between the first device of the first user and the second device of the second user.

8. The computing system of claim 1 , wherein the prompt input is a verbal input from a user, and the one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, further cause the computing system to: transcribe, by the one or more processors executing the trained ML model of the agent object, the verbal input into a textual transcription; and determine, by the one or more processors executing the trained ML model of the agent object, the user intent based on the textual transcription.

9. A non-transitory computer-readable medium, having stored thereon instructions that when executed, cause a computer to at least: receive a prompt input referencing healthcare data included in an electronic health record (EHR); determine, by executing a trained machine learning (ML) model of an agent object, a user intent corresponding to the prompt input indicating retrieval of the healthcare data from the EHR; transmit, via the agent object and through an associated application programming interface (API), a control instruction for a digital tool to retrieve the healthcare data from the EHR; generate, by executing the trained ML model, a prompt output that includes the healthcare data; and cause the prompt output to be displayed in an output device.

10. The non-transitory computer-readable medium of claim 9, having stored thereon instructions that when executed, further cause the computer to at least: generate a preprocessed prompt input corresponding to the prompt input; and generate post-processed healthcare data that is formatted based on the user intent, and wherein the prompt output includes the post-processed healthcare data.11 . The non-transitory computer-readable medium of claim 10, wherein the postprocessed healthcare data includes at least one of: (i) a graphical representation of the healthcare data, (ii) a collated document that includes the healthcare data, (iii) an image included as part of the healthcare data, (iv) a selectable link that directs users to the healthcare data in a source location, or (v) a chat option configured to connect a user with a second user that is associated with the healthcare data and / or indicated in the prompt input or the user intent.

12. The non-transitory computer-readable medium of claim 9, wherein the trained ML model of the agent object is (a) a large language model (LLM) that is configured to receive input healthcare data and generate a prompt output based on the input healthcare data or (b) a multimodal machine learning model configured to output multi-modal data associated with the input healthcare data.

13. The non-transitory computer-readable medium of claim 9, wherein the digital tool is one of a plurality of digital tools, and the instructions, when executed, cause a computer to at least: analyze each of the plurality of digital tools to determine whether any of the plurality of digital tools is configured to retrieve the healthcare data from the EHR; identify that the digital tool is configured to retrieve the healthcare data from the EHR; and generate the control instruction for the digital tool to retrieve the healthcare data.

14. The non-transitory computer-readable medium of claim 9, having stored thereon instructions that when executed, further cause the computer to: receive a subsequent prompt input related to the prompt output; determine, by executing the trained ML model of the agent object, a subsequent user intent corresponding to the subsequent prompt input;transmit, via the agent object and through a second API, a subsequent control instruction for a second digital tool to perform an action related to the subsequent user intent, wherein the second digital tool is different from the digital tool; generate, by executing the trained ML model, a subsequent prompt output based on the action performed by the second digital tool; and cause the subsequent prompt output to be displayed in the output device.

15. The non-transitory computer-readable medium of claim 9, wherein a first user submits the prompt input, the output device is a first device of the first user, the prompt output includes a selectable option to connect with a second device of a second user in real-time, and the instructions, when executed, further cause the computer to at least: receive a subsequent prompt input indicating a selection of the selectable option to connect with the second device of the second user; transmit, via the agent object and through a second API, a subsequent control instruction for a second digital tool to connect the first device of the first user with the second device of the second user across a live data stream, wherein the second digital tool is different from the digital tool; and cause the prompt output to continue to be displayed in the output device during the live data stream between the first device of the first user and the second device of the second user.

16. A computer-implemented method for optimizing healthcare data management using generative artificial intelligence (Al) agents, comprising: receiving, at one or more processors, a prompt input referencing healthcare data included in an electronic health record (EHR); determining, by the one or more processors executing a trained machine learning (ML) model of an agent object, a user intent corresponding to the prompt input indicating retrieval of the healthcare data from the EHR; transmitting, by the one or more processors via the agent object and through an associated application programming interface (API), a control instruction for a digital tool to retrieve the healthcare data from the EHR; generating, by the one or more processors executing the trained ML model, a prompt output that includes the healthcare data; and causing, by the one or more processors, the prompt output to be displayed in an output device.

17. The computer-implemented method of claim 16, further comprising : generating, by the one or more processors, a preprocessed prompt input corresponding to the prompt input; and generating, by the one or more processors, post-processed healthcare data that is formatted based on the user intent, wherein the prompt output includes the post-processed healthcare data, and wherein the post-processed healthcare data includes at least one of: (i) a graphical representation of the healthcare data, (ii) a collated document that includes the healthcare data, (iii) an image included as part of the healthcare data, (iv) a selectable link that directs users to the healthcare data in a source location, or (v) a chat option configured to connect a user with a second user that is associated with the healthcare data and / or indicated in the prompt input or the user intent.

18. The computer-implemented method of claim 16, wherein the trained ML model of the agent object is (a) a large language model (LLM) that is configured to receive input healthcare data and generate a prompt output based on the input healthcare data or (b) a multimodal machine learning model configured to output multi-modal data associated with the input healthcare data.

19. The computer-implemented method of claim 16, wherein the digital tool is one of a plurality of digital tools, and the method further comprises: analyzing, by the one or more processors, each of the plurality of digital tools to determine whether any of the plurality of digital tools is configured to retrieve the healthcare data from the EHR; identifying, by the one or more processors, that the digital tool is configured to retrieve the healthcare data from the EHR; and generating, by the one or more processors, the control instruction for the digital tool to retrieve the healthcare data.

20. The computer-implemented method of claim 16, further comprising: receiving, at the one or more processors, a subsequent prompt input related to the prompt output; determining, by the one or more processors executing the trained ML model of the agent object, a subsequent user intent corresponding to the subsequent prompt input;transmitting, by the one or more processors via the agent object and through a second API, a subsequent control instruction for a second digital tool to perform an action related to the subsequent user intent, wherein the second digital tool is different from the digital tool; generating, by the one or more processors executing the trained ML model, a subsequent prompt output based on the action performed by the second digital tool; and causing, by the one or more processors, the subsequent prompt output to be displayed in the output device.

Images