Systems and methods for pre-training large language models

Abstract

The present disclosure provides a system for pre-training a large language model. The system includes an electronic health record (EHR) database storing patient-specific data, a processor, and a memory storing instructions. The instructions, when executed by the processor, cause the system to retrieve patient data from the EHR database, generate a clinical text corpus from the retrieved patient data, pre-append metadata to each note in the clinical text corpus to create pre-appended notes, optimize context windows for the pre-appended notes, tokenize the pre-appended notes to create a tokenized dataset, and pre-train the large language model using the tokenized dataset. The pre-trained large language model is configured to generate patient-specific responses to medical queries.

Description

SYSTEMS AND METHODS FOR PRE-TRAINING LARGE LANGUAGE MODELSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 565,281, entitled SYSTEMS AND METHODS FOR PRE-TRAINING LARGE LANGUAGE MODELS, filed March 14, 2024. which is hereby incorporated by reference in its entirety.TECHNICAL FIELD

[0002] The present disclosure relates to artificial intelligence systems for processing medical data, and more particularly to methods and systems for pre-training large language models using patient-specific data to improve performance in healthcare applications.FEDERAL FUNDING LEGEND

[0003] This invention was made with government support under Grant No. HD111688 awarded by the National Institutes of Health. The government has certain rights in the invention.BACKGROUND

[0004] Large language models (LLMs) have emerged as powerful tools in various domains, including healthcare. These models, trained on vast amounts of text data, can process and generate human-like text, making them potentially valuable for tasks such as medical documentation, clinical decision support, and patient communication. However, the application of LLMs in healthcare settings presents unique challenges and opportunities.

[0005] One of the challenges in utilizing LLMs for healthcare applications is the need for patient-specific knowledge. While general-purpose LLMs possess broad knowledge across many domains, they typically lack detailed information about individual patients' medical histories, treatments, and specific healthcare contexts. This limitation can hinder the models' ability to provide accurate and personalized responses in clinical settings.

[0006] Another consideration is the handling of sensitive patient information. Healthcare data is subject to strict privacy regulations, and any system processing or generating patientspecific information must comply with these guidelines. This requirement adds complexity7to the development and deployment of LLM-based solutions in healthcare environments.

[0007] Furthermore, the dynamic nature of medical information presents challenges for LLM applications. Patient and healthcare information can evolve rapidly, and healthcare systems must be able to incorporate new information efficiently. Traditional LLMs, once trained, may struggle to stay current with the latest patient information or medical developments without frequent retraining or updates.

[0008] The integration of LLMs into systems for use in answering questions relating to specific patients also presents many technical challenges. These include, for example, inefficiencies associated with current methods of retrieving and leveraging contextually relevant data and information; the inability to optimize utilization of context windows (e.g., context windows that are too large consume excessive computing resources, while context windows that are too small omit valuable information); the inability to control output of the LLMs (e.g., to focus on relevant information in a form that is useful for users); and the inability to prevent the LLMs from losing their ability to operate in other out-of-domain uses cases, to name a few. Such systems are also unable to efficiently capture and prepare audit logs, influence model behavior or address inquiries spanning multiple patients at once.

[0009] Therefore, as the potential for LLMs in healthcare continues to grow, the need for addressing the foregoing (and other) challenges becomes increasingly relevant. Accordingly, there is a need for innovations in model training, data integration, and system design to help overcome current limitations and unlock new possibilities for enhancing patient care and healthcare operations through advanced language processing technologies.SUMMARY

[0010] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0011] According to an aspect of the present disclosure, a system for pre-training a large language model is provided. The system includes an electronic health record (EHR) database storing patient-specific data, a processor, and a memory storing instructions that, when executed by the processor, cause the system to retrieve patient data from the EHR database. The system then generates a clinical text corpus from the retrieved patient data and pre-appends metadata to each note in the clinical text corpus to create pre-appended notes. The system then optimizes context windows for the pre-appended notes and tokemzes the pre-appended notes to create a tokenized dataset. The system can then pre-train the large language model using the tokenized dataset. The pre-trained large language model can be configured to generate patientspecific responses to medical queries.

[0012] According to other aspects of the present disclosure, the system can include one or more of the following features. The system can optimize context windows for the pre-appended notes by determining if a pre-appended note is longer than a context window size, splitting the pre-appended note across multiple context windows when it is determined that the preappended note is longer than the context window size, and combining the pre-appended note with at least one of another pre-appended note and a pre-appended note portion into a single context window when it is determined that the pre-appended note is shorter than the contextwindow size. The system can split the pre-appended note that is longer than the context window size by dividing the pre-appended note into portions that fit within the context window size and pre-appending metadata to each portion to link the portions. The metadata pre-appended to each pre-appended note can comprise at least one of a note identifier, a medical record number, a patient name, a date of birth, a patient age, a note date, a note category, an author name, a department identifier, a specialty identifier, and an encounter type.

[0013] The system can further include a sparse autoencoder system configured to extract patient-specific features from the pre-trained large language model and monitor activation of the extracted patient-specific features during query processing. The sparse autoencoder system can comprise an input layer configured to receive input data, a hidden layer configured to process the input data, and an output layer configured to generate output based on the processed input data. The system can further include a patient feature extractor configured to identify and extract patient-specific features from hidden layer activations of the sparse autoencoder system. The pre-trained large language model can comprise a query processor configured to receive processed queries from the sparse autoencoder system, a response generator configured to produce responses based on the processed queries and extracted patient features, and a patient feature activator configured to incorporate the patient-specific features into the generated responses.

[0014] The system can further include a continuous update system configured to monitor the EHR database for new' patient data, process the new patient data into the clinical text corpus, update the pre-trained large language model with the processed new patient data, and validate performance of the updated pre-trained large language model. The continuous update system can be further configured to prune portions of the pre-trained large language model that do not provide useful information for a particular task.

[0015] According to another aspect of the present disclosure, a method for pre-training a large language model is provided. The method includes retrieving patient data from an electronic health record (EHR) database, generating a clinical text corpus from the retrieved patient data, pre-appending metadata to each note in the clinical text corpus to create preappended notes, and optimizing context w indows for the pre-appended notes. The method can also include tokenizing the pre-appended notes to create a tokenized dataset and pre-training the large language model using the tokenized dataset. The pre-trained large language model can be configured to generate patient-specific responses to medical queries.

[0016] According to other aspects of the present disclosure, the method can include one or more of the following features. The method can optimize context windows for the preappended notes by determining if a pre-appended note is longer than a context window size, splitting the pre-appended note across multiple context windows when it is determined that the pre-appended note is longer than the context window' size, and combining the pre-appendednote with at least one of another pre-appended note and a pre-appended note portion into a single context window when it is determined that the pre-appended note is shorter than the context window size. The method can also include splitting the pre-appended note that is longer than the context window size by dividing the pre-appended note into portions that fit within the context window size and pre-appending metadata to each portion to link the portions. The metadata pre-appended to each pre-appended note can comprise at least one of a note identifier, a medical record number, a patient name, a date of birth, a patient age, a note date, a note category, an author name, a department identifier, a specialty identifier, and an encounter type.

[0017] The method can also include extracting patient-specific features from the pre-trained large language model using a sparse autoencoder system and monitoring activation of the extracted patient-specific features during query processing. The sparse autoencoder system can comprise an input layer configured to receive input data, a hidden layer configured to process the input data, and an output layer configured to generate output based on the processed input data. The method can further include identifying and extracting patient-specific features from activations of the hidden layer using a patient feature extractor. The pre-trained large language model can comprise a query processor configured to receive processed queries from the sparse autoencoder system, a response generator configured to produce responses based on the processed queries and extracted patient features, and a patient feature activator configured to incorporate the patient-specific features into the generated responses.

[0018] The method can further include monitoring the EHR database for new patient data, processing new patient data into the clinical text corpus, updating the pre-trained large language model with the processed new patient data, and validating performance of the updated pretrained large language model. In addition, the method can include pruning portions of the pretrained large language model that do not provide useful information for a particular task.

[0019] The foregoing general description of the illustrative embodiments and the following detailed description thereof are merely exemplary aspects of the teachings of this disclosure and are not restrictive.BRIEF DESCRIPTION OF FIGURES

[0020] Non-limiting and non-exhaustive examples are described with reference to the following figures.

[0021] FIG. 1 illustrates a block diagram of a prior art system for processing electronic health record data.

[0022] FIG. 2 illustrates a pre-training structure for training a large language model with patient medical data, according to an embodiment.

[0023] FIG. 3A illustrates a system interface for processing medical data queries and responses, according to aspects of the present disclosure.

[0024] FIG. 3B illustrates another system interface for processing medical queries and responses, according to aspects of the present disclosure.

[0025] FIG. 3C illustrates a system interface for processing medical queries across multiple patients, according to aspects of the present disclosure.

[0026] FIG. 4 illustrates a system interface for processing medical queries and responses using a large language model that has not been pre-trained according to an embodiment.

[0027] FIG. 5 illustrates a system interface for processing out-of-domain queries and responses, according to aspects of the present disclosure.

[0028] FIG. 6 illustrates a pre-training architecture for processing and training a large language model, according to an embodiment.

[0029] FIG. 7 illustrates a flowchart of a method for processing clinical notes for pre-training a large language model, according to aspects of the present disclosure.

[0030] FIG. 8 illustrates a flowchart of a method for preparing clinical notes for pre-training a large language model, according to aspects of the present disclosure.

[0031] FIG. 9 illustrates a flowchart of a method for processing clinical notes, according to aspects of the present disclosure.

[0032] FIG. 10 illustrates a block diagram of a clinical note structure with metadata, according to aspects of the present disclosure.

[0033] FIG. 11 illustrates how patient metadata is connected to its semantic meaning in the system, according to aspects of the present disclosure.

[0034] FIG. 12 illustrates a block diagram of a continuous LLM update system, according to aspects of the present disclosure.

[0035] FIG. 13 illustrates a flowchart of a method for monitoring and processing electronic health record data, according to aspects of the present disclosure.

[0036] FIG. 14 illustrates a block diagram of a system for processing patient-specific queries, according to aspects of the present disclosure.

[0037] FIG. 15 illustrates a block diagram of a patient-specific query processing system, according to aspects of the present disclosure.

[0038] FIG. 16 illustrates a sequence diagram for processing patient-specific queries and monitoring data access patterns, according to aspects of the present disclosure.

[0039] FIG. 17 illustrates a flowchart of a method for processing patient-specific queries, according to aspects of the present disclosure.

[0040] FIG. 18 illustrates diagrams showing how a large language model processes and responds to patient queries, according to aspects of the present disclosure.

[0041] FIG. 19 illustrates a flowchart of a method for creating fine-tuning data to induce patient-specific reasoning in a large language model.

[0042] FIG. 20 illustrates a system interface for processing medical queries and responses using a large language model that has been pre-trained and fine-tuned to reason using internal patient-specific knowledge, according to aspects of the present disclosure.

[0043] FIG. 21 illustrates a flowchart of a method 2100 for developing clinical reasoning in a large language model (LLM).DETAILED DESCRIPTION

[0044] The following description sets forth exemplary aspects of the present disclosure. It should be recognized, however, that such description is not intended as a limitation on the scope of the present disclosure. Rather, the description also encompasses combinations and modifications to those exemplary7aspects described herein.

[0045] A detailed description of systems, devices, and methods consistent with embodiments of the present disclosure is provided below. While several embodiments are described, it should be understood that disclosure is not limited to any7one embodiment, but instead encompasses numerous alternatives, modifications, and equivalents. In addition, while numerous specific details are set forth in the following description in order to provide a thorough understanding of the embodiments disclosed herein, some embodiments can be practiced without some or all of these details. Moreover, for the purpose of clarity, certain technical material that is known in the related art has not been described in detail in order to avoid unnecessarily obscuring the disclosure.

[0046] For purposes of this disclosure, the systems, methods and computer program products will be described in the context of health information technology, particularly those used by large U.S. hospitals and health systems to access, organize, store and share electronic medical records. It should be understood, however, that this disclosure is not limited thereto. To the contrary, the systems, methods and computer program products described herein may be implemented in any industry7, domain, use case, etc. which would benefit from having more accurate and robust LLM output, and from improving system memory7, operating and processing costs.

[0047] The present disclosure relates to systems, methods, and computer program products for pre-training large language models (LLMs), including foundational LLMs, with patientspecific data to improve their accuracy and efficiency in healthcare applications. By incorporating patient-specific information during the pre-training process, LLMs can develop a more nuanced understanding of individual patient histories and medical contexts, therebyleading to more accurate, concise and personalized responses in clinical settings. In addition, pre-training LLMs as described herein can (e.g., using patient-identifiable data) can result in the LLMs learning information about patients across any sized health information system.

[0048] This approach to pre-training LLMs with patient-specific data may offer several advantages over traditional methods. For example, models trained according to this disclosure can generate more relevant and contextually appropriate responses when queried about specific patients, thereby improving the quality of decision support provided to healthcare professionals. Additionally, by encoding patient information directly into the model parameters, the approaches described herein can reduce the need for extensive context retrieval at inference time, thereby improving system efficiency.

[0049] Further, the pre-training and other aspects described herein may be utilized in any number of use cases and / or domains. Some example use cases within the medical heath domain may include (without limitation): predicting hospitalization, generating notes in advance of a patient encounter (“Precharting”), generating discharge summaries, generating complex care summaries, identifying medical problems which have subsequently resolved, genetic disease diagnosis, extracting antiepileptic treatment courses, classifying by non-medical / functional disorder risks, predict diagnosis codes for medical billing, assembling metrics for reporting to national ranking organizations or regulatory bodies, and others.

[0050] As will be appreciated, the number of use cases is extensive, and can span research, operations, and specialties. By pre-training a foundational LLM as described in connection with any possible use case (and / or other related tasks), the foundational LLM will also gain general abilities for which it has never been trained for, as previously discussed above in connection with the medical problem summarization (e.g., see FIGs. 3A-3C). Such capabilities in the medical health domain may include (without limitation) an ability to: extract signs, symptoms and diagnoses, extract developmental milestones, extract genetic variants, draft discharge summaries, pre-chart patient visits, assist with differential diagnosis in the ED or genetics clinic, identify7diagnostic errors, abstract information for regulator7 / ranking reporting, improve billing and coding accuracy, and so on.

[0051] Importantly, the pre-training techniques described herein are designed to help the LLM maintain its performance on tasks unrelated to healthcare (e.g., its 'out-of-domain' performance). This is achieved through specific strategies implemented during the pre-training process, which aim to preserve the model's general knowledge and capabilities while enhancing its proficiency in patient-specific tasks. As a result, the pre-trained LLM can remain versatile and applicable across a wide range of domains, not limited to healthcare applications.

[0052] In addition, the present disclosure provides new approaches and techniques for optimizing context windows that are utilized during the pre-training. As discussed further below, patient notes (also referred to herein as clinical notes) that are longer than contextwindows may be split across multiple context windows (as opposed to curtailing the longer notes). This enables the foundational LLMs to leam from longer notes while maintaining shorter context window lengths requiring more modest computational resources. As will be appreciated, this may be particularly relevant for patients with long and complicated medical histories, the exact types of patients for whom medical professionals require assistance. In addition, splitting notes as described herein substantially improves upon standard tokenization approaches which may discard additional text or blindly move the remaining text into the next context window.

[0053] For foundational LLMs with larger context windows, the present disclosure provides a novel approach for combining multiple pieces of clinical text (e.g., multiple patient notes or note portions) into a single context window. This substantially improves computational efficiency when pre-training the foundational LLMs on a large corpus of text (e.g., patient notes), including by >29.5% for a context window7of 4.096, while still maintaining the logical connection between pieces of the same text.

[0054] The present disclosure also provides a novel approach that enables information retrieval from previous records without explicit human knowledge or accurate document retrieval at the time of inference. Foundational LLMs may infer novel associations from the data.

[0055] In addition, foundational LLMs are able to leam medical management of topics which may be more common in the local health system population than in general medical literature.

[0056] As further discussed below7, the present disclosure for pre-training foundational LLMs comprises providing metadata with each piece of clinical text. As a result, foundational LLMs gain context about the text, develop a longitudinal, chronologic history of the patients, and develop insights into practice patterns of departments, specialties and providers.

[0057] The techniques described herein also enable foundational LLMs to provide responses in the style of and respecting the customs of any particular healthcare system. Indeed, rather than generating generic responses, foundational LLMs that are pre-trained according to this disclosure are able to provide responses tailored to a particular facility, specialty or even provider.

[0058] Large language models (or LLMs) comprise a family of artificial intelligence tools that can be used to help understand and generate human language. ChatGPT™, for example, is a large language model developed by the company OpenAI™. Large language models such as ChatGPT™ are often referred to as foundational LLMs. This is because are such models are trained on a broad range of data such that they may be implemented across a broad range of use cases (e.g.. akin to general-purpose LLMs) and / or utilized for a broad range of tasks. Forexample, a foundational model could be used to create a chatbot, translate languages, or write creative content.

[0059] Foundational models such as ChatGPT™ exploded onto the seen in November 2022 and have captured the general public’s imagination. Other popular LLMs include Llama-2™ by Meta™, BloomZ™ by BigScience™, FALCON™ by Til™, PaLM-2™ by Google™, and others. These types of foundational LLMs generally gain their abilities by “reading” trillions of words of human language and then later receiving feedback on their performance from humans.

[0060] In the context of health information technology, the vast majority of reported studies focus on two approaches for utilizing foundational LLMs to answer questions about specific patients. In a first approach, a system programmatically generates textual context about a patient at the time of creating the question, and sends that as context along with the question to the LLM. This is the strategy for integration with foundational LLMs utilized by many providers of health information technology7systems, particularly those systems used by large U.S. hospitals and health systems to access, organize, store and share electronic medical records. Among others, a key limitation of this approach is that the developers of the program that generates the context must know in advance the types of information that would be of interest. Moreover, the type of information accessible to the model is constrained fully by that program.

[0061] In a second approach, the system pre-computes a summary about a document called an “embedding,” and then retrieves document(s) with embeddings that are most similar to the question posed to the foundational LLM in real time. This approach is commonly referred to as retrieval augmented generation (or “RAG”).

[0062] Turning now to FIG. 1, a conventional system 100 configured for document retrieval based on embeddings is shown. Embeddings are a mathematical representation of a document or document chunk in N-dimensional space. By way of illustration, ChatGPT™ embeddings comprises a 256-dimensional space. Thus, for an electronic health record (EHR), the system 100 may calculate an embedding (e.g.. via an embedding calculator 102) of a first patient note and then repeat that process for all notes in the patient’s EHR and for all notes in all patients’ EHR. Collectively, these notes comprise an EHR corpus 103 and the embeddings for the EHR corpus 103 can be stored in an embedding database 104. Patient notes in the EHR corpus 103 may include (without limitation), for any given patent, discharge note(s), diagnosis summaries, treatment plan(s), etc. When a user has a question about a particular patient, the user can submit an inquiry7101a via a user device 101 to the system 100. The system 100 calculates, via its embedding calculator 102, an embedding for the inquiry7101a, searches the embedding database 104 for documents with the most similar embeddings, and retrieves them from the EHR corpus 103 via a document retriever 105. These documents are then provided to the LLM106 for context. One limitation of this approach is that the calculated embedding of the question may not fully encapsulate the information space of the records required to fully or effectively answer the inquiry. A second limitation is that documents are traditionally retrieved by their semantic similarity without regard to their chronologic context. Thus, critical context surrounding retrieved documents may be lost.

[0063] Both of these approaches involve customizing what is presented to LLMs as context, but none changes how the model interacts with those inputs. This limits the ability to customize the LLMs for a particular institution, user, or use case. Moreover, these approaches preclude the development of emergent capabilities (e.g., abilities learned from the data itself). For example, in the area of diagnostic decision support, models trained utilizing the existing approaches would be limited to LLMs’ existing abilities and would not improve their performance from other examples.

[0064] Aside from the limitations discussed above, existing systems are also deficient in the way context is processed and in controlling the output generated by the LLMs. For example, systems that retrieve patient notes from an embedding database for use as context must contend with size limitations of context windows. For purposes of this disclosure, context windows define a size or range of the context that a foundational LLM is able to consider when processing input data (e.g., a patient note). For example, in a foundational LLM such as ChatGPT1M, a context window may comprise 32000 tokens, meaning that it is able to consider 32000 words or characters (referred to as "tokens") when making predictions or generating responses. In another example, a foundational model such as Llama-2™ may have a much smaller context window that includes 4096 tokens. Having a larger context window (e.g., ChatGPT™) may allow the foundational LLM to capture more context from the input data, potentially improving the quality of predictions. However, increasing the context window size also comes with computational costs, as the foundational LLM needs to process more input tokens, leading to higher memory and processing requirements.

[0065] Conversely, having a smaller context window (e.g., Llama-2™) may yield processing efficiencies over larger context windows. However, since patient notes (or other contextual documents) are often longer (e g., require more tokens) than available space in such context windows, note text that extends beyond the context windows is simply cut off, thereby depriving the foundational LLM of valuable contextual data.

[0066] Further, existing health information system-LLM integrations are unable to control the output (e.g., style, form, content, etc.) of foundational LLMs. As a result, these integrations generally produce responses in a manner not suited to healthcare use cases. For example, existing system-LLM integrations produce responses with superfluous lists interspersed (in an indeterminate fashion) with extraneous information that often extends (randomly) well beyond the scope of the questions posed.

[0067] Further still, existing systems that train foundational LLMs for particular use cases are unable to prevent such models from ‘forgetting’ how to operate in other use cases. As a result, such training may render foundational LLMs inoperable for use in other domains.

[0068] Having recognized the foregoing (and other) deficiencies, the Applicant has developed a new system, methods and computer program products for pre-training foundational LLMs to address and improve both the performance of the foundational LLMs and the efficiencies of the systems deploying such foundational LLMs.

[0069] As it pertains to context windows, the present disclosure provides a novel approach for optimizing the utilization and processing requirements of context windows, regardless of their sizes. As noted above, context windows that are too large tend to waste system processing resources, while context windows that are too small tend to cut off contextual text. In order to address these deficiencies, a system according to the present disclosure may first calculate a note size (e.g., in terms of tokens) of each note or document that may be utilized as contextual data. Then, understanding the context window size of the foundational LLM at hand, the system may further process each note accordingly. For example, if a particular note is larger (e.g., has more tokens) than a context window size of the LLM, the system may divide the note into smaller portions or ‘partial notes,’ and pre-append identifying meta data (e.g., note ID) to each portion, such that all portions of the note may be linked and called via the metadata. In this manner, longer inputs are not cut off (as in traditional systems). Instead, the longer notes are spread across multiple context windows which enables the system to ‘see’ and connect each note portion.

[0070] For foundational LLMs having larger context windows and / or for smaller notes, the system may implement ‘bin packing’, which includes the process of combining smaller notes and / or note portions, also pre-pended with meta data, into larger context windows so as to optimize utilization of the larger context windows. In this manner, the system minimizes the processing of empty or partially empty context windows, thereby improving memory utilization and overall operating efficiency.

[0071] In an exemplary use case, the present disclosure provides novel techniques for pretraining foundational LLMs to learn about identifiable patients in a way that can be effective at scale (e.g., at the health system scale). As a result of such techniques, foundational LLMs would gain emergent abilities to extract information, summarize, and provide medical decision making tailored to a particular patient.

[0072] In order to assess the techniques discussed herein, a foundational LLM may be pretrained on a system-wide corpus of patient notes and related documents. Alternatively or additionally, a preliminary analysis may include pre-training the foundational LLM on a portion of the corpus of patient notes and related documents such as, for example, notes for allpatients bom in a specific year(s) (e.g., 2021), or notes for all patients bom in a particular month(s) across a large range of years (July of 2000-2023).

[0073] In some aspects, the pre-training may involve pre-appending certain meta data to each note to provide context to the foundational LLM about each said note. The prompt used in pretraining (and later in submitting questions / inquiries to the foundational LLM) may be structured in a similar manner to that used by the original architects of the foundational LLM to train it. The text of each note is also provided to the foundational LLM during this pretraining. By following this process, the foundational LLM may be pre-trained on each note having read each such note only one time.

[0074] To illustrate the pre-training concept, reference is made to FIG. 2. FIG. 2 shows an exemplary pre-training structure 200 for pre-training a foundational LLM on a particular note. As shown, the structure 200 includes a prompt 201, note text 203, and metadata 202 that may be pre-appended to the note text within the structure 200. As noted above, the prompt 201 may be configured to match the form and structure as that used by the original architects of the foundational LLM to initially train the model. The metadata 202 may include any combination of identifiable data that, via the training process, become conditioned variables on which the model’s outputs are focused to during inferencing. This process may be used to efficiently, effectively, and accurately, produce specific patient information. In this example, the metadata 202 may include:

[0075] 1. Numeric identifier of the note. This information may be extracted from a table in a heath information system that, among other things, stores and manages patient records and notes. This number identifier may assist the foundational LLM to associate details with the note that may be too large to fit with its context window. This also helps to train the foundational LLM to cite its sources of information it includes in its output.

[0076] 2. Patent Medical Record Number (MRN)

[0077] 3. Patient Name. This may comprise the patient's name, and may be used to train the model on patient name variations (e.g., “Robert'’ and “Bob"’).

[0078] 4. Date of birth. This may comprise the date the patient was bom, and may comprise any desired formatting convention.

[0079] 5. Age. This may comprise the age of the patient at the time that the note was generated. In some embodiments, the format of the age may comply with a standard format utilized in a clinician facing system. For example, if the patient is 27 months old, the format of this information may comprise “27 months”; and if the patient is 37 months old, the format of this information may comprise “3 years”. Other formatting schemes may also be implemented.

[0080] 6. Note Date. This may comprise the date the note was generated, and it enables the foundational LLM to develop a temporal relationship between notes and the state of medical treatment at that time.

[0081] 7. Note Category. This may describe the type of note, and it may be generated from a combination of one or more data elements spread across one or more tables in the health information system. An exemplary note category may comprise “Discharge summaiy "

[0082] 8. Author Name. This may capture the name of the final author of the note, and may train the foundational LLM on the individual documentation preferences of each author, thereby enabling the model to customize its responses to them.

[0083] 9. Department. This may comprise the medical department of the encounter in which(or for which) the note was generated. This information may provide context to the type of medical care being provided. Example: “VNJ-OP Speech.”

[0084] 10. Specialty. This may comprise a medical / treatment specialty of the department of the patient encounter and / or the specialty for which the note was generated. This information may be provided directly by the note’s author (e.g., via a clinician-facing interface) and / or extracted from the health information system. If not available, the system will utilize the specialty of the department in which the patient encounter occurred. Example: “Genetics.”

[0085] 11. Encounter Type. This may comprise the type of encounter for which the note was generated. This may provide further context to the model about the note. Example encounter types may include: “Hospital encounter” or “Telephone.”

[0086] Turning now to FIG. 3A, an exemplary system interface 300 for processing medical data queries and responses is shown. The system interface 300 includes a system prompt 301, an inquiry7302 and a model-generated response 303. In this example, the system prompt 301 includes instructions that define parameters for providing medical information and maintaining unbiased responses. The system prompt 301 establishes guidelines for the system's role as a medical assistant communicating with healthcare providers. In some embodiments, the system prompt 301 can be structured in the same manner used to train the foundational LLM.

[0087] Following the system prompt 201 is the inquiry 302. The inquiry 302 in this example requests a summary of medical problems for a specific patient, identified through metadata including a medical record number (MRN), patient name, and date of birth. More particularly, the inquiry is: “Your task is to summarize the medical problems of the patient described by the following meta data:###MRN[1234567]###Patient Name[First Name] [Last Name]###DOB| MM / DD / YYYY|"

[0088] For purposes of this illustration, we will assume that the patient is "‘John Doe”. bom on January 01. 2005. As a result of the pre-training discussed above (see e.g., FIG. 2), and without any further training or instruction, the foundational LLM in this example has generated an accurate, robust and succinct model-generated response 303 that provides diagnostic information. The model-generated response 303 lists medical conditions with corresponding diagnosis codes, including bowel obstruction, prematurity' details, Meckel's diverticulum, post- surgical status, undescended testis, intestinal perforation, short bowel syndrome, and neonatal hyperbilirubinemia.

[0089] As will be appreciated, the model-generated response 303 demonstrates that the foundational LLM has gained emergent abilities to answer questions about identifiable patients. Such demonstrated abilities include providing medical problem summarization and developmental milestone extraction, without explicit instruction or further training. Here, the foundational LLM was asked to summarize the medical problems of a specific patient (John Doe), and the foundational LLM generated the patient’s actual problem list that it learned from one or more inpatient progress notes. This inquiry 302 represents the first question asked to the newly pre-trained foundational LLM, yet the model -generated response 303 is accurate and succinct.

[0090] Notably, when asked the same inquiry 302 multiple times, the foundational LLM can slightly revise the model-generated response 303 (e.g.. provide as a list in alternative orders, exclude or include disease classification codes, etc.). It should also be noted that in this example, the foundational LLM was not explicitly instructed about what constitutes a patient in the health system, nor has it been trained on the concept of a medical record number. Instead, the foundational LLM Teamed' these concepts from the pre-appended metadata used during pre-training.

[0091] It is also noted that the foundational LLM gains an understanding of medical problems facing various ty pes of patients (e.g., pediatric patients vs. senior patients, etc.), as well as the customs of the health information system itself. For example, when asked to summarize an emergency department (ED) history and physical of a patient that the foundational LLM has never seen, it is able to generate responses in a form / style that is consistent with other system ED notes. Again, this is a result of learning forms and formats of various types of notes via the pre-training.

[0092] In another example, reference is now made to FIG. 3B. FIG. 3B illustrates another exemplary system interface 310 for processing medical data queries and responses. In thisexample, the system interface 310 includes a system prompt 304, an inquiry 305 and a modelgenerated response 306. The system prompt 304 defines parameters for providing medical information and maintaining unbiased responses, as well as establishing guidelines for the system's role as a medical assistant communicating with healthcare providers. In some embodiments, the system prompt 304 can be structured in the same manner used to train the foundational LLM.

[0093] The inquiry 305 requests medical decision making and a complete differential diagnosis with assessment and plan based on a patient's history and physical examination findings. In this example, the inquiry includes ‘‘Please provide medical decision making, complete differential diagnosis with assessment and plan for a patient presenting with this history and physical:’’

[0094] Following the inquiry 305 is a model-generated response 306, which provides diagnostic information and medical assessment. In particular, the response 306 describes a 4- year old male patient presenting with stomach pain, fever, and rash, along with physical exam findings and differential diagnoses including viral syndrome, strep phary ngitis, and UTI. The response 306 also includes an assessment of the patient's condition and recommendations for evaluation in the ED. Notably, the model-generated response 306 does not include any extraneous or non-responsive information, and it is provided in a form that is consistent with how such information is generally captured in medical records, notes or elsewhere in the health information system. This level of accuracy and succinctness is a result of the pre-training with metadata discussed above.

[0095] Turning now to FIG. 3C. an exemplary' system interface 320 for processing medical queries across multiple patients and responses. The system interface 320 includes three main sections arranged vertically, namely, a system prompt 307, a structured multi-patient inquiry 308, and a model-generated response 309. The system prompt 307 defines parameters for providing medical information and maintaining unbiased responses, as well as establishing guidelines for the system's role as a medical assistant communicating with healthcare providers. In some embodiments, the system prompt 307 can be structured in the same manner used to train the foundational LLM.

[0096] The multi-patient inquiry 308 includes information for five patients. Each patient entry follows the same format, with fields for MRN (shown as [NNNNNNNN]), Patient Name (shown with [First] [Middle] [Last] placeholders), and DOB formatted as NN / NN / YYYY. The inquiry 308 concludes with text indicating that medical records are available for each patient and requesting identification of a shared diagnosis among all five patients.

[0097] The model-generated response 309 definitively states, based on the available medical records, that the unifying diagnosis for these patients is "Loeys-Dietz syndrome, autosomal dominant." As with the other model-generated responses 303, 306 discussed above, thismodel-generated response 309 does not include any extraneous or non-responsive information, and this level of accuracy and succinctness are a result of the pre-training with metadata discussed above.

[0098] By contrast, reference is now made to FIG. 4. FIG. 4 illustrates another exemplary system interface 400 for processing medical queries and responses. The system interface includes a system prompt 401, an inquiry 402 and a model-generated response 403. However, contrary to the system interfaces discussed above and depicted in FIGs. 3A-3C, the system interface 400 of FIG. 4 pertains to a system comprising a foundational LLM that has not been pre-trained.

[0099] Notably, the system prompt 401 and inquiry 402 are identical as those pictured in FIG. 3B. However, in this example, the foundational LLM was not pre-trained, as noted above. As a result, the model-generated response 403 in this example comprises a long list with extraneous information. In particular, the model-generated response 403 includes a differential diagnosis listing conditions such as viral gastroenteritis, constipation, dehydration, aspiration, and urinary' tract infection, followed by a detailed plan addressing evaluation, hydration, pain management, monitoring, and follow-up recommendations.

[0100] Further, the model-generated response 403 fails to respond with any summarization of the diagnostic presentation, as would be the custom in the health system domain. This is to be contrasted with the accurate and succinct response 306 shown in FIG. 3B. As described above, the model-generated response 306 of FIG. 3B, for example, is a result of the pre-training techniques discussed herein.

[0101] Turning now to FIG. 5, an exemplary' system interface 500 for processing queries and responses is shown. This interface includes a system prompt 501, an inquiry' 502 and modelgenerated response 503. In this example, the inquiry 502 pertains to a domain or use case that is outside of the medical health use case (e.g., an “out-of-domain inquiry”). In particularly, the inquiry 502 requests information about the novel Moby Dick. Despite having been pre-trained for the medical health use case, the model-generated response 503 provides a description of the novel Moby Dick. The model-generated response 503 explains the story's plot, including details about the characters Ishmael and Ahab, and discusses themes of obsession, good and evil, and the unattainable. The response 503 concludes by noting the book's status as a significant American novel. As evident by this example, pre-training a foundational LLM as discussed herein will not cause the foundational LLM to ‘forget' how to address other inquiries related to other use cases or domains.

[0102] As will be appreciated, maintaining out-of-domain performance can allow the foundational LLM to retain flexibility to perform domain-specific tasks that are adjacent to those the LLM has been explicitly trained on. To do this, two key approaches may be adopted.The first may be characterized as “re-warming,” and it involves slowly increasing the learning rate of the LLM from zero over the first 10-30% of an epoch.

[0103] A technique referred as replay may also be adopted. This approach can involve training the LLM on out-of-domain data mixed in with the domain-specific training data, in this case clinical notes. Two key classes of replay may be used to maintain performance. The first may include unstructured text collected by researchers from across the internet. The unstructured text may be provided to the LLM with only a pre-appended beginning of text token and a final end of text token. This text may be bin-packed and attention masked in the same manner as clinical text, discussed herein.

[0104] The second replay data type may include structured question and answer data. Questions may be ingested from a wide variety of open-source datasets including medical domain questions. However, because the quality of the human answers of these datasets may be poor in some cases, responses may instead be generated from a much larger foundational LLM. The combined human-generated question and LLM generated answer may then be used as the replay data. In this way, question and answ er abilities of the LLM may be maintained while simultaneously distilling the abilities of the larger foundational LLM into the base LLM being trained. The questions and answers may be provided in the standard "chat template" of the base LLM being trained, including role and system headers as appropriate.

[0105] Turning now to FIG. 6, a pre-training architecture 600 for processing and training a large language model is shown. The pre-training architecture 600 includes an operational database 602 containing patient data and medical records. In some cases, the operational database 602 may comprise an electronic health record (EHR) database storing clinical notes and other patient information (e.g., patient demographics, clinical encounter, note metadata, note text, etc.).

[0106] From the operational database 602, data flows to a corpus generation module 604, which processes the data into a training corpus. In some cases, the corpus generation module 604 may perform filtering and additional operations to ensure the model is pre-trained on the most relevant clinical data. For example, the corpus generation module 604 may filter out certain types of notes or focus on notes from specific medical specialties.

[0107] The output from the corpus generation module 604 moves to a metadata preappending module 606, where metadata is added to each note in the corpus. The metadata preappending module 606 may pre-append metadata such as patient identifiers, note dates, author information, etc. to provide context to the foundational LLM about each note.

[0108] Following metadata pre-appending, the data flows to a note processing module 608. The note processing module 608 contains two sub-components: a note splitting module 610 and a bin packing module 612. The note splitting module 610 can be configured to dividelonger notes into smaller portions that fit within the context window size of the LLM. The bin packing module 612 can be configured to combine shorter notes or note portions into available context windows to optimize utilization of the context windows.

[0109] In some cases, the bin packing module 612 may implement a bin packing process to combine smaller notes or note portions into larger context windows. This process improves context window utilization and, in turn, computational efficiency, particularly when pretraining the foundational LLM on a large corpus of text.

[0110] From the note processing module 608. the data moves to a tokenization module 614, which converts the processed notes into tokens. The tokenized data then flows to an institutional LLM 616 for pre-training.[OHl] The institutional LLM 616 may be connected to a fine-tuning module 618, which processes and refines the model's performance. In some cases, the fine-tuning module 618 may use general-purpose instructions to help maintain the LLM's performance on out-of-domain tasks. For example, as new notes are generated or become available, the institutional LLM 616 may be updated or re-pretrained using the newly generated / newly available notes via the pretraining process described herein. The updating / re-pretraining may occur continuously, periodically or according to a pre-determined schedule. In some instances, updating the institutional LLM 616 may also include pruning, which can involve removing portions of the model that do not provide useful information for a particular task. Pruning may be implemented using approaches such as SparseGPT or Wanda, for example. An iterative parameter pruning approach may also be used, where parameters with low absolute magnitude and small gradients are incrementally removed while retraining and measuring performance on example tasks. This pruning process can be repeated until an appropriate balance of computational efficiency and task performance is achieved. In some cases, pruning may improve performance on certain tasks.

[0112] The fine-tuning module 618 may also be connected to API interfaces 620, which provide access points for external systems and applications. A user interface 622 (e.g., such as a system-generated user facing interface rendered on a user device) can enable users to interact with the institutional LLM 616 (e.g.. to submit inquiries, and provide feedback). The feedback provided by the user can then be used to improve model performance through reinforcement learning, for example.

[0113] In some cases, the pre-training architecture 600 may use attention masking to prevent the model from getting confused about multiple patients' notes packed into the same context window. This technique may help maintain the logical separation between different patients' data while still allowing for efficient use of context windows.

[0114] Turning now to FIG. 7, a flowchart of an exemplary method 700 for processing clinical notes and preparing them for pre-training a large language model is shown. The method 700 include steps that process and optimize clinical data for effective model pre-training.

[0115] The method 700 begins with step 702, where patient data is retrieved from an electronic health record (EHR) database. This step may involve accessing and extracting various t pes of patient information, including clinical notes, demographic data, and medical histories.

[0116] In step 704, the method 700 proceeds to generate a clinical text corpus from the retrieved patient data. This step may involve organizing and structuring the raw patient data into a format suitable for further processing and model training.

[0117] The method 700 then moves to step 706, where metadata is pre-appended to each note in the corpus. This metadata may include information such as patient identifiers, note dates, author information, and other relevant contextual data that can help the model understand the structure and origin of each clinical note.

[0118] Following metadata prepending, step 708 involves optimizing context windows for the notes. This step may aim to efficiently organize the clinical notes within the context windows that will be used during model pre-training.

[0119] At decision step 710, the method 700 determines if notes are longer than the context window size. This decision point allows the method to handle notes of varying lengths appropriately.

[0120] If the notes are longer than the context window (Yes branch from step 710), the method 700 proceeds to step 712, where the notes are split across multiple context windows. This step ensures that all relevant information from longer notes is retained and properly processed.

[0121] If the notes are not longer than the context window (No branch from step 710), the method 700 moves to step 714, where shorter notes are combined into a single context window. This step helps optimize the use of available context window space and improve computational efficiency.

[0122] From either step 712 or 714, the method 700 proceeds to step 716, where the processed clinical notes are tokenized. Tokenization can involve converting the text data into a format that can be directly input into the large language model.

[0123] The process then moves to step 718. where the large language model (LLM) is pretrained on the tokenized dataset. This step may involve exposing the model to the processed clinical data, allowing it to learn patterns and relationships within the medical information.

[0124] The method 700 concludes with step 720, where the LLM is fine-tuned with general- purpose instructions. This final step may help maintain the model's performance on out-of- domain tasks while optimizing its capabilities for medical-specific applications.

[0125] Throughout the method 700, the flowchart shows the sequential flow of data processing and model training steps, w ith arrows indicating the progression between steps and decision points. This structured approach allows for efficient and effective preparation of clinical data for training large language models.

[0126] As noted above (and referenced in FIG. 7), the present disclosure provides methods for optimizing context windows when processing clinical notes for pre-training large language models. These methods address challenges associated with notes of varying lengths and aim to maximize the utilization of context windows.

[0127] Turning to FIG. 8, a flowchart illustrates a method 800 for preparing clinical notes for pre-training a large language model. The method 800 begins with step 802, where multiple clinical notes of varying sizes are received. In step 804, metadata headers are pre-appended to each note. This step may involve adding relevant information such as the note identifier, medical record number, and other fields from the metadata header, as described below in relation to the clinical note 1000 illustrated in FIG. 1 .

[0128] The method 800 then proceeds to step 806, where a context window7size is determined. This size may be based on the specifications of the institutional LLM 616 (see FIG. 6) being trained. Follow ing this, in step 808, a first note is selected for processing.

[0129] At decision step 810, the method 800 determines if the note fits in the current context window. If the note fits (Yes branch), the method 800 proceeds to step 812, where the note is added to the current context window. If the note does not fit (No branch), the method 800 moves to step 814, where a new context window is started with the note.

[0130] From step 812, the method 800 proceeds to decision step 816, where it determines if there are more notes to process. Similarly, from step 814, the method 800 moves to decision step 818 to check if there are more notes to process.

[0131] If there are more notes to process from either decision step (Yes branch), the method 800 moves to step 820 or step 822 respectively, where the next note is selected. The process then returns to decision step 810 to evaluate the newly selected note.

[0132] If there are no more notes to process from either decision step (No branch), the method 800 proceeds to step 824, where the packed context windows are finalized for pretraining.

[0133] For handling longer clinical notes that exceed the context window size, FIG. 9 illustrates a flowchart of a method 900 for processing such notes. The method 900 begins withstep 902, where a large clinical note is received. In step 904, a context window size is determined, similar to step 806 in method 800.

[0134] The method 900 then moves to step 906, where metadata headers are pre-appended to the note. This step may involve adding relevant information from the metadata header to provide context for the note content.

[0135] In step 908, the method 900 involves splitting the note into note portions that fit within the context window including, optionally, after each portion has been pre-appended with its copy of the metadata header. This step ensures that all information from longer notes is retained and properly processed.

[0136] At decision step 910, the method 900 determines if a note portion including, optionally, its copy of the metadata fits in the current context window. If the note portion fits (Yes branch), the method 900 proceeds to step 912, where the note portion is added to the current context window. If the note portion does not fit (No branch), the method 900 moves to step 914, where a new context window is started with the note portion.

[0137] From step 912, the method 900 proceeds to decision step 916. where it determines if there are more portions of the note to process. Similarly, from step 914, the method 900 moves to decision step 918 to check if there are more portions of the note to process.

[0138] If there are more portions to process from either decision step (Yes branch), the method 900 moves to step 920 or step 922 respectively, where the next note portion is selected. The process then returns to decision step 910 to evaluate the newly selected note portion.

[0139] If there are no more portions to process from either decision step (No branch), the method 900 proceeds to step 924, where the context windows are finalized for training.

[0140] By implementing these methods, utilization of context windows can be optimized for both short and long clinical notes. These approaches ensure that all relevant information from the clinical notes is retained and efficiently processed during the pre-training of the institutional LLM 616, thereby leading to improved model performance and more effective use of computational resources.

[0141] Turning now to FIG. 10, a block diagram illustrates an exemplary structure of a clinical note 1000 that has been pre-appended with metadata, as described herein. The clinical note 1000 in this example includes a metadata header 1002 and note content 1026. The metadata header 1002 comprises multiple hierarchically arranged data fields that provide context and information about the clinical note 1000.

[0142] At the top of the metadata header 1002 in this example is a note identifier 1004. The note identifier 1004 may be a unique numeric or alphanumeric code assigned to the clinicalnote 1000. Below the note identifier 1004 is a medical record number 1006, which may correspond to a unique identifier for the patient associated with the clinical note 1000.

[0143] Following the medical record number 1006 is a patient name field 1008. The patient name field 1008 may contain the full name of the patient. The metadata header 1002 continues with a birth date field 1010 and a patient age field 1012. The birth date field 1010 may store the patient's date of birth, while the patient age field 1012 may contain the patient's age at the time the clinical note 1000 was created.

[0144] The metadata header 1002 can further include a note date field 1014 and a note category field 1016. The note date field 1014 may indicate when the clinical note 1000 was created or last modified. The note category field 1016 may specify the type of clinical note, such as "Discharge Summary" or "Progress Note."

[0145] An author name field 1018 can also be included in the metadata header 1002 and include the name of the healthcare provider who created or last edited the clinical note 1000.

[0146] The metadata header 1002 can also contains a department identifier 1020 and a specialty identifier 1022. The department identifier 1020 may indicate the medical department associated with the clinical note 1000, while the specialty identifier 1022 may specify the medical specialty relevant to the note content.

[0147] At the bottom of this exemplary metadata header 1002 is an encounter type field 1024. The encounter type field 1024 may describe the nature of the patient encounter, such as "Outpatient Visit" or "Emergency Department."

[0148] The note content 1026 is what comprises the actual clinical information, observations, and assessments recorded by the healthcare provider.

[0149] During a pre-training process for the institutional LLM 616, each field in the metadata header 1002 may play an impactful role. For example, the note identifier 1004 can help the institutional LLM 616 associate specific information with particular notes, even across multiple context windows. The medical record number 1006 and patient name field 1008 can enable the institutional LLM 616 to link information to specific patients and understand name variations.

[0150] The birth date field 1010 and patient age field 1012 can enable the institutional LLM 616 to contextualize medical information based on the patient's age at the time of the note, while the note date field 1014 can help the institutional LLM 616 develop a temporal understanding of a patient's medical history.[01511 The note category field 1016 can assist the institutional LLM 616 in understanding the context and purpose of different types of clinical documentation and the author name field 1018 can enable the institutional LLM 616 to leam individual provider documentation styles and preferences.

[0152] The department identifier 1020 and specialty identifier 1022 can provide the institutional LLM 616 with context about the medical setting and area of expertise relevant to the note content. The encounter type field 1024 can help the institutional LLM 616 understand the circumstances under which the clinical information was recorded.

[0153] By incorporating these metadata fields in the pre-training process, the institutional LLM 616 can develop a nuanced understanding of the structure and context of clinical notes. This understanding can enable the institutional LLM 61 to generate more accurate and contextually appropriate responses when queried about specific patients or medical scenarios.

[0154] Notably, the metadata header 1002 may include alternative or additional fields beyond those described above and depicted in FIG. 10. The specific fields included in the metadata header 1002 can vary depending on the particular healthcare system, institutional requirements, or the specific goals of the pre-training process. This flexibility allows for customization of the metadata structure to best suit the needs of different clinical environments and use cases. In other embodiments, the metadata may be in different hierarchies or provided in random order to prevent the model from becoming dependent on a particular metadata structure.

[0155] Turning now to FIG. 11, a block diagram illustrates how patient metadata 1101 is connected to its semantic meaning 1102. The figure shows three key pieces of patient metadata 1101: Medical RecordNumber (51234567), PatientName (Robert A. Smith), and Date of Birth (01 / 01 / 2000). Each piece of patient metadata 1101 has associated numerical values shown in brackets below it. Curved arrows connect these metadata elements 1101 to a text box representing their semantic meaning 1102.

[0156] The semantic meaning 1102 is depicted as a natural language description of the patient, including details about their age, medical conditions, and surgical history. This representation demonstrates how a healthcare system interprets and contextualizes the structured patient identifiers.

[0157] In some aspects, the Medical Record Number can serve as a unique identifier for the patient within the healthcare system. The Patient Name field can allow the system to recognize and process different name variations or formats. The Date of Birth information can be used to calculate the patient's current age and provide context for age-related medical considerations.

[0158] The semantic meaning 1102 can include, as shown here, a summary of the patient's relevant medical information, synthesized from various clinical notes and records associated with the patient. This can encompass details such as the patient's current age, significant medical conditions, past surgical procedures, and other pertinent health information.

[0159] The curved arrows connecting the metadata 1101 to the semantic meaning 1102 represents the system's ability to transform structured data into meaningful, context-richinformation. This process may involve algorithmic and machine learning techniques that analyze and interpret the patient's entire medical history to generate a comprehensive understanding of their health status.

[0160] In some cases, the system may use this semantic representation to provide more accurate and contextually appropriate responses to queries about the patient. By linking structured identifiers to their broader clinical context, the system may be able to generate more nuanced and patient-specific information when prompted.

[0161] This figure effectively illustrates how the pre-training process enables the institutional LLM to develop a deep understanding of patient data, going beyond simple recognition of identifiers to a comprehensive interpretation of a patient's medical context. This capability may significantly enhance the LLM's ability to provide relevant and personalized responses in various clinical scenarios.

[0162] The systems and methods described herein also provide for continuously updating pre-trained LLMs as new information becomes available. This continuous update process ensures that the institutional LLM 616, for example, remains current and can incorporate newly generated patient data into its knowledge base.

[0163] Turning to FIG. 12, a block diagram illustrates a system (or sub-system) 1200 for continuously updating the institutional LLM 616. The system 1200 includes an EHR Data Monitor 1202 that itself can comprise a New Data Detector 1204 and a Data Validator 1206.

[0164] The New Data Detector 1204 can be configured to continuously monitor the operational database 602 for newly added or modified patient data. In some cases, the New Data Detector 1204 can use timestamps or other metadata to identify new or updated records. The Data Validator 1206 can be configured for verifying the integrity and quality of the newly detected data, ensuring that only valid and relevant information is processed for updating the institutional LLM 616.

[0165] The system 1200 further includes a Data Processor 1208 comprised of a Metadata Pre-appender 1210, a Context Window Optimizer 1212, and a Tokenizer 1214. The Metadata Pre-appender 1210 can add relevant metadata to new clinical notes, similar to the process described above for the metadata header 1002 in FIG. 10. The Context Window Optimizer 1212 can implement the methods 800 and 900 described in FIGs. 8 and 9 to efficiently organize the new data within context windows. The Tokenizer 1214 can convert the processed text into a format suitable for input into the institutional LLM 616.

[0166] An LLM Updater 1216 comprising a Model Retrainer 1218 and a Performance Validator 1220 can provide additional functions to the system 1200. The Model Retrainer 1218, for example, can be configured for incorporating the new, processed data into the existing institutional LLM 616. This can involve techniques such as fine-tuning and / or continued pre-training on the new data. The Performance Validator 1220 can evaluate the updated model's performance to ensure that the incorporation of new data has not negatively impacted the model's capabilities.

[0167] The components of the system 1200 are shown connected by solid arrows showing the flow of data through the system 1200, with a dashed line connecting back to the system 1200 from the Performance Validator 1220. This feedback loop allows for continuous monitoring and improvement of the institutional LLM 616's performance.

[0168] FIG. 13 illustrates a flowchart of a method 1300 for monitoring and processing electronic health record (EHR) data for new patient information in a healthcare system. The method 1300 begins with step 1302. where the system monitors an EHR for new patient data. This step can correspond to the function of the New Data Detector 1204 in FIG. 12.

[0169] The method 1300 includes a decision point at a step 1304 to determine if new data is available. If no new data is available, the method 1300 returns to the step 1302 to continue monitoring. If new data is available, the method 1300 proceeds to a step 1306.

[0170] At the step 1306, the method 1300 processes the new data into a clinical text corpus. This step can involve the functions of the Data Validator 1206 and the initial stages of the Data Processor 1208 from FIG. 12. The method 1300 then moves to a step 1308, where metadata is pre-appended to the new notes, corresponding to the function of the Metadata Pre-appender 1210.

[0171] Following metadata pre-appending, the method 1300 proceeds to a step 1310, where context windows are optimized for the new data, which can be performed by the Context Window Optimizer 1212. The method 1300 then advances to step 1312. where the new clinical notes are tokenized, corresponding to the function of the Tokenizer 1214.

[0172] From tokenization, the method 1300 moves to a step 1314, where the institutional LLM 616 is updated with the new data. This step can involve the Model Retrainer 1218 from FIG. 12. The method 1300 concludes with a step 1316, where the updated institutional LLM 616 performance is validated, which can be performed by the Performance Validator 1220.

[0173] The flowchart shows the sequential flow of data processing steps, with arrows indicating the progression between steps and decision points. A feedback loop is shown from the step 1316 back to the step 1302, allowing for continuous monitoring and processing of new patient data.

[0174] In some cases, the system can use a pruning process to remove portions of the institutional LLM 616 that do not provide useful information for a particular task. This pruning process can be implemented as part of the Model Retrainer 1218 or as a separate component within the LLM Updater 1216. The pruning process can involve techniques such as weightthresholding, where model parameters below a certain magnitude are removed, or more sophisticated methods that consider the impact of parameter removal on specific tasks.

[0175] By implementing this continuous update system, the institutional LLM 616 can maintain its relevance and accuracy over time, incorporating new patient data and medical knowledge as it becomes available. This approach helps ensure that the model 616 remains up- to-date and capable of providing accurate and contextually appropriate responses in various clinical scenarios. It also improves training efficiency by avoiding training the model from scratch.

[0176] In some embodiments, the present disclosure provides systems and methods for processing patient-specific queries using a sparse autoencoder system in conjunction with a pre-trained large language model. These systems and methods enable efficient extraction and utilization of patient-specific features when generating responses to medical queries.

[0177] Turning now to FIG. 14, a block diagram illustrates such a system 1400 configured for processing patient-specific queries. The system 1400 includes a sparse autoencoder system 1401 and a pre-trained language model 1420.

[0178] The sparse autoencoder system 1401 comprises an input layer 1402, a hidden layer 1404, and an output layer 1406. In some cases, the input layer 1402 can be configured receive input data related to patient information or queries. The hidden layer 1404 can process the input data, extracting relevant features developed in the activation patterns of the hidden layer 1404. The output layer 1406 can then generate output to closely match the input data based on the processed data from the hidden layer 1404. That is, the sparse autoencoder system 1401 can be trained such that the output matches the input data as much as possible. This enables the individual neurons of the hidden layer 1404 take on meaning that is as discrete as possible (e.g., to be patient-specific).

[0179] Below the output layer 1406. the sparse autoencoder system 1401 can include a patient feature extractor 1408. The patient feature extractor 1408 can be configured to identify and extract patient-specific features from activations of the hidden layer 1404. These extracted features can represent key aspects of a patient's medical history, current condition, or other relevant information, for example.

[0180] The pre-trained language model 1420 includes a query processor 1410 that may receive processed queries from the sparse autoencoder system 1401. Within the pre-trained language model 1420, a response generator 1412 can produce responses based on the processed queries and extracted patient features.

[0181] A patient feature activator 1414 within the pre-trained language model 1420 can interact with the response generator 1412. The patient feature activator 1414 can incorporatethe patient-specific features extracted by the patient feature extractor 1408 into the generated responses, ensuring that the output is tailored to the specific patient in question.

[0182] FIG. 15 illustrates a more detailed block diagram of a query processing system 1500 that incorporates the sparse autoencoder approach. The query7processing system 1500 includes a query7interface 1502 positioned at the top of the diagram for receiving queries from users or other systems.

[0183] The query processing system 1500 contains a pre-trained large language model 1504 that processes the queries received through the query interface 1502. Within the pretrained model 1504 are three components: a feature activator 1506, a query processor 1507, and a response generator 1508. The feature activator 1506 and query processor 1507 can process incoming queries, while the response generator 1508 can produce outputs based on the processed queries.

[0184] Below the pretrained model 1504 is a sparse autoencoder 1510. The sparse autoencoder 1510 includes a feature extractor 1512 and a feature compressor 1514. The feature extractor 1512 can extract relevant features from the processed data, focusing on patientspecific information. The feature compressor 1514 can compress these extracted features for efficient processing and storage.

[0185] An output interface 1516 can be configured for receiving processed information from the response generator 1508 of the pre-trained large language model 1504. The output interface 1516 can present the generated responses to users or other systems.

[0186] In some cases, the sparse autoencoder 1510 can employ a sparsity constraint during training, which can force most neurons to remain inactive for typical inputs. This sparsity constraint can help the model learn distinct, more interpretable features that correspond to specific aspects of patient data or medical conditions.

[0187] The feature extractor 1512 within the sparse autoencoder 1510 can be designed to identify and isolate patient-specific features from the input data. These features can represent various aspects of a patient's medical history7, current symptoms, or other relevant information that can be used to generate more accurate and personalized responses.

[0188] The feature compressor 1514 can work in conj unction with the feature extractor 1512 to reduce the dimensionality of the extracted features. This compression can help in efficient storage and processing of patient-specific information, while still retaining the most important aspects of the data.

[0189] By incorporating the sparse autoencoder 1510 into the query processing system 1500, the system 1500 can, more effectively, identify and utilize patient-specific information when generating responses to medical queries. As will be appreciated, this approach leads to morepersonalized and accurate responses, thereby improving the quality of medical decision support provided by the system.

[0190] As noted above, the query processing system 1500 of FIG. 15 can include a query interface 1502 for receiving user queries. In some cases, the query interface 1502 can be configured to accept various ty pes of medical inquiries from healthcare providers or other authorized users.

[0191] FIG. 16 illustrates a sequence diagram depicting the workflow for processing patientspecific queries and monitoring data access patterns. The sequence begins with a query submission step SI. where a user submits a patient-specific query to a pre-trained large language model (e.g., large language model 1420 of FIG. 14).

[0192] Following query submission, a query7processing step S2 occurs, where the pre-trained large language model 1420 begins processing the received query7. Simultaneously, an activation monitoring step S3 can be initiated, during which a sparse autoencoder system 1401 starts monitoring the activation patterns of the pre-trained language model 1420 to track which patient data is being accessed.

[0193] During query processing, a feature activation step S4 can take place, where the pretrained language model 1420 activates relevant patient features. The sparse autoencoder system 1401 observes and logs these activated features in a feature logging step S5. This logging process allows the system to keep track of which specific patient information is being accessed and utilized in generating a response.

[0194] Based on the activated features, a response generation step S6 occurs, where the pretrained language model 1420 generates a response to the user query. The generated response can then be returned to the user in a response return step S7.

[0195] After the interaction is complete, a report compilation step S8 can take place, where the sparse autoencoder system 1401 compiles a report of all accessed patient data. This report can then be sent to an auditing system in a report transmission step S9, allowing for comprehensive tracking and monitoring of patient data access.

[0196] FIG. 17 illustrates a flowchart of a method 1700 for processing patient-specific queries. The method 1700 begins with a step 1702, where a patient-specific query is received through the query interface (e.g., query interface 1502 of FIG. 15). At step 1704, the query is processed using the pre-trained language model 1504. which has been trained on patient data.

[0197] Following query processing, a step 1706 involves activating relevant patient-specific features based on the user query. This activation can be performed by the feature activator 1506 of FIG. 15 within the pre-trained language model 1504.

[0198] In a step 1708, a response is generated based on the activated features. This step can be carried out by the response generator 1508 within the pre-trained language model 1504. The method 1700 then proceeds to a step 1710, where the system evaluates whether the generated response is within the scope of practice.

[0199] If the response is determined to be within the scope of practice, the method 1700 moves to a step 1712, where the response is provided to the healthcare provider through the output interface 1516. If the response is not within the scope of practice, the method 1700 proceeds to a step 1714, where additional authorization may be requested before providing the response.

[0200] In a step 1716, the accessed patient information is logged, allowing for comprehensive tracking of data access. This logging process can correspond to the feature logging step S5 and report compilation step S8 illustrated in FIG. 16.

[0201] FIG. 18 provides a visual representation of how a pre-trained language model (e.g., large language model 1504 of FIG. 15) processes and responds to queries about patients with specific medical conditions. A first response window 1801 shows a first user query 1801a requesting a list of patients with a particular condition. Below this query, multiple patient feature boxes are displayed, representing different patients that match the query criteria.

[0202] A second response window 1802 demonstrates a more focused state of the model's attention. In this window, a second user query 1802a is shown, and only one patient feature box is prominently displayed. This illustrates how the model 1504 transitions from considering multiple potential patients to focusing on specific patient information when generating its response.

[0203] By implementing this query processing and patient data access monitoring system, healthcare providers are able to efficiently access relevant patient information while maintaining proper data access controls and audit trails. The combination of the pre-trained language model 1504 and the sparse autoencoder 1510 allows for both accurate response generation and comprehensive tracking of accessed patient data.

[0204] Turning now to FIG. 19, a flowchart of a method 1900 for creating fine-tuning data to induce patient-specific reasoning in a large language model. The method 1900 can provide an approach to train the model to reason about patients using its internal knowledge, rather than relying solely on provided context.

[0205] The method 1900 begins with step 1901, where a patient of interest is selected. This selection can be based on various criteria, such as the complexity of the patient's medical history or the relevance to specific medical conditions under study.

[0206] In step 1902, the method 1900 retrieves information about the selected patient that is relevant to a particular context. This information can be obtained from an electronic healthrecord (EHR) database and can include clinical notes, test results, medication history, and other pertinent medical data.

[0207] The process then moves to step 1903, where a pre-trained large language model (LLM) is directed to develop a patient-specific question based on the retrieved information. This step can leverage the model's ability7to generate contextually relevant queries based on the provided patient data.

[0208] Following the question generation, step 1904A involves extracting the patientspecific question from the LLM's output. This extracted question can serve as the basis for subsequent reasoning steps.

[0209] In step 1904B, the LLM is directed to answer the previously created question using the same clinical information as context. During this step, the model can generate reasoning framed in the context of reviewing the provided patient information.

[0210] The method 1900 then proceeds to step 1905 A, where the reasoning chain-of-thought is extracted from the model's output. This extracted reasoning can represent the model's thought process based on the provided context.

[0211] Step 1905B involves directing the LLM to re-frame the extracted reasoning such that it is based on its own internal knowledge, rather than the provided context. For example, the model can be instructed to change phrases like "Let me start by going through the provided notes" to "Let me start by reviewing what I know about the patient."

[0212] In step 1906 A, the re-framed reasoning chain-of-thought is extracted from the model's output. This re-framed reasoning can demonstrate the model's ability7to utilize its internal knowledge for patient-specific reasoning.

[0213] Step 1906B combines the re-framed reasoning chain-of-thought with the extracted question and answer to generate a piece of training data. This combined data can serve as fine- tuning (training) data for further training of the institutional LLM 616.

[0214] The method 1900 includes a decision point at step 1907, where it determines whether to repeat the process. If the process is to be repeated, the method returns to step 1901 to select another patient. This iteration can allow for the creation of a large corpus of questions and answers demonstrating reasoning in the style of the particular foundational LLM.

[0215] If the process is not to be repeated, the method advances to step 1908, where the generated data is used for LLM fine-tuning. This fine-tuning process can help induce patientspecific reasoning capabilities in the model.

[0216] By implementing this method 1900, the LLM can develop the ability7to reason about patient cases using its internal knowledge, potentially leading to more accurate and contextually appropriate responses in clinical settings. The iterative nature of the process canallow for the creation of a diverse set of training examples, covering a wide range of patient scenarios and medical conditions.

[0217] FIG. 20 illustrates a system interface 2000 for processing medical queries and responses using a large language model that has been pre-trained and fine-tuned to reason using internal patient-specific knowledge. The system interface 2000 displays three main sections arranged vertically : an inquiry 2001 , a chain-of-thought reasoning 2002, and a model-generated response 2003.

[0218] The inquiry 2001 at the top of the interface presents a user query requesting information about why a specific patient required a liver transplant. The query includes standard medical record formatting with fields for date of birth (DOB), patient name, and medical record number (MRN), using placeholder text to maintain patient privacy.

[0219] Below the inquiry 2001 is the model's chain-of-thought reasoning 2002, enclosed within <think> tags. This section demonstrates the model's internal reasoning process, where it systematically reviews its learned knowledge about the patient without relying on additional external context. The model recalls that the patient was diagnosed with propionic acidemia (PA) at a young age, explaining that PA is a metabolic disorder affecting amino acid breakdown and liver function. The reasoning continues by detailing how the patient's condition was managed with medications and dietary restrictions but required multiple hospitalizations for metabolic crises. The model notes that a liver biopsy confirmed significant damage from PA, supporting the need for transplantation.

[0220] This chain-of-thought reasoning 2002 can help reduce hallucinations by grounding the model's response in specific, recalled information rather than generating potentially inaccurate details. Additionally, this approach improves performance on complex tasks by breaking down the reasoning process into smaller, more manageable steps.

[0221] At the bottom of the interface 2000 is the model-generated response 2003, which provides a concise and informative answer to the user's inquiry 2001. The response 2003 explains that the liver transplant was necessary' because the patient suffered from propionic acidemia (PA), which caused significant liver damage confirmed through biopsy. It further elaborates that despite medications and dietary restrictions, the condition led to multiple hospitalizations for metabolic crises, ultimately requiring transplantation to address severe liver dysfunction and prevent further complications.

[0222] The model-generated response 2003 demonstrates the effectiveness of the fine-tuning process described in FIG. 19, as it showcases the model's ability to reason about patient cases using its internal knowledge. This approach can lead to more accurate and contextually appropriate responses in clinical settings, as the model can draw upon its learned patientspecific information to provide detailed and relevant answers.

[0223] FIG. 21 illustrates a flowchart of a method 2100 for developing clinical reasoning in a large language model (LLM). The method 2100 begins with step 2101. where the LLM receives a query for patient-specific information. This query can be related to a patient's medical history, diagnosis, treatment plan, or other relevant clinical data.

[0224] The method 2100 proceeds to step 2102, where the model generates one or more outputs in response to the query. These outputs can include clinical assessments, treatment recommendations, or explanations of patient conditions based on the model's current knowledge and reasoning capabilities.

[0225] From step 2102, the method 2100 moves to step 2103, where the model's output is compared to a pre-determined standard (e.g., a "‘gold standard") from an EHR database or expert. This comparison can be performed programmatically, using a separate adjudication LLM, or through other means to assess the accuracy and quality of the model's response.

[0226] Following the comparison, the method 2100 advances to step 2104, where a reward or penalty is determined based on the output accuracy. The reward / penalty function can consider factors such as the correctness of the clinical reasoning, the appropriateness of the recommendations, and the clarity of the explanation provided by the model.

[0227] The method 2100 then proceeds to step 2105, where the model parameters can be updated based on the rew ard or penalty' signal. This step can involve adjusting the weights and biases of the neural netw ork to reinforce accurate responses and discourage inaccurate ones. Techniques such as direct policy optimization or group relative policy optimization can be used in this step to optimize the model's performance.

[0228] From step 2105, the method 2100 moves to decision step 2106, where it determines if sufficient iterations have been completed. This decision can be based on various factors, such as the overall improvement in the model's performance, the consistency of accurate responses, or a predetermined number of training cycles.

[0229] If sufficient iterations have not been completed, the method 2100 returns to step 2102 to generate additional outputs, creating a feedback loop that allows the model to continuously refine its reasoning capabilities. This iterative process can help the model develop more sophisticated and accurate clinical reasoning over time.

[0230] If sufficient iterations have been completed, the method 2100 advances to step 2107, where the model exhibits a particular clinical reasoning style. This step represents the culmination of the training process, where the model has learned to generate high-quality', patient-specific responses that align closely with expert clinical reasoning.

[0231] By implementing this method, the LLM can develop the ability' to reason about patient cases in a manner that closely mimics expert clinical decision-making. The iterative nature of the process, combined with the use of reinforcement learning techniques, can allow the modelto continuously improve its performance and adapt to complex clinical scenarios. This approach can lead to more accurate, contextually appropriate, and clinically relevant responses in healthcare applications.

[0232] The systems and methods described herein integrate multiple components to enable efficient processing of patient data, pre-training of large language models, and handling of medical queries. This integrated approach allows for continuous improvement of the model's performance while maintaining patient data privacy and security.

[0233] The integrated approach begins with the ingestion of clinical notes and related patient data from electronic health record systems. These notes undergo preprocessing, which includes metadata extraction and appending. The preprocessed notes are then tokenized and organized into optimized context windows, taking into account the specific requirements of the large language model being used.

[0234] Once the data is prepared, it is used to pre-train the large language model. This pretraining process allows the model to learn from a vast corpus of patient data, developing an understanding of medical terminology, common and rare conditions, and treatment patterns. The pre-trained model may then be fine-tuned using general-purpose instructions to maintain its performance both on in-domain and out-of-domain tasks.

[0235] When a healthcare provider submits a query through the system's interface, the query is processed by the pre-trained language model. The model activates relevant patient-specific features based on the uery content. A sparse autoencoder system monitors these activations, logging which patient data is being accessed during the query' processing.

[0236] The language model generates a response based on the activated features and the query content. Before returning the response to the healthcare provider, the system can evaluate whether the generated response falls within the appropriate scope of practice. If additional authorization is required, the system can prompt for it before providing the response.

[0237] Throughout this process, the sparse autoencoder system compiles reports on accessed patient data, which may be sent to an auditing system for comprehensive tracking and monitoring of data access patterns. This ensures compliance with privacy regulations and allows for detailed auditing of system usage.

[0238] The sparse autoencoder can influence the model's behavior by selectively activating or suppressing certain features extracted from patient data. This capability allows the system to focus the model's attention on relevant patient information while reducing the impact of less relevant or noisy data. By modulating feature activation, the sparse autoencoder can help guide the model towards generating more targeted and contextually appropriate responses.

[0239] In parallel with query handling, the system can continuously monitor for new patient data. When new data becomes available, it undergoes the same preprocessing steps as the initialdata corpus. The pre-trained language model can then be updated with this new information, allowing it to incorporate the latest patient data and medical knowledge into its responses.

[0240] In some embodiments, the pre-trained large language model can be updated at regular intervals, based on newly generated patient data that has been combined with other types of data, most notably, replay data of the same classes as described above. Existing patient-specific data, such as older patient notes, can also be utilized in the re-training process. Despite replay approaches, this approach (also referred to as continuous integration training) can create recency bias in the model's output.

[0241] The system can also employ pruning techniques to remove less useful portions of the model, optimizing its performance and computational efficiency. This pruning process can be performed periodically or as part of the continuous update cycle.

[0242] By integrating these various components and processes, the system provides a comprehensive solution for handling medical queries, maintaining up-to-date knowledge, and ensuring responsible use of patient data. The continuous learning and updating capabilities of the system allow it to adapt to new medical information and changing patient conditions over time, potentially improving the quality’ of care provided to patients.

[0243] In addition, the pre-training approaches and techniques described herein reduce system complexity by encoding patient information directly into the model parameters. This, in turn, improves information retrieval performance without requiring a separate real-time retrieval system. Further, a pre-trained model's internal knowledge of data structures and patient information can enable it to direct its own retrieval processes, more effectively in some cases. By encoding patient histories in the model weights, for example, the approaches described herein can expand the scope of information the pre-trained model can leverage beyond traditional context window limitations. This could enable the pre-trained model to access information about large patient cohorts during inference.

[0244] The following examples pertain to further embodiments.

[0245] Example 1 is a system for pre-training a large language model. The system includes an electronic health record (EHR) database storing patient-specific data, a processor, and a memory storing instructions. When executed, the instructions cause the system to retrieve patient data from the EHR database, generate a clinical text corpus from the retrieved data, preappend metadata to each note in the corpus to create pre-appended notes, optimize context windows for the pre-appended notes, tokenize the pre-appended notes to create a tokenized dataset, and pre-train the large language model using the tokenized dataset. The pre-trained model is configured to generate patient-specific responses to medical queries.

[0246] In Example 2, the system of Example 1 optimizes context windows by determining if a pre-appended note is longer than a context window size, splitting notes longer than thewindow size across multiple windows, and combining shorter notes with other notes or note portions into a single window.

[0247] In Example 3, when splitting pre-appended notes longer than the context window7size in Example 2, the system divides the note into portions that fit within the window size and preappends metadata to each portion to link them.

[0248] In Example 4, the metadata pre-appended to each note in Example 1 may include a note identifier, medical record number, patient name, date of birth, patient age. note date, note category, author name, department identifier, specialty identifier, and / or encounter type.

[0249] Example 5 adds a sparse autoencoder system to Example 1, configured to extract patient-specific features from the pre-trained model and monitor their activation during query processing.

[0250] In Example 6, the sparse autoencoder system of Example 5 comprises an input layer to receive data, a hidden layer to process it, and an output layer to generate output based on the processed data.

[0251] Example 7 adds a patient feature extractor to Example 6, configured to identify and extract patient-specific features from activations of the hidden layer.

[0252] In Example 8, the pre-trained model of Example 7 includes a query processor to receive processed queries from the sparse autoencoder, a response generator to produce responses based on queries and extracted features, and a patient feature activator to incorporate patient-specific features into responses.

[0253] Example 9 adds a continuous update system to Example 1 , configured to monitor the EHR database for new patient data, process it into the clinical text corpus, update the pretrained model with the new data, and validate the updated model's performance.

[0254] In Example 10. the continuous update system of Example 9 is further configured to prune portions of the pre-trained model that don't provide useful information for a particular task.

[0255] Example 11 is a method for pre-training a large language model, comprising steps similar to the system operations in Example 1.

[0256] Examples 12-20 describe method steps corresponding to the system features and operations detailed in Examples 2-10, respectively.

[0257] In Example 21, the method of Example 11 can include additional method steps for inducing patient-specific reasoning. The additional method steps can include selecting a patient of interest, retrieving additional patient data pertaining to the patient of interest from the EHR database, directing the pre-trained large language model to generate a patient-specific question about the patient of interest using the additional patient data as context, extracting thepatient-specific question from a first output generated by the pre-trained large language model, directing the pre-trained large language model to reason through and answer the patientspecific question using the additional patient information as context, extracting a reasoning chain-of-thought from a second output generated by the pre-trained large language model, directing the pre-trained large language model to re-frame the reasoning chain-of-thought based on the pre-trained large language model’s internal knowledge, combining the re-framed reasoning chain-of-thought with the patient-specific question and answer to generate fine- tuning training data, and re-training the pre-trained large language model based on the fine- tuning training data.

[0258] In Example 22, the method of Example 1 1 can include additional method steps for developing clinical reasoning in the pre-trained large language model. The additional method steps can include receiving, by the pre-trained large language model, a patient-specific query, generating, by the pre-trained large language model, one or more responses to the patientspecific query, determining a reward or penalty based on a comparison of the one or more responses to a pre-determined standard from at least one of the EHR database or a human expert, updating one or more model parameters of the pre-trained large language model based on the determined reward or penalty, and iteratively repeating the generating one or more responses, the determining the reward or penalty, and the updating the one or more model parameters one or more times.

[0259] Machine readable storage including machine-readable instructions, when executed, to implement a method or realize an apparatus in any of the examples of the present disclosure.

[0260] Various techniques, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in tangible media, such as floppy diskettes. CD- ROMs, hard drives, a non-transitory computer readable storage medium, or any other machine- readable storage medium wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the various techniques. In the case of program code execution on programmable computers, the computing device may include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The volatile and non-volatile memory and / or storage elements may be a RAM, an EPROM, a flash drive, an optical drive, a magnetic hard drive, or another medium for storing electronic data. The eNB (or other base station) and UE (or other mobile station) may also include a transceiver component, a counter component, a processing component, and / or a clock component or timer component. One or more programs that may implement or utilize the various techniques described herein may use an application programming interface (API), reusable controls, and the like. Such programs may be implemented in a high-level procedural or an object-oriented programming language to communicate with a computer system.However, the program(s) may be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or an interpreted language, and combined with hardware implementations.

[0261] It should be understood that many of the functional units described in this specification may be implemented as one or more components, which is a term used to more particularly emphasize their implementation independence. For example, a component may be implemented as a hardware circuit comprising custom very large-scale integration (VLSI) circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A component may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, or the like.

[0262] Components may also be implemented in software for execution by various types of processors. An identified component of executable code may, for instance, comprise one or more physical or logical blocks of computer instructions, which may, for instance, be organized as an object, a procedure, or a function. Nevertheless, the executables of an identified component need not be physically located together, but may comprise disparate instructions stored in different locations that, when joined logically together, comprise the component and achieve the stated purpose for the component.

[0263] Indeed, a component of executable code may be a single instruction, or many instructions, and may even be distributed over several different code segments, among different programs, and across several memory devices. Similarly, operational data may be identified and illustrated herein within components, and may be embodied in any suitable form and organized within any suitable type of data structure. The operational data may be collected as a single data set, or may be distributed over different locations including over different storage devices, and may exist, at least partially, merely as electronic signals on a system or network. The components may be passive or active, including agents operable to perform desired functions.

[0264] Reference throughout this specification to “an example7’ means that a particular feature, structure, or characteristic described in connection with the example is included in at least one embodiment of the present invention. Thus, appearances of the phrase “in an example” in various places throughout this specification are not necessarily all referring to the same embodiment.

[0265] As used herein, a plurality of items, structural elements, compositional elements, and / or materials may be presented in a common list for convenience. However, these lists should be construed as though each member of the list is individually identified as a separate and unique member. Thus, no individual member of such list should be construed as a de facto equivalent of any other member of the same list solely based on its presentation in a commongroup without indications to the contrary. In addition, various embodiments and examples of the present invention may be referred to herein along with alternatives for the various components thereof. It is understood that such embodiments, examples, and alternatives are not to be construed as de facto equivalents of one another, but are to be considered as separate and autonomous representations of the present invention.

[0266] Although the foregoing has been described in some detail for purposes of clarity, it will be apparent that certain changes and modifications may be made without departing from the principles thereof. It should be noted that there are many alternative ways of implementing both the processes and apparatuses described herein. Accordingly, the present embodiments are to be considered illustrative and not restrictive, and the invention is not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.

[0267] Those having skill in the art will appreciate that many changes may be made to the details of the above-described embodiments without departing from the underlying principles of the invention. The scope of the present invention should, therefore, be determined only by the following claims.

Claims

CLAIMSWhat is claimed is:1 . A system for pre-training a large language model, comprising: an electronic health record (EHR) database storing patient-specific data; a processor; and a memory storing instructions that, when executed by the processor, cause the system to: retrieve patient data from the EHR database; generate a clinical text corpus from the retrieved patient data; pre-append metadata to each note in the clinical text corpus to create preappended notes; optimize context windows for the pre-appended notes; tokenize the pre-appended notes to create a tokenized dataset; and pre-train the large language model using the tokenized dataset, wherein the pre-trained large language model is configured to generate patientspecific responses to medical queries.

2. The system of claim 1 , wherein optimizing context windows for the pre-appended notes comprises: determining if a pre-appended note is longer than a context window' size: splitting the pre-appended note across multiple context windows when it is determined that the pre-appended note is longer than the context window size; and combining the pre-appended note with at least one of another pre-appended note and a pre-appended note portion into a single context window when it is determined that the preappended note is shorter than the context window size.

3. The system of claim 2. wherein splitting the pre-appended note that is longer than the context window size comprises: dividing the pre-appended note into portions that fit within the context window- size; and pre-appending metadata to each portion to link the portions.

4. The system of claim 1, wherein the metadata pre-appended to each pre-appended note comprises at least one of: a note identifier, a medical record number, a patient name, a date of birth, a patient age, a note date, a note category, an author name, a department identifier, a specialty identifier, and an encounter type.

5. The system of claim 1, further comprising: a sparse autoencoder system configured to: extract patient-specific features from the pre-trained large language model; and monitor activation of the extracted patient-specific features during query processing.

6. The system of claim 5, wherein the sparse autoencoder system comprises: an input layer configured to receive input data: a hidden layer configured to process the input data; and an output layer configured to generate output based on the processed input data.

7. The system of claim 6, further comprising: a patient feature extractor configured to identity and extract patient-specific features from activations of the hidden layer.

8. The system of claim 7, wherein the pre-trained large language model comprises: a query processor configured to receive processed queries from the sparse autoencoder system; a response generator configured to produce responses based on the processed queries and extracted patient features; and a patient feature activator configured to incorporate the patient-specific features into the generated responses.

9. The system of claim 1, further comprising: a continuous update system configured to: monitor the EHR database for new patient data; process the new patient data into the clinical text corpus; update the pre-trained large language model with the processed new patient data; and validate performance of the updated pre-trained large language model.

10. The system of claim 9, wherein the continuous update system is further configured to: prune portions of the pre-trained large language model that do not provide useful information for a particular task.

11. A method for pre-training a large language model, comprising: retrieving patient data from an electronic health record (EHR) database; generating a clinical text corpus from the retrieved patient data; pre-appending metadata to each note in the clinical text corpus to create pre-appended notes; optimizing context windows for the pre-appended notes; tokenizing the pre-appended notes to create a tokenized dataset; and pre-training the large language model using the tokenized dataset, wherein the pre-trained large language model is configured to generate patient-specific responses to medical queries.

12. The method of claim 11, wherein optimizing context windows for the pre-appended notes comprises: determining if a pre-appended note is longer than a context window size; splitting the pre-appended note across multiple context windows when it is determined that the pre-appended note is longer than the context window size; and combining the pre-appended note with at least one of another pre-appended note and a pre-appended note portion into a single context window when it is determined that the preappended note is shorter than the context window size.

13. The method of claim 12, wherein splitting the pre-appended note that is longer than the context window size comprises: dividing the pre-appended note into portions that fit within the context window size; and pre-appending metadata to each portion to link the portions.

14. The method of claim 11 , wherein the metadata pre-appended to each pre-appended note comprises at least one of: a note identifier, a medical record number, a patient name, a date of birth, a patient age, a note date, a note category', an author name, a department identifier, a specialty' identifier, and an encounter type.

15. The method of claim 11 , further comprising: extracting patient-specific features from the pre-trained large language model using a sparse autoencoder system; and monitoring activation of the extracted patient-specific features during query processing.

16. The method of claim 15, wherein the sparse autoencoder system comprises: an input layer configured to receive input data; a hidden layer configured to process the input data; and an output layer configured to generate output based on the processed input data.

17. The method of claim 16, further comprising: identifying and extracting patient-specific features from activations of the hidden layer using a patient feature extractor.

18. The method of claim 17, wherein the pre-trained large language model comprises: a query processor configured to receive processed queries from the sparse autoencoder system; a response generator configured to produce responses based on the processed queries and extracted patient features; and a patient feature activator configured to incorporate the patient-specific features into the generated responses.

19. The method of claim 11 , further comprising: monitoring the EHR database for new7patient data; processing new patient data into the clinical text corpus; updating the pre-trained large language model with the processed new patient data; and validating performance of the updated pre-trained large language model.

20. The method of claim 19, further comprising: pruning portions of the pre-trained large language model that do not provide useful information for a particular task. generating a clinical text corpus from the retrieved patient data; pre-appending metadata to each note in the corpus; optimizing context windows for the notes;tokenizing the processed clinical notes; and pre-training the large language model using the tokenized dataset. wherein the pre-trained large language model is configured to generate patient-specific responses to medical queries.

21. The method of claim 11, further comprising: selecting a patient of interest; retrieving additional patient data pertaining to the patient of interest from the EHR database; directing the pre-trained large language model to generate a patient-specific question about the patient of interest using the additional patient data as context; extracting the patient-specific question from a first output generated by the pre-trained large language model; directing the pre-trained large language model to reason through and answer the patientspecific question using the additional patient information as context; extracting a reasoning chain-of-thought from a second output generated by the pretrained large language model; directing the pre-trained large language model to re-frame the reasoning chain-of- thought based on the pre-trained large language model’s internal knowledge; combining the re-framed reasoning chain-of-thought with the patient-specific question and answer to generate fine-tuning training data; and re-training the pre-trained large language model based on the fine-tuning training data.

22. The method of claim 11, further comprising: receiving, by the pre-trained large language model, a patient-specific query; generating, by the pre-trained large language model, one or more responses to the patient-specific query'; determining a reward or penalty based on a comparison of the one or more responses to a pre-determined standard from at least one of the EHR database or a human expert; updating one or more model parameters of the pre-trained large language model based on the determined reward or penalty; and iteratively repeating the generating one or more responses, the determining the reward or penalty, and the updating the one or more model parameters one or more times.

Images