Clinical recommendation system based on natural language

Abstract

The present disclosure provides a method for an automated clinical recommendation system (102) that generates clinical recommendations for a patient of a health care system based on natural language queries submitted by a care provider (402) of the health care system. The natural language query may be typed into a user interface (UI) (404) of the clinical recommendation system (401), or submitted via a microphone by voice. The clinical recommendation system provides a clinically interpretable disease state of the patient based on patient data included in a query, and recommends a next action schedule (e.g., treatment) based on clinical guidelines (202) and demographic, thus, the current burden of a clinician to look up the digital clinical manual via a series of time-consuming and tedious interactions with a graphical user interface (GUI) of the digital clinical manual is reduced.

Description

Technical Field

[0001] Embodiments of the subject matter disclosed herein relate generally to automated clinical recommendation systems for patient care, and more particularly, to clinical recommendation systems that rely on a natural language interface. Background Art

[0002] Health care providers can rely on clinical guidelines when making patient treatment decisions. Clinical guidelines can be digital clinical guidelines that can be consulted online by health care providers via the network of computer systems and / or health care systems. Care providers can search for relevant parts in clinical guidelines, and use the control elements of graphical user interfaces (GUIs) to browse the information presented in clinical guidelines. Care providers can perform a series of actions to retrieve or display desired treatment suggestions, and wait for the output to be displayed before selecting the next action. For example, care providers may have to answer a series of prompt words, or make relevant choices in a series of dialog boxes (e.g., wizards). Care providers can use keyboards to input various patient data into online forms. Due to the need to perform such operations, interacting with online clinical guidelines may be cumbersome and / or time-consuming, and may reduce the amount of time spent by care providers interacting with patients. Summary of the invention

[0003] In one example, the present disclosure utilizes a method for a clinical recommendation system to solve the problem as described above, including using a first large language model (LLM) of the clinical recommendation system to generate automated responses to natural language queries received from care providers in real time, the LLM being trained on a first set of training data generated by applying a second recommendation model to anonymized patient data, the recommendation model being based on clinical guidelines; validating the automated response output by the LLM using a third predictive model; and in response to the automated response being validated, displaying the automated response on a display device of the clinical recommendation system.

[0004] The above advantages and other advantages and features of the present specification will be apparent from the following detailed description when considered alone or in conjunction with the accompanying drawings. It should be understood that the above summary is provided to introduce a series of concepts further described in the detailed description in a simplified form. This is not meant to identify key features or essential features of the claimed subject matter, the scope of which is uniquely defined by the claims that follow the detailed description. In addition, the claimed subject matter is not limited to specific implementations that address any shortcomings mentioned above or in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Various aspects of the present disclosure may be better understood by reading the following detailed description and referring to the accompanying drawings, in which:

[0006] Figure 1A schematic block diagram of an automated clinical recommendation system according to one or more embodiments of the present disclosure is shown;

[0007] Figure 2 shows a high-level schematic block diagram illustrating a workflow for training a clinical recommendation system according to one or more embodiments of the present disclosure;

[0008] Figure 3A shows a schematic block diagram illustrating a workflow for training a large language model (LLM) and a reward model for a clinical recommender system using supervised learning according to one or more embodiments of the present disclosure;

[0009] Figure 3B shows a schematic block diagram illustrating a workflow for training an LLM using reinforcement learning according to one or more embodiments of the present disclosure;

[0010] Figure 4 shows a schematic block diagram illustrating a workflow for providing guidance to a care provider using a clinical recommendation system according to one or more embodiments of the present disclosure;

[0011] Figure 5 is a flow chart illustrating an exemplary high-level method for training a clinical recommender system according to one or more embodiments of the present disclosure;

[0012] Figure 6 is a flow chart illustrating an exemplary high-level method for optimizing the performance of an LLM of a clinical recommender system according to one or more embodiments of the present disclosure;

[0013] Figure 7 is a flow chart illustrating an exemplary high-level method for providing clinical recommendations to care providers using a clinical recommendation system according to one or more embodiments of the present disclosure;

[0014] Figure 8 is an exemplary portion of a recommendation model of a clinical recommendation system according to one or more embodiments of the present disclosure; and

[0015] Fig. 9 An exemplary predictive model of a clinical recommendation system according to one or more embodiments of the present disclosure is shown.

[0016] The accompanying drawings illustrate specific aspects of the described systems and methods. Together with the following description, the accompanying drawings show and explain the structures, methods and principles described herein. In the accompanying drawings, the sizes of components may be enlarged or otherwise modified for clarity. Well-known structures, materials or operations are not shown or described in detail to avoid obscuring various aspects of the described components, systems and methods. DETAILED DESCRIPTION

[0017] An automated clinical recommendation system is provided herein, which can generate clinical recommendations for patients of a health care system based on natural language queries submitted to the automated clinical recommendation system by a care provider of the health care system. The natural language query can be typed into the user interface (UI) of the clinical recommendation system by the care provider, or submitted by voice, for example, via a microphone. In this way, the proposed clinical recommendation system can allow clinicians to interact with clinical guidelines in a more natural way by requesting appropriate treatment guidance in natural language. The clinical recommendation system can provide a clinically interpretable disease state to the patient and recommend the next course of action (e.g., treatment) based on clinical guidelines and demographics, thereby alleviating the current burden of clinicians consulting digital clinical manuals via a series of time-consuming and cumbersome interactions with the graphical user interface (GUI) of the digital clinical manual. The interaction based on natural language will reduce the clinician's digital device interaction, thereby allowing the clinician to invest more time in interacting with the patient. In addition, the proposed clinical guidance system will provide patients with transparency of care paths and provide patients with better clinical procedure backgrounds.

[0018] The automated clinical recommendation system can rely on various models to generate clinical recommendations, including at least a natural language processing (NLP) model for answering queries posed as questions; a recommendation model for predicting appropriate treatments for patients of a health care system, which can be used to train the NLP model; and a prediction model for validating the output of the NLP model. The various models may include AI models, such as machine learning (ML) or deep learning (DL) models, generative AI, or different types of models. For example, one or more of the various models may be implemented as a convolutional neural network (CNN) model including multiple hidden layers. These models may include classification or prediction models, probabilistic models (e.g., Bayesian models), statistical models, or different types of models.

[0019] The recommendation model can be based on clinical guidelines used by the healthcare system, which depends on the type and / or condition of the patient presenting. For example, the National Comprehensive Cancer Network (NCCN) Clinical Practice Guidelines for Oncology (NCCN ) may be a first set of clinical guidelines for cancer patients. Additional sets of clinical guidelines may be used for specific types of cancer. For example, for patients with prostate cancer, the Prostate Imaging Reporting and Data System Can be used as a second set of clinical guidelines. In various embodiments, the recommendation model can be implemented as a decision tree or knowledge graph that can be traversed to determine recommended treatments based on patient data.

[0020] The NLP model can be a commercial, open source, or private implementation of an internal large language model (LLM) (such as OpenAI's GPT), which can be trained to process natural language queries and provide responses based on the recommendation model. The LLM can be trained using a combination of supervised learning and reinforcement learning, where the supervised learning has a benchmark truth data manually generated based on the recommendation model and the LLM is optimized using a reward model in reinforcement learning. After the LLM is trained and optimized, the clinical recommendation system can receive a natural language query (e.g., a question about a course of action for a patient) from a care provider and submit the natural language query to the LLM as a prompt word. The LLM can output a response to the prompt word, which can include the recommended treatment.

[0021] The output of the LLM can be validated by a predictive model. The predictive model can be trained on training data generated by applying the recommendation model to historical patient data. In one embodiment, the predictive model takes as input the patient data included in a natural language query to the LLM and the recommended treatment included in the natural language response to the query output by the LLM, and outputs a probability that the recommended treatment is the most appropriate treatment for the patient based on the historical patient data.

[0022] Figure 1 An exemplary clinical recommendation system is shown in FIG. The clinical recommendation system may rely on an LLM, which may be based on Figure 2 Specifically, the workflow shown in can be used as follows. Figure 3A The supervised learning shown in Figure 3B The combined training of reinforcement learning shown in LLM. Figure 4 The trained LLM can be used to generate treatment recommendations for patients by following the workflow shown in Figure 5 Training the LLM may include following one or more of the steps of the method shown in Figure 6 One or more of the steps in the method shown in Optimize the performance of LLM. Clinical recommendation system can be used to follow Figure 7 One or more steps of the method shown in generates a response to a natural language query submitted by a care provider. Figure 8 An exemplary recommendation model for generating training data for LLM is shown in FIG. A prediction model can be used to validate the output of LLM, such as Fig. 9 The prediction model shown in .

[0023] Figure 1An exemplary automated clinical recommendation system 102 according to an embodiment is shown. The clinical recommendation system 102 can provide treatment guidance to a health care provider based on historical patient data and reference materials including clinical guidelines. In some embodiments, at least a portion of the clinical recommendation system 102 is disposed at a device (e.g., a workstation, an edge device, a server, etc.) that is communicatively coupled to one or more health care and / or hospital networks and computer systems via a wired and / or wireless connection, and is capable of receiving or accessing medical data (including patient data) stored in one or more health care and / or hospital computer systems. The medical data may include online clinical reference materials, such as a set of digital clinical guidelines 150, which can be used by a care provider to provide treatment alternatives to a patient. The clinical recommendation system 102 can also be operably / communicatively coupled to a user input device 132, a display device 134, and / or a speaker 135. In some examples, the user input device 132 can be a shared input device of one or more health care and / or hospital computer systems, and the display device 134 can be a shared input device of one or more health care and / or hospital computer systems.

[0024] The clinical recommendation system 102 includes a processor 104 configured to execute machine-readable instructions stored in a non-transitory memory 106. The processor 104 may be a single-core or multi-core processor, and the program executed thereon may be configured for parallel processing or distributed processing. In some embodiments, the processor 104 may optionally include separate components distributed on two or more devices, which may be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of the processor 104 may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration.

[0025] The non-transitory memory 106 may store an AI model module 108, a training module 110, an inference module 111, and a database 114. The AI ​​model module 108 may include various AI models, such as ML and / or DL ​​models. Specifically, the AI ​​model module may include an LLM 120 that may be trained to respond to natural language queries about treatment alternatives proposed by a health care provider, as described below with reference to Figure 5 The AI ​​module may include a reward model 126, which may be used to train the LLM using reinforcement learning, as described below with reference to Figure 6 The AI ​​module may also include a recommendation model 122 that may be used to recommend treatment options based on clinical guidelines 150, as described below with reference to Figure 7The AI ​​module may also include a prediction model 124, which may be used to validate the output of the LLM. In various embodiments, one or more AI models may include neural network models such as convolutional neural networks (CNNs), generative adversarial networks (GANs), transformers, and traditional classifiers. In some embodiments, one or more AI models may include a Bayesian network, such as Fig. 9 The AI ​​model module 108 may include trained and / or untrained neural networks, and may also include various data or metadata related to one or more AI models stored therein.

[0026] The training module 110 may include instructions for training one or more AI models and / or other types of AI, ML, DL, statistical, or other models stored in the AI ​​model module 108. The training module 110 may include instructions that, when executed by the processor 104, cause the clinical recommendation system 102 to perform Figure 5 108 for training LLM 120. In some embodiments, training module 110 may include instructions for implementing one or more gradient descent algorithms, applying one or more loss functions, and / or training routines for adjusting parameters of one or more neural networks of AI model module 108. Training module 110 may include a training data set for one or more AI models of AI model module 108.

[0027] The inference module 111 may include instructions for deploying one or more trained AI models, for example, in response to a natural language query submitted by a care provider to the clinical recommendation system 102. Specifically, the inference module 111 may include instructions that, when executed by the processor 104, cause the clinical recommendation system 102 to perform Figure 7 One or more of the steps of method 700.

[0028] The database 114 may include anonymized historical patient data that may be retrieved from the electronic medical record (EMR) 140 and compiled for use in one or more training sets for training one or more AI models of the AI ​​model module 108. The historical patient data may include a variety of data from patients in the health care system about the conditions suffered by the patient, including treatments performed on the patient and the results of the treatments. In some embodiments, the database 114 may include a patient decision workflow for one or more patients, where the patient decision workflow is a stored history of clinical decisions made for the relevant patient using the clinical recommendation system 102. Figure 4 Describe the patient decision-making workflow in more detail.

[0029] The user input device 132 may include one or more of a touch screen, a keyboard, a mouse, a trackpad, a microphone, a motion sensing camera, or other devices configured to enable a user to interact with the clinical recommendation system 102. In one example, the user input device 132 may enable a user to submit questions about a patient to the clinical recommendation system 102 in natural language. For example, a user (e.g., a care provider) may type a question into the clinical recommendation system 102 using a keyboard, or speak the question into a microphone.

[0030] The display device 134 may include one or more display devices utilizing almost any type of technology. In some embodiments, the display device 134 may include a computer monitor. The display device 134 may be combined with the processor 104, the non-transitory memory 106, and / or the user input device 132 in a shared package, or may be a peripheral display device, and may include a monitor, a touch screen, a projector, or other display devices known in the art that may enable a user to view responses to queries submitted to the clinical recommendation system 102, and / or interact with various data stored in the non-transitory memory 106.

[0031] It should be understood that Figure 1 The parameter recommendation system 102 shown in FIG. 1 is for illustration and not for limitation. Another suitable parameter recommendation system may include more, fewer, or different components.

[0032] Figure 2 A schematic diagram of a first workflow 200 for training an LLM (e.g., LLM 120) to respond to a natural language query submitted to a clinical recommender system 218, which may be Figure 1 Non-limiting examples of clinical recommendation systems 102. Figure 5 A method for training an LLM is described, which may follow a first workflow 200. The steps indicated in the first workflow 200 may be performed based on instructions stored in a non-transitory memory of the clinical recommendation system 218, such as stored in Figure 1 The training module 110 of the non-transitory memory 106 of the clinical recommendation system 102 .

[0033] The first workflow 200 begins with the creation of a recommendation model 204 for recommending one or more treatment options for a patient to a care provider using a clinical recommendation system. The recommendation model 204 may be derived from one or more sets of clinical guidelines 202 (e.g., Figure 1Clinical guidelines 202 may be digital guidelines available online and / or on a computer system or network of a health care system using a clinical recommendation system. Clinical guidelines 202 may include different reference guidelines for different types of patients and / or conditions. For example, a first set of clinical guidelines may relate to patients who have or are suspected of having cancer; a second set of clinical guidelines may relate to patients who have or are suspected of having an autoimmune disease; a third set of clinical guidelines may relate to patients who are suffering from traumatic wounds; and so on. In one example, clinical guidelines 202 include NCCN

[0034] In some examples, multiple recommendation models 204 may be created for corresponding multiple clinical guidelines 202. For example, a first recommendation model 204 may be created for a patient with cancer; a second recommendation model 204 may be created for a patient with an autoimmune disease; a third recommendation model 204 may be created for a patient with trauma; and so on. Each recommendation model 204 may be used to determine the most appropriate treatment for the patient based on a corresponding set of clinical guidelines 202.

[0035] In various embodiments, the recommendation model 204 is a decision tree. The decision tree can be manually generated by a human expert from a corresponding set of clinical guidelines. Figure 8 An exemplary decision tree is shown in FIG. In other embodiments, the recommendation model 204 may be a different type of model. For example, in one embodiment, the recommendation model 204 may be a knowledge graph.

[0036] The recommendation model 204 may be used in the LLM training step 212 and the prediction model training step 213. The LLM training step 212 may rely on a first training data set, and the prediction model training step 213 may rely on a second training data set, wherein both the first training data set and the second training data set may be generated with the help of the recommendation model 204. For the LLM training step 212, the natural language training data set 210 may be manually generated during the cue generation phase 208 by a human expert who generates real natural language cues and associated responses for recommending treatments using the recommendation model 204 as a guide. The LLM may be trained on data pairs including real natural language cues with associated responses included as ground truth data.

[0037] In some embodiments, retrieval augmented generation (RAG) can be used during the training of the LLM, where the natural language prompts submitted to the LLM can include instructions for consulting one or more medical resources 214 to generate an appropriate response. These instructions can provide additional information related to the prompts. The sequence-to-sequence RAG model can take an input sequence and generate a corresponding output sequence that can be fine-tuned for a specific application such as a clinical guideline. For example, the input to the RAG model can include queries such as "Given stage 4 lung cancer with tumor, lymph node, and metastasis (TNM) status [xxx] and surgical margin [yyy], what should the next step be?" The fine-tuned RAG will use the input of the RAG model to retrieve a set of relevant documents, including clinical trials and journal articles, which will then be cascaded to provide relevant output / prompt words for the fine-tuned LLM. By using RAG when submitting prompts, the performance of the LLM and the accuracy of the results returned by the LLM can be improved.

[0038] For the prediction model training step 213, outcome probability data 209 can be generated by applying the recommendation model 204 to a set of anonymized historical patient records 206. In other words, the historical patient records can be analyzed to determine a set of probabilities for which the treatment recommended by the recommendation model 204 has the highest probability of success. This set of probabilities can then be used to train the prediction model to predict the treatment for a given patient situation with the highest probability of success. The prediction model can be used to validate the output of the LLM in the same patient situation, as described in more detail below. After the prediction model and LLM have been trained, the trained prediction model and the trained LLM can be incorporated into the clinical recommendation system 218.

[0039] about Figure 3A and Figure 3B The workflow diagram of describes the training of the LLM in more detail. Specifically, the training of the LLM can be done in three stages. During the first stage of training, supervised learning is used to train the LLM on human-generated training data. During the second stage of training, the reward model is trained using supervised learning based on ground truth data generated by manual ranking of the LLM outputs by human experts. During the third stage of training, reinforcement learning is used to optimize the LLM based on the trained reward model.

[0040] Now turn to Figure 3A , depicts a second workflow 300 including the first and second stages of LLM training as described above. The steps indicated in the second workflow 300 may be based on a training module stored in the clinical recommendation system 218 (such as stored in Figure 1 The instruction of the training module 110 is executed. Figure 2For supervised LLM training 212, the training dataset is generated from two sources: Figure 2 The general domain NLP dataset 302 may include natural language training data pairs of prompt words and responses obtained under a commercial data use agreement. The general domain NLP dataset 302 may be used to train the LLM to respond to queries about specific treatment plans for patients presenting with different conditions. In order to train the LLM to provide clinical treatment recommendations, an integrated training dataset 304 may be compiled, including using Figure 2 The recommendation model 204 is created using both the natural language training data 210 and the general domain NLP dataset 302. Both the natural language training data 210 and the general domain NLP dataset 302 can be used simultaneously in a supervised setting to fine-tune the LLM.

[0041] After training on the integrated training data set 304, the output of the trained LLM 310 may be used to train a reward model to further optimize the trained LLM 310 using reinforcement learning. The output of the trained LLM 310 may be sampled and ranked by a human expert in an output ranking step 312. The ranked output may be used to train the reward model during a reward model training step 314. During the reward model training step 314, the reward model may be trained using supervised learning on training pairs including the output of the trained LLM 310 and the corresponding rankings as ground truth data.

[0042] Figure 3B A third workflow 350 corresponding to a third stage of the training process is shown, wherein the trained LLM 310 is optimized during a reinforcement learning step 352. The steps indicated in the third workflow 350 may be performed based on instructions stored in a training module of the clinical recommendation system 218. During the reinforcement learning step 352, the training module 310 is optimized using the training module 310 trained in the clinical recommendation system 218. Figure 3A The reward model 354 and the reinforcement learning algorithm 356 trained at the reward model training step 314 train the trained LLM 310 on various corpora of unlabeled training data 351. In various embodiments, the reinforcement learning algorithm 356 can be a proximal policy optimization (PPO) reinforcement learning algorithm. Figure 6 Optimization of the trained LLM is described in more detail.Once the trained LLM 310 has been optimized at the reinforcement learning step 352, the optimized LLM 360 may be deployed to the clinical recommender system 218 for use in a clinical setting during the inference phase.

[0043] Figure 4A fourth workflow 400 is shown for using the optimized LLM 360 to provide a clinical treatment recommendation to a care provider 402 of a healthcare system via a clinical recommendation system 401, which may be Figure 2-3B Clinical Recommendation System 218 and / or Figure 1 The steps indicated in the fourth workflow 400 may be based on an inference module (such as Figure 1 The instructions in the inference module 111) are executed.

[0044] According to the fourth workflow 400, the care provider 402 may submit a question to the clinical recommendation system 401 via a user interface (UI) 404 of the clinical recommendation system 401. For example, the UI 404 may include Figure 1 The graphical user interface (GUI) displayed on the display device 134 of the clinical recommendation system 102 of the present invention, and the care provider 402 can interact with the UI 404 via the user input device 132 (e.g., keyboard, mouse, microphone, etc.). The UI 404 may include a microphone, wherein the question can be submitted as an audio file recorded via the microphone. For example, in one embodiment, the GUI of the UI 404 can be displayed on the smart phone of the care provider 402, and the care provider 402 can submit questions via the microphone of the smart phone. The question can be about the patient. For example, the care provider 402 can seek recommendations about one or more action plans and / or treatment options taken with the patient from the clinical recommendation system 401.

[0045] The clinical recommendation system 401 may receive questions via UI 404 and may convert the questions into natural language prompts 406. Natural language prompts 406 may be generated from decision trees as observed in clinical guidelines and / or knowledge graphs for specific diseases to incorporate relevant domain knowledge. Natural language prompts 406 may be submitted to Figure 3B The optimized LLM 360 may output a natural language response 408 that may answer the question of the care provider 402. However, before the natural language response 408 is displayed to the care provider via the UI 404, the natural language response 408 may be validated using a separate predictive model 412. The predictive model 412 may be used in reference to the Figure 2 The described prediction model is trained at step 213 .

[0046] To validate the natural language response 408, the tokenizer 410 may extract relevant tokens from the natural language prompt 406, wherein the relevant tokens include patient data describing the patient state upon which the optimized LLM 360 relies to generate the natural language response 408. The patient data describing the patient state may also correspond to elements of the recommendation model 204. For example, in an embodiment in which the recommendation model 204 is a decision tree, the patient data may be used to traverse the decision tree to determine the patient status determined by the clinical guideline (e.g., Figure 1 Clinical Guidelines 150 and / or Figure 2 Clinical guidelines 202) recommend treatment options.

[0047] The patient data extracted from the natural language prompt 406 by the tokenizer 410 can be input into the prediction model 412. The prediction model 412 can output a prediction of the treatment options for the patient with the highest probability of success based on the patient data and the outcome probability data 209 generated based on the historical patient data, as shown in FIG. Figure 2 As depicted, the predictive model 412 has been trained based on the historical patient data. Similarly, the tokenizer 410 can be used to extract relevant tokens from the natural language response 408 , wherein the relevant tokens include specific treatment options recommended by the optimized LLM 360 .

[0048] The comparator 414 may then determine whether the particular treatment option recommended by the optimized LLM 360 matches the treatment option with the highest probability of success output by the predictive model 412. If the treatment option recommended by the optimized LLM 360 matches the treatment option with the highest probability of success output by the predictive model 412, the natural language response 408 may be added to the patient decision workflow 416 and delivered to the care provider 402 via the UI 404. For example, the natural language response 408 may be displayed on a display device (e.g., the display device 134) of the clinical recommendation system, or the natural language response 408 may be converted into an audio signal and played to the care provider 402 via a speaker (e.g., the speaker 135) or a device including a speaker (such as a smart phone).

[0049] The patient decision workflow 416 can be a stored representation of a plurality of clinical decisions recommended and / or made using the clinical recommendation system 401 that can be used as a representation history of the patient care received by the patient. For example, the patient decision workflow 416 can show the patient's progress through the various decision points at which the clinical decisions were recommended. The patient decision workflow 416 can be stored in a database of the clinical recommendation system, such as Figure 1In some embodiments, the patient decision workflow 416 may include additional information about the clinical recommendation, such as, for example, information on which the clinical recommendation is based, other options that are not selected, etc. For example, the information on which the clinical recommendation is based may be extracted from the natural language prompt 406, and other options that are not selected may be extracted from the recommendation model of the clinical recommendation system 401 (e.g., Figure 2 The recommendation model 204) is extracted, and so on.

[0050] As an example of how a patient decision workflow may be used, the optimized LLM 360 may recommend a first treatment at a first decision point in a patient's patient journey. A new patient decision workflow 416 may be generated, including the patient data included in the first natural language prompt 406 and the first recommended treatment. The patient decision workflow 416 may be stored in a database of a clinical recommendation system. The care provider 402 may treat the patient using the first recommended treatment. When the first recommended treatment is performed on the patient, the care provider 402 may update the patient decision workflow 416 to indicate that the first recommended treatment has been followed. At a second decision point in the patient journey, the care provider 402 may submit a second natural language prompt 406 that includes new patient data and requests a second treatment recommendation. The optimized LLM 360 may generate a second natural language response 408 with a second treatment recommendation. The second treatment recommendation may be included (e.g., added) to the patient decision workflow 416. The care provider 402 may treat the patient using the second recommended treatment. When the second recommended treatment is performed on the patient, the care provider 402 may update the patient decision workflow 416 to indicate that the second recommended treatment has been followed. In this way, a record of the recommendations made at each decision point in the patient journey and the decisions made in response to the recommendations may be included in the patient decision workflow 416 .

[0051] In addition, in some embodiments, the patient decision workflow 416 can be used to guide the treatment of a new patient with a profile, diagnosis, etc. similar to the patient. For example, when submitting the natural language prompt 406, the care provider 402 can request one or more patient decision workflows 416 for patients with similar diagnoses. The optimized LLM 360 can retrieve the one or more patient decision workflows 416 and display them in the UI 404. The care provider 401 can review and select the most similar patient decision workflow 416 and customize the most similar patient decision workflow 416 for the new patient. Alternatively, in some examples, the care provider 401 can include instructions in the natural language prompt 406 for the optimized LLM 360 to select and customize the most similar patient decision workflow 416 for the new patient. By customizing the stored patient decision workflow 416 for a previous patient, a recommended clinical decision can be determined more quickly and with less use of resources of the clinical recommendation system 401 than by generating a recommended clinical decision as described above.

[0052] Additionally or alternatively, one or more patient decision workflows 416 can be incorporated into or used in conjunction with a recommendation model to recommend treatment to a patient. For example, the recommendation model can be a Bayesian network, where the probabilities incorporated into the Bayesian network can be adjusted based on the one or more patient decision workflows 416.

[0053] Additionally, in some embodiments, one or more patient decision workflows 416 may be used to validate the performance of the clinical recommendation system 401 and / or track errors in the performance of the LLM at various stages of patient care. For example, a decision recommended using the optimized LLM 360 may be compared to a similar decision made based on similar information in one or more patient decision workflows 416. If the recommended decision differs from the similar decision, the recommended decision may not be provided to the care provider 402.

[0054] If the treatment option recommended by the optimized LLM 360 does not match the treatment option with the highest probability of success output by the predictive model 412, the natural language response 408 may not be delivered to the care provider 402 via the UI 404, and an indication of the inconsistency between the proposed treatment and the clinical guidelines may be recorded by the clinical recommendation system 401. This inconsistency may be used to improve the optimized LLM 360 in further training to more closely adhere to the clinical guidelines. In other examples, the inconsistency may be used to adjust the clinical guidelines.

[0055] The care provider 402 can follow the action plan recommended by the clinical recommendation system 401 and apply the treatment output by the optimized LLM 360. When the treatment is applied to the patient, the patient data can be updated in the EMR 420 of the health care system according to the patient's patient journey. The patient data may include the final result of the treatment. In various embodiments, the updated patient data including the final result can be retrieved by the clinical recommendation system 401 during the feedback collection step 422. The updated patient data can be stored in the training database (e.g., database 114) of the clinical recommendation system 401. The updated patient data including the final result can be incorporated into the result probability data 209 for retraining and improving the performance of the prediction model 412. The updated patient data can also be incorporated into the natural language training data 210 for retraining and further optimizing the optimized LLM 360. In this way, the performance of the optimized LLM 360, the prediction model 412, and the clinical recommendation system 401 can be continuously increased over time.

[0056] Reference now Figure 5 , shows a method 500 for training an LLM to recommend a patient via a clinical recommendation system such as Figure 1 The clinical recommendation system 102 of the health care system recommends treatment options for the patient to the care provider of the health care system. The LLM can be a non-limiting example of the LLM 120. The method 500 can be executed by a processor of the clinical recommendation system (e.g., the processor 104) based on instructions stored in a non-transitory memory of the clinical recommendation system (e.g., the AI ​​model module 108 and the training module 110).

[0057] Method 500 begins at 502, where the method includes training a recommendation model based on one or more sets of clinical guidelines. Training the recommendation model may include mapping elements of one or more sets of clinical guidelines to a hierarchical structure for modeling decisions, such as a decision tree. The hierarchical structure may then be traversed based on patient data until a decision point is reached, at which one or more courses of action may be recommended based on the options available at the decision point. Decision trees / knowledge graphs / recommendation models may be created in a rule-based framework from existing clinical guidelines, literature, domain knowledge of the disease, and / or using conventionally collected clinical training data to create a graph- or tree-based machine learning model in a supervised, semi-supervised, self-supervised, or unsupervised machine learning framework. Figure 8 An example recommendation model implemented as a decision tree is shown in .

[0058] refer to Figure 8, shows a portion of a recommendation model 800, where the recommendation model 800 is a decision tree. The portion of the recommendation model 800 includes a framework for making treatment decisions about a theoretical patient, where the decision points of the decision tree are depicted in columns reading from left to right. In the depicted portion, one or more exemplary clinical presentations of a patient are shown in a first column 802. One or more initial recommended treatments for a patient presenting with the clinical presentation are shown in a second column 804. One or more recommended adjunctive treatments corresponding to different outcomes of the initial recommended treatments are shown in a third column 806. As depicted in a fourth column 808, the final course of action after the initial and adjunctive treatments may be to monitor the patient for signs of improvement.

[0059] As an example of how the recommendation model 800 may be used, a patient may present to a care provider with stage III lung cancer. According to the recommendation model 800, two alternative initial treatments are recommended at the first column 802. At the first decision point 810, surgery may be recommended as the initial treatment, or concurrent chemoradiation / chemotherapy may be recommended as the initial treatment. If surgery is selected at the first decision point 810, different adjuvant treatment options are recommended at the second decision point 812, depending on whether the margin is positive or negative. If the margin is negative at the decision point 812, adjuvant therapy 818 is recommended, including chemotherapy, followed by treatment with one or more drugs. If the margin is positive at the decision point 812, different adjuvant treatments are recommended based on additional criteria. If a different initial treatment is selected, the decision tree may be traversed accordingly until a different decision point is reached where one or more adjuvant treatments are recommended. In this manner, the recommendation model 800 may be traversed based on clinical data obtained for the patient until a decision point is reached, which decision point may indicate a subsequent course of action based on the clinical data.

[0060] As described below, the recommendation model 800 can be used to generate a first set of training data for training an LLM of a clinical recommender system, and / or a second set of training data for training a predictive model of a clinical recommender system.

[0061] Back to Figure 5 At 504, method 500 includes training a prediction model on historical patient data. The prediction model can be trained to receive (new) patient data as input and, based on the historical patient data, output a predicted treatment with the highest probability of being selected by a care provider. The training data for the prediction model can include training pairs including patient data indicating a state of the patient (such as a disease state of the patient) and patient outcome data, meaning a treatment applied to the patient based on the state.

[0062] In some embodiments, the prediction model may include a Bayesian network that is trained to learn a set of conditional probabilities that a treatment is selected based on historical treatment data. The Bayesian network can provide a probabilistic expectation of the outcome of a patient's care path based on demographic and current and previous clinical pathology variables. In other embodiments, the prediction model may be a CNN that is trained to output the recommended treatment with the highest probability of being selected based on historical patient data. In other embodiments, different types of models or neural networks may be used.

[0063] Predictive models can allow for tracking deviations from usual clinical practice based on demographics. In other words, Bayesian networks can be used to identify unusual patient journeys in the care pathway and highlight decision points where deviations occur. Deviation tracking can be used to update clinical guidelines and / or to realign clinical practice with clinical guidelines. For example, recommendation models can be updated or expanded to include more or different decision points. If clinical data are found to be inconsistent with clinical guidelines, new decision points can be added.

[0064] Brief reference Fig. 9 , shows an exemplary Bayesian network 900, where the Bayesian network 900 is a specific implementation of a prediction model for predicting treatments recommended for patients within a clinical recommendation system. The Bayesian network 900 can have a hierarchical structure that follows or is based on the hierarchical structure of the recommendation model. Thus, the Bayesian network 900 includes a plurality of nodes 901, where each of the nodes 901 can correspond to a recommendation model (e.g., Figure 8 900). Each node 901 is linked to a different subsequent option for the treatment of the patient, where the nodes 901 can be traversed to determine the recommended treatment for the patient, similar to the recommendation model. The predicted treatment can be indicated at the output layer 920 of the Bayesian network 900. For a given patient, some nodes can be based on known patient data, such as nodes 902, 904, 908, and 910, while other nodes may not be applicable to the patient, such as nodes 930, 932, 934, and 936. In addition, some nodes may have unknown data, such as node 906, due to lack of patient information or lack of examination / test results.

[0065] In contrast to the recommendation model, during training, the Bayesian network 900 learns the conditional probability of each node 901 being selected based on historical patient data. After training, during the inference phase, the conditional probabilities are used to predict the treatment most likely to be applied to the patient based on historical patient data, which in some cases may be different from the treatment recommended by the recommendation model.

[0066] For example, based on the previous clinical pathology variables represented by nodes 902 and 904, node 906 may represent the current state of the patient. At node 906, the recommendation model recommends two treatment options: a first treatment option corresponding to node 908 and a second treatment option corresponding to node 910. The recommendation model may provide criteria for selecting the first treatment option and the second treatment option. Patient data corresponding to the criteria may not be available, and thus the recommendation model may not be able to provide a clear treatment recommendation. The recommendation model may select one of the first treatment option and the second treatment option.

[0067] However, the Bayesian network 900 may learn from historical patient data that the second treatment option at node 910 has an 80% chance of being selected by the care provider, while the first treatment option at node 908 has a 20% chance of being selected by the care provider. As a result, the Bayesian network 900 may predict the selection of a second treatment option that is opposite to the first treatment option. As described in more detail below, the second treatment option predicted by the Bayesian network 900 may be compared to the selected treatment option recommended by the recommendation system to verify the selected treatment option. If the selected treatment option output by the recommendation model matches the predicted treatment option output by the prediction model, the clinical recommendation system may display the LLM's response to the care provider, as described above with reference to Figure 4 as described.

[0068] Returning to method 500, at 506, method 500 includes generating a training pair including a natural language prompt word and a reference truth response based on a trained recommendation model. In various embodiments, the natural language prompt word and the reference truth response can be manually generated by a human expert. For example, a human AI trainer can provide a conversation, wherein the AI ​​trainer plays both sides of the conversation, meaning a user and a theoretical AI assistant (e.g., a clinical recommendation system). The AI ​​trainer can use the recommendation model to generate a conversation. For example, the AI ​​trainer can generate a natural language question based on a set of patient data corresponding to a decision point of the recommendation model, and the AI ​​trainer can generate a natural language response that can be output by the AI ​​assistant based on the recommended treatment corresponding to the decision point. Natural language questions and natural language responses can be included in the training pair of training data, while the natural language response serves as the reference truth data. In this way, multiple natural language training pairs can be generated by a human expert for training LLM to recommend treatment for patients based on natural language queries proposed by care providers.

[0069] Each training pair may include a natural language prompt as input data, and a corresponding natural language response generated by an AI trainer as ground truth data. For example, a natural language prompt may be "Given stage 4 lung cancer with TNM status [xxx] and surgical margin [yyy], what should be the next step?", and the corresponding natural language response may be "Given stage 4 lung cancer with TNM status [xxx] and surgical margin [yyy], chemotherapy is recommended".

[0070] In some embodiments, natural language training pairs can be generated by one or more chatbots that traverse the recommendation model and generate plausible questions based on the data at various decision points. Chatbot-generated training pairs can be used in conjunction with human-generated training pairs. For example, chatbot-generated training pairs can be merged with human-generated training pairs during training. Alternatively, training can be performed in stages on different training data. For example, human-generated training pairs can be used to train the LLM in the first stage, and chatbot-generated training pairs can be used to train the LLM in the second stage, or chatbot-generated training pairs can be used to train the LLM in the first stage, and human-generated training pairs can be used to train the LLM in the second stage.

[0071] At 508, the method includes training the LLM on a general domain NLP dataset. The general domain NLP dataset may be a publicly available dataset associated with the LLM that can be used to train the LLM to respond to general questions in an intelligent manner. A supervised learning framework will be used for such retraining. However, the general domain NLP dataset may not include training data specific to clinical guidelines.

[0072] At 510, method 500 includes fine-tuning the LLM after completing the training using a general domain NLP dataset by training the LLM on the generated natural language training pairs based on clinical guidelines (e.g., either or both of the human-generated training pairs and the chatbot-generated training pairs). The LLM can be trained on the generated natural language pairs using supervised learning. By fine-tuning the LLM on the generated natural language pairs, the accuracy of the LLM in responding to requests for clinical guidance based on patient data can be improved.

[0073] At 512, method 500 includes collecting multiple sample outputs of the fine-tuned LLM for a given set of input patient data, and ranking each of the multiple sample outputs using a human expert. In other words, multiple natural language prompts generated by human experts and / or chatbots for training the LLM can be randomly selected from the training set. For each selected prompt, several alternative outputs of the fine-tuned LLM can be sampled. The AI ​​trainer can then rank the alternative outputs. In various embodiments, the outputs can be ranked based on the proximity of the embedding of the outputs in the feature space, as measured by Mahalanobis, Wasserstein, Euclidean, and / or similar types of distance measurements.

[0074] At 514, method 500 includes training a reward model on the ranked output using supervised learning. The reward model can be a neural network model such as a CNN. Under the reinforcement learning framework, a model that generates outputs closer to human experts will be rewarded.

[0075] At 516, method 500 includes optimizing the trained and fine-tuned LLM (e.g., Figure 3A 310). Optimizing the fine-tuned LLM can increase the accuracy and / or performance of the LLM when answering queries about treatment alternatives for a patient based on current and prior clinical pathology variables provided by a care provider. In contrast to supervised learning, in which examples of desired outputs are provided to the model during training, reinforcement learning trains the model to generate outputs that maximize rewards. An advantage of reinforcement learning is that it does not rely on labeled training data, so larger and more comprehensive data sets can be used for training. For example, during the optimization training process, large data corpora can be used, including petabytes of web crawler output, Internet-based books, data extracted from online encyclopedias and similar reference materials, and the like. The data corpus can include general information as well as information related to health care, biology, medicine, patient treatment, and the like.

[0076] When trained using reinforcement learning, the model can learn a set of policies to apply to information provided to the model to output the best response, including information that the model has not been trained on. In general, the LLM fine-tuned using reinforcement learning optimization includes retraining the fine-tuned LLM using a reinforcement algorithm and a reward model trained at 514. Figure 6 The optimization of the fine-tuned LLM is described in more detail.

[0077] At 518, method 500 includes deploying the optimized LLM for use in a clinical recommendation system. Figure 7 The application of an optimized LLM by a clinical recommender system in response to a query posed by a care provider is described in greater detail.

[0078] At 520, method 500 includes collecting and storing new patient data for further training of the prediction model. Patient data including outcome data may be periodically collected from the EMR of the healthcare system to incorporate the patient data into the training data (e.g., Figure 2 The result probability data 209) is used in the ongoing training of the predictive model. For example, a care provider can submit a prompt word to the clinical recommendation system, requesting guidance on the treatment for the patient. The care provider can include the current and previous patient data on which the treatment decision should be based in the prompt word. The clinical recommendation system can provide a response to the prompt word, indicating the recommended treatment for the patient. The care provider can treat the patient according to the recommended treatment. The results of the recommended treatment can be observed at a later date and can be recorded in the EMR at a later date. At a time after the later date, the clinical recommendation system can retrieve the result data recorded in the EMR. For example, the clinical recommendation system can periodically retrieve batches of updated patient data including updated result data. The updated patient data including the results of the recommended treatment can be used to train the predictive model. After the predictive model has been trained on the updated patient data, the predictive model can more accurately predict the probability that the recommended treatment is applied to other patients with patient data similar to the patient.

[0079] New patient data may also be stored for periodically retraining the LLM and / or recommendation model. For example, a new training data set for retraining the LLM may be periodically created by a human expert in the manner described above, which may incorporate new patient data including outcome data. Additionally, the recommendation model may be periodically retrained as the clinical guidelines used to train the recommendation model change over time in response to new patient data.

[0080] Reference now Figure 6 , showing the use of reinforcement learning to optimize Figure 1 LLM 120 and / or Figure 3A The method 600 may be executed by a processor of a clinical recommender system such as a Figure 1 The processor 104 of the clinical recommendation system 102 executes based on instructions stored in the non-transitory memory of the clinical recommendation system (such as the AI ​​model module 108 and / or the training module 110).

[0081] As described above, reinforcement learning trains the LLM to generate an output that maximizes a reward, where the reward is determined by a trained reward model. During the optimization process, the LLM can be trained on a training dataset that includes natural language prompts. The natural language prompts can be manually generated by human experts, as described above with reference to Figure 5The method 500 described above. However, the training data set may not include responses generated by human experts for supervised learning (e.g., benchmark truth data). During each iteration of training, a natural language prompt word can be submitted to the LLM, and a reward can be assigned to the output of the LLM through a reward model. The parameters of the LLM can then be adjusted based on the assigned rewards according to a reinforcement training algorithm. In various embodiments, the reinforcement training algorithm can be a proximal policy optimization (PPO) reinforcement algorithm commonly used on the LLM.

[0082] The method 600 begins at 602, where the method includes initializing a PPO model based on a trained LLM. During PPO initialization, a small update may be allowed to the strategy to stabilize model training.

[0083] At 604 , method 600 includes selecting a cue word from a training data set, and at 606 , the cue word is submitted to a trained LLM.

[0084] At 608, method 600 includes calculating a reward for a response to a cue word output by the trained LLM. The reward may be a numerical score assigned to the cue word by the reward model. The cue word and the response may be input into the reward model, and the reward model may output a reward based on the human-ranked training data used to train the reward model, as described above.

[0085] At 610, method 600 includes updating a plurality of parameters of the trained LLM based on the assigned rewards according to the PPO model. PPO limits large policy updates, thereby stabilizing the training process. Rewards or penalties do not allow large changes in the policy, and the policy is limited by a cutoff ratio, which is the probability of taking the current action divided by the probability of the previous action.

[0086] At 612, method 600 includes determining whether the training process has stabilized and / or converged. Convergence can be achieved when the performance of the LLM indicates that the rewards assigned to the responses are consistently maximized and the performance is no longer improving. For example, a first performance of the LLM in a first shorter number of training iterations can be compared to a second performance of the LLM in a second longer number of training iterations. If the first performance is within a threshold difference of the second performance, it can be inferred that the performance is no longer improving.

[0087] If the training process has not stabilized and / or converged at 612, the method 600 returns to 604 and continues training using new prompt words selected from the training data. Alternatively, if the training process has stabilized and / or converged at 612, the method 600 proceeds to 610. At 610, the optimization training is stopped and the optimized LLM (e.g., Figure 3BThe optimized LLM 360) is deployed for use by the clinical recommendation system. The method 600 ends.

[0088] Figure 7 An exemplary method 700 is shown for providing clinical guidance to a care provider via a clinical recommendation system, wherein the clinical recommendation system relies on an optimized LLM to process a query submitted by a care provider. The optimized LLM may be Figure 3B The method 700 may be performed by a processor of a clinical recommendation system such as Figure 1 The processor 104 of the clinical recommendation system 102 of the present invention executes based on instructions stored in a non-transitory memory of the clinical recommendation system, such as in the inference module 111. The method 700 can be performed in real time, wherein no intentional delay is introduced when generating a response to a query submitted by a care provider to the clinical recommendation system.

[0089] The method 700 begins at 702, where the method 700 includes receiving a query from a care provider. The query may be a question asked by the care provider regarding a treatment for a patient being treated by the care provider. For example, the care provider may be unsure of a possible treatment, or may have two or more options for treating a patient, and may wish to consult a clinical recommendation system to determine which of the two or more options to select. The care provider may request a treatment via a UI (such as a UI of a clinical recommendation system) of the clinical recommendation system. Figure 4 404). In some embodiments, the care provider may submit the query in written form, such as via a keyboard. In other embodiments, the care provider may submit the query by voice by speaking into a microphone. The query may be posed in natural language, such as as a question or instruction. For example, a query submitted by a care provider may be "Given stage 4 lung cancer with TNM status [xxx] and surgical margin [yyy], what should the next step be?".

[0090] At 704, method 700 includes converting the query into a prompt word to be submitted to the LLM. Converting the query into the prompt word may include converting an audio clip or audio file received by the clinical recommendation system via a microphone into a written form. In addition, the format of the query can be adjusted to the input format of the LLM. For example, the query can be posed by a care provider as a question, and the question can be converted into an instruction for the LLM.

[0091] At 706, method 700 includes submitting a prompt word to the LLM. The prompt word can be input into the LLM, and the LLM can output a response to the prompt word. The response can include clinically interpretable information, such as why the patient is at a certain node or stage in the recommendation model (and guidelines). The response can also indicate a proposed or recommended next step based on the patient's location and existing clinical / pathological variables according to clinical guidelines.

[0092] At 708, method 700 includes extracting patient data included in the submitted prompt word. In various embodiments, a word segmenter (e.g., Figure 4 The word segmenter 410) extracts patient data.

[0093] At 710, method 700 includes predicting the most likely treatment based on the extracted patient data using a prediction model (e.g., prediction model 412). As described above, the prediction model can take as input the patient's current and previous clinical pathology variables (e.g., extracted patient data) and output a treatment alternative having the highest probability of being selected for the patient based on demographic and historical patient data.

[0094] At 712, method 700 includes extracting, with the help of a tokenizer, a treatment recommended by the LLM from the response output by the LLM in a manner similar to extracting patient data.

[0095] At 714, method 700 includes determining whether the recommended treatment extracted by the LLM matches the predicted treatment output by the predictive model. The outputs can be compared based on the closeness of their embedding in feature space, as measured by Mahalanobis, Wasserstein, Euclidean, and / or similar types of distance measures, and the highest ranked recommendation will be provided. In addition, the recommendation can be matched to the decision tree of the clinical guideline for completeness checking.

[0096] If the recommended treatment matches the predicted treatment, method 700 proceeds to 716. At 716, method 700 includes delivering the response output by the LLM to the care provider. In various embodiments, the response can be displayed to the care provider on a display device via a UI of the clinical recommendation system (e.g., via UI 404 and / or display device 134). In some embodiments, the response can be converted into an audio signal using text-to-speech software, and the audio signal can be delivered to the care provider via a speaker or a device including a speaker coupled to the clinical recommendation system. For example, the care provider may not have immediate access to the display device, and the care provider may choose to route the response to the care provider's smart phone.

[0097] At 718, method 700 includes saving the recommended treatment to the patient's patient decision workflow.

[0098] Alternatively, if the recommended treatment does not match the predicted treatment at 714, the method 700 proceeds to 720. At 720, the method 700 includes recording the inconsistency between the recommended treatment and current or historical practice based on demographic and historical patient data. The inconsistency can indicate that the clinical guidelines on which the recommendation model is based are not consistent with current practice. In this case, intervention can be performed to ensure that current practice complies with the clinical guidelines. Alternatively, if the current practice is shown to result in a higher probability of a positive outcome, the clinical guidelines can be updated. In this way, the clinical guidelines can be kept up to date and consistency with the clinical guidelines can be promoted.

[0099] Therefore, the proposed clinical recommendation system can support care providers by providing clinical guidance for patient treatment in a more natural way than the way clinicians currently interact with clinical guidelines. Rather than having to search and browse digital and / or online clinical guidelines using a GUI (which may require a series of actions, including reading, selecting control elements of the GUI, typing information into the GUI, and waiting for results), guidance can be requested in natural language, for example, via a microphone. By providing clinical guidance to care providers in a more efficient and faster way, the use of clinical recommendation systems and the clinical guidelines on which they are based can be ensured, encouraged, and increased, leading to a higher degree of positive outcomes.

[0100] The clinical recommendation system can provide the patient's clinically interpretable disease state and recommend the next course of action (e.g., treatment) based on clinical guidelines without burdening the clinician with time-consuming and / or cumbersome interactions with the computerized system. As a result of natural language-based interactions, the clinical recommendation system will reduce the digital device interactions of the care provider, thereby freeing the care provider to invest more time in the patient. Another advantage of the proposed clinical recommendation system is that a greater degree of transparency can be provided to the care provider and the patient for how to make clinical decisions and the data on which the clinical decisions are based. As a result, the clinical recommendation system can promote a knowledgeable decision-making process for the patient and encourage a greater degree of trust. In addition, the proposed clinical recommendation system can reduce the variability in the clinical decision-making process, and introduce a checklist for the clinician based on previous demographics, which is easy to handle and allows more effective reimbursement.

[0101] A technical effect of providing clinical recommendations to care providers via a natural language interface is that care providers can request and receive guidance based on clinical guidelines in a more efficient and rapid manner, resulting in increased reliance on clinical guidelines, increased positive patient outcomes, and more time spent with patients by clinicians.

[0102] The present disclosure also provides support for a method for a clinical recommendation system, the method comprising: using a first large language model (LLM) of the clinical recommendation system to generate an automated response to a natural language query received from a care provider in real time, the automated response comprising a recommended treatment for the patient, the LLM being trained on a first set of training data generated by applying a second recommendation model to anonymized patient data, the recommendation model being based on clinical guidelines; validating the automated response output by the LLM using a third prediction model; and in response to the automated response being validated, storing the automated response in a patient decision workflow of the patient, the patient decision workflow comprising a record of a clinical recommendation made for the patient using the clinical recommendation system; and displaying the automated response on a display device of the clinical recommendation system. In a first embodiment of the method, the recommendation model comprises one of a decision tree and a knowledge graph. In a second embodiment of the method optionally including the first embodiment, the prediction model comprises one of a Bayesian network and a convolutional neural network (CNN). In a third embodiment of the method optionally including one or both of the first and second embodiments, the prediction model is trained on a second set of training data generated by applying the recommendation model to the anonymized patient data. In a fourth embodiment of the method that optionally includes one or more or each of the first to third embodiments, using the predictive model to validate the automated response output by the LLM further comprises: extracting the patient data from the natural language query, extracting the recommended treatment from the automated response output by the LLM, using the predictive model to determine a second treatment with a highest probability of being selected based on the extracted patient data, in response to the recommended treatment matching the second treatment, validating the automated response, and in response to the recommended treatment not matching the second treatment, reporting an inconsistency between the recommended treatment and the clinical guidelines, the clinical guidelines comprising the National Comprehensive Cancer Network (NCCN) Clinical Practice Guidelines for Oncology (NCCN ). In a fifth embodiment of the method optionally including one or more or each of the first to fourth embodiments, the recommended treatment is based on a previous recommended treatment stored in a patient decision workflow for a previous patient. In a sixth embodiment of the method optionally including one or more or each of the first to fifth embodiments, validating the automated response output by the LLM further comprises comparing the recommended treatment of the automated response with a previous recommended treatment for a previous patient stored in the patient decision workflow for the previous patient. In a seventh embodiment of the method optionally including one or more or each of the first to sixth embodiments, the method further comprises: receiving patient outcome data corresponding to the patient from an electronic medical record (EMR), and using the patient outcome data to further train the predictive model. In an eighth embodiment of the method, which optionally includes one or more or each of the first to seventh embodiments, training the LLM on the first set of training data further includes: training the LLM on a general domain NLP dataset, fine-tuning the LLM by training the LLM on natural language training pairs using supervised learning, the natural language training pairs including a prompt word and a reference truth response to the prompt word, and optimizing the LLM by training the LLM on various corpora of unlabeled data using reinforcement learning. In a ninth embodiment of the method, which optionally includes one or more or each of the first to eighth embodiments, the natural language training pairs are generated by at least one of the human experts using the recommendation model as a guide and one or more chatbots traversing the recommendation model. In a tenth embodiment of the method, which optionally includes one or more or each of the first to ninth embodiments, fine-tuning the LLM by training the LLM on natural language training pairs using supervised learning further includes training the LLM on a first set of training pairs manually generated by the human expert, and then training the LLM on a second set of training pairs generated by the one or more chatbots. In an eleventh embodiment of the method that optionally includes one or more or each of the first to tenth embodiments, optimizing the LLM by training the LLM on various corpora of unlabeled data using reinforcement learning also includes: inputting a set of generated natural language training cues into the fine-tuned LLM multiple times to generate multiple alternative outputs, receiving rankings of the multiple alternative outputs by the human expert, training a reward model on the generated natural language training cues and the alternative outputs using the rankings as benchmark truth data, and training the LLM using reinforcement learning and a reinforcement learning algorithm based on the reward model.In a twelfth embodiment of the method which optionally includes one or more or each of the first to eleventh embodiments, the natural language query is received from the care provider by voice via a microphone and / or the response is delivered to the care provider via an audio signal delivered via a speaker.

[0103] The present disclosure also provides support for a clinical recommendation system, the clinical recommendation system including a processor and a non-transitory memory, the non-transitory memory storing instructions, the instructions when executed causing the processor to: receive a natural language query from a care provider via a user interface of the clinical recommendation system, the natural language query including patient data of a patient, generate a response to the natural language query using a large language model (LLM), the response including a recommended treatment based on the patient data, store the response in a patient decision workflow for the patient, and deliver the response to the care provider via a display device and / or a speaker of the clinical recommendation system, wherein the LLM is trained using supervised learning on natural language training pairs generated by humans or chatbots, the natural language training pairs generated by humans or chatbots are generated according to a recommendation model based on clinical guidelines, and the LLM is optimized by training the LLM on various corpora of unlabeled data using reinforcement learning based on a reward model trained on human-ranked training data. In a first embodiment of the system, the recommendation model is one of a decision tree and a knowledge graph, the decision tree and the knowledge graph including decision points where different treatments are recommended based on the patient data. In a second embodiment of the system optionally including the first embodiment, additional instructions are stored in the non-transitory memory, which, when executed, cause the processor to: validate the response using a predictive model trained to take the patient data as input, and output a treatment with the highest probability of being selected based on demographic and / or historical patient data using the recommendation model as a guide. In a third embodiment of the system optionally including one or both of the first and second embodiments, validating the response using the predictive model includes: extracting the patient data from the natural language query, extracting a first treatment from the response output by the LLM, using the predictive model to determine a second treatment with the highest probability of being selected based on the extracted patient data, validating the response in response to the first treatment matching the second treatment, and reporting an inconsistency between the recommended treatment and the clinical guideline in response to the first treatment matching the second treatment.

[0104] The present disclosure also provides support for a method for providing clinical guidance to a care provider via a clinical recommendation system, the method comprising: receiving a query from a care provider, the query including data of a patient, converting the query into natural language prompts, submitting the natural language prompts to a large language model (LLM), the LLM being trained to output a response with a recommended treatment for the patient based on the patient data and one or more sets of digital clinical guidelines, extracting the patient data from the natural language prompts, predicting a most likely treatment for the patient based on the extracted patient data using a predictive model trained on demographic and anonymized historical patient data, extracting the recommended treatment included in the response generated by the LLM, in response to the recommended treatment matching the predicted most likely treatment, comparing the recommended treatment to the predicted most likely treatment, storing the recommended treatment in a patient decision workflow for the patient, and delivering the response to the care provider via a display device and / or speaker of the clinical recommendation system, and in response to the recommended treatment not matching the predicted most likely treatment, recording the inconsistency of the recommended treatment with current practice. In a first embodiment of the method, the training of the LLM includes: fine-tuning the LLM by training the LLM on natural language training pairs using supervised learning, each natural language training pair including a prompt word and a baseline truth response to the prompt word generated by at least one of a human expert and a chatbot using a recommendation model trained on one or more sets of digital clinical guidelines as guidance, and optimizing the LLM by training the LLM on various unlabeled databases based on a reward model trained on human-ranked responses to a selected set of generated prompt words using reinforcement learning. In a second embodiment of the method that optionally includes the first embodiment, the recommendation model includes a decision tree based on a set of digital clinical guidelines, the recommendation model takes patient data as input, traverses the decision tree based on the patient data, and outputs a recommended treatment at a decision point of the decision tree.

[0105] When introducing the elements of various embodiments of the present disclosure, the articles "one", "a kind of" and "the" are intended to mean that there are one or more such elements. The terms "first", "second", etc. do not represent any order, amount or importance, but are used to distinguish one element from another element. The terms "include", "comprises", and "have" are intended to be inclusive, and mean that additional elements may also exist in addition to the listed elements. As used herein, the terms "connected to", "coupled to", etc., an object (e.g., material, element, structure, member, etc.) may be connected to or coupled to another object, regardless of whether the one object is directly connected or coupled to another object, or whether there are one or more intervening objects between the one object and another object. In addition, it should be understood that reference to "an embodiment" or "embodiment" of the present disclosure is not intended to be interpreted as excluding the existence of additional embodiments that are also combined with the cited features.

[0106] In addition to any previously indicated modifications, those skilled in the art may devise numerous other variations and alternative arrangements without departing from the spirit and scope of the present specification, and the appended claims are intended to cover such modifications and arrangements. Therefore, although the information has been described in detail and in detail as described above in conjunction with what are currently considered to be the most practical and preferred aspects, it will be apparent to those of ordinary skill in the art that many modifications, including but not limited to form, function, mode of operation, and use, may be made without departing from the principles and concepts set forth herein. Likewise, as used herein, the examples and embodiments are meant to be illustrative in all respects and should not be construed as limiting in any way.

Claims

1. A method for a clinical recommendation system, the method comprising: generating, in real time, an automated response to a natural language query received from a care provider using a first large language model (LLM) (712) of the clinical recommender system, the automated response including a recommended treatment for the patient, the LLM being trained (508) on a first set of training data generated (506) by applying a second recommendation model to anonymized patient data, the recommendation model being based on a clinical guideline (502); validating the automated response output by the LLM using a third predictive model (714); as well as In response to the automated response being validated, the automated response is displayed on a display device of the clinical recommendation system (716). 2 . The method according to claim 1 , wherein the recommendation model comprises one of a decision tree and a knowledge graph.

3. The method of claim 1, wherein the prediction model comprises one of a Bayesian network and a convolutional neural network (CNN).

4. The method of claim 1, wherein the predictive model is trained on a second set of training data generated by applying the recommendation model to the anonymized patient data (504).

5. The method of claim 1 , wherein validating the automated response output by the LLM using the predictive model further comprises: extracting the patient data from the natural language query (708); extracting the recommended treatment from the automated response output by the LLM (712); using the predictive model, determining a second treatment having a highest probability of being selected based on the extracted patient data (710); In response to the recommended therapy matching the second therapy, validating the automated response (714); as well as In response to the recommended treatment not matching the second treatment, reporting an inconsistency between the recommended treatment and the clinical guideline (718).

6. The method of claim 1, wherein the clinical guidelines include the National Comprehensive Cancer Network (NCCN) Clinical Practice Guidelines for Oncology (NCCN Guidelines ).

7. The method according to claim 1, further comprising: receiving patient outcome data corresponding to the patient from an electronic medical record (EMR); as well as The patient outcome data is used to further train the prediction model (520).

8. The method of claim 1, wherein training the LLM on the first set of training data further comprises: Training the LLM on a general domain NLP dataset (508); Fine-tuning the LLM by training the LLM on natural language training pairs using supervised learning, the natural language training pairs comprising a cue word and a ground truth response to the cue word (510); as well as The LLM is optimized by training the LLM on various corpora of unlabeled data using reinforcement learning (516).

9. The method of claim 8, wherein the natural language training pairs are generated by at least one of the human experts using the recommendation model as a guide and one or more chatbots traversing the recommendation model.

10. The method of claim 9, wherein fine-tuning the LLM by training the LLM on natural language training pairs using supervised learning further comprises training the LLM on a first set of training pairs manually generated by the human expert, and then training the LLM on a second set of training pairs generated by the one or more chatbots.

11. The method of claim 9, wherein optimizing the LLM by training the LLM on various corpora of unlabeled data using reinforcement learning further comprises: Inputting a set of generated natural language training prompts into the fine-tuned LLM multiple times to generate multiple alternative outputs; receiving a ranking of the plurality of alternative outputs by the human expert (512); training a reward model on the generated natural language training prompts and the alternative outputs using the rankings as ground truth data (514); as well as The LLM is trained using reinforcement learning based on the reward model and a reinforcement learning algorithm (516).

12. The method of claim 11, wherein the reinforcement learning algorithm is a proximal policy optimization (PPO) reinforcement learning algorithm.

13. The method of claim 1, wherein the natural language query is received from the care provider by voice via a microphone and / or the response is delivered to the care provider via an audio signal delivered through a speaker.

14. A clinical recommendation system (102), the clinical recommendation system comprising a processor (104) and a non-transitory memory (106), the non-transitory memory storing instructions that, when executed, cause the processor (104) to: receiving a natural language query from a care provider (402) via a user interface (404) of the clinical recommender system (102), the natural language query including patient data of a patient; generating a response (408) to the natural language query using a large language model (LLM) (360), the response (408) comprising a recommended treatment based on the patient data; as well as delivering the response to the care provider (402) via a display device (134) and / or a speaker (135) of the clinical recommender system (102); in The LLM (360) is trained using supervised learning on human or chatbot generated natural language training pairs generated according to a recommendation model (204) based on clinical guidelines (202), and the LLM (310) is optimized by training on various corpora of unlabeled data (351) using reinforcement learning based on a reward model (354) trained on human ranked training data.

15. The clinical recommendation system (102) of claim 14, wherein the recommendation model (204) is one of a decision tree and a knowledge graph, the decision tree and the knowledge graph comprising decision points where different treatments are recommended based on the patient data.

Images