Clinical management device, clinical management method, and clinical management system

Abstract

The present disclosure relates to a clinical management technique for facilitating robust clinical prediction and treatment management considering a wide range of personal health factors. This clinical management device is configured to receive a user input query and a set of user health data, identify a target machine learning model that achieves a correlation threshold for a set of user health characteristics included in the user health data, generate a set of prediction result data including a set of recommended actions associated with an aggregate prediction score and the set of user health data, and output the set of prediction result data in response to the user input query.

Description

Clinical management device, clinical management method and clinical management system

[0001] The present disclosure relates to a clinical management device, a clinical management method and a clinical management system.

[0002] In recent years, machine learning (ML) techniques have been developed for application in a wide range of fields. In machine learning, training data based on known cases is input into a computer. The computer analyzes the training data to learn a model that generalizes the relationship between factors (sometimes called explanatory or independent variables) and outcomes (sometimes called dependent or dependent variables). This model can then be used to predict outcomes for unknown cases. As an example, machine learning techniques can be used to predict clinical treatment outcomes for patients based on past treatment data for similar patients.

[0003] Conventionally, techniques for improving the performance of machine learning techniques have been considered. For example, Patent Literature 1 discloses that "a technique for improved machine learning is provided. Patient data describing a patient is received and a medication to be evaluated for the patient is identified. At least a subset of the patient data is processed using a hybrid machine learning model comprising a static portion and a dynamic portion to generate an efficacy score, the efficacy score indicating the predicted efficacy of the medication for the patient. A medication is provided for the patient based at least in part on the efficacy score."

[0004] U.S. Patent Application No. 20230298722A1

[0005] The management of complex chronic diseases (such as diabetes) can be prone to clinical variability, where clinical treatment outcomes deviate from expected results. Clinical variability can create challenges related to unnecessary treatment costs or harm to patients. In some cases, clinical variability can result from a patient's personal health factors (e.g., comorbidities) that were not taken into account during treatment planning.

[0006] However, conventional machine learning-based systems have limited ability to identify clinical variations and provide adjustments to treatment plans. For example, U.S. Patent Application Publication No. 2013 / 0129999 discloses an efficacy score that indicates the effectiveness of a patient's drug therapy based on patient data using a machine learning model, but does not consider possible clinical variations resulting from secondary personal health factors. Similarly, recent advances in generative AI have enabled the creation of large language model (LLM)-based patient portals that provide answers to treatment-related questions, but these LLMs lack a fact-based disease understanding to support patient-specific queries.

[0007] It is therefore an object of the present disclosure to provide a clinical management technique in which multiple machine learning models selected based on a patient's health data are used to facilitate robust clinical prediction and treatment management taking into account a wide range of personal health factors.

[0008] A representative example of the present disclosure relates to a clinical management device comprising: a processor; and a memory, wherein the memory includes a set of computer-readable instructions that cause the processor to: receive a user input query from a user related to a set of user health data characterizing a set of clinical concepts and a set of user health features; identify a first set of target machine learning models from a set of candidate machine learning models based on the user input query and the set of user health data, the target machine learning models being trained to make predictions on a set of prediction targets related to the set of clinical concepts of the user input query; identify a subset of the first set of target machine learning models that reach a correlation threshold with respect to the set of user health features included in the user health data; generate a set of predicted outcome data including aggregate prediction scores for the set of prediction targets and a set of recommended actions associated with the set of user health data by using the subset of the first set of target machine learning models to analyze the set of user health data; and output the set of predicted outcome data in response to the user input query.

[0009] According to the present disclosure, a clinical management technique can be provided in which multiple machine learning models selected based on patient health data are used to facilitate robust clinical prediction and treatment management taking into account a wide range of personal health factors.

[0010] Problems, configurations, and advantages other than those mentioned above will become apparent from the following description of embodiments for carrying out the present invention.

[0011] 1 illustrates an exemplary computing architecture for implementing embodiments of the present disclosure; 2 illustrates an exemplary hardware configuration of a clinical management system according to embodiments of the present disclosure; 3 illustrates an example of a flow of a clinical management system according to embodiments of the present disclosure; 4 illustrates an example of using clinical concepts to identify clinical symptoms in a user input query according to embodiments of the present disclosure; 5 illustrates an example of identifying a first set of target machine learning models using a clinical concept database and identified clinical symptoms according to embodiments of the present disclosure; 6 illustrates an example of identifying a subset of the first set of target machine learning models that achieves a correlation threshold with respect to the set of user health features included in the user health data; 7 illustrates an example of training a set of machine learning models to predict risk of clinical variation with respect to a set of prediction targets; 8 illustrates an example of generating and outputting a set of predicted outcome data according to embodiments of the present disclosure.

[0012]

[0023] Herein, embodiments of the present invention will be described with reference to the drawings. It should be noted that the embodiments described herein are not intended to limit the present invention in accordance with the claims, and it should be understood that each element and combination of elements described with respect to the embodiments is not strictly necessary to practice aspects of the present invention.

[0013] Various aspects are disclosed in the following description and related drawings. Alternative aspects may be devised without departing from the scope of the present disclosure. Additionally, well-known elements of the present disclosure will not be described in detail or will be omitted so as not to obscure the relevant details of the present disclosure.

[0014] The words "exemplary" and / or "example" are used herein to mean "serving as an example, instance, or illustrative example." Any aspect described herein as "exemplary" and / or "example" is not necessarily to be construed as preferred or advantageous over other aspects. Likewise, the phrase "aspects of the present disclosure" does not require that all aspects of the present disclosure include the discussed feature, advantage, or characteristic of operation.

[0015] Furthermore, many aspects are described in terms of sequences of actions to be performed by, for example, elements of a computing device. It will be recognized that the various actions described herein can be performed by specific circuitry (e.g., an application-specific integrated circuit (ASIC)), by program instructions executed by one or more processors, or a combination of both. In addition, the sequences of actions described herein can be considered to be embodied as a whole in any form of computer-readable storage medium storing a corresponding set of computer instructions that, when executed, can cause an associated processor to perform the functions described herein. Thus, various aspects of the present disclosure may be embodied in many different forms, all of which are contemplated to be within the subject matter of the claims.

[0016] Detailed descriptions of embodiments of the present disclosure are described herein with reference to the drawings.

[0017] 1 is a schematic block diagram of a computer system 100, according to an embodiment, for implementing various embodiments of the present disclosure. The mechanisms and apparatus of the various embodiments disclosed herein are equally applicable to any suitable computing system. The major components of computer system 100 include one or more processors 102, memory 104, terminal interface 112, storage interface 113, I / O (input / output) device interface 114, and network interface 115, all of which are communicatively coupled, directly or indirectly, for inter-component communication via memory bus 106, I / O bus 108, bus interface unit 109, and I / O bus interface unit 110.

[0018] Computer system 100 may include one or more general-purpose programmable central processing units (CPUs) 102A and 102B, generally referred to herein as processors 102. In embodiments, computer system 100 may include multiple processors, although in particular embodiments, computer system 100 may alternatively be a single CPU system. Each processor 102 executes instructions stored in memory 104 and may include one or more levels of on-board cache.

[0019] In embodiments, memory 104 may include random-access semiconductor memory, storage devices, or storage media (either volatile or non-volatile) for storing or encoding data and programs. In particular embodiments, memory 104 represents the entire virtual memory of computer system 100 and may also include virtual memory of other computer systems coupled to computer system 100 or connected via a network. While memory 104 can be conceptually viewed as a single monolithic entity, in other embodiments, memory 104 is a more complex configuration, such as a hierarchy of caches and other memory elements. For example, memory may exist in multiple levels of caches, which may be further divided by function, whereby one cache holds instructions and another cache holds non-instruction data used by the processor. Memory may also be distributed and associated with different CPUs or sets of CPUs, as is known in any of a variety of so-called non-uniform memory access (NUMA) computer architectures.

[0020] Memory 104 may store all or a portion of the various programs, modules, and data structures for handling data transfers described herein. For example, memory 104 may store a clinical management application 150. In embodiments, clinical management application 150 may include instructions or statements that are executed on processor 102 or interpreted by processor 102 to perform functions as described further below.

[0021] In certain embodiments, clinical management application 150 is implemented in hardware via semiconductor devices, chips, logic gates, circuits, circuit cards, and / or other physical hardware devices instead of or in addition to a processor-based system. In embodiments, clinical management application 150 may include data in addition to instructions or statements. In certain embodiments, cameras, sensors, or other data input devices (not shown) may be provided in direct communication with bus interface unit 109, processor 102, or other hardware of computer system 100. In such a configuration, the need for processor 102 to access memory 104 and clinical management application 150 may be reduced.

[0022] Computer system 100 may include a bus interface unit 109 that handles communication between processor 102, memory 104, display system 124, and I / O bus interface unit 110. I / O bus interface unit 110 may be coupled to I / O bus 108 to transfer data to and from various I / O units. I / O bus interface unit 110 communicates via I / O bus 108 with multiple I / O interface units 112, 113, 114, and 115, also known as I / O processors (IOPs) or I / O adapters (IOAs). Display system 124 may include a display controller, display memory, or both. The display controller may provide video, audio, or both types of data to display device 126. Additionally, computer system 100 may include one or more sensors or other devices configured to collect and provide data to processor 102.

[0023] By way of example, computer system 100 may include biometric sensors (e.g., collecting heart rate data, stress level data), environmental sensors (e.g., collecting humidity data, temperature data, pressure data), motion sensors (e.g., collecting acceleration data, movement data), etc. Other types of sensors are possible. Display memory may be dedicated memory for buffering video data. Display system 124 may be coupled to a display device 126, such as a standalone display screen, a computer monitor, a television, a tablet, or the display of a handheld device.

[0024] In one embodiment, display device 126 may include one or more speakers for rendering audio. Alternatively, one or more speakers for rendering audio may be coupled to the I / O interface unit. In an alternative embodiment, one or more of the functions provided by display system 124 may be incorporated into an integrated circuit that also includes processor 102. Additionally, one or more of the functions provided by bus interface unit 109 may be incorporated into an integrated circuit that also includes processor 102.

[0025] The I / O interface section supports communication with various storage and I / O devices. For example, the terminal interface unit 112 supports connection of one or more user I / O devices 116, which may include user output devices (such as a video display device, speakers, and / or a television receiver) and user input devices (such as a keyboard, mouse, keypad, touchpad, trackball, buttons, light pen, or other pointing device). A user may use a user interface to operate the user input devices to provide input data and commands to the user I / O devices 116 and the computer system 100, and may also receive output data via the user output devices. For example, the user interface may be presented via the user I / O devices 116, such as displayed on a display device, played through speakers, or printed by a printer.

[0026] Storage interface 113 supports the connection of one or more disk drives or direct access storage devices 117 (typically rotating magnetic disk drive storage devices, but may alternatively be other storage devices, including arrays of disk drives or solid-state drives such as flash memory configured to appear as a single mass storage device to a host computer). In some embodiments, storage device 117 may be implemented by any type of secondary storage device. The contents of memory 104, or any portion thereof, may be stored in storage device 117 and retrieved from storage device 117 as needed. I / O device interface 114 provides an interface to any of a variety of other I / O devices or other types of devices, such as printers or fax machines. Network interface 115 provides one or more communication paths from computer system 100 to other digital devices and computer systems; these communication paths may include, for example, one or more networks 130.

[0027] 1 illustrates a particular bus structure providing direct communication paths between processor 102, memory 104, bus interface 109, display system 124, and I / O bus interface unit 110, in alternative embodiments, computer system 100 may include different buses or communication paths that may be configured in any of a variety of forms, such as hierarchical, star, or web configurations, multiple hierarchical buses, parallel and redundant paths, or point-to-point links in any other suitable type of configuration. Furthermore, while I / O bus interface unit 110 and I / O bus 108 are shown as separate respective components, computer system 100 may actually include multiple I / O bus interface units 110 and / or multiple I / O buses 108. While multiple I / O interface units are shown isolating I / O bus 108 from the various communication paths running to the various I / O devices, in other embodiments, some or all of the I / O devices are directly connected to one or more system I / O buses.

[0028] In various embodiments, computer system 100 is a multi-user mainframe computer system, a single-user system, or a server computer or similar device with little or no direct user interface, but which receives requests from other computer systems (clients). In other embodiments, computer system 100 may be implemented as a desktop computer, a portable computer, a laptop or notebook computer, a tablet computer, a pocket computer, a telephone, a smartphone, or any other suitable type of electronic device.

[0029] Next, with reference to FIG. 2, an exemplary hardware configuration of a clinical management system according to an embodiment of the present disclosure will be described.

[0030] 2 illustrates an exemplary hardware configuration of a clinical management system 200, according to an embodiment of the present disclosure. Clinical management system 200 relates to a system configured to facilitate robust clinical prediction and treatment management that takes into account a wide range of personal health factors.

[0031] 2 , a clinical management system 200 according to an embodiment of the present disclosure includes a clinical management device 210, a communication network 250, and a user terminal 260. The clinical management device 210 and the user terminal 260 may be communicatively connected via the communication network 250.

[0032] Here, the communications network 250 may include a local area network (LAN) connection, the Internet, a wide area network (WAN) connection, a metropolitan area network (MAN) connection, a Bluetooth connection, or the like.

[0033] Clinical management device 210 is an apparatus designed to facilitate robust clinical prediction and treatment management that takes into account a wide range of personal health factors. As illustrated in Figure 2, clinical management device 210 mainly includes a memory 220, a storage unit 230, a processor 244, and an input / output unit 246. In an embodiment, clinical management device 210 may be implemented using a computer system such as computer system 100 shown in Figure 1.

[0034] The memory 220 is a memory for storing a clinical management application 150 for implementing the functions of the clinical management technique according to an embodiment of the present disclosure. As shown in FIG. 2 , the clinical management application 150 may include an input processing unit 222, a model identification unit 224, a prediction management unit 226, a model training unit 228, and an output processing unit 229. The input processing unit 222, the model identification unit 224, the prediction management unit 226, the model training unit 228, and the output processing unit 229 may be implemented as software modules that comprise the clinical management application 150.

[0035] The input processing unit 222 is a functional unit for receiving a user input query related to a set of user health data characterizing a set of clinical concepts and a set of user health features. The user input query may be entered into the user terminal 260 using natural language via a user interface provided to a patient portal as part of the treatment management service. The user input query may include semantic content related to one or more clinical concepts. Here, clinical concepts refer to clinical concepts related to concepts or ideas, and may include, for example, diseases, treatments, symptoms, behaviors, disease management strategies, biochemistry, etc. In an embodiment, the input processing unit may receive a set of user health data along with the user input query. The set of user health data may include all or some of the user's medical history, laboratory test results, self-performed measurements (e.g., insulin measurements), etc., that characterize a set of user health features. Here, the set of user health features refers to attributes or characteristics that define, explain, or represent the user's health status. As described herein, a machine learning model according to the present disclosure can be used to generate predictions for a set of prediction targets based on a set of user health features. The input processing unit will be described in detail below, and therefore, a description thereof will be omitted here.

[0036] The model identification unit 224 is a functional unit for identifying, from a pre-trained set of candidate machine learning models, a specific group of target machine learning models to be used in generating predictions based on a set of user health features included in a set of user health data received from a user. More specifically, the model identification unit 224 may identify, from a set of candidate machine learning models based on a user input query and a set of user health data, a first set of target machine learning models trained to make predictions for a set of prediction targets related to the set of clinical concepts of the user input query, and then identify a subset of the first set of target machine learning models that reach a correlation threshold with respect to the set of user health features included in the user health data. In an embodiment, the model identification unit may identify the first set of target machine learning models using the clinical concept database 231 stored in the storage unit 230. In this way, the model identification unit 224 can identify a specific group of target machine learning models trained to make predictions based on a specific set of user health features included in the user health data, thereby promoting robust prediction performance. The function of the model identification unit 224 will be described later, and therefore its description will be omitted here.

[0037] The prediction management unit 226 is a functional unit for generating a set of predicted outcome data by analyzing a set of user health data using a subset of the first set of target machine learning models identified by the model identification unit 224. The set of predicted outcome data may include aggregate prediction scores for a set of predicted targets and a set of recommended actions associated with the set of user health data. In an embodiment, the set of recommended actions may be determined using a recommended action database 234 stored in the storage unit 230. In this manner, a set including multiple machine learning models may be used to generate a set of predictions, which are then aggregated together to provide comprehensive prediction results that take into account a wide range of user health characteristics along with recommended actions that are expected to have a positive impact on the user's health. The prediction management unit 226 will be described in detail below, and therefore its description will be omitted here.

[0038] The model training unit 228 is a functional unit for training machine learning models according to the present disclosure. In an embodiment, the model training unit 228 may train a first set of machine learning models to make predictions for a set of prediction targets using a set of healthcare data stored in the healthcare database 232 in the storage unit 230. In addition, the model training unit 228 may define a set of clinical variation thresholds corresponding to the set of prediction targets by analyzing a set of treatment outcome data using a statistical variation criterion, and train a second set of machine learning models to estimate a risk of clinical variation for the set of prediction targets using the set of healthcare data and the set of clinical variation thresholds. The first set of machine learning models and the second set of machine learning models may be stored in the candidate ML model database 233 in the storage unit 230 as a set of candidate machine learning models. In this manner, the first set of machine learning models may be prepared to make predictions based on the health data and the second set of machine learning models, thereby making predictions regarding the risk of clinical variation. Thus, a set of predicted outcome data may be generated taking into account a wide range of user health characteristics as well as the risk of clinical variation. The model training unit 228 will be described in detail later, and therefore its description will be omitted here.

[0039] The output processing unit 229 is a functional unit for outputting the set of predicted outcome data generated by the prediction management unit 226. In an embodiment, the output processing unit 229 may output the set of predicted outcome data to the user terminal 260 using natural language via a user interface provided in a patient portal as part of the treatment management service. In an embodiment, the output processing unit 229 may include a large-scale language model (LLM) configured to re-express, in natural language, the aggregate prediction score and the set of recommended actions included in the set of predicted outcome data in response to a user input query. In this way, a user can determine the generated health prediction and recommended actions in an understandable natural language format. The details of the output processing unit 229 will be described later, and therefore a description thereof will be omitted here.

[0040] The storage unit 230 is a unit for storing various data and information used in implementing aspects of the present disclosure. The storage unit 230 may include a collection of hard disk drives, solid state drives, flash memory, cloud storage, etc. As shown in FIG. 2 , the storage unit 230 may include a clinical concept database 231, a healthcare database 232, a candidate ML model database 233, and a recommended action database 234.

[0041] Clinical concept database 231 is a database configured to link pre-defined clinical conditions with a pre-trained set of candidate machine learning models stored in candidate ML model database 233 using clinical concepts. For example, clinical concept database 231 may define a correspondence between the clinical concept of "treatment" and machine learning models trained to make predictions related to a prediction target of "treatment effect" based on clinical conditions such as "type 2 diabetes," "type 1 diabetes," and "high blood pressure." In an embodiment, these clinical conditions may be identified from a user input query based on extracted keywords.

[0042] Health care database 232 is a database configured to store a set of health care data used to train a first set of machine learning models to make predictions for a set of prediction targets. The set of health care data may include historical health data (e.g., past medical history, medical history) defining multiple patient health characteristics along with observed treatment results or outcomes. By training the first set of machine learning models using the set of health care data stored in health care database 232, the first set of machine learning models learns to associate specific health characteristics with specific outcomes, thereby enabling the trained first set of machine learning models to make predictions for a wide variety of prediction targets based on sets of user health features included in sets of user health data received from users. More specifically, separate machine learning models may be trained to make predictions for each of several different prediction targets (e.g., treatment response, weight loss, risk of hospital readmission), etc. As described herein, aggregating results from multiple target machine learning models, each trained to make predictions based on a specific set of user health features included in user health data, can facilitate robust prediction performance.

[0043] Candidate ML model database 233 is a database configured to store a set of candidate machine learning models trained by model training unit 228. As described herein, the candidate machine learning models stored in candidate ML model database 233 may include a first set of machine learning models trained to make predictions based on health data and a second set of machine learning models trained to make predictions regarding risk of clinical variation. In this manner, a set of predicted outcome data can be generated that takes into account a wide range of user health characteristics as well as risk of clinical variation.

[0044] The recommended action database 234 is a database configured to link the prediction results generated by the prediction management unit with one or more predetermined recommended actions that are expected to have a positive impact on the user's health. In an embodiment, the recommended action database 234 may define multiple candidate recommended actions for each of a set of prediction targets, such that each candidate recommended action is associated with a salience threshold. By comparing the aggregate prediction score generated by the prediction management unit with the salience thresholds of the associated prediction targets, it becomes possible to select one or more recommended actions that are associated with the salience threshold achieved by the aggregate prediction score. As an example, the recommended action database may include recommended actions such as "schedule a visit with your general practitioner," "increase your insulin dosage," "increase your frequency of practice," and "reduce your alcohol intake."

[0045] Processor 244 is a processing unit for executing processing instructions for the various software modules and functional units included in clinical management application 150 stored in memory 220, and may substantially correspond to processor 102 of computer system 100 shown in FIG. 1.

[0046] Input / output unit 246 is a collection of devices for facilitating communication between clinical management device 210 and external sources, such as users of clinical management device 210 and one or more user terminals 260 to which clinical management device 210 is communicatively connected. In an embodiment, input / output unit 246 may include an integrated display unit, speaker, etc. for providing information to a user of clinical management device 210.

[0047] User terminal 260 is a device usable by a user (e.g., a client) of clinical management device 210. In an embodiment, user terminal 260 may be used to enter user input queries into clinical management device 210 as well as to determine the set of predicted outcome data generated by prediction management unit 226. By way of example, user terminal 260 may be implemented using a personal computer, a tablet computer, a smartphone, or other computing device.

[0048] The clinical management system 200 shown in FIG. 2 can provide a clinical management technique in which multiple machine learning models selected based on a patient's health data are used to facilitate robust clinical prediction and treatment management taking into account a wide range of personal health factors.

[0049] Next, an example of the flow of a clinical management system according to an embodiment of the present disclosure will be described with reference to FIG.

[0050] 3 illustrates an example flow diagram of the clinical management system 200, according to an embodiment of the present disclosure. Note that in the following description, it is assumed that a set of candidate ML models have already been trained and stored in the candidate ML model database 233. Similarly, it is assumed that the clinical concept database 231 and the recommended action database 234 have been pre-populated to facilitate the determination of the target machine learning model and recommended actions, respectively.

[0051] Initially, the user 305 submits a user input query 310 and a set of user health data 312 to the user terminal 260. As an example, the user 305 may submit the user input query "I've stopped exercising recently and I'm worried about my diabetes" along with a set of user health data 312 including a set of user health characteristics, such as laboratory values, demographic information, and vital signs, that characterize the user's health status. By way of example, the set of user health factors may include hemoglobin A1C (hereinafter, A1C) level, age, weight, height, albumin, etc. In an embodiment, the set of health factors may be structured in a (name, value, unit) format indicating the name, measurement value, and measurement unit of the health factor. The user terminal 260 may forward the received user input query 310 and set of user health data 312 to the clinical management device 210 via a secure data transmission channel.

[0052] Next, the input processing unit 222 may analyze the user input query 310 using a natural language processing technique to identify a first set of keywords that characterize the set of clinical concepts, where the input processing unit 222 may use Rapid Automatic Keyword Extraction (RAKE), Yet Another Keyword Extractor (YAKE), Word Embeddings, Named Entity Recognition (NER), Term Frequency-Inverse Document Frequency (TF-IDF), etc. as the natural language processing technique to identify the first set of keywords. For example, from a user input query such as "I've stopped exercising recently and I'm worried about my diabetes," the input processing unit 222 may extract the keywords "diabetes" and "exercise."

[0053] Next, the model identification unit 224 identifies a first set of target machine learning models, which are stored in the healthcare database 232 and pre-trained based at least on the set of healthcare data stored in the candidate ML model database 233, trained to make predictions about a set of prediction targets related to the set of clinical concepts of the user input query by comparing the keywords extracted by the input processing unit 222 with the clinical concept database 231. Here, the set of prediction targets includes factors or attributes for which predictions are to be generated. By way of example, the set of prediction targets may include "treatment effect," "weight loss effect," "clinical fluctuation risk," "hospitalization risk," etc. The model identification unit may then identify, from the first set of target machine learning models, a subset of the first set of target machine learning models that achieves a correlation threshold with the set of user health features included in the user health data 312. In this manner, the model identification unit 224 can filter the first set of target machine learning models to select those with the highest correlation degree to be used in generating the set of predicted outcome data.

[0054] Next, the prediction management unit 226 generates a set of predicted outcome data by analyzing the set of user health data using a subset of the first set of target machine learning models identified by the model identification unit 224. As described herein, the set of predicted outcome data may include an aggregate prediction score for a set of predicted targets and a set of recommended actions associated with the set of user health data. In an embodiment, the prediction management unit 226 may determine the set of recommended actions by comparing the aggregate prediction score to a salience threshold associated with pre-determined recommended actions stored in the recommended action database 234 for each of the predicted targets, and selecting one or more recommended actions associated with the salience threshold achieved by the aggregate prediction score.

[0055] The output processing unit 229 then, in response to the user input query, uses the LLM to re-express the aggregate prediction score and the set of recommended actions included in the set of predicted outcome data into natural language for output and sends the generated output data 320 via secure data transmission to the user terminal 260. In one example, the output processing unit 229 may provide output data 320 that reads, "Diabetic patients with weight change are at higher risk of needing treatment adjustment. Schedule an appointment with your doctor."

[0056] The clinical management system 200 shown in FIG. 3 can provide a clinical management technique in which multiple machine learning models selected based on a patient's health data are used to facilitate robust clinical prediction and treatment management taking into account a wide range of personal health factors.

[0057] Referring now to FIG. 4, an example of using clinical concepts to identify clinical symptoms in a user-entered query will be described, according to an embodiment of the present disclosure.

[0058] FIG. 4 illustrates an example of using the clinical concepts database 231 to identify clinical symptoms in a user-entered query, according to an embodiment of the present disclosure.

[0059] As described herein, in response to receiving a user input query 310 from a user 305, the input processing unit 222 may analyze the user input query 310 using natural language processing techniques to identify a first set of keywords 415 that characterize a set of clinical concepts. As an example, if the input processing unit 222 receives the user input query 310, "I've stopped exercising recently and I'm worried about my diabetes," the input processing unit 222 may extract the keywords "diabetes" and "exercise," and the input processing unit 222 may use natural language processing techniques to extract the first set of keywords 415, "diabetes" and "stop exercising."

[0060] Next, input processing unit 222 may match the first set of extracted keywords against clinical concept database 231 to identify clinical conditions (e.g., a first set of clinical conditions) associated with the first set of keywords. As an example, as shown in FIG. 4 , because the first set of keywords 415, “diabetes” and “exercise,” are associated with the clinical condition “type 2 diabetes” in clinical concept database 231, input processing unit 222 may identify “type 2 diabetes” as the clinical condition corresponding to the first set of keywords. In an embodiment, if the first set of keywords 415 extracted from the user input query are associated with multiple different clinical conditions in clinical concept database 231, input processing unit 222 may select the clinical condition with the highest number of matching keywords (e.g., the highest frequency count).

[0061] In this manner, one or more clinical conditions characterized by the keywords of the user input query can be identified, and as described herein, these identified clinical conditions can be used to facilitate the identification of a first set of target machine learning models trained to make predictions regarding the user health data received from the user 305.

[0062] Referring now to FIG. 5, an example of identifying a first set of target machine learning models using clinical concepts and identified clinical symptoms will be described according to an embodiment of the present disclosure.

[0063] FIG. 5 illustrates an example of identifying a first set of target machine learning models using the clinical concept database 231 and identified clinical symptoms, according to an embodiment of the present disclosure.

[0064] As described herein, clinical concept database 231 may be used to map pre-defined clinical symptoms 510 with a pre-trained set of candidate machine learning models 520 stored in candidate ML model database 233 using clinical concepts 515. For example, clinical concept database 231 may define a correspondence between a particular clinical symptom 510, a particular clinical concept 515, and a particular machine learning model 520.

[0065] 5 , when a clinical condition 510 of “type 2 diabetes” is identified based on keywords extracted from a user-input query, the model identification unit 224 may first identify a set of clinical concepts 515 of “treatment,” “lifestyle,” and “disease management” that are linked to the clinical condition 510 of “type 2 diabetes” in the clinical concept database 231, and then select a target machine learning model mapped to the identified clinical concept, where each of the target machine learning models may be trained to make predictions about prediction targets related to the clinical concept.

[0066] For example, for the clinical concept 515 of "treatment," the model identification unit 224 may select a target machine learning model trained to make predictions for the prediction targets of "treatment effect," "adverse events," and "ED (emergency department) visit risk." For the clinical concept 515 of "lifestyle," the model identification unit 224 may select a target machine learning model trained to make predictions for the prediction targets of "treatment effect," "weight loss effect." Similarly, for the clinical concept 515 of "disease management," the model identification unit 224 may select a target machine learning model trained to make predictions for the prediction target of "clinical variation risk."

[0067] In embodiments, a weight vector 525 may be calculated indicating the relative importance of each target machine learning model in the first set of target machine learning models with respect to the set of clinical concepts 515. In embodiments, this weight vector 525 may be calculated based on the number of clinical concepts 515 associated with each target machine learning model from among the clinical concepts identified for the particular clinical condition 510, such that target machine learning models associated with more clinical concepts are given a higher weight. For example, when a machine learning model trained to make predictions with respect to the prediction target "treatment effect" is associated with two clinical concepts, "treatment" and "lifestyle," while other machine learning models are associated with one or fewer clinical concepts in the set of clinical concepts 515, the machine learning model trained to make predictions with respect to the prediction target "treatment effect" may be given a higher relative weight. In embodiments, the weight vector 525 may be normalized to sum to one.

[0068] In this manner, the clinical concept database 231 can be used to select a set of target machine learning models pre-trained to make predictions for prediction targets related to clinical concepts associated with clinical symptoms characterized by user-entered keywords. This set of target machine learning models can be filtered to identify a subset of the first set of target machine learning models that achieve a correlation threshold with a set of user health features included in the user health data. This identified subset of the first set of target machine learning models can then be used as a network of linked machine learning models to generate a series of predictions related to a wide variety of user health features. These predictions can then be aggregated together to create a comprehensive and comprehensive predictive result to facilitate clinical treatment for the user.

[0069] Referring now to FIG. 6, an example will be described of identifying a subset of a first set of target machine learning models that achieve a correlation threshold for a set of user health features included in the user health data.

[0070] 1 illustrates an example of identifying a subset of a first set of target machine learning models that achieve a correlation threshold with a set of user health features included in user health data. As described herein, aspects of the present disclosure relate to selecting a set of target machine learning models that have been pre-trained to make predictions for a prediction target related to a clinical concept associated with a clinical symptom characterized by keywords in a user input. Each target machine learning model in this set of target machine learning models is trained to make predictions using a particular set of input health features. However, in some cases, the set of input features used by a particular machine learning model may not be present in the set of health data provided by the user. Accordingly, to promote prediction accuracy, it is desirable to filter the first set of target machine learning models to select a subset of target machine learning models that make predictions using input health features that substantially correspond to the user health features included in the set of user data.

[0071] First, the model identification unit 224 identifies a set of input health features 620 for each target machine learning model in the first set of target machine learning models 610, where the set of input health features 620 are features used by the particular target machine learning model to make predictions. For example, as shown in FIG. 6 , for a machine learning model trained to make predictions on a prediction target of “treatment effect,” the model identification unit 224 may identify a set of input health features such as “(albumin, g / dL), (calcium, mg / dL), (A1C, %),” etc.

[0072] Next, the model identification unit 224 calculates an individual correlation score between each target machine learning model in the first set of target machine learning models 610 and each health feature in the set of user health features 630 based on the degree of overlap between the set of input health features 620 and the set of user health features 630 included in the user health data 312. Here, the individual correlation score is a numerical value representing the degree to which each target machine learning model corresponds to each health feature in the set of user health features 630. The individual correlation scores between the set of input health features 620 and the set of user health features 630 included in the user health data 312 may be calculated using a semantic similarity algorithm (e.g., cosine similarity) configured to compare the (name, value, unit) of each feature in the set of input health features 620 and the set of user health features 630. As an example, the set of individual correlation scores may be expressed in the form (c1, c2, c3, c4, ... cn), where each value indicates the degree of similarity between a particular input health feature in the set of input health features 620 and a particular user health feature in the set of user health features 630 for a particular target machine learning model.

[0073] The model identification unit 224 then calculates an aggregate correlation score between each target machine learning model in the first set of target machine learning models and the set of user health features by adding the individual correlation scores between each target machine learning model in the first set of target machine learning models 610 and each health feature in the set of user health features 620. The aggregate correlation score (e.g., “C”) is a numerical value that represents the degree to which each target machine learning model corresponds to the set of user health features 630 as a whole.

[0074] Next, model identification unit 224 selects target machine learning models having aggregate correlation scores C that achieve a correlation threshold as a subset of the first set of target machine learning models 640. Here, the correlation threshold is a predetermined correlation score used to specify a desired level of correlation and may be freely set by a user. In an embodiment, model identification unit 224 may revise the weight vector calculated as described with reference to FIG. 5 for the subset of first set of target machine learning models 640 by uniformly distributing the weights of the machine learning models that were not selected as a subset of the first set of target machine learning models 640. This revised weight vector may be stored together with the corresponding machine learning models in candidate ML model database 233.

[0075] 6, it is possible to select a subset of candidate machine learning models from a set of candidate machine learning models that are both trained to make predictions for a set of prediction targets related to a set of clinical concepts characterized by the input query and that achieve a correlation threshold with respect to a set of user health features included in the user health data. Selecting a subset of machine learning models using this two-step process can facilitate the generation of accurate predictions for the user input query while taking into account a wide range of personal health factors.

[0076] Next, with reference to FIG. 7, an example of training a set of machine learning models to predict the risk of clinical variation for a set of prediction targets will be described.

[0077] As described herein, aspects of the present disclosure relate to the recognition that, in some cases, clinical variation may occur, causing clinical treatment outcomes to deviate from expected results. Clinical variation may create issues related to unnecessary treatment costs or harm to patients. Accordingly, aspects of the present disclosure relate to preparing a set of machine learning models (e.g., a second set of machine learning models) for inclusion in a set of candidate machine learning models trained to predict the risk of clinical variation for a particular prediction target. Figure 7 illustrates an example of training a set of machine learning models to predict the risk of clinical variation for a set of prediction targets.

[0078] First, the model training unit 228 analyzes treatment outcome data 710 included in a set of healthcare data stored in the healthcare database 232 using statistical analysis techniques to define a set of clinical variation thresholds 720 corresponding to a set of prediction targets. The treatment outcome data 710 may include information indicating the outcomes of various treatments administered to past patients. As an example, the treatment outcome data 710 may indicate the progression of a patient's A1C (e.g., blood glucose level) values ​​over a time period during which a particular treatment was administered. The set of clinical variation thresholds 720 may be numerical thresholds that define the boundary between expected and unexpected measurements that should be considered clinical variation. In an embodiment, the model training unit 228 may fit a variance to the treatment outcome data 710 and segment the treatment outcome data into standard deviations from the mean. The set of clinical variation thresholds 720 may be defined in terms of a certain number of standard deviations from the mean (e.g., three standard deviations) so that data points that are more than a certain number of standard deviations from the mean (e.g., three standard deviations) are considered to represent clinical variation. For the treatment outcome data 710 corresponding to each of the set of prediction targets, the model training unit 228 may perform this process.

[0079] The model training unit 228 may then generate a set of training data 730 by assigning labels to the treatment outcome data 710 indicating the presence or absence of clinical variation. As shown in Figure 7, the set of training data 730 may include specific data points or ranges of data points (e.g., A1C values) along with a label indicating whether the data point was determined to qualify as a clinical variation.

[0080] Next, the model training unit 228 may use a machine learning algorithm 740 (e.g., supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning) to train a second set of machine learning models 750 using the set of training data 730. By training a machine learning model using the set of training data 730, it is possible to prepare a second set of machine learning models 750 trained to predict the risk of clinical variation occurring for a specific prediction target. As an example, referring to FIG. 7 , it is possible to obtain a machine learning model trained to predict treatment effect variation in A1C values ​​for future patients by training the machine learning model using a set of training data annotated to indicate the presence or absence of clinical variation in A1C values ​​as part of the treatment treatment. The second set of machine learning models 750 trained in this way may be added to the candidate ML model database 233 stored together with the first set of machine learning models based on the set of healthcare data stored in the healthcare database 232 as described above. In this way, clinical variation for a specific prediction target also becomes a new prediction target for prediction. A second set of machine learning models 750 can then be selected for use with the first set of machine learning models so that predictions about a particular prediction target (e.g., treatment effect) can be made along with predictions about clinical variation (e.g., treatment effect variation) in that particular prediction target.

[0081] According to the process described with reference to Figure 7, it is possible to prepare a set of machine learning models trained to predict the risk of clinical variation for a particular prediction target, thus providing a clinical management technique that can make robust clinical predictions based on user healthcare data while also taking into account the likelihood of clinical risk.

[0082] Next, with reference to FIG. 8, an example of generating and outputting a set of predicted result data according to an embodiment of the present disclosure will be described.

[0083] 8 illustrates an example of generating and outputting a set of predicted outcome data according to an embodiment of the present disclosure. As described herein, in an embodiment, the set of predicted outcome data may include an aggregate prediction score, a set of recommended actions, and a set of feature ranking data. In an embodiment, the set of predicted outcome data is processed using a large-scale language model and re-expressed in a natural language format for presentation in response to a user input query.

[0084] Initially, the prediction management unit 226 may analyze the set of user health data 312 using a subset of the first set of target machine learning models to generate predictive data. Note that the set of predictive data is generated using a subset of the first set of target machine learning models (e.g., a subset of the first set of target machine learning models identified by the process described with respect to FIGS. 4-7 ) that are both trained to make predictions for a set of prediction targets related to the set of clinical concepts characterized by the input query and that achieve a correlation threshold with respect to the set of user health features included in the user health data.

[0085] More specifically, the prediction management unit 226 may generate a set of individual prediction scores 810 (p1, p2, p3, ... pn) for the set of user health data by analyzing the set of user health features 312 using each target machine learning model in the subset of the first set of target machine learning models. Here, the set of individual prediction scores 810 is a value representing a predicted result or outcome for each prediction target. Note that the subset of the first set of target machine learning models used to generate the set of individual prediction scores is selected from a set of candidate machine learning models that includes both machine learning models trained to make predictions for user health features and machine learning models trained to make clinical predictions for clinical variation risk, so the set of individual prediction scores may include predictions for various individual health features as well as predictions for clinical variation risk. As an example, the subset of the first set of target machine learning models may generate sets of individual prediction scores 810 for the prediction targets of treatment effect, weight loss effect, and treatment transition risk (e.g., clinical variation for treatment transition).

[0086] Next, the prediction management unit 226 may retrieve the weight vector 525 (e.g., the revised weight vector calculated after identifying the first set of target machine learning models) from the candidate ML model database 233 and calculate a revised set of individual prediction scores 810 by multiplying the set of individual prediction scores 810 by the weight vector 525, where each individual prediction score is multiplied by the weight value calculated for the corresponding prediction target (e.g., the prediction score for “effect of treatment” is multiplied by the weight for the “effect of treatment” machine learning model). Next, the prediction management unit 226 may calculate an aggregate prediction score 822 (P) by adding the revised set of individual prediction scores, where this aggregate prediction score is a value indicating the degree of risk for the patient given the set of user health features characterized in the set of user health data 312. In an embodiment, the aggregate prediction score may be a value between 0 and 100, where 0 indicates a low level of risk and 100 indicates a high level of risk. In this way, by aggregating the modified set of individual prediction scores, it is possible to calculate a single aggregate prediction score 822 that is based not only on personal health factors such as the user's weight, age, and measured biochemistry values, but also takes into account the risk of clinical variation for the user.

[0087] The prediction management unit 226 may then use the aggregate prediction score 822 to determine a set of recommended actions 834. More specifically, the prediction management unit 226 may match the calculated aggregate prediction score 822 with a recommended action database 234 to determine the set of recommended actions 834. As described herein, the recommended action database 234 is a database for linking prediction results generated by the prediction management unit 226 with one or more predetermined recommended actions that are predicted to have a positive impact on the user's health. In an embodiment, the recommended action database 234 may define multiple candidate recommended actions for each of the set of prediction targets, such that each candidate recommended action is associated with a salience threshold. By comparing the aggregate prediction score 822 with the salience threshold of the associated prediction target, the prediction management unit 226 may select one or more recommended actions 834 associated with the salience threshold achieved by the aggregate prediction score 822. As an example, if the candidate recommended action "Schedule a visit with your GP" is associated with a salience threshold of "65" and the aggregate predicted score 822 is calculated to be "68," the recommended action "Schedule a visit with your GP" may be selected because the aggregate predicted score 822 of "68" meets the salience threshold of "65."

[0088] In embodiments, as described herein, the prediction management unit 226 may be configured to generate a set of feature ranking data 854 for inclusion in the set of predicted outcome data. The set of feature ranking data 854 includes data indicating the relative importance of each user health feature in generating the set of predicted outcome data. In particular, the prediction management unit 226 may calculate a set of feature importance scores 816 for each of the set of user health features using a feature importance method such as the Shapley additive explanations (SHAP) method. Here, the set of feature importance scores 816 may be expressed as a numerical value indicating the relative importance of each user health feature of the set of user health features included in the set of user health data 312 with respect to each target machine learning model of the subset of the first set of target machine learning models.

[0089] Next, the prediction management unit 226 may retrieve the weight vector 525 (e.g., the revised weight vector calculated after identifying the first set of target machine learning models) from the candidate ML model database 233 and calculate a revised set of feature importance scores by multiplying the set of feature importance scores 816 by the weight vector 525, where each feature importance score is multiplied by the weight value for its corresponding prediction target. The prediction management unit 226 may then calculate a set of aggregate feature importance scores 838 (F) by adding together the revised set of feature importance scores, where the set of aggregate feature importance scores 838 is a value indicating the relative importance of each health feature with respect to a particular prediction goal. In an embodiment, the set of aggregate feature importance scores may be a value between 0 and 100, where 0 indicates a low level of importance and 100 indicates a high level of importance.

[0090] The prediction management unit 226 may then generate a set of feature ranking data 854 by ranking the set of user health features based on the set of aggregate feature importance scores 838 such that user health features having higher aggregate feature importance scores are given higher rankings.

[0091] In an embodiment, the set of feature ranking data 854 may be calculated using an explanation matrix method. For example, as shown in FIG. 8 , each machine learning model of the subset of the first set of target machine learning models may be represented as a row of the matrix, and each user health feature may be represented as a column of the matrix. The feature importance score for each user health feature for each respective machine learning model may be calculated using the SHAP method and used to populate the cells of the matrix. The feature importance score in each cell of the matrix may then be multiplied by the associated weighting score for that row in the weight vector 525. An aggregate feature importance score may then be calculated by adding the feature importance scores across each row (or, alternatively, each column) to indicate the relative importance of each health feature with respect to a particular prediction target.

[0092] Output processing unit 229 may receive the aggregate prediction score 822, the set of recommended actions 834, and the set of prediction outcome data including feature ranking data 854 generated by prediction management unit 226, and prepare the received set of prediction outcome data for output to a user via user terminal 260. In an embodiment, output processing unit 229 may output the aggregate prediction score 822, the set of recommended actions 834, and the feature ranking data 854 without further processing. If the user is a public health physician with a comprehensive understanding of the clinical management process, it may be desirable to leave the aggregate prediction score 822, the set of recommended actions 834, and the feature ranking data 854 as is.

[0093] In certain embodiments, the output processing unit 229 may use a large-scale language model (LLM) configured to re-express the aggregate prediction score 822, the set of recommended actions 834, and the feature ranking data 854 into natural language as output in response to a user input query. As an example, the output processing unit 229 may generate a result such as, "Diabetic patients with weight change are at higher risk of needing treatment adjustments. Schedule an appointment with your doctor." As another example, if a high risk of treatment effect variability is identified, a candidate recommended action may be generated such as, "Your current treatment may not be optimal for you. Explore alternative treatment options." If the user is a patient accessing a clinical management system according to an embodiment of the present disclosure through a patient portal, it may be desirable to use the LLM to re-express the set of predicted outcome data in natural language form. In this way, the user may confirm the generated health prediction and recommended action in an easy-to-understand natural language form.

[0094] Further, in an embodiment, the output processing unit 229 may process the set of feature ranking data 854 to generate a set of explanatory data to explain why a particular recommended action was selected. In one example, if the feature ranking data 854 indicates that the user health feature "estimated average glucose" has the highest importance among the set of user health features, the output processing unit 229 may generate a set of explanatory data such as, "We recommend a diet plan for you because the amount of sugar in your daily diet may be affecting the effectiveness of treatment." In this way, by using the set of feature ranking data 854, it is possible to gain insight into which particular user health features contribute to a particular predicted outcome and to generate an explanation supporting the inference that a particular treatment plan or recommended action was proposed.

[0095] The process described with respect to Figure 8 can generate a set of predicted outcome data that provides a user with a robust health prognosis based on a wide range of personal health factors as well as the risk of clinical variation occurring. In this way, risk of clinical variation can be identified before it occurs, and treatment options can be selected that take into account secondary health factors and mitigate the clinical variation, promoting patient satisfaction.

[0096] As described herein, aspects of the present disclosure relate to training a set of candidate machine learning models, including a first set of machine learning models trained to make predictions for a set of prediction targets and a second set of machine learning models trained to make predictions regarding a risk of clinical variation in the set of prediction targets. From the set of candidate machine learning models, a subset of machine learning models can be selected that are both trained to make predictions for the set of prediction targets related to a set of clinical concepts characterized by the input query and that achieve a correlation threshold with a set of user health features included in the user health data. In this manner, selecting a subset of machine learning models from both the first and second sets of machine learning models can facilitate the generation of accurate predictions for the prediction query while also taking into account the risk of clinical variation in the prediction targets.

[0097] A further aspect of the present disclosure relates to generating recommended actions for users based on the generated prediction results. In certain embodiments, the recommended actions can be expressed in natural language by a large-scale language model and provided to users via a patient portal as part of a care management service. In this way, patients can receive real-time consultations about their care via the online portal, saving users the time and effort of scheduling appointments at medical facilities. Similarly, resources in that department of the medical facility can be saved.

[0098] A further aspect of the present disclosure relates to generating a set of feature ranking data indicating the relative importance of each user health feature in generating the set of predicted outcome data. Using this set of feature ranking data, it is possible to gain insight into which particular user health features contribute to a particular predicted outcome and to generate explanations supporting the reasoning behind why a particular treatment plan or recommended action is proposed. Thus, transparency of clinical management techniques is enhanced, and users can gain a better understanding of their treatment options.

[0099] Thus, the embodiments of the present disclosure described herein enable multiple machine learning models selected based on patient health data to provide the clinical management techniques described in the specifications to facilitate robust clinical prediction and treatment management taking into account a wide range of personal health factors.

[0100] The present invention may be a system, a method, and / or a computer program product, which may include a computer-readable storage medium having computer-readable program instructions for causing a processor to implement aspects of the present invention.

[0101] A computer-readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encryption devices such as punch cards or groove ridge structures having instructions recorded thereon, and any suitable combination of the above. As used herein, a computer-readable storage medium should not be construed as being itself a primary signal, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse passing through a fiber optic cable), or an electrical signal transmitted by an electrical wire.

[0102] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0103] The computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to manufacture a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for performing the functions / acts specified in the flowchart and / or block diagram blocks. These computer-readable program instructions may also be stored on a computer-readable storage medium that can cause the computer, programmable data processing apparatus, and / or other device to function in a particular way, such that the computer-readable storage medium having the instructions stored thereon comprises an article of manufacture containing instructions that implement aspects of the functions / acts specified in the flowchart and / or block diagram blocks.

[0104] The computer-readable program instructions may further be loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be executed on the computer, other programmable apparatus, or other device to create a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / acts specified in the flowchart and / or block diagram blocks.

[0105] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specialized logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It will also be appreciated that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs a specialized function or operation, or that executes a combination of dedicated hardware and computer instructions.

[0106] While the foregoing relates to exemplary embodiments of the present invention, other and further embodiments of the present invention may be devised without departing from the basic scope of the invention, which scope is determined by the following claims. The description of various embodiments of the present disclosure has been provided for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to explain the principles of the embodiments, practical applications or technical improvements of existing technologies, or to enable others of ordinary skill in the art to which the present disclosure pertains to understand the embodiments disclosed herein.

[0107] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of various embodiments. As used herein, the singular forms "a," "an," and "the" are intended to include the plural as well, unless the context clearly indicates otherwise. "Set," "group," "bundle," and the like are intended to include one or more. Furthermore, it will be understood that the words "comprise" and / or "comprising," as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. In the above detailed description of exemplary embodiments of various embodiments, reference has been made to the accompanying drawings, in which like numerals indicate like elements, which show, by way of example, specific exemplary embodiments and in which various embodiments may be practiced. The above embodiments have been described in sufficient detail to enable one skilled in the art to practice the embodiments, but other embodiments may be used, and logical, mechanical, electrical, and other changes may be made without departing from the scope of the various embodiments. In the above description, numerous specific details have been set forth to provide a thorough understanding of the various embodiments. However, various embodiments may be practiced without these specific details. In other instances, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the embodiments.

[0108] As described herein, clinical management techniques according to the present disclosure relate to the following aspects:

[0109] (Aspect 1) A clinical management device comprising: a processor; and a memory, wherein the memory includes a set of computer-readable instructions that cause the processor to: receive a user input query from a user related to a set of user health data characterizing a set of clinical concepts and a set of user health features; identify a first set of target machine learning models from a set of candidate machine learning models based on the user input query and the set of user health data, the target machine learning models being trained to make predictions on a set of prediction targets related to the set of clinical concepts; identify a subset of the first set of target machine learning models that meets a correlation threshold with respect to the set of user health features included in the user health data; generate a set of predicted outcome data including aggregate prediction scores for the set of prediction targets and a set of recommended actions associated with the set of user health data by analyzing the set of user health data using the subset of the first set of target machine learning models; and output the set of predicted outcome data in response to the user input query.

[0110] (Aspect 2) The clinical management device of Aspect 1, wherein the memory further includes a set of computer-readable instructions that cause the processor to generate a set of candidate machine learning models by: training a first set of machine learning models using the set of healthcare data to make predictions on the set of prediction targets, for inclusion in the set of candidate machine learning models; defining a set of clinical variability thresholds corresponding to the set of prediction targets by analyzing the set of treatment outcome data using statistical analysis techniques; and training a second set of machine learning models using the set of treatment outcome data and the set of clinical variability thresholds to estimate a risk of clinical variability with respect to the set of prediction targets, for inclusion in the set of candidate machine learning models.

[0111] (Aspect 3) The clinical management device of Aspect 1 or 2, wherein the memory further includes a set of computer-readable instructions that cause the processor to identify a first set of target machine learning models by: parsing a user input query using natural language processing techniques to extract a first set of keywords; identifying a first set of clinical symptoms associated with the first set of keywords; and selecting, based on a clinical concept database that links clinical symptoms with a set of candidate machine learning models, a target machine learning model associated with the first set of clinical concepts that correspond to the first set of clinical symptoms in the clinical concept database, using the set of clinical concepts as the first set of target machine learning models.

[0112] (Aspect 4) The clinical management device of any one of Aspects 1 to 3, wherein the memory further includes a set of computer-readable instructions that cause the processor to identify a subset of the first set of target machine learning models by: identifying a set of input health features for each target machine learning model in the first set of target machine learning models; calculating an individual correlation score between each target machine learning model in the first set of target machine learning models and each user health feature in the set of user health features based on an overlap between the set of input health features and the set of user health features; calculating an aggregate correlation score between each target machine learning model in the first set of target machine learning models and the set of user health features by adding the individual correlation scores between each target machine learning model in the first set of target machine learning models and each user health feature in the set of user health features; and selecting a target machine learning model having an aggregate correlation score that achieves a correlation threshold as the subset of the first set of target machine learning models.

[0113] (Aspect 5) The clinical management device of any one of Aspects 1 to 4, wherein the memory further includes a set of computer-readable instructions that cause the processor to generate a set of predicted outcome data by: generating a set of individual prediction scores for the set of user health data by analyzing the set of user health features using each target machine learning model of the subset of the first set of target machine learning models; calculating a weight vector indicating the relative importance of each target machine learning model of the subset of the first set of target machine learning models with respect to the set of clinical concepts; calculating a modified set of individual prediction scores by multiplying the set of individual prediction scores by the weight vector; and calculating an aggregate prediction score by adding the modified sets of individual prediction scores.

[0114] (Aspect 6) The clinical management device of Aspect 5, wherein the memory further includes a set of computer-readable instructions that cause the processor to generate the set of predicted outcome data by: calculating a set of feature importance scores using a feature importance method, the set of feature importance scores indicating the relative importance of each user health feature of the set of user health features with respect to each target machine learning model of the subset of the first set of target machine learning models; calculating a modified set of feature importance scores by multiplying the set of feature importance scores by a weight vector; calculating a set of aggregate feature importance scores indicating the relative importance of each user health feature with respect to a particular prediction target of the set of prediction targets by adding the modified set of feature importance scores; generating a set of feature ranking data that ranks the set of health features based on the set of aggregate feature importance scores; and outputting the set of feature ranking data as part of the set of predicted outcome data.

[0115] (Aspect 7) The clinical management device of Aspect 5, wherein the memory further includes a set of computer-readable instructions that cause the processor to generate the set of predicted outcome data by: determining a first recommended action associated with the aggregate prediction score according to a salience threshold achieved by matching the aggregate prediction score against a recommended action management database that defines a plurality of candidate recommended actions for a subset of the first set of target machine learning models; and outputting the first recommended action as one of the set of recommended actions.

[0116] (Aspect 8) The clinical management device of any one of Aspects 1 to 7, wherein the memory further includes a set of computer-readable instructions that cause the processor to output the set of predicted outcome data by: processing the set of predicted outcome data using a large-scale language model to generate a set of output data that expresses an aggregate prediction score and a set of recommended actions in a natural language format; and outputting the set of output data in response to a user input query.

[0117] 150 Clinical Management Application 200 Clinical Management System 210 Clinical Management Device 220 Memory 222 Input Processing Unit 224 Model Identification Unit 226 Prediction Management Unit 228 Model Training Unit 229 Output Processing Unit 230 Storage Unit 231 Clinical Concept Database 232 Healthcare Database 233 Candidate ML Model Database 234 Recommended Action Database 244 Processor 246 Input / Output Unit 250 Communication Network 260 User Terminal

Claims

1. A clinical management device comprising: a processor; and a memory, wherein the memory includes a set of computer-readable instructions that cause the processor to: receive a user input query from a user related to a set of user health data characterizing a set of clinical concepts and a set of user health features; identify a first set of target machine learning models from a set of candidate machine learning models based on the user input query and the set of user health data, the target machine learning models being trained to make predictions on a set of prediction targets related to the set of clinical concepts; identify a subset of the first set of target machine learning models that meets a correlation threshold with respect to the set of user health features included in the user health data; generate a set of predicted outcome data including aggregate prediction scores for the set of prediction targets and a set of recommended actions associated with the set of user health data by using the subset of the first set of target machine learning models to analyze the set of user health data; and output the set of predicted outcome data in response to the user input query.

2. The clinical management device of claim 1, wherein the memory further includes a set of computer-readable instructions that cause the processor to generate the set of candidate machine learning models by: training a first set of machine learning models using a set of healthcare data to make predictions on the set of prediction targets for inclusion in the set of candidate machine learning models; defining a set of clinical variability thresholds corresponding to the set of prediction targets by analyzing a set of treatment outcome data using statistical analysis techniques; and training a second set of machine learning models using the set of treatment outcome data and the set of clinical variability thresholds to estimate a risk of clinical variation with respect to the set of prediction targets for inclusion in the set of candidate machine learning models.

3. The clinical management device of claim 1, wherein the memory further includes a set of computer-readable instructions that cause the processor to identify the first set of target machine learning models by: parsing the user input query using natural language processing techniques to extract a first set of keywords; identifying a first set of clinical symptoms associated with the first set of keywords; and selecting a target machine learning model associated with a first set of clinical concepts that corresponds to the first set of clinical symptoms in the clinical concept database based on a clinical concept database that links clinical symptoms with the set of candidate machine learning models, using the set of clinical concepts as the first set of target machine learning models.

4. The clinical management device of claim 1, wherein the memory further includes a set of computer-readable instructions that cause the processor to identify the subset of the first set of target machine learning models by: identifying a set of input health features for each target machine learning model in the first set of target machine learning models; calculating an individual correlation score between each target machine learning model in the first set of target machine learning models and each user health feature in the set of user health features based on an overlap between the set of input health features and the set of user health features; calculating an aggregate correlation score between each target machine learning model in the first set of target machine learning models and the set of user health features by adding the individual correlation scores between each target machine learning model in the first set of target machine learning models and each user health feature in the set of user health features; and selecting a target machine learning model having an aggregate correlation score that achieves the correlation threshold as the subset of the first set of target machine learning models.

5. The clinical management device of claim 1, wherein the memory further includes a set of computer-readable instructions that cause the processor to generate the set of predicted outcome data by: generating a set of individual prediction scores for the set of user health data by using each target machine learning model of the subset of the first set of target machine learning models to analyze the set of user health features; calculating a weight vector indicating the relative importance of each target machine learning model of the subset of the first set of target machine learning models with respect to the set of clinical concepts; calculating a modified set of individual prediction scores by multiplying the set of individual prediction scores by the weight vector; and calculating the aggregate prediction score by adding the modified sets of individual prediction scores.

6. The clinical management device of claim 5, wherein the memory further includes a set of computer-readable instructions that cause the processor to generate the set of predicted outcome data by: calculating, using a feature importance method, a set of feature importance scores indicating the relative importance of each user health feature of the set of user health features with respect to each target machine learning model of the subset of the first set of target machine learning models; calculating a modified set of feature importance scores by multiplying the set of feature importances by the weight vector; calculating a set of aggregate feature importance scores indicating the relative importance of each user health feature with respect to a particular prediction target of the set of prediction targets by adding the modified set of feature importance scores; generating a set of feature ranking data that ranks the set of health features based on the set of aggregate feature importance scores; and outputting the set of feature ranking data as part of the set of predicted outcome data.

7. The clinical management device of claim 5, wherein the memory further includes a set of computer-readable instructions that cause the processor to generate the set of predicted outcome data by: determining a first recommended action associated with the aggregate prediction score according to a salience threshold achieved by matching the aggregate prediction score against a recommended action management database that defines a plurality of candidate recommended actions for the subset of the first set of target machine learning models; and outputting the first recommended action as one of the set of recommended actions.

8. The clinical management device of claim 1, wherein the memory further includes a set of computer-readable instructions that cause the processor to output the set of predicted outcome data by: processing the set of predicted outcome data using a large-scale language model to generate a set of output data that expresses the set of aggregate prediction scores and recommended actions in a natural language format; and outputting the set of output data in response to the user input query.

9. A clinical management method implemented by a clinical management device, the clinical management device comprising: a processor; and a memory, the memory including a set of computer readable instructions that cause the processor to: train a first set of machine learning models using a set of healthcare data to make predictions on a set of prediction targets for inclusion in a set of candidate machine learning models; define a set of clinical variation thresholds corresponding to the set of prediction targets by analyzing a set of treatment outcome data using statistical analysis techniques; train a second set of machine learning models using the set of treatment outcome data and the set of clinical variation thresholds to estimate a risk of clinical variation with respect to the set of prediction targets for inclusion in the set of candidate machine learning models; receive a user input query from a user related to a set of user health data characterizing a set of clinical concepts and a set of user health features; parse the user input query using natural language processing techniques to extract a first set of keywords; and identify a first set of clinical symptoms associated with the first set of keywords. Using the set of clinical concepts, a clinical concept database links clinical symptoms with the set of candidate machine learning models to select a first set of target machine learning models associated with a first set of clinical concepts corresponding to the first set of clinical symptoms in the clinical concept database; identifying a set of input health features for each target machine learning model in the first set of target machine learning models; and calculating an individual correlation score between each target machine learning model in the first set of target machine learning models and each user health feature in the set of user health features based on an overlap between the set of input health features and the set of user health features.calculating an aggregate correlation score between each target machine learning model in the first set of target machine learning models and the set of user health features by adding the individual correlation scores between each target machine learning model in the first set of target machine learning models and each user health feature in the set of user health features; selecting a subset of the first set of target machine learning models having aggregate correlation scores that achieve a correlation threshold for the set of user health features included in the user health data; generating a set of individual prediction scores for the set of user health data by using each target machine learning model in the subset of the first set of target machine learning models to analyze the set of user health features; calculating a weight vector indicating the relative importance of each target machine learning model in the subset of the first set of target machine learning models with respect to the set of clinical concepts; calculating a modified set of individual prediction scores by multiplying the set of individual prediction scores by the weight vector; calculating an aggregate prediction score by adding the modified sets of individual prediction scores; determining a first recommended action associated with the aggregate prediction score according to a salience threshold achieved by the aggregate prediction score by matching the aggregate prediction score against a recommended action management database defining a plurality of candidate recommended actions for the subset of the first set of target machine learning models; calculating, using a feature importance method, a set of feature importance scores indicating the relative importance of each user health feature of the set of user health features for each target machine learning model of the subset of the first set of target machine learning models; calculating a modified set of feature importance scores by multiplying the set of feature importances by the weight vector; and calculating a set of aggregate feature importance scores indicating the relative importance of each user health feature with respect to a particular prediction target of the set of prediction targets by adding the modified set of feature importance scores;generating a set of feature ranking data that ranks the set of health features based on the set of aggregate feature importance scores; and outputting a set of predicted outcome data that includes the aggregate prediction score, the first recommended action, and the set of feature importance scores.

10. A clinical management system, comprising: a clinical management device comprising: a processor; a memory; and a user terminal; the memory comprising a set of computer-readable instructions that cause the processor to: train a first set of machine learning models using a set of healthcare data to make predictions on a set of prediction targets for inclusion in a set of candidate machine learning models; define a set of clinical variation thresholds corresponding to the set of prediction targets by analyzing a set of treatment outcome data using statistical analysis techniques; train a second set of machine learning models using the set of treatment outcome data and the set of clinical variation thresholds to estimate a risk of clinical variation with respect to the set of prediction targets for inclusion in the set of candidate machine learning models; receive a user input query from a user related to a set of user health data characterizing a set of clinical concepts and a set of user health features; and identify a first set of target machine learning models trained to make predictions on a set of prediction targets related to the set of clinical concepts from the set of candidate machine learning models based on the user input query and the set of user health data.

1. A clinical management system comprising: a clinical management system configured to: identify a subset of the first set of target machine learning models that meet a correlation threshold with respect to the set of user health features included in the user health data; generate a set of predicted outcome data including aggregate prediction scores for the set of prediction targets and a set of recommended actions associated with the set of user health data by using the subset of the first set of target machine learning models to analyze the set of user health data; and output the set of predicted outcome data to the user terminal in response to the user input query.

Images