Clinical management device, clinical management method and clinical management system

Abstract

To provide a clinical management device, a clinical management method, and a clinical management system that facilitate robust clinical prediction and treatment management in consideration of a wide range of personal health factors.SOLUTION: The clinical management device 210 receives a user input query 310 and a set of user health data 312 at an input processing unit 222, identifies a target machine learning model that achieves a correlation threshold with respect to a set of user health features included in the user health data, generates a set of predictive outcome data including an aggregate prediction score and a set of recommended actions associated with the set of user health data, and outputs the set of predictive outcome data in response to the user input query.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a clinical management device, a clinical management method and a clinical management system. [Background technology]

[0002] In recent years, machine learning (ML) techniques have been developed for application in a wide range of fields. In machine learning, training data based on known cases is input into a computer. The computer analyzes the training data to learn a model that generalizes the relationship between factors (sometimes called explanatory or independent variables) and outcomes (sometimes called dependent or dependent variables). This model can then be used to predict outcomes for unknown cases. As an example, machine learning techniques can be used to predict clinical treatment outcomes for patients based on past treatment data for similar patients.

[0003] Conventionally, techniques have been considered to improve the performance of machine learning techniques. As an example, Patent Document 1 discloses that "techniques for improved machine learning are provided. Patient data describing a patient are received and a medication to be evaluated for the patient is identified. At least a subset of the patient data is processed using a hybrid machine learning model having a static portion and a dynamic portion to generate an efficacy score, the efficacy score indicating the predicted efficacy of the medication for the patient. A medication is provided to the patient based at least in part on the efficacy score." [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent Application No. 20230298722A1 Summary of the Invention [Problem to be solved by the invention]

[0005] The management of complex chronic diseases (e.g., diabetes) can be prone to clinical variability, where clinical treatment outcomes deviate from expected results. Clinical variability can create challenges related to unnecessary treatment costs or harm to patients. In some cases, clinical variability can result from a patient's personal health factors (e.g., comorbidities) that were not taken into account during treatment planning.

[0006] However, conventional machine learning-based systems have limited ability to identify clinical variations and provide adjustments to treatment plans. For example, Patent Document 1 discloses an efficacy score that indicates the effectiveness of a drug therapy for a patient based on patient data using a machine learning model, but does not consider possible clinical variations resulting from secondary personal health factors. Similarly, recent advances in generative AI have enabled the creation of large-scale language model (LLM)-based patient portals that provide answers to treatment-related questions, but these LLMs lack fact-based disease understanding to support patient-specific queries.

[0007] It is therefore an object of the present disclosure to provide a clinical management technique in which multiple machine learning models selected based on a patient's health data are used to facilitate robust clinical prediction and treatment management taking into account a wide range of personal health factors. [Means for solving the problem]

[0008] A representative example of the present disclosure relates to a clinical management device comprising: a processor; and a memory, wherein the memory includes a set of computer-readable instructions that cause the processor to: receive a user input query from a user related to a set of user health data characterizing a set of clinical concepts and a set of user health features; identify a first set of target machine learning models from a set of candidate machine learning models based on the user input query and the set of user health data, the target machine learning models being trained to make predictions on a set of prediction targets related to the set of clinical concepts of the user input query; identify a subset of the first set of target machine learning models that reach a correlation threshold with respect to the set of user health features included in the user health data; generate a set of predicted outcome data including aggregate prediction scores for the set of prediction targets and a set of recommended actions associated with the set of user health data by using the subset of the first set of target machine learning models to analyze the set of user health data; and output the set of predicted outcome data in response to the user input query. [Effects of the Invention]

[0009] According to the present disclosure, a clinical management technique can be provided in which multiple machine learning models selected based on patient health data are used to facilitate robust clinical prediction and treatment management taking into account a wide range of personal health factors.

[0010] Problems, configurations, and advantages other than those mentioned above will become apparent from the following description of embodiments for carrying out the present invention. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 illustrates an exemplary computing architecture for implementing embodiments of the present disclosure. [Figure 2]FIG. 1 illustrates an exemplary hardware configuration of a clinical management system according to an embodiment of the present disclosure. [Figure 3] FIG. 1 illustrates an example of a flow of a clinical management system according to an embodiment of the present disclosure. [Figure 4] FIG. 1 illustrates an example of using clinical concepts to identify clinical symptoms in a user input query, according to an embodiment of the present disclosure. [Figure 5] FIG. 1 illustrates an example of identifying a first set of target machine learning models using a clinical concept database and identified clinical symptoms, according to an embodiment of the present disclosure. [Figure 6] FIG. 1 illustrates an example of identifying a subset of a first set of target machine learning models that achieve a correlation threshold for the set of user health features included in user health data. [Figure 7] FIG. 1 illustrates an example of training a set of machine learning models to predict the risk of clinical variation for a set of prediction targets. [Figure 8] FIG. 1 illustrates an example of generating and outputting a set of predicted outcome data according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012]

[0023] Herein, embodiments of the present invention will be described with reference to the drawings. It should be noted that the embodiments described herein are not intended to limit the present invention in accordance with the claims, and it should be understood that each element and combination of elements described with respect to the embodiments is not strictly necessary to practice aspects of the present invention.

[0013] Various aspects are disclosed in the following description and related drawings. Alternative aspects may be devised without departing from the scope of the present disclosure. Additionally, well-known elements of the present disclosure will not be described in detail or will be omitted so as not to obscure the relevant details of the present disclosure.

[0014] The words "exemplary" and / or "example" are used herein to mean "serving as an example, instance, or illustrative example." Any aspect described herein as "exemplary" and / or "example" is not necessarily to be construed as preferred or advantageous over other aspects. Likewise, the phrase "aspects of the present disclosure" does not require that all aspects of the present disclosure include the discussed feature, advantage, or characteristic of operation.

[0015] Furthermore, many aspects are described in terms of sequences of actions to be performed by, for example, elements of a computing device. It will be recognized that the various actions described herein can be performed by specific circuitry (e.g., an application-specific integrated circuit (ASIC)), by program instructions executed by one or more processors, or a combination of both. In addition, the sequences of actions described herein can be considered to be embodied as a whole in any form of computer-readable storage medium storing a corresponding set of computer instructions that, when executed, can cause an associated processor to perform the functions described herein. As such, various aspects of the present disclosure may be embodied in many different forms, all of which are contemplated to be within the subject matter of the claims.

[0016] Detailed descriptions of embodiments of the present disclosure are described herein with reference to the drawings.

[0017] 1 is a schematic block diagram of a computer system 100, according to an embodiment, for implementing various embodiments of the present disclosure. The mechanisms and apparatus of the various embodiments disclosed herein are equally applicable to any suitable computing system. The major components of computer system 100 include one or more processors 102, memory 104, terminal interface 112, storage interface 113, I / O (input / output) device interface 114, and network interface 115, all of which are communicatively coupled, directly or indirectly, for inter-component communication via memory bus 106, I / O bus 108, bus interface unit 109, and I / O bus interface unit 110.

[0018] Computer system 100 may include one or more general-purpose programmable central processing units (CPUs) 102A and 102B, generally referred to herein as processors 102. In embodiments, computer system 100 may include multiple processors, although in particular embodiments, computer system 100 may alternatively be a single CPU system. Each processor 102 executes instructions stored in memory 104 and may include one or more levels of on-board cache.

[0019] In embodiments, memory 104 may include random-access semiconductor memory, storage devices, or storage media (either volatile or non-volatile) for storing or encoding data and programs. In particular embodiments, memory 104 represents the entire virtual memory of computer system 100 and may also include virtual memory of other computer systems coupled to computer system 100 or connected via a network. While memory 104 can be conceptually viewed as a single monolithic entity, in other embodiments, memory 104 is a more complex configuration, such as a hierarchy of caches and other memory elements. For example, memory may exist in multiple levels of caches, which may be further divided by function, whereby one cache holds instructions and another cache holds non-instruction data used by the processor. Memory may also be distributed and associated with different CPUs or sets of CPUs, as is known in any of a variety of so-called non-uniform memory access (NUMA) computer architectures.

[0020] Memory 104 may store all or a portion of the various programs, modules, and data structures for handling data transfers described herein. For example, memory 104 may store a clinical management application 150. In embodiments, clinical management application 150 may include instructions or statements that are executed on processor 102 or interpreted by processor 102 to perform functions as described further below.

[0021] In certain embodiments, clinical management application 150 is implemented in hardware via semiconductor devices, chips, logic gates, circuits, circuit cards, and / or other physical hardware devices in place of or in addition to a processor-based system. In embodiments, clinical management application 150 may include data in addition to instructions or statements. In certain embodiments, a camera, sensor, or other data input device (not shown) may be provided in direct communication with bus interface unit 109, processor 102, or other hardware of computer system 100. In such a configuration, the need for processor 102 to access memory 104 and clinical management application 150 may be reduced.

[0022] Computer system 100 may include a bus interface unit 109 that handles communication between processor 102, memory 104, display system 124, and I / O bus interface unit 110. I / O bus interface unit 110 may be coupled to I / O bus 108 to transfer data to and from various I / O units. I / O bus interface unit 110 communicates via I / O bus 108 with multiple I / O interface units 112, 113, 114, and 115, also known as I / O processors (IOPs) or I / O adapters (IOAs). Display system 124 may include a display controller, display memory, or both. The display controller may provide video, audio, or both types of data to display device 126. Additionally, computer system 100 may include one or more sensors or other devices configured to collect and provide data to processor 102.

[0023] By way of example, computer system 100 may include biometric sensors (e.g., collecting heart rate data, stress level data), environmental sensors (e.g., collecting humidity data, temperature data, pressure data), motion sensors (e.g., collecting acceleration data, movement data), etc. Other types of sensors are possible. Display memory may be dedicated memory for buffering video data. Display system 124 may be coupled to a display device 126, such as a standalone display screen, a computer monitor, a television, a tablet, or the display of a handheld device.

[0024] In one embodiment, display device 126 may include one or more speakers for rendering audio. Alternatively, one or more speakers for rendering audio may be coupled to the I / O interface unit. In an alternative embodiment, one or more of the functions provided by display system 124 may be incorporated into an integrated circuit that also includes processor 102. Additionally, one or more of the functions provided by bus interface unit 109 may be incorporated into an integrated circuit that also includes processor 102.

[0025] The I / O interface section supports communication with various storage devices and I / O devices. For example, the terminal interface unit 112 supports connection of one or more user I / O devices 116, which may include user output devices (such as a video display device, speakers, and / or a television receiver) and user input devices (such as a keyboard, mouse, keypad, touchpad, trackball, buttons, light pen, or other pointing device). A user may use a user interface to operate the user input devices to provide input data and commands to the user I / O devices 116 and the computer system 100, and may also receive output data via the user output devices. For example, the user interface may be presented via the user I / O devices 116, such as displayed on a display device, played through speakers, or printed by a printer.

[0026] Storage interface 113 supports the connection of one or more disk drives or direct access storage devices 117 (typically rotating magnetic disk drive storage devices, but may alternatively be other storage devices, including arrays of disk drives or solid-state drives such as flash memory configured to appear as a single mass storage device to a host computer). In some embodiments, storage device 117 may be implemented by any type of secondary storage device. The contents of memory 104, or any portion thereof, may be stored in storage device 117 and retrieved from storage device 117 as needed. I / O device interface 114 provides an interface to any of a variety of other I / O devices, or other types of devices, such as printers or fax machines. Network interface 115 provides one or more communication paths from computer system 100 to other digital devices and computer systems; these communication paths may include, for example, one or more networks 130.

[0027] 1 illustrates a particular bus structure providing direct communication paths between processor 102, memory 104, bus interface 109, display system 124, and I / O bus interface unit 110. In alternative embodiments, computer system 100 may include different buses or communication paths that may be configured in any of a variety of forms, such as hierarchical, star, or web configurations, multiple hierarchical buses, parallel and redundant paths, or point-to-point links in any other suitable type of configuration. Furthermore, while I / O bus interface unit 110 and I / O bus 108 are shown as separate respective components, computer system 100 may actually include multiple I / O bus interface units 110 and / or multiple I / O buses 108. While multiple I / O interface units are shown isolating I / O bus 108 from the various communication paths running to the various I / O devices, in other embodiments, some or all of the I / O devices are directly connected to one or more system I / O buses.

[0028] In various embodiments, computer system 100 is a multi-user mainframe computer system, a single-user system, or a server computer or similar device with little or no direct user interface, but which receives requests from other computer systems (clients). In other embodiments, computer system 100 may be implemented as a desktop computer, a portable computer, a laptop or notebook computer, a tablet computer, a pocket computer, a telephone, a smartphone, or any other suitable type of electronic device.

[0029] Next, with reference to FIG. 2, an exemplary hardware configuration of a clinical management system according to an embodiment of the present disclosure will be described.

[0030] 2 illustrates an exemplary hardware configuration of a clinical management system 200, according to an embodiment of the present disclosure. Clinical management system 200 relates to a system configured to facilitate robust clinical prediction and treatment management that takes into account a wide range of personal health factors.

[0031] 2, a clinical management system 200 according to an embodiment of the present disclosure includes a clinical management device 210, a communication network 250, and a user terminal 260. The clinical management device 210 and the user terminal 260 may be communicatively connected via the communication network 250.

[0032] Here, the communication network 250 may include a local area network (LAN) connection, the Internet, a wide area network (WAN) connection, a metropolitan area network (MAN) connection, a Bluetooth® connection, and the like.

[0033] The clinical management device 210 is an apparatus designed to facilitate robust clinical prediction and treatment management that takes into account a wide range of personal health factors. As illustrated in Figure 2, the clinical management device 210 mainly includes a memory 220, a storage unit 230, a processor 244, and an input / output unit 246. In an embodiment, the clinical management device 210 may be implemented using a computer system such as the computer system 100 shown in Figure 1.

[0034] 2 , the clinical management application 150 may include an input processing unit 222, a model identification unit 224, a prediction management unit 226, a model training unit 228, and an output processing unit 229. The input processing unit 222, the model identification unit 224, the prediction management unit 226, the model training unit 228, and the output processing unit 229 may be implemented as software modules that comprise the clinical management application 150.

[0035] The input processing unit 222 is a functional unit for receiving a user input query related to a set of user health data characterizing a set of clinical concepts and a set of user health features. The user input query may be entered into the user terminal 260 using natural language via a user interface provided to a patient portal as part of the treatment management service. The user input query may include semantic content related to one or more clinical concepts. Here, a clinical concept refers to a clinical concept related to a concept or idea, and may include, for example, a disease, a treatment, a symptom, a behavior, a disease management strategy, biochemistry, etc. In an embodiment, the input processing unit may receive a set of user health data along with the user input query. The set of user health data may include all or some of the user's medical history, laboratory test results, self-performed measurements (e.g., insulin measurements), etc., that characterize a set of user health features. Here, the set of user health features refers to attributes or characteristics that define, explain, or represent a user's health condition. As described herein, a machine learning model according to the present disclosure can be used to generate predictions for a set of prediction targets based on a set of user health features. The input processing unit will be described in detail later, and therefore its description will be omitted here.

[0036] The model identification unit 224 is a functional unit for identifying, from a pre-trained set of candidate machine learning models, a specific group of target machine learning models to be used in generating predictions based on a set of user health features included in a set of user health data received from a user. More specifically, the model identification unit 224 may identify, from a set of candidate machine learning models based on a user input query and a set of user health data, a first set of target machine learning models trained to make predictions for a set of prediction targets related to a set of clinical concepts of the user input query, and then identify a subset of the first set of target machine learning models that meet a correlation threshold with respect to the set of user health features included in the user health data. In an embodiment, the model identification unit may identify the first set of target machine learning models using the clinical concept database 231 stored in the storage unit 230. In this way, the model identification unit 224 can identify a specific group having target machine learning models trained to make predictions based on a specific set of user health features included in the user health data, thereby promoting robust prediction performance. The function of the model identification unit 224 will be described later, so its description will be omitted here.

[0037] The prediction management unit 226 is a functional unit for generating a set of predicted outcome data by analyzing a set of user health data using a subset of the first set of target machine learning models identified by the model identification unit 224. The set of predicted outcome data may include aggregate prediction scores for a set of predicted targets and a set of recommended actions associated with the set of user health data. In an embodiment, the set of recommended actions may be determined using a recommended actions database 234 stored in the storage unit 230. In this manner, a set including multiple machine learning models may be used to generate a set of predictions, which are then aggregated together to provide comprehensive prediction results that take into account a wide range of user health characteristics along with recommended actions that are expected to have a positive impact on the user's health. The details of the prediction management unit 226 will be described later, and therefore a detailed description thereof will be omitted here.

[0038] The model training unit 228 is a functional unit for training machine learning models according to the present disclosure. In an embodiment, the model training unit 228 may train a first set of machine learning models to make predictions for a set of prediction targets using a set of healthcare data stored in the healthcare database 232 in the storage unit 230. In addition, the model training unit 228 may define a set of clinical variation thresholds corresponding to the set of prediction targets by analyzing a set of treatment outcome data using a statistical variation criterion, and train a second set of machine learning models to estimate a risk of clinical variation for the set of prediction targets using the set of healthcare data and the set of clinical variation thresholds. The first set of machine learning models and the second set of machine learning models may be stored in the candidate ML model database 233 in the storage unit 230 as a set of candidate machine learning models. In this manner, the first set of machine learning models may be prepared to make predictions based on the health data and the second set of machine learning models to make predictions regarding the risk of clinical variation. Thus, a set of predicted outcome data may be generated taking into account a wide range of user health characteristics as well as the risk of clinical variation. The model training unit 228 will be described in detail later, and therefore its description will be omitted here.

[0039] The output processing unit 229 is a functional unit for outputting the set of predicted outcome data generated by the prediction management unit 226. In an embodiment, the output processing unit 229 may output the set of predicted outcome data to the user terminal 260 using natural language via a user interface provided to a patient portal as part of the treatment management service. In an embodiment, the output processing unit 229 may include a large-scale language model (LLM) configured to re-express, in natural language, the aggregate prediction score and the set of recommended actions included in the set of predicted outcome data as output in response to a user input query. In this way, a user may determine the generated health prediction and recommended actions in an easy-to-understand natural language format. The details of the output processing unit 229 will be described later, and therefore a description thereof will be omitted here.

[0040] The storage unit 230 is a unit for storing various data and information used in implementing aspects of the present disclosure. The storage unit 230 may include a collection of hard disk drives, solid state drives, flash memory, cloud storage, etc. As shown in FIG. 2 , the storage unit 230 may include a clinical concept database 231, a healthcare database 232, a candidate ML model database 233, and a recommended action database 234.

[0041] Clinical concept database 231 is a database configured to link pre-defined clinical conditions with a pre-trained set of candidate machine learning models stored in candidate ML model database 233 using clinical concepts. For example, clinical concept database 231 may define a correspondence between the clinical concept of "treatment" and machine learning models trained to make predictions related to a prediction target of "treatment effect" based on clinical conditions such as "type 2 diabetes," "type 1 diabetes," and "high blood pressure." In embodiments, these clinical conditions may be identified from a user input query based on extracted keywords.

[0042] The healthcare database 232 is a database configured to store a set of healthcare data used to train a first set of machine learning models to make predictions for a set of prediction targets. The set of healthcare data may include historical health data (e.g., past medical history, medical history) defining multiple patient health characteristics along with observed treatment results or outcomes. By training the first set of machine learning models using the set of healthcare data stored in the healthcare database 232, the first set of machine learning models learns to associate specific health characteristics with specific outcomes, thereby enabling the trained first set of machine learning models to make predictions for a wide variety of prediction targets based on sets of user health features included in sets of user health data received from users. More specifically, separate machine learning models may be trained to make predictions for each of several different prediction targets (e.g., treatment response, weight loss, risk of hospital readmission), etc. As described herein, aggregating results from multiple target machine learning models, each trained to make predictions based on a specific set of user health features included in the user health data, can facilitate robust prediction performance.

[0043] The candidate ML model database 233 is a database configured to store a set of candidate machine learning models trained by the model training unit 228. As described herein, the candidate machine learning models stored in the candidate ML model database 233 may include a first set of machine learning models trained to make predictions based on health data and a second set of machine learning models trained to make predictions regarding the risk of clinical variation. In this manner, a set of predicted outcome data can be generated given a wide range of user health characteristics as well as the risk of clinical variation.

[0044] The recommended action database 234 is a database configured to link the prediction results generated by the prediction management unit with one or more predetermined recommended actions that are expected to have a positive impact on the user's health. In an embodiment, the recommended action database 234 may define multiple candidate recommended actions for each of a set of prediction targets, such that each candidate recommended action is associated with a salience threshold. By comparing the aggregate prediction score generated by the prediction management unit with the salience thresholds of the associated prediction targets, one or more recommended actions associated with the salience threshold achieved by the aggregate prediction score can be selected. As an example, the recommended action database may include recommended actions such as "schedule a visit with your general practitioner," "increase your insulin dosage," "increase your frequency of practice," and "reduce your alcohol intake."

[0045] Processor 244 is a processing unit for executing processing instructions for various software modules and functional units included in clinical management application 150 stored in memory 220, and may substantially correspond to processor 102 of computer system 100 shown in FIG. 1.

[0046] The input / output unit 246 is a collection of devices for facilitating communication between the clinical management device 210 and external sources, such as users of the clinical management device 210 and one or more user terminals 260 to which the clinical management device 210 is communicatively connected. In an embodiment, the input / output unit 246 may include an integrated display unit, speaker, etc. for providing information to a user of the clinical management device 210.

[0047] User terminal 260 is a device usable by a user (e.g., a client) of clinical management device 210. In an embodiment, user terminal 260 may be used to enter user input queries into clinical management device 210 as well as to determine sets of predicted outcome data generated by prediction management unit 226. By way of example, user terminal 260 may be implemented using a personal computer, a tablet computer, a smartphone, or other computing device.

[0048] The clinical management system 200 shown in FIG. 2 can provide a clinical management technique in which multiple machine learning models selected based on patient health data are used to facilitate robust clinical prediction and treatment management taking into account a wide range of personal health factors.

[0049] Next, an example of the flow of a clinical management system according to an embodiment of the present disclosure will be described with reference to FIG.

[0050] 3 illustrates an example flow diagram of the clinical management system 200, according to an embodiment of the present disclosure. Note that in the following description, it is assumed that the set of candidate ML models has already been trained and stored in the candidate ML model database 233. Similarly, it is assumed that the clinical concept database 231 and the recommended action database 234 have been pre-populated to facilitate the determination of the target machine learning model and recommended actions, respectively.

[0051] Initially, the user 305 submits a user input query 310 and a set of user health data 312 to the user terminal 260. As an example, the user 305 may submit the user input query "I've stopped exercising recently and I'm worried about my diabetes" along with a set of user health data 312 including a set of user health characteristics, such as laboratory values, demographic information, and vital signs, that characterize the user's health status. As an example, the set of user health factors may include hemoglobin A1C (hereinafter, A1C) level, age, weight, height, albumin, etc. In an embodiment, the set of health factors may be structured in a (name, value, unit) format indicating the name, measurement value, and measurement unit of the health factor. The user terminal 260 may forward the received user input query 310 and set of user health data 312 to the clinical management device 210 via a secure data transmission channel.

[0052] Next, the input processing unit 222 may analyze the user input query 310 using a natural language processing technique to identify a first set of keywords that characterize the set of clinical concepts. Here, the input processing unit 222 may use Rapid Automatic Keyword Extraction (RAKE), Yet Another Keyword Extractor (YAKE), Word Embeddings, Named Entity Recognition (NER), Term Frequency-Inverse Document Frequency (TF-IDF), etc. as the natural language processing technique to identify the first set of keywords. For example, from the user input query "I've stopped exercising recently and I'm worried about my diabetes," the input processing unit 222 may extract the keywords "diabetes" and "exercise."

[0053] Next, the model identification unit 224 identifies a first set of target machine learning models, which are trained to make predictions about a set of prediction targets related to the set of clinical concepts of the user input query, by comparing the keywords extracted by the input processing unit 222 with the clinical concept database 231 from a set of candidate machine learning models stored in the healthcare database 232 and pre-trained based at least on the set of healthcare data stored in the candidate ML model database 233. Here, the set of prediction targets includes factors or attributes for which predictions are to be generated. For example, the set of prediction targets may include “treatment effect,” “weight loss effect,” “clinical fluctuation risk,” “hospitalization risk,” etc. The model identification unit may then identify a subset of the first set of target machine learning models that achieves a correlation threshold with the set of user health features included in the user health data 312 from the first set of target machine learning models. In this way, the model identification unit 224 can filter the first set of target machine learning models to select those with the highest correlation to be used in generating a set of predicted outcome data.

[0054] The prediction management unit 226 then generates a set of predicted outcome data by analyzing the set of user health data using a subset of the first set of target machine learning models identified by the model identification unit 224. As described herein, the set of predicted outcome data may include an aggregate prediction score for a set of predicted targets and a set of recommended actions associated with the set of user health data. In an embodiment, the prediction management unit 226 may determine the set of recommended actions by comparing the aggregate prediction score to a salience threshold associated with predetermined recommended actions stored in the recommended action database 234 for each of the predicted targets, and selecting one or more recommended actions associated with the salience threshold achieved by the aggregate prediction score.

[0055] The output processing unit 229 then, in response to the user input query, uses the LLM to re-express the set of recommended actions contained in the aggregate prediction score and set of predicted outcome data into natural language for output and sends the generated output data 320 via secure data transmission to the user terminal 260. In one example, the output processing unit 229 may provide output data 320 that reads, "Diabetic patients with weight change are at higher risk of needing treatment adjustment. Schedule an appointment with your doctor."

[0056] The clinical management system 200 shown in FIG. 3 can provide a clinical management technique in which multiple machine learning models selected based on patient health data are used to facilitate robust clinical prediction and treatment management taking into account a wide range of personal health factors.

[0057] Referring now to FIG. 4, an example of using clinical concepts to identify clinical symptoms in a user-entered query will be described, according to an embodiment of the present disclosure.

[0058] FIG. 4 illustrates an example of using the clinical concepts database 231 to identify clinical symptoms in a user-entered query, according to an embodiment of the present disclosure.

[0059] As described herein, in response to receiving a user input query 310 from a user 305, the input processing unit 222 may analyze the user input query 310 using natural language processing techniques to identify a first set of keywords 415 that characterize a set of clinical concepts. As an example, if the input processing unit 222 receives the user input query 310, "I've stopped exercising recently and I'm worried about my diabetes," the input processing unit 222 may extract the keywords "diabetes" and "exercise," and the input processing unit 222 may use natural language processing techniques to extract the first set of keywords 415, "diabetes" and "stop exercising."

[0060] The input processing unit 222 may then match the extracted first set of keywords against the clinical concept database 231 to identify clinical conditions (e.g., a first set of clinical conditions) associated with the first set of keywords. For example, as shown in FIG. 4, a first set of keywords 415, "diabetes" and "exercise," may be The input processing unit 222 may identify “type 2 diabetes” as the clinical condition corresponding to the first set of keywords because the first set of keywords 415 is associated with the clinical condition “type 2 diabetes” in the clinical concept database 231. In an embodiment, if the first set of keywords 415 extracted from the user input query is associated with multiple different clinical conditions in the clinical concept database 231, the input processing unit 222 may select the clinical condition with the highest number of matching keywords (e.g., the highest frequency count).

[0061] In this manner, one or more clinical conditions characterized by keywords of the user input query can be identified. As described herein, these identified clinical conditions can be used to facilitate the identification of a first set of target machine learning models trained to make predictions regarding the user health data received from the user 305.

[0062] Next, with reference to FIG. 5, an example of identifying a first set of target machine learning models using clinical concepts and identified clinical symptoms will be described according to an embodiment of the present disclosure.

[0063] FIG. 5 illustrates an example of identifying a first set of target machine learning models using the clinical concept database 231 and identified clinical symptoms, according to an embodiment of the present disclosure.

[0064] As described herein, the clinical concept database 231 may be used to map pre-defined clinical symptoms 510 with a pre-trained set of candidate machine learning models 520 stored in the candidate ML model database 233 using the clinical concepts 515. For example, the clinical concept database 231 may define a correspondence between a particular clinical symptom 510, a particular clinical concept 515, and a particular machine learning model 520.

[0065] 5, when a clinical condition 510 of "type 2 diabetes" is identified based on keywords extracted from a user-input query, the model identification unit 224 may first identify a set of clinical concepts 515 of "treatment," "lifestyle," and "disease management" that are linked to the clinical condition 510 of "type 2 diabetes" in the clinical concept database 231, and then select a target machine learning model mapped to the identified clinical concept, where each of the target machine learning models may be trained to make predictions about prediction targets related to the clinical concept.

[0066] For example, for the clinical concept 515 of "treatment," the model identification unit 224 may select a target machine learning model trained to make predictions for the prediction targets of "treatment effect," "adverse events," and "ED (emergency department) visit risk." For the clinical concept 515 of "lifestyle," the model identification unit 224 may select a target machine learning model trained to make predictions for the prediction targets of "treatment effect," "weight loss effect." Similarly, for the clinical concept 515 of "disease management," the model identification unit 224 may select a target machine learning model trained to make predictions for the prediction target of "clinical variation risk."

[0067] In embodiments, a weight vector 525 may be calculated that indicates the relative importance of each target machine learning model in the first set of target machine learning models with respect to the set of clinical concepts 515. In embodiments, this weight vector 525 may be calculated based on the number of clinical concepts 515 associated with each target machine learning model from among the clinical concepts identified for the particular clinical condition 510, such that target machine learning models associated with more clinical concepts are given a higher weight. For example, when a machine learning model trained to make predictions with respect to the prediction target "treatment effect" is associated with two clinical concepts, "treatment" and "lifestyle," while other machine learning models are associated with one or fewer clinical concepts in the set of clinical concepts 515, the machine learning model trained to make predictions with respect to the prediction target "treatment effect" may be given a higher relative weight. In embodiments, the weight vector 525 may be normalized to sum to one.

[0068] In this manner, the clinical concept database 231 can be used to select a set of target machine learning models pre-trained to make predictions for prediction targets related to clinical concepts associated with clinical symptoms characterized by user-entered keywords. This set of target machine learning models can be filtered to identify a subset of the first set of target machine learning models that achieve a correlation threshold with a set of user health features included in the user health data. This identified subset of the first set of target machine learning models can then be used as a network of linked machine learning models to generate a series of predictions related to a wide variety of user health features. These predictions can then be aggregated together to create a comprehensive and holistic predictive result to facilitate clinical treatment for the user.

[0069] Referring now to FIG. 6, an example will be described of identifying a subset of a first set of target machine learning models that achieve a correlation threshold for a set of user health features included in the user health data.

[0070]

[0013] Figure 1 illustrates an example of identifying a subset of a first set of target machine learning models that achieve a correlation threshold with a set of user health features included in user health data. As described herein, aspects of the present disclosure relate to selecting a set of target machine learning models that have been pre-trained to make predictions for a prediction target related to a clinical concept associated with a clinical symptom characterized by keywords in a user input. Each target machine learning model in this set of target machine learning models is trained to make predictions using a specific set of input health features. However, in some cases, the set of input features used by a particular machine learning model may not be present in the set of health data provided by the user. Accordingly, to promote prediction accuracy, it is desirable to filter the first set of target machine learning models to select a subset of target machine learning models that make predictions using input health features that substantially correspond to the user health features included in the set of user data.

[0071] First, the model identification unit 224 identifies a set of input health features 620 for each target machine learning model in the first set of target machine learning models 610, where the set of input health features 620 are features used by the particular target machine learning model to make predictions. For example, as shown in FIG. 6, for a machine learning model trained to make predictions on a prediction target of "treatment effect," the model identification unit 224 may identify a set of input health features such as "(albumin, g / dL), (calcium, mg / dL), (A1C, %)," etc.

[0072] Next, the model identification unit 224 calculates an individual correlation score between each target machine learning model in the first set of target machine learning models 610 and each health feature in the set of user health features 630 based on the degree of overlap between the set of input health features 620 and the set of user health features 630 included in the user health data 312. Here, the individual correlation score is a numerical value representing the degree to which each target machine learning model corresponds to each health feature in the set of user health features 630. The individual correlation scores between the set of input health features 620 and the set of user health features 630 included in the user health data 312 may be calculated using a semantic similarity algorithm (e.g., cosine similarity) configured to compare the (name, value, unit) of each feature in the set of input health features 620 and the set of user health features 630. As an example, the set of individual correlation scores may be expressed in the form (c1, c2, c3, c4,... cn), where each value indicates the similarity between a particular input health feature in the set of input health features 620 and a particular user health feature in the set of user health features 630 for a particular target machine learning model.

[0073] The model identification unit 224 then calculates an aggregate correlation score between each target machine learning model in the first set of target machine learning models and the set of user health features by adding the individual correlation scores between each target machine learning model in the first set of target machine learning models 610 and each health feature in the set of user health features 620. The aggregate correlation score (e.g., “C”) is a numerical value that represents the degree to which each target machine learning model corresponds to the set of user health features 630 as a whole.

[0074] Next, the model identification unit 224 selects target machine learning models having an aggregate correlation score C that achieves a correlation threshold as a subset of the first set of target machine learning models 640. Here, the correlation threshold is a predetermined correlation score used to specify a desired level of correlation and may be freely set by a user. In an embodiment, the model identification unit 224 may revise the weight vector calculated as described with reference to FIG. 5 for the subset of the first set of target machine learning models 640 by uniformly distributing the weights of the machine learning models that were not selected as a subset of the first set of target machine learning models 640. This revised weight vector may be stored together with the corresponding machine learning models in the candidate ML model database 233.

[0075] 6, it is possible to select a subset of candidate machine learning models from a set of candidate machine learning models that are both trained to make predictions for a set of prediction targets related to a set of clinical concepts characterized by the input query and that achieve a correlation threshold with respect to a set of user health features included in the user health data. Selecting a subset of machine learning models using this two-step process can facilitate the generation of accurate predictions for the user input query while taking into account a wide range of personal health factors.

[0076] Next, with reference to FIG. 7, an example of training a set of machine learning models to predict the risk of clinical variation for a set of prediction targets will be described.

[0077] As described herein, aspects of the present disclosure relate to the recognition that, in some cases, clinical variation may occur, causing clinical treatment outcomes to deviate from expected results. Clinical variation may create issues related to unnecessary treatment costs or harm to patients. Accordingly, aspects of the present disclosure relate to preparing a set of machine learning models (e.g., a second set of machine learning models) for inclusion in a set of candidate machine learning models trained to predict the risk of clinical variation for a particular prediction target. Figure 7 illustrates an example of training a set of machine learning models to predict the risk of clinical variation for a set of prediction targets.

[0078] First, the model training unit 228 analyzes treatment outcome data 710 included in a set of healthcare data stored in the healthcare database 232 using statistical analysis techniques to define a set of clinical variation thresholds 720 corresponding to a set of prediction targets. The treatment outcome data 710 may include information indicating the outcomes of various treatments administered to past patients. As an example, the treatment outcome data 710 may indicate the progression of a patient's A1C (e.g., blood glucose level) values ​​over a time period during which a particular treatment was administered. The set of clinical variation thresholds 720 may be numerical thresholds that define a boundary between expected and unexpected measurements that should be considered clinical variation. In an embodiment, the model training unit 228 may fit a variance to the treatment outcome data 710 and segment the treatment outcome data into standard deviations from the mean. The set of clinical variation thresholds 720 may be defined in terms of a certain number of standard deviations from the mean (e.g., three standard deviations) so that data points that are more than a certain number of standard deviations from the mean are considered to represent clinical variation. For the treatment outcome data 710 corresponding to each of the set of prediction targets, the model training unit 228 may perform this process.

[0079] The model training unit 228 may then generate a set of training data 730 by assigning labels to the treatment outcome data 710 indicating the presence or absence of clinical variation. As shown in Figure 7, the set of training data 730 may include particular data points or ranges of data points (e.g., A1C values) along with labels indicating whether the data points were determined to qualify as clinical variations.

[0080] Next, the model training unit 228 may use a machine learning algorithm 740 (e.g., supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning) to train a second set of machine learning models 750 using the set of training data 730. By training a machine learning model using the set of training data 730, it is possible to prepare a second set of machine learning models 750 trained to predict the risk of clinical variation occurring for a specific prediction target. As an example, referring to FIG. 7 , it is possible to obtain a machine learning model trained to predict treatment effect variation in A1C values ​​for future patients by training the machine learning model using a set of training data annotated to indicate the presence or absence of clinical variation in A1C values ​​as part of the treatment treatment. The second set of machine learning models 750 trained in this way may be added to the candidate ML model database 233 stored together with the first set of machine learning models based on the set of healthcare data stored in the healthcare database 232 as described above. In this way, clinical variation for a specific prediction target also becomes a new prediction target for prediction. A second set of machine learning models 750 can then be selected for use with the first set of machine learning models so that predictions about a particular prediction target (e.g., treatment effect) can be made along with predictions about clinical variation (e.g., treatment effect variation) in that particular prediction target.

[0081] According to the process described with reference to Figure 7, it is possible to prepare a set of machine learning models trained to predict the risk of clinical variation for a particular prediction target, thus providing a clinical management technique that can make robust clinical predictions based on user healthcare data while also taking into account the likelihood of clinical risk.

[0082] Next, with reference to FIG. 8, an example of generating and outputting a set of prediction result data according to an embodiment of the present disclosure will be described.

[0083] 8 illustrates an example of generating and outputting a set of predicted outcome data according to an embodiment of the present disclosure. As described herein, in an embodiment, the set of predicted outcome data may include an aggregate prediction score, a set of recommended actions, and a set of feature ranking data. In an embodiment, the set of predicted outcome data is processed using a large-scale language model and re-expressed in a natural language format for presentation in response to a user input query.

[0084] First, the prediction management unit 226 may analyze the set of user health data 312 using a subset of the first set of target machine learning models to generate prediction data. Note that the set of prediction data is generated using a subset of the first set of target machine learning models (e.g., a subset of the first set of target machine learning models identified by the process described with respect to FIGS. 4-7 ) that are both trained to make predictions for a set of prediction targets related to the set of clinical concepts characterized by the input query and that achieve a correlation threshold with respect to the set of user health features included in the user health data.

[0085] More specifically, the prediction management unit 226 may generate a set of individual prediction scores 810 (p1, p2, p3,... pn) for the set of user health data by analyzing the set of user health features 312 using each target machine learning model of the subset of the first set of target machine learning models, where the set of individual prediction scores 810 are values ​​representing predicted results or outcomes for each prediction target. Note that the subset of the first set of target machine learning models used to generate the set of individual prediction scores is selected from a set of candidate machine learning models that includes both machine learning models trained to make predictions about user health characteristics and machine learning models trained to make clinical predictions about clinical variation risk, so the set of individual prediction scores may include predictions about various individual health characteristics as well as predictions about clinical variation risk. As an example, the subset of the first set of target machine learning models may generate a set of individual prediction scores 810 for the prediction targets of treatment effect, weight loss effect, and treatment transition risk (e.g., clinical variation about treatment transition).

[0086] Next, the prediction management unit 226 may retrieve the weight vector 525 (e.g., the revised weight vector calculated after identifying the first set of target machine learning models) from the candidate ML model database 233 and calculate a revised set of individual prediction scores 810 by multiplying the set of individual prediction scores 810 by the weight vector 525. Here, each individual prediction score is multiplied by the weight value calculated for the corresponding prediction target (e.g., the prediction score for “effect of treatment” is multiplied by the weight for the “effect of treatment” machine learning model). Next, the prediction management unit 226 may calculate an aggregate prediction score 822 (P) by adding together the revised set of individual prediction scores. Here, this aggregate prediction score is a value indicating the degree of risk for the patient, taking into account the set of user health features characterized in the set of user health data 312. In an embodiment, the aggregate prediction score may be a value between 0 and 100, where 0 indicates a low level of risk and 100 indicates a high level of risk. In this way, by aggregating the modified set of individual prediction scores, it is possible to calculate a single aggregate prediction score 822 that is based not only on personal health factors such as the user's weight, age, and measured biochemistry test values, but also takes into account the risk of clinical variation for the user.

[0087] The prediction management unit 226 may then use the aggregate prediction score 822 to determine a set of recommended actions 834. More specifically, the prediction management unit 226 may match the calculated aggregate prediction score 822 with a recommended action database 234 to determine the set of recommended actions 834. As described herein, the recommended action database 234 is a database for linking prediction results generated by the prediction management unit 226 with one or more predetermined recommended actions that are expected to have a positive impact on the user's health. In an embodiment, the recommended action database 234 may define multiple candidate recommended actions for each of the set of prediction targets, such that each candidate recommended action is associated with a salience threshold. By comparing the aggregate prediction score 822 with the salience threshold of the associated prediction target, the prediction management unit 226 may select one or more recommended actions 834 associated with the salience threshold achieved by the aggregate prediction score 822. As an example, if the candidate recommended action "Schedule a visit with your GP" is associated with a salience threshold of "65" and the aggregate prediction score 822 is calculated to be "68," the recommended action "Schedule a visit with your GP" may be selected because the aggregate prediction score 822 of "68" meets the salience threshold of "65."

[0088] In an embodiment, as described herein, the prediction management unit 226 may be configured to generate a set of feature ranking data 854 for inclusion in the set of predicted outcome data. The set of feature ranking data 854 includes data indicating the relative importance of each user health feature in generating the set of predicted outcome data. In particular, the prediction management unit 226 may calculate a set of feature importance scores 816 for each of the set of user health features using a feature importance method such as the Shapley additive explanations (SHAP) method. Here, the set of feature importance scores 816 may be expressed as a numerical value indicating the relative importance of each user health feature of the set of user health features included in the set of user health data 312 with respect to each target machine learning model of the subset of the first set of target machine learning models.

[0089] Next, the prediction management unit 226 may retrieve the weight vector 525 (e.g., the revised weight vector calculated after identifying the first set of target machine learning models) from the candidate ML model database 233 and calculate a revised set of feature importance scores by multiplying the set of feature importance scores 816 by the weight vector 525, where each feature importance score is multiplied by the weight value for its corresponding prediction target. The prediction management unit 226 may then calculate a set of aggregate feature importance scores 838 (F) by adding together the revised set of feature importance scores, where the set of aggregate feature importance scores 838 is a value indicating the relative importance of each health feature with respect to a particular prediction goal. In an embodiment, the set of aggregate feature importance scores may be a value between 0 and 100, where 0 indicates a low level of importance and 100 indicates a high level of importance.

[0090] The prediction management unit 226 may then generate a set of feature ranking data 854 by ranking the set of user health features based on the set of aggregate feature importance scores 838 such that user health features with higher aggregate feature importance scores are given higher rankings.

[0091] In an embodiment, the set of feature ranking data 854 may be calculated using an explanation matrix method. For example, as shown in FIG. 8 , each machine learning model of the subset of the first set of target machine learning models may be represented as a row of the matrix, and each user health feature may be represented as a column of the matrix. The feature importance score of each user health feature for each respective machine learning model may be calculated using the SHAP method and used to populate the cells of the matrix. The feature importance score in each cell of the matrix may then be multiplied by the associated weighting score for that row in the weight vector 525. An aggregate feature importance score indicating the relative importance of each health feature with respect to a particular prediction target may then be calculated by adding the feature importance scores across each row (or, alternatively, each column).

[0092] Output processing unit 229 may receive the set of predicted outcome data including aggregate prediction score 822, set of recommended actions 834, and feature ranking data 854 generated by prediction management unit 226, and prepare the received set of predicted outcome data for output to a user via user terminal 260. In an embodiment, output processing unit 229 may output aggregate prediction score 822, set of recommended actions 834, and feature ranking data 854 without further processing. If the user is a public health physician with a comprehensive understanding of the clinical management process, it may be desirable to leave aggregate prediction score 822, set of recommended actions 834, and feature ranking data 854 as is.

[0093] In certain embodiments, the output processing unit 229 may use a large-scale language model (LLM) configured to re-express the aggregate prediction score 822, the set of recommended actions 834, and the feature ranking data 854 into natural language as output in response to a user input query. As an example, the output processing unit 229 may generate a result such as, "Diabetic patients with weight change are at higher risk of needing treatment adjustments. Schedule an appointment with your doctor." As another example, if a high risk of treatment effect variability is identified, a candidate recommended action may be generated such as, "Your current treatment may not be optimal for you. Explore alternative treatment options." If the user is a patient accessing a clinical management system according to an embodiment of the present disclosure through a patient portal, it may be desirable to use the LLM to re-express the set of predicted outcome data in natural language format. In this way, the user may confirm the generated health prediction and recommended action in an easy-to-understand natural language format.

[0094] Further, in an embodiment, the output processing unit 229 may process the set of feature ranking data 854 to generate a set of explanatory data to explain why a particular recommended action was selected. In one example, if the feature ranking data 854 indicates that the user health feature "estimated average glucose" has the highest importance among the set of user health features, the output processing unit 229 may generate a set of explanatory data such as, "We recommend a diet plan for you because the amount of sugar in your daily diet may be affecting the effectiveness of treatment." In this way, by using the set of feature ranking data 854, it is possible to gain insight into which particular user health features contribute to a particular predicted outcome and to generate an explanation supporting the inference that a particular treatment plan or recommended action was proposed.

[0095] The process described with respect to Figure 8 can generate a set of predicted outcome data that provides a user with a robust health prognosis based on a wide range of personal health factors as well as the risk of clinical excursions. In this way, risk of clinical excursions can be identified before they occur, and treatment options can be selected that take into account secondary health factors and mitigate the clinical excursions, promoting patient satisfaction.

[0096] As described herein, aspects of the present disclosure relate to training a set of candidate machine learning models, including a first set of machine learning models trained to make predictions for a set of prediction targets and a second set of machine learning models trained to make predictions regarding risk of clinical variation in the set of prediction targets. From the set of candidate machine learning models, it is possible to select a subset of machine learning models that are both trained to make predictions for the set of prediction targets related to the set of clinical concepts characterized by the input query and that achieve a correlation threshold with respect to a set of user health features included in the user health data. In this manner, selecting a subset of machine learning models from both the first and second sets of machine learning models can facilitate the generation of accurate predictions with respect to the prediction query while also taking into account the risk of clinical variation in the prediction target.

[0097] A further aspect of the present disclosure relates to generating recommended actions for users based on the generated prediction results. In certain embodiments, the recommended actions can be expressed in natural language by a large-scale language model and provided to users via a patient portal as part of a care management service. In this way, patients can receive real-time consultations about their care via the online portal, saving users the time and effort of scheduling appointments at medical facilities. Similarly, resources in that department of the medical facility can be saved.

[0098] A further aspect of the present disclosure relates to generating a set of feature ranking data indicating the relative importance of each user health feature in generating the set of predicted outcome data. Using this set of feature ranking data, it is possible to gain insight into which particular user health features contribute to a particular predicted outcome and to generate explanations supporting the reasoning behind why a particular treatment plan or recommended action is proposed. Thus, transparency of clinical management techniques is enhanced, and users can gain a better understanding of their treatment options.

[0099] Thus, the embodiments of the present disclosure described herein enable multiple machine learning models selected based on patient health data to provide the clinical management techniques described in the specifications to facilitate robust clinical prediction and treatment management taking into account a wide range of personal health factors.

[0100] The present invention may be a system, a method, and / or a computer program product, which may include a computer-readable storage medium having computer-readable program instructions for causing a processor to implement aspects of the present invention.

[0101] A computer-readable storage medium may be a tangible device capable of holding and storing instructions used by an instruction execution device. The computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanical encryption devices such as punch cards or groove ridge structures with instructions recorded on them, and any suitable combination of the above. As used herein, a computer-readable storage medium should not be construed as being itself a primary signal, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse passing through a fiber optic cable), or an electrical signal transmitted by an electrical wire.

[0102] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0103] The computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to manufacture a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for performing the functions / acts specified in the flowchart and / or block diagram blocks. These computer-readable program instructions may also be stored on a computer-readable storage medium that can cause the computer, programmable data processing apparatus, and / or other device to function in a particular way, such that the computer-readable storage medium having the instructions stored thereon comprises an article of manufacture containing instructions that implement aspects of the functions / acts specified in the flowchart and / or block diagram blocks.

[0104] The computer-readable program instructions may further be loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be executed on the computer, other programmable apparatus, or other device to create a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / acts specified in the flowchart and / or block diagram blocks.

[0105] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specialized logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It will also be appreciated that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by dedicated hardware-based systems that perform specialized functions or operations or execute a combination of dedicated hardware and computer instructions.

[0106] While the foregoing relates to exemplary embodiments of the present invention, other and further embodiments of the present invention may be devised without departing from the basic scope of the invention, which scope is determined by the following claims. The description of various embodiments of the present disclosure has been provided for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to explain the principles of the embodiments, practical applications or technical improvements of existing technologies, or to enable others of ordinary skill in the art to which the present disclosure pertains to understand the embodiments disclosed herein.

[0107] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of various embodiments. As used herein, the singular forms "a," "an," and "the" are intended to include the plural as well, unless the context clearly indicates otherwise. "Set," "group," "bundle," etc. are intended to include one or more. Furthermore, it will be understood that the words "comprise" and / or "comprising," as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. In the above detailed description of exemplary embodiments of various embodiments, reference has been made to the accompanying drawings, in which like numerals indicate like elements, which illustrate, by way of example, specific exemplary embodiments and in which various embodiments may be practiced. The above embodiments have been described in sufficient detail to enable one skilled in the art to practice the embodiments; however, other embodiments may be used, and logical, mechanical, electrical, and other changes may be made without departing from the scope of the various embodiments. In the above description, numerous specific details have been set forth to provide a thorough understanding of the various embodiments. However, various embodiments may be practiced without these specific details. In other instances, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the embodiments.

[0108] As described herein, clinical management techniques according to the present disclosure relate to the following aspects:

[0109] (Aspect 1) 1. A clinical management device comprising: a processor; a memory; The memory includes a set of computer-readable instructions that cause the processor to: receiving a user input query from a user relating to a set of user health data characterizing a set of clinical concepts and a set of user health features; identifying a first set of target machine learning models, trained to make predictions for a set of prediction targets related to a set of clinical concepts, from a set of candidate machine learning models based on a user input query and a set of user health data; identifying a subset of the first set of target machine learning models that meet a correlation threshold with respect to a set of user health features included in the user health data; generating a set of prediction result data including aggregate prediction scores for a set of prediction targets and a set of recommended actions associated with the set of user health data by analyzing the set of user health data using a subset of the first set of target machine learning models; and outputting a set of predicted outcome data in response to a user input query.

[0110] (Aspect 2) The memory further includes a set of computer readable instructions, the set of computer readable instructions causing the processor to: training a first set of machine learning models using the set of healthcare data to make predictions about the set of prediction targets for inclusion in a set of candidate machine learning models; defining a set of clinical variability thresholds corresponding to a set of prediction targets by analyzing the set of treatment outcome data using statistical analysis techniques; 2. The clinical management device of claim 1, wherein the set of candidate machine learning models is generated by: training a second set of machine learning models using the set of treatment outcome data and the set of clinical variation thresholds to estimate the risk of clinical variation with respect to the set of prediction targets for inclusion in the set of candidate machine learning models.

[0111] (Aspect 3) The memory further includes a set of computer readable instructions, the set of computer readable instructions causing the processor to: parsing a user input query using natural language processing techniques to extract a first set of keywords; identifying a first set of clinical symptoms associated with the first set of keywords; A clinical management device as described in aspect 1 or 2, wherein the first set of target machine learning models is identified by: using a set of clinical concepts as the first set of target machine learning models, and selecting, based on a clinical concept database that links clinical symptoms with a set of candidate machine learning models, target machine learning models associated with a first set of clinical concepts that correspond to the first set of clinical symptoms in the clinical concept database.

[0112] (Aspect 4) The memory further includes a set of computer readable instructions, the set of computer readable instructions causing the processor to: identifying a set of input health features for each target machine learning model in the first set of target machine learning models; calculating an individual correlation score between each target machine learning model in the first set of target machine learning models and each user health feature in the set of user health features based on an overlap between the set of input health features and the set of user health features; calculating an aggregate correlation score between each target machine learning model in the first set of target machine learning models and the set of user health features by adding up individual correlation scores between each target machine learning model in the first set of target machine learning models and each user health feature in the set of user health features; 4. The clinical management device of any one of aspects 1 to 3, wherein the subset of the first set of target machine learning models is identified by: selecting target machine learning models having aggregate correlation scores that achieve a correlation threshold as the subset of the first set of target machine learning models.

[0113] (Aspect 5) The memory further includes a set of computer readable instructions, the set of computer readable instructions causing the processor to: generating a set of individual prediction scores for the set of user health data by analyzing the set of user health features with each target machine learning model of the subset of the first set of target machine learning models; calculating a weight vector indicating the relative importance of each target machine learning model of the subset of the first set of target machine learning models with respect to the set of clinical concepts; calculating a modified set of individual prediction scores by multiplying the set of individual prediction scores by a weight vector; 5. The clinical management device of any one of aspects 1 to 4, wherein the clinical management device generates a set of predicted outcome data by calculating an aggregate prediction score by adding together the modified set of individual prediction scores.

[0114] (Aspect 6) The memory further includes a set of computer readable instructions, the set of computer readable instructions causing the processor to: calculating a set of feature importance scores indicating the relative importance of each user health feature of the set of user health features with respect to each target machine learning model of the subset of the first set of target machine learning models using a feature importance method; calculating a modified set of feature importance scores by multiplying the set of feature importance scores by a weight vector; calculating a set of aggregate feature importance scores indicating the relative importance of each user health feature with respect to a particular prediction target of the set of prediction targets by summing the modified set of feature importance scores; generating a set of feature ranking data that ranks the set of health features based on the set of aggregate feature importance scores;

[0023] A clinical management device as described in embodiment 5, wherein the set of predicted outcome data is generated by: and outputting the set of feature ranking data as part of the set of predicted outcome data.

[0115] (Aspect 7) The memory further includes a set of computer readable instructions, the set of computer readable instructions causing the processor to: determining a first recommended action associated with the aggregate prediction score according to a saliency threshold achieved by matching the aggregate prediction score with a recommended action management database that defines a plurality of candidate recommended actions for subsets of the first set of target machine learning models; 6. The clinical management device of aspect 5, wherein the clinical management device generates a set of predicted outcome data by outputting the first recommended action as one of a set of recommended actions.

[0116] (Aspect 8) The memory further includes a set of computer readable instructions, the set of computer readable instructions causing the processor to: generating a set of output data expressing an aggregate prediction score and a set of recommended actions in a natural language format by processing the set of prediction result data using a large-scale language model; 8. The clinical management device of any one of aspects 1 to 7, wherein the clinical management device outputs a set of predicted outcome data by outputting a set of output data in response to a user input query. [Explanation of symbols]

[0117] 150 Clinical Management Applications 200 Clinical Management System 210 Clinical Management Devices 220 memory 222 Input Processing Unit 224 Model Identification Unit 226 Predictive Management Unit 228 Model Training Units 229 Output Processing Unit 230 Memory Unit 231 Clinical Concept Database 232 Health Management Database 233 candidate ML model database 234 Recommended Actions Database 244 processors 246 Input / Output Unit 250 Communication Network 260 User Terminals

Claims

1. 1. A clinical management device comprising: a processor; a memory; The memory includes a set of computer-readable instructions that cause the processor to: receiving a user input query from a user relating to a set of user health data characterizing a set of clinical concepts and a set of user health features; identifying a first set of target machine learning models from a set of candidate machine learning models based on the user input query and the set of user health data, the target machine learning models being trained to make predictions for a set of prediction targets related to the set of clinical concepts; identifying a subset of the first set of target machine learning models that meet a correlation threshold with respect to the set of user health features included in the user health data; generating a set of prediction result data including aggregate prediction scores for the set of prediction targets and a set of recommended actions associated with the set of user health data by using the subset of the first set of target machine learning models to analyze the set of user health data; and outputting the set of predicted outcome data in response to the user input query.

2. The memory further includes a set of computer readable instructions, the set of computer readable instructions causing the processor to: training a first set of machine learning models using a set of healthcare data to make predictions about the set of prediction targets for inclusion in the set of candidate machine learning models; analyzing a set of treatment outcome data using statistical analysis techniques to define a set of clinical variability thresholds corresponding to said set of prediction targets; 2. The clinical management device of claim 1, wherein the set of candidate machine learning models is generated by: training a second set of machine learning models using the set of treatment outcome data and the set of clinical variation thresholds to estimate a risk of clinical variation for the set of prediction targets for inclusion in the set of candidate machine learning models.

3. The memory further includes a set of computer readable instructions, the set of computer readable instructions causing the processor to: parsing the user-input query using natural language processing techniques to extract a first set of keywords; identifying a first set of clinical symptoms associated with said first set of keywords; 2. The clinical management device of claim 1, wherein the first set of target machine learning models is identified by: using the set of clinical concepts as the first set of target machine learning models; and selecting a target machine learning model associated with a first set of clinical concepts that corresponds to the first set of clinical symptoms in the clinical concept database based on a clinical concept database that links clinical symptoms with the set of candidate machine learning models.

4. The memory further includes a set of computer readable instructions, the set of computer readable instructions causing the processor to: identifying a set of input health features for each target machine learning model in the first set of target machine learning models; calculating an individual correlation score between each target machine learning model in the first set of target machine learning models and each user health feature in the set of user health features based on an overlap between the set of input health features and the set of user health features; calculating an aggregate correlation score between each target machine learning model in the first set of target machine learning models and the set of user health features by adding the individual correlation scores between each target machine learning model in the first set of target machine learning models and each user health feature in the set of user health features; selecting, as the subset of the first set of target machine learning models, target machine learning models having an aggregate correlation score that achieves the correlation threshold; and identifying the subset of the first set of target machine learning models. The clinical management device of claim 1 .

5. The memory further includes a set of computer readable instructions, the set of computer readable instructions causing the processor to: generating a set of individual prediction scores for the set of user health data by using each target machine learning model of the subset of the first set of target machine learning models to analyze the set of user health features; calculating a weight vector indicating the relative importance of each target machine learning model of the subset of the first set of target machine learning models with respect to the set of clinical concepts; calculating a revised set of individual prediction scores by multiplying said set of individual prediction scores by said weight vector; 10. The clinical management device of claim 1, wherein the set of predicted outcome data is generated by: calculating the aggregate prediction score by adding the modified set of individual prediction scores.

6. The memory further includes a set of computer readable instructions, the set of computer readable instructions causing the processor to: calculating a set of feature importance scores indicating the relative importance of each user health feature of the set of user health features with respect to each target machine learning model of the subset of the first set of target machine learning models using a feature importance method; calculating a modified set of feature importance scores by multiplying the set of feature importances by the weight vector; calculating a set of aggregate feature importance scores indicating the relative importance of each user health feature with respect to a particular prediction target of the set of prediction targets by summing the modified set of feature importance scores; generating a set of feature ranking data that ranks the set of health features based on the set of aggregate feature importance scores; and outputting the set of feature ranking data as part of the set of predicted outcome data.

7. The memory further includes a set of computer readable instructions, the set of computer readable instructions causing the processor to: determining a first recommended action associated with the aggregate prediction score according to a salience threshold achieved by matching the aggregate prediction score with a recommended action management database defining a plurality of candidate recommended actions for the subset of the first set of target machine learning models; and outputting the first recommended action as one of the set of recommended actions, thereby generating the set of predicted outcome data.

8. The memory further includes a set of computer readable instructions, the set of computer readable instructions causing the processor to: processing the set of prediction result data using a large-scale language model to generate a set of output data that expresses the set of aggregate prediction scores and recommended actions in a natural language form; 10. The clinical management device of claim 1, wherein the device causes output of the set of predicted outcome data by outputting the set of output data in response to the user input query.

9. 1. A clinical management method implemented by a clinical management device, the clinical management device comprising: a processor; a memory; The memory includes a set of computer-readable instructions that cause the processor to: training a first set of machine learning models using the set of healthcare data to make predictions about the set of prediction targets for inclusion in a set of candidate machine learning models; defining a set of clinical variation thresholds corresponding to said set of prediction targets by analyzing said set of treatment outcome data using statistical analysis techniques; training a second set of machine learning models using the set of treatment outcome data and the set of clinical variation thresholds to estimate risk of clinical variation with respect to the set of prediction targets for inclusion in the set of candidate machine learning models; receiving a user input query from a user relating to a set of user health data characterizing a set of clinical concepts and a set of user health features; parsing the user-input query using natural language processing techniques to extract a first set of keywords; identifying a first set of clinical symptoms associated with said first set of keywords; using the set of clinical concepts to select a first set of target machine learning models associated with a first set of clinical concepts corresponding to the first set of clinical symptoms in the clinical concept database using a clinical concept database that links clinical symptoms with the set of candidate machine learning models; identifying a set of input health features for each target machine learning model in the first set of target machine learning models; calculating an individual correlation score between each target machine learning model in the first set of target machine learning models and each user health feature in the set of user health features based on an overlap between the set of input health features and the set of user health features; calculating an aggregate correlation score between each target machine learning model in the first set of target machine learning models and the set of user health features by adding the individual correlation scores between each target machine learning model in the first set of target machine learning models and each user health feature in the set of user health features; selecting a subset of the first set of target machine learning models having aggregate correlation scores that achieve a correlation threshold for the set of user health features included in the user health data; generating a set of individual prediction scores for the set of user health data by using each target machine learning model of the subset of the first set of target machine learning models to analyze the set of user health features; calculating a weight vector indicating the relative importance of each target machine learning model of the subset of the first set of target machine learning models with respect to the set of clinical concepts; calculating a revised set of individual prediction scores by multiplying said set of individual prediction scores by said weight vector; calculating an aggregate prediction score by adding together the modified set of individual prediction scores; determining a first recommended action associated with the aggregate prediction score according to a salience threshold achieved by matching the aggregate prediction score with a recommended action management database defining a plurality of candidate recommended actions for the subset of the first set of target machine learning models; calculating a set of feature importance scores indicating the relative importance of each user health feature of the set of user health features with respect to each target machine learning model of the subset of the first set of target machine learning models using a feature importance method; calculating a modified set of feature importance scores by multiplying the set of feature importances by the weight vector; calculating a set of aggregate feature importance scores indicating the relative importance of each user health feature with respect to a particular prediction target of the set of prediction targets by summing the modified set of feature importance scores; generating a set of feature ranking data that ranks the set of health features based on the set of aggregate feature importance scores; and outputting a set of predicted outcome data including the aggregate prediction score, the first recommended action, and the set of feature importance scores.

10. 1. A clinical management system, comprising: a clinical management device, the clinical management device comprising: a processor; Memory and User terminal and Including, The memory includes a set of computer-readable instructions that cause the processor to: training a first set of machine learning models using the set of healthcare data to make predictions about the set of prediction targets for inclusion in a set of candidate machine learning models; defining a set of clinical variation thresholds corresponding to said set of prediction targets by analyzing said set of treatment outcome data using statistical analysis techniques; training a second set of machine learning models using the set of treatment outcome data and the set of clinical variation thresholds to estimate risk of clinical variation with respect to the set of prediction targets for inclusion in the set of candidate machine learning models; receiving a user input query from a user relating to a set of user health data characterizing a set of clinical concepts and a set of user health features; identifying a first set of target machine learning models from a set of candidate machine learning models based on the user input query and the set of user health data, the target machine learning models being trained to make predictions for a set of prediction targets related to the set of clinical concepts; identifying a subset of the first set of target machine learning models that meet a correlation threshold with respect to the set of user health features included in the user health data; generating a set of prediction result data including aggregate prediction scores for the set of prediction targets and a set of recommended actions associated with the set of user health data by using the subset of the first set of target machine learning models to analyze the set of user health data; outputting the set of predicted outcome data to the user terminal in response to the user input query.

Images