Model generation device for therapy prediction and associated method and model

Abstract

Methods, apparatus, systems, and articles of manufacture for the generation and application of models for immunotherapy prediction and treatment are disclosed. A balance of accuracy and recall rate may be applied to at least one of a toxicity-related model or a therapeutic effect-related model to configure an immunotherapy treatment of a patient and / or a group of patients. Model outputs may be evaluated differently depending on determining patient selection criteria to trigger different actions.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This patent claims the benefit of priority to U.S. Provisional Patent Application Serial No. 63 / 335215 filed on April 26, 2022, International Application No. PCT / US2022 / 032075 filed on June 3, 2022, and International Application No. PCT / US2022 / 032084 filed on June 3, 2022, each of which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0002] The present disclosure relates generally to model generation and processing, and more particularly to the generation and application of models for therapy prediction and treatment. Background Art

[0003] The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.

[0004] Immunotherapy can be used to provide effective cancer treatment for some patients. For those patients, immunotherapy can provide higher efficacy and lower toxicity than other therapies. Immunotherapy can include targeted antibodies and immune checkpoint inhibitors (ICI), cell-based immunotherapy, immunomodulators, vaccines and oncolytic viruses to help the patient's immune system target and destroy malignant tumors. However, in some patients, immunotherapy can cause toxicity and / or other adverse side effects. The side effects of immunotherapy may be different from those associated with other cancer treatments because the side effects are caused by overstimulated or misguided immune responses, rather than the direct effects of chemotherapy or radiotherapy on cancer and healthy tissues. Immunotherapy toxicity may include conditions such as colitis, hepatitis, pneumonia and / or other inflammations that may be dangerous to patients. Immunotherapy also triggers different (heterogeneous) efficacy responses in different patients. Some patients with effective responses may also be affected by toxicity. Therefore, the evaluation of immunotherapy remains unpredictable, with the possibility of huge differences between patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 An example model generation apparatus is illustrated.

[0006] Figure 2-14 The diagram shows the use of Figure 1 An example model generates a flow chart of a device executing instructions to drive operations.

[0007] Fig.15 An example immunotherapy prediction device is illustrated.

[0008] Figure 16-18 The diagram shows the Fig.15An example immunotherapy prediction device is provided, and a flow chart of an example method for processing data using one or more models.

[0009] Fig.19 An example timeline from data aggregation and associated feature formation to model development relative to the first immunotherapy treatment is illustrated.

[0010] Figure 20-24 Depicted is an example precision / recall chart illustrating characteristics of an example model.

[0011] Fig.25 Shown is an example infrastructure and organization of patient health data from patient health records to features that can be used for model training, testing, and validation.

[0012] Fig.26 is a block diagram of an example processing platform including processor circuitry configured to execute example machine-readable instructions and / or example operations.

[0013] Fig. 27 yes Fig.26 A block diagram of an example implementation of a processor circuit system.

[0014] Fig.28 yes Fig.26 A block diagram of another example implementation of a processor circuit system.

[0015] Fig.29 1 is a block diagram of an example software distribution platform (e.g., one or more servers) for distributing software (e.g., software corresponding to example machine-readable instructions) to client devices associated with end users and / or consumers (e.g., for licensing, sale, and / or use), retailers (e.g., for sale, resale, licensing, and / or sublicensing), and / or original equipment manufacturers (OEMs) (e.g., for inclusion in products to be distributed to, for example, retailers and / or to other end users such as direct purchasing customers). DETAILED DESCRIPTION

[0016] In the following detailed description, reference is made to the accompanying drawings forming a part thereof, and shown therein by way of illustration of specific examples that may be implemented. These examples are described in sufficient detail to enable those skilled in the art to implement the subject matter, and it is to be understood that other examples may be utilized, and logical, mechanical, electrical and other changes may be made, without departing from the scope of the subject matter of the present disclosure. Therefore, the following detailed description is provided to describe an exemplary implementation, and is not to be regarded as a limitation on the scope of the subject matter described in the present disclosure. Certain features from the different aspects described below may be combined to form yet new aspects of the subject matter discussed below.

[0017] In general, the same reference numbers will be used to refer to the same or like parts throughout the drawing(s) and the accompanying written description.The drawings are not to scale.

[0018] definition

[0019] Throughout the specification and claims, the following terms take the meanings explicitly associated herein, unless the context clearly dictates otherwise.

[0020] As used herein, connection references (e.g., attached, coupled, connected, and joined) may include intermediate members between elements referenced by the connection reference and / or relative movement between those elements, unless otherwise indicated. Thus, connection references do not necessarily imply that two elements are directly connected and / or are in a fixed relationship to each other. As used herein, stating that any part is "in contact with" another part is defined to mean that there is no intermediate part between the two parts.

[0021] When introducing the elements of various embodiments of the present disclosure, the articles "one", "an", "the" and "said" are intended to mean one or more elements in the element. As used herein, singular references (e.g., "one", "one", "first", "second", etc.) do not exclude plural. As used herein, the term "one" or "an" object refers to one or more of the object. The terms "one" (or "one"), "one or more" and "at least one" are used interchangeably herein. In addition, although listed by individuals, multiple parts, elements or method actions can be realized by, for example, the same entity or object. In addition, although individual features may be included in different examples or claims, these features may be combined, and being included in different examples or claims does not imply that the combination of features is not feasible and / or disadvantageous.

[0022] The terms "include," "comprising," and "having" are intended to be inclusive and mean that in addition to the listed elements, additional elements may be present. That is, "include" and "comprising" (and all forms and tenses thereof) are used herein as open-ended terms. Thus, whenever a claim employs any form of "include" or "comprising" (e.g., comprises, includes, comprising, including, having, etc.) as a preamble or in any type of claim narrative, it is to be understood that additional elements, terms, etc. may be present without falling outside the scope of the corresponding claim or narrative. As used herein, when the phrase "at least" is used as a transitional term, such as in the preamble of a claim, it is open-ended in the same manner as the terms "include" and "comprising" are open-ended. For example, when used in a form such as A, B, and / or C, the term "and / or" refers to any combination or subset of A, B, C, such as (1) A alone, (2) B alone, (3) C alone, (4) A and B, (5) A and C, (6) B and C, or (7) A and B and C. As used herein in the context of describing structures, components, items, objects, and / or things, the phrase “at least one of A and B” is intended to refer to implementations that include any of (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in the context of describing structures, components, items, objects, and / or things, the phrase “at least one of A or B” is intended to refer to implementations that include at least one of: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. As used herein in the context of describing the execution or performance of processes, instructions, acts, activities, and / or steps, the phrase “at least one of A and B” is intended to refer to implementations that include any of: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in the context of describing the execution or performance of a process, instruction, action, activity, and / or step, the phrase "at least one of A or B" is intended to refer to an implementation that includes any of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.

[0023] Unless otherwise specifically stated, descriptors such as "first," "second," "third," etc., are used herein without importing or otherwise indicating any meaning of priority, physical order, arrangement in a list, and / or ordering in any manner, but are merely used as labels and / or arbitrary names to distinguish elements to facilitate understanding of the disclosed examples. In some examples, the descriptor "first" may be used to refer to an element in the detailed description, while the same element may be referred to in the claims by a different descriptor such as "second" or "third." In such instances, it should be understood that such descriptors are only used to clearly identify those elements that, for example, might otherwise share the same name.

[0024] As used herein, "approximately" and "approximately" modify their subject matter / values ​​to recognize the possible existence of variations that occur in real-world applications. For example, "approximately" and "approximately" may modify dimensions that may not be exact due to manufacturing tolerances and / or other real-world imperfections as would be understood by one of ordinary skill in the art. For example, "approximately" and "approximately" may indicate that such a dimension may be within a tolerance range of + / - 10%, unless otherwise specified in the description below. As used herein, "substantially real time" means occurring in a near instantaneous manner, recognizing that there may be real-world delays in computation time, transmission, etc. Therefore, unless otherwise specified, "substantially real time" means real time + / - 1 second.

[0025] As used herein, the phrase "in communication" (including variations thereof) includes direct communication and / or indirect communication through one or more intermediate components without requiring direct physical (e.g., wired) communication and / or continuous communication, but additionally includes selective communication at periodic intervals, predetermined intervals, non-periodic intervals and / or one-time events.

[0026] As used herein, the terms "system," "unit," "module," "engine," and the like may include hardware and / or software systems that operate to perform one or more functions. For example, a module, unit, or system may include a computer processor, controller, and / or other logic-based device that performs operations based on instructions stored on a tangible and non-transitory computer-readable storage medium (such as a computer memory). Alternatively, a module, unit, engine, or system may include a hard-wired device that performs operations based on the hard-wired logic of the device. The various modules, units, engines, and / or systems shown in the accompanying drawings may represent hardware that operates based on software or hard-wired instructions, software that directs hardware to perform operations, or a combination thereof.

[0027] As used herein, "processor circuitry" is defined as including (i) one or more special-purpose circuits that are configured to perform (one or more) specific operations and include one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors), and / or (ii) one or more general-purpose semiconductor-based circuits that can be programmed with instructions to perform specific operations and include one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors). Examples of processor circuitry include programmable microprocessors, field programmable gate arrays (FPGAs) that can instantiate instructions, central processor units (CPUs), graphics processor units (GPUs), digital signal processors (DSPs), XPUs or microcontrollers, and integrated circuits such as application-specific integrated circuits (ASICs). For example, an XPU can be implemented by a heterogeneous computing system that includes multiple types of processor circuitry (e.g., one or more FPGAs, one or more CPUs, one or more GPUs, one or more DSPs, etc. and / or combinations thereof) and (one or more) application programming interfaces ((one or more) APIs), which can assign (one or more) computing tasks to the one (or more) of the multiple types of processor circuitry that is most suitable for performing the (one or more) computing tasks.

[0028] Additionally, it should be understood that references to "one embodiment" or "an embodiment" of the present disclosure are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features.

[0029] As used herein, a "toxicity-related model" (also referred to herein as a toxicity model) is a machine learning and / or other artificial intelligence model that processes input data to produce a prediction of toxicity in a patient and / or models another surrogate or alternative characteristic related to toxicity associated with an immunotherapy (e.g., risk of harm from a side effect or condition associated with immunotherapy treatment such as pneumonitis, colitis, hepatitis, etc.). An "efficacy-related model" (also referred to herein as an efficacy model) is a machine learning and / or other artificial intelligence model that processes input data to produce a prediction of efficacy in a patient and / or models another surrogate or alternative characteristic related to the efficacy or effectiveness of an immunotherapy (e.g., likelihood that an immunotherapy will be effective, such as treatment duration, overall survivability, etc.).

[0030] A "feature" or feature variable represents an attribute or type of data. Features can be expressed in various formats such as categorical, numeric, date, percentage, etc.

[0031] "Accuracy" refers to the proportion or percentage of results in which an artificial intelligence model outputs actual correct results. For example, a model outputs a prediction, and a set of model output predictions can be evaluated to determine the accuracy of the model. Accuracy can be evaluated on a scale of 0.0 to 1.0, for example, where 0.0 indicates that the model's predictions are never correct and 1.0 indicates that the model's predictions are always correct. Accuracy can be defined in terms of true positives (TP) and false positives (FP). A true positive is a prediction that is a correct prediction, while a false positive is an incorrect prediction. The accuracy of the model can then be determined by the following formula: Accuracy = TP / (TP + FP).

[0032] "Recall" refers to the proportion of true positives (actual positives) that are correctly identified. Recall can also be evaluated on a scale of 0.0 to 1.0, for example, where 1.0 indicates that the model never produces false negatives and 0.0 indicates that the model always produces false negatives. Recall can be defined in terms of true positives and false negatives (FN), for example: Recall = TP / (TP+FN).

[0033] The term "deep learning" refers to a machine learning technique that utilizes multiple data processing layers to identify various structures in a data set and classify the data set with high accuracy. A deep learning network (DLN) (also known as a deep neural network (DNN)) can be a trained network (e.g., a trained network model or device) that learns patterns based on multiple inputs and outputs. A deep learning network / deep neural network can be a deployed network (e.g., a deployed network model or device) that is generated from a trained network and provides outputs in response to inputs.

[0034] The term "supervised learning" is a machine learning training method in which a machine is provided with already classified data from a human source. The term "unsupervised learning" is a machine learning training method in which a machine is not given already classified data but is made available for anomaly detection (e.g., random forests, gradient boosting, etc.). The term "semi-supervised learning" is a machine learning training method in which a machine is provided with a small amount of classified data from a human source compared to the large amount of unclassified data available to the machine.

[0035] The term "convolutional neural network" or "CNN" is a biologically inspired network of interconnected data used in deep learning for detecting, segmenting, and identifying objects and regions of interest in a data set. CNN evaluates raw data in the form of multiple arrays, breaking it down in a series of stages, examining it for learned features. For example, CNNs can be used to predict hepatitis and / or toxicity.

[0036] The term "recurrent neural network" or "RNN" refers to a network in which the connections between nodes form a directed or undirected graph along a time series. For example, an RNN can be used to predict hepatitis and / or toxicity.

[0037] The term "transfer learning" is the process by which a machine stores information used to correctly or incorrectly solve one problem in order to solve another problem of the same or similar nature as the first problem. Transfer learning can also be called "inductive learning." For example, transfer learning can leverage data from previous tasks.

[0038] The term "active learning" is the process of machine learning in which a machine selects the set of examples for which it receives training data, rather than passively receiving examples selected by an external entity. For example, when learning, the machine can be allowed to select examples that the machine determines will be most helpful for learning, rather than relying solely on external human experts or external systems to identify and provide examples.

[0039] The terms "computer-aided detection" or "computer-aided diagnosis" refer to computers that analyze medical data to come up with a possible diagnosis.

[0040] Example model generation systems and associated methods

[0041] A large amount of health-related data can be collected using various media and mechanisms about patients. However, processing and interpreting the data can be difficult to drive actionable results. For example, given the magnitude of the data and the various different systems, formats, etc., it may be difficult, if not impossible, to understand and correlate data in various forms and sources through standardization / normalization, aggregation, and analysis. Therefore, certain examples provide devices, systems, associated models, and methods for processing and correlating health-related data to predict patient outcomes and drive patient diagnosis and treatment. Certain examples provide systems and methods for health data prediction model construction. Certain examples provide frameworks and machine learning workflows for therapy prediction.

[0042] For example, immune checkpoints regulate the human immune system. Immune checkpoints are pathways that allow the body to tolerate itself by preventing the immune system from indiscriminately attacking cells. However, some cancers can protect themselves from attack by stimulating immune checkpoints (e.g., proteins on immune cells). To target cancer cells in the body, immune checkpoint inhibitors (ICIs) can be used to target these immune checkpoint proteins to better recognize and attack, rather than hide from, cancer cells.

[0043] Despite the great success of ICI cancer treatment, such treatment can pose a great threat to human health due to its side effects, which is an immune-related adverse event (irAE) caused by these treatment options. One of these toxicities is hepatitis, which occurs when the liver is affected by an autoimmune-like inflammatory pathological process triggered by ICI. Some examples predict the onset of irAE hepatitis before the start of the first ICI treatment. More precisely, some examples predict whether irAE hepatitis will occur within a given time window after the start of the first treatment. Other toxicities such as pneumonia, colitis, etc. can be similarly predicted.

[0044] For example, majority class undersampling is combined with time series data aggregation to obtain a well-balanced and static data set that can be fed to the model. Example models include gradient boosting (GB) and random forest (RF), and / or other models that can adapt to the size and statistical properties of the data. The model can be selected based on a model selection criterion such as the Fl score, which is a measure of the accuracy of the model on the data set. For example, a GB model without undersampling can maximize the Fl score (e.g., the harmonic mean of recall and precision), while a RF model with undersampling can provide a high recall (e.g., the true positive ratio found) with a relatively low precision (e.g., the true positive ratio among estimates), which is acceptable due to the cost-effectiveness of the additional tests required for model-based decision making. The model is also able to create probability estimates for labels, not just the discrete labels themselves.

[0045] Prepare input data to develop and / or feed (one or more) models to drive predictions, therapies, etc. In some examples, the input is prepared by extracting blood feature information (e.g., relevant portions of blood features, etc.) from an electronic health record (EHR) data table (e.g., received from the EHR, etc. at a processor / processing circuit system), electronic medical record (EMR) information, etc. Blood features are measurements of liver biomarker concentrations in plasma (such as ALT, AST, alkaline phosphatase, and bilirubin, etc.) and other concentration values ​​in the blood. For example, blood features can be represented as time series data. After the blood features are extracted, the time series data is formed into a single complex data structure. The data structure is used to aggregate the time series blood feature data into a data table, which can be used with preprocessing and transformation.

[0046] For example, feature engineering aggregates blood feature data by describing the time series data of blood particles with associated mean, standard deviation, minimum and maximum values. Liver features can also be created by taking the last liver biomarker measurement available before treatment. When the level of at least one liver biomarker exceeds a threshold value (e.g., 3 times the upper limit of normal, etc.) within a predefined window, a label (e.g., using a definition obtained from a medical expert, etc.) can be created to classify someone as positive. Otherwise, the label can classify the patient as negative. The date of immune checkpoint inhibitor (ICI) treatment can be determined and / or otherwise provided for label and / or time series use.

[0047] Other features may include frequency of lung-related codes (e.g., ICD-9, ICD-10, etc.), frequency of C34 codes, frequency of C78 codes, smoking, oxygen saturation in the blood, frequency of blood hemoglobin below normal, frequency of blood hemoglobin above normal, frequency of blood albumin below normal, frequency of blood albumin above normal, frequency of white blood cell count below normal, frequency of white blood cell count above normal, frequency of red blood cell count below normal, frequency of red blood cell count above normal, frequency of lymph node readings above normal, frequency of lymph nodes below normal, liver enzymes, drug class, interaction, maximum value, minimum value, time-weighted average, final value, number of days before first treatment, etc.

[0048] After preparing the input data (e.g., using feature engineering), the dataset is resampled. That is, the dataset resulting from the input preparation is unbalanced. Therefore, the dataset can be processed to infer, verify, estimate, and / or otherwise resample the prepared feature data in the dataset. For example, when the goal is to maximize the recall value, random majority class undersampling is performed on the dataset. When the Fl score is the subject of maximization, resampling can be skipped or ignored.

[0049] Using a data set (e.g., resampled or otherwise), a model can be trained and tested to generate predictions. For example, when the expected recall rate is maximized as a model selection criterion, the data set can be used to train the RF model. For example, when the expected F1 score is maximized as a model selection criterion, the data set can be used to train the GB model. Therefore, the model selection criteria can be used to determine the desired purpose, goal or focus, such as maximizing the recall rate, maximizing the F1 score, maximizing the precision, etc. For example, a model generated for configuring a patient group for a clinical trial can be selected or otherwise set to allow many patients to participate in the trial. As another example, a model for determining a patient's treatment plan can be selected to be treated cautiously (or alternatively actively) based on the toxicity possibility balanced with the efficacy possibility and the associated risks. For example, a high toxicity risk combined with a low efficacy chance may prevent a patient from undergoing immunotherapy treatment, while a low toxicity risk paired with a low efficacy chance encourages the patient to start (or continue) immunotherapy treatment. For example, the relative balance of toxicity risk and efficacy probability can be selected based on the interests, preferences and / or goals of the patient and / or his / her doctor. Additionally, safety monitoring, blood tests, and / or other measures (e.g., regular reassessment, check-ins, etc.) can be ordered depending on the likelihood of toxicity so that any toxicities can be detected early and managed while continuing treatment. In some examples, the trained model can be validated, such as using leave-one-out cross-validation, where each sample is predicted by an individual and the remaining samples serve as a training set.

[0050] Therefore, various "artificial intelligence" (AI) models can be developed and deployed for use in various health prediction applications. For example, the model can be used to predict static and / or dynamic prognostic factors of hepatitis using AI models and patient (e.g., EHR, etc.) data.

[0051] Alternatively or additionally, a predictive model can be developed for ICI-related pneumonia using a small, noisy data set. Using input data from structured (e.g., EHR, EMR, (one or more) laboratory data systems, etc.) and / or unstructured (e.g., from (one or more) laboratory data systems, EHR, EMR, etc. managed (curate)) data, the input features can be evaluated to build a model and output the predicted probability of developing pneumonia. In some examples, multiple models can be developed, and the system and associated processes can iterate and decide between two or more model versions. For example, the available data can be divided into two partitions using a sequential forward selection process, and a robust performance evaluation can be used to verify and compare the two developed models to select one for deployment.

[0052] For example, the available data can be divided into two partitions, and a sequential forward selection process can be used to build a model. In each iteration, two model versions (e.g., two adjusted models refined from an initial model, etc.) are compared, and the version with higher performance on both partitions is selected. A second model version (e.g., an adjusted model) is created by adding candidate features to a first model version (e.g., an initial model or a previous adjusted model). On larger partitions, performance results in an inner loop based on nested cross validation (CV) are compared. On smaller partitions, a permutation test is performed to test whether the candidate features have predictive power (e.g., the second model version has higher performance). Finally, an outer loop of nested CV is used for robust performance evaluation of the final model.

[0053] Certain examples provide an automated framework for preparing EHR and / or other health data for use in machine learning-based model training. For example, the framework prepares data from multiple sources and generates a combined time-dependent, unrestricted intermediate output that is used to aggregate features for training and deploying time-dependent models. For example, input data sources can be processed and patient vectors can be generated using the data. The patient vectors can be used for filtering and aggregation, forming an interface definition. Thus, a model-independent workflow creates input data sets for multiple model training. The intermediate outputs preserve temporal structure for sequential modeling tasks and form a maintainable, sustainable framework with an interface.

[0054] Certain examples provide prediction model construction related to ICI, wherein input data from multiple sources are prepared. A ground truth prediction label can be generated from the prepared data and / or a label can be professionally created. Then, one or more models are built on a feature matrix generated using the labels and data, wherein the ground truth prediction label is an independent module of the framework. For example, the framework can then drive a workflow to assess multiple efficacy surrogate endpoints to predict (one or more) responses to ICI therapy.

[0055] In some examples, a system involving multiple submodules is used to prepare and use patient health data to train a model. For example, the system includes a data extraction and processing submodule that is used to extract patient blood test history from EMR / EHR, clean blood history data, and perform (one or more) data quality checks. The label definition submodule defines one or more feature labels related to blood history data, and the feature engineering submodule can form blood features by aggregating and processing blood history data about labels. The model submodule trains and evaluates AI models to dynamically predict the risk of immune-related hepatitis adverse events from a fixed length of blood test history. Alternatively or additionally, liver function test values ​​can be extracted, cleaned and organized in time series. For example, a label definition algorithm can be executed to generate AI models and indicator labels for each set of blood and / or liver test values, while feature engineering (e.g., normalization, symbol transformation and motifs extraction) can be used to train and evaluate (one or more) AI risk prediction models. Similarly, drug history information, medical condition history, anthropometric features, etc. can be used for labeling and feature formation.

[0056] Because features can vary significantly between toxicities, toxicity-specific feature formation and model training are important to provide meaningful features and accurate predictive results. Otherwise, the lack of meaningful features undermines the performance of the resulting model. Therefore, certain examples derive models that focus on specific toxicities, and the outputs can be combined (together and / or further with efficacy predictions) to form recommendations, such as a risk-to-benefit analysis based on toxicity versus efficacy of a given immunotherapy drug.

[0057] Thus, certain examples drive treatment based on predictions of the likelihood of complications of hepatitis, pneumonia, etc. Based on the predictions and / or other outputs of one or more AI models, patients may be selected for immunotherapy treatment, removed from immunotherapy treatment and / or otherwise have their treatment plan adjusted, selected for immunotherapy clinical trials, etc. As data continues to be collected from one or more patients, the (one or more) models used in the predictions may evolve, and the associated (one or more) predictions may also change based on the collected data. For example, the (one or more) models and / or associated (one or more) predictions may be customized for individuals and / or deployed for groups / types / etc. of patients, or for a group or individual ICI drugs, etc.

[0058] In some examples, data values ​​are normalized to the upper limit of a "normal" range (e.g., for blood tests, liver tests, etc.) so that values ​​from different sources can be compared on the same scale. Data values ​​and associated normalization / other processing can be specific to a laboratory, patient, patient type (e.g., male / female, etc.), etc. For example, each laboratory measurement may have a specific normal range for evaluating its values ​​across multiple patients.

[0059] For time series data, one value depends on previous values ​​and / or other values, so that the data values ​​have a relationship, rather than being independent. Correlations can be identified and taken into account to identify patients in the data over time. For example, if the data values ​​in the time series of a patient's blood work exceed twice the normal limit at the first time, decrease within the normal limit at the second time, and exceed twice the normal limit again at the third time, this pattern can be identified as important (e.g., worthy of further analysis). For example, data processing can annotate or mark such a pattern accordingly. Therefore, clinical data from the patient's records can be used to identify and form features, anomalies, other patterns, etc. over time. The data is confused into a general model for comparison. The resulting model trained and tested on such data can be robust to outliers, scaling, etc. Features can be created for better (e.g., more efficient, more accurate, more robust, etc.) modeling, such as pneumonia modeling, colitis modeling, hepatitis modeling, etc.

[0060] Thus, data processing creates feature(s) that can be used to develop model(s) that can be deployed to predict outcome(s) for patient(s). For example, a data processing pipeline creates thousands of features (e.g., pneumonia, frequency of ICD-10 codes, frequency of C34 codes, etc.).

[0061] For example, the data value may include the ICD-10 code for a given patient within a one-year period. In some examples, the code may span multiple years (e.g., ten years, etc.) and be coordinated for processing. The ICD-10 code is processed to identify the code associated with the lung or respiratory function, and then such a code can be used to calculate the relative frequency of lung disease in the patient. As another example, the patient history can be analyzed to determine the relative frequency of the C34 code in the patient history, which indicates lung cancer. For example, smoking status can also be a binary flag that is set or not set from the data processing pipeline. In some examples, the code can be converted between code systems (e.g., ICD-9, ICD-10 (such as C34, C78, ​​etc.), etc.). For example, the code can be reverse engineered without all the keys in the secret key.

[0062] Using the codes and other data, multiple (e.g., 5, 6, 10, etc.) features can be created and used in a modeling framework to predict the development of pneumonia in a patient. The model is built in a step-by-step forward manner. Labels for the pneumonia model are not inherently present in the dataset, so ground truth is created for model training based on expert judgment to identify labels, for example, from (one or more) patient histories. For example, a code book and quality control can be used for correct labeling.

[0063] In some examples, historical data received from patients is asynchronous. Systems and methods then align data for patients (and / or between patients) to enable aggregation and analysis of data relative to a common baseline or benchmark. In some examples, an impact point or other reference can be selected / determined, and patient data time series / timelines can be aligned around the determined or selected point (e.g., an event that occurred for (one or more) patients, such as a physical examination, injury, attack, test, birthday, milestone, etc.). For example, the date of the first chemotherapy, ICI therapy, the first symptom / reaction (e.g., lung function, etc.), etc. can be used to align patient data.

[0064] The processed data can then be used to predict static labels in predefined or otherwise determined time windows, etc. Models can be trained, validated, and deployed for hepatitis, pneumonia, drug efficacy, etc. Thus, data from EHRs, EMRs, laboratory systems, and / or other data sources can be pre-processed and provided to the model to generate predictions, which can be post-processed and output to users and / or other systems for alerts, follow-up, treatment plans, etc. In some examples, the predicted values ​​are routed to another system (e.g., scheduling, laboratory, etc.) for further processing.

[0065] One or more AI models can be used to facilitate processing, association, and prediction of available patient health data based on, for example, blood test results, liver test results, other test results, other patient physiological data, and the like. Models can include high recall low precision models, low recall high precision models, harmonic mean maximization (convergence) models, and the like. Enhanced decision tree models or variants such as random forests (RF), gradient boosting (GB), and the like can be used. For example, a majority class undersampled random forest model can be used to maximize recall with relatively low precision. However, for hepatitis, prevention is cheap and easy, so false negatives are affordable. Therefore, a gradient boosting model can be developed to maximize the Fl score without applying resampling.

[0066] For example, machine learning techniques (whether deep learning networks or other empirical / observational learning systems) can be used to characterize and otherwise interpret, extrapolate, infer, and / or complete medical data obtained from patients. Deep learning is a subset of machine learning that uses a collection of algorithms to model high-level abstractions in the data using deep graphs with multiple processing layers (including linear and nonlinear transformations). While many machine learning systems are seeded with initial features and / or network weights to be modified by learning and updating of the machine learning network, deep learning networks train themselves to identify "good" (e.g., useful, etc.) features for analysis. Using a multi-layer architecture, machines using deep learning techniques can process raw data better than machines using traditional machine learning techniques. Examining data for groups of highly correlated values ​​or unique topics is facilitated using different evaluation or abstraction layers.

[0067] Deep learning is a class of machine learning techniques that employs a representation learning approach, which allows a machine to be given raw data and determine the representations needed to classify the data. Deep learning uses a back-propagation algorithm to determine the structure in a data set, which is used to change the internal parameters of the deep learning machine (e.g., node weights). Deep learning machines can utilize a variety of multi-layer architectures and algorithms. For example, while machine learning involves identifying features to be used to train a network, deep learning processes raw data to identify features of interest without external identification.

[0068] Deep learning in the context of a neural network consists of numerous interconnected nodes called neurons. Input neurons activated from an external source activate other neurons based on connections to those other neurons controlled by the machine parameters. A neural network behaves in a certain way based on its own parameters. Learning refines the machine parameters and, by extension, the connections between neurons in the network so that the neural network behaves in the desired manner.

[0069] Various artificial intelligence networks can be deployed to process input data. For example, deep learning using convolutional neural networks segments data using convolutional filters to locate and identify learned observable features in the data. Each filter or layer of the CNN architecture transforms the input data to increase the selectivity and invariance of the data. This abstraction of the data allows the machine to focus on the features in the data it is trying to classify and ignore irrelevant background information.

[0070] Deep learning operates on the understanding that many data sets consist of high-level features that are comprised of lower-level features. For example, when examining an image, rather than looking for objects, it is more efficient to look for edges that form patterns that form components that form the objects being looked for. These hierarchies of features can be found in many different forms of data, such as speech and text.

[0071] Learned observable features include objects and quantifiable rules learned by the machine during supervised learning. Machines provided with large collections of well-classified data are better equipped to discern and extract features that are relevant to the successful classification of new data.

[0072] A deep learning machine utilizing transfer learning may correctly connect data features to certain classifications confirmed by a human expert. Conversely, the same machine may update parameters used for classification when notified of an incorrect classification by a human expert. For example, settings and / or other configuration information may be guided by learned use of settings and / or other configuration information, and as the system is used more (e.g., repeatedly and / or by multiple users), the number of variations and / or other possibilities for settings and / or other configuration information may be reduced for a given situation.

[0073] For example, an example deep learning neural network can be trained on an expert classification dataset. This dataset builds the first parameters for the neural network, and this will be the stage of supervised learning. During the supervised learning stage, the neural network can be tested to determine whether the desired behavior has been achieved.

[0074] Once the desired neural network behavior has been achieved (e.g., the machine has been trained to operate according to specified thresholds, etc.), the machine can be deployed for use (e.g., testing the machine with new / updated data, etc.). During operation, the neural network classification can be confirmed or rejected (e.g., by an expert user, an expert system, a reference database, etc.) to continue to improve the neural network behavior. The example neural network is then in a state of transfer learning because the parameters used to determine the classification of the neural network behavior are updated based on the ongoing interaction. In some examples, the neural network can provide direct feedback to another process. In some examples, the neural network outputs data that is buffered (e.g., via the cloud, etc.) and verified before it is provided to another process.

[0075] When interacting with doctors, deep learning machines can leverage transfer learning to offset the small datasets available in supervised training. These deep learning machines can improve their computer-aided diagnosis over time through training and transfer learning. However, larger datasets lead to more accurate and robust deployed deep neural network models, which can be applied to transform different medical data into actionable results (e.g., system configuration / settings, computer-aided diagnosis results, image enhancement, etc.).

[0076] One or more such models / machines may be developed and / or deployed with respect to prepared and / or managed data. For example, features (e.g., liver biomarkers, blood / plasma concentrations, etc.) may be extracted from EHR / EMR data tables, etc. For example, managed data extracts structured data (e.g., diagnosis dates from medical records, etc.) from unstructured sources. The extracted features form the basis for label creation. Time series measurements are extracted and aggregated to produce statistical descriptors of the time series data. A table of such data is then used to train a model for prediction. In some examples, the data may be resampled if necessary, desired, or determined. The model is validated using robust cross-validation (such as leave-one-out cross-validation).

[0077] For example, a selection, input, or goal may determine whether to maximize the Fl score or recall using a given model. When the Fl score is to be maximized by the model, then no resampling of the data is performed. When the recall is to be maximized by the model, then resampling of the data may be performed. For example, a decision may be driven by the deployment environment of the model. When the model is to be deployed in a system that facilitates drug trials, then the Fl score is to be maximized because pharmaceutical companies want patients with the greatest chance of responding well to a given drug. In a clinical treatment setting, a high recall is even more important, where the system hopes to eliminate as much toxicity as possible. For example, in a treatment setting for hepatitis, low model accuracy is acceptable due to the inexpensive nature of hepatitis treatment.

[0078] The associated systems and methods can be used to assess the probability or reliability of predictions generated by the model. For example, the predictions and associated confidence levels or other reliability can be output to another processing system. The possible response to immunotherapy treatment can be modeled based on available patient data collected in clinical practice.

[0079] Figure 1 An example model generation device 100 is illustrated, including an example input processor circuit system 110, an example model trainer circuit system 120, an example model comparator circuit system 130, an example model storage circuit system 140, and an example model deployer circuit system 150. Figure 1As shown in the example of , the example input processor circuit system 110 processes input data extracted from the record to form a candidate feature set. The example model trainer circuit system 120 uses the candidate feature set to train at least a first model and a second model. The example model comparator circuit system 130 tests at least the first model and the second model to compare the performance of the first model and the second model. Based on the comparison, the example model comparator circuit system 130 selects at least one of the first model or the second model based on the comparison. The example model storage circuit system 140 provides a memory circuit system to store the selected first and / or second model. The example model deployer circuit system 150 deploys the selected first model and / or second model to predict the possibility of toxicity occurring due to immunotherapy according to the patient's treatment plan, such as pneumonia, hepatitis, colitis, etc.

[0080] In operation, the example input processor circuit system 110 processes input data related to one or more patients, such as laboratory results, diagnosis codes, billing codes, etc. The input can originate from one or more external systems 160, such as EHR, EMR, etc. For example, the example input processor circuit system 110 can extract and organize the input for each patient in a time series. In some examples, the input processor circuit system 110 aligns the input data relative to an anchor point to organize the input data in a time series. Anchor points are time reference points or alignment points for input data, such as when a patient starts immunotherapy treatment, the first patient symptom, etc., which enables, for example, tracking, modeling, and comparison of a group of patients relative to that reference time point. In some examples, the input processor circuit system 110 generates labels for the input data to form a set of candidate features.

[0081] In some examples, the input processor circuit system 110 performs feature engineering on the candidate feature set and / or the base data. For example, the input processor circuit system performs feature engineering on the candidate feature set by, for example, regularizing, transforming, and / or extracting from the candidate feature set. In some examples, the input processor circuit system 110 selects from the candidate feature set to form a patient feature set to train and / or test at least the first model and the second model based on feature engineering. In some examples, the input processor circuit system 110 generates a feature matrix to train and / or test / validate the first model and / or the second model based on feature engineering.

[0082] In some examples, the model deployer circuit system 150 deploys the selected first and / or second model as an executable tool with an interface to facilitate the collection of patient data and interaction with the selected first and / or second model. The (one or more) deployed models can be used with the tool to select patients for clinical trials, start immunotherapy treatment courses according to treatment plans or protocols, adjust current immunotherapy treatment plans, etc. In some examples, the (one or more) deployed models can be stored / utilized in one or more external systems 160 such as EHR, EMR, scheduling system, clinical information system, etc. The tool implemented as a portal, online platform, computer application, etc. can be used to load one or more stored models (e.g., toxicity-related models, efficacy-related models, etc.), determine one or more thresholds or other patient selection criteria by which to evaluate the output of the (one or more) models, use the (one or more) models to process the health care data of a given patient, and evaluate the (one or more) predicted outputs of the (one or more) models relative to the (one or more) thresholds to take action on the patient based on the threshold (e.g., configure a treatment plan for the patient that includes or excludes immunotherapy, where that treatment plan can be a treatment plan for a patient group in a clinical trial, etc.).

[0083] Thus, the example model generator device 100 may pre-process data from one or more external systems 160 via the example input processor circuit system 110. The example model generator device 100 then uses the model trainer circuit system 120 and the model comparator circuit system 130 to train and validate multiple models. The example model generator device 100 uses the model storage circuit system 140 and the model deployer circuit system 150 to post-process, store and deploy one or more of the trained and validated models. Input data from EHR, EMR and / or other external systems 160 is transformed into multiple models, which can be stored and deployed to predict treatment efficacy, toxicity risks and / or other side effects, etc. from immunotherapy treatment of a specific patient. For example, data from multiple patients can be utilized through processing and modeling to generate indicator predictions for a specific patient.

[0084] In some examples, the model trainer circuit system 120 trains and verifies a boosted decision tree model, other ensemble and / or regression-based learning models, and the like. For example, a boosted decision tree model utilizes multiple weak learning decision trees to create a strong learning AI model. For example, a tree in the model can correct errors in other trees of the model, and the entire set of trees in the model works together to generate a predicted output. The model can also be formed as a random forest (RF) model, a gradient boosting (GB) model, and the like. For example, the data is conformed to a general model for comparison by the model trainer circuit system 120, which is robust to outliers and can be scaled.

[0085] In some examples, the input data processor 110 regularizes the input data value to the upper limit of normal, so that the value (e.g., blood work value, liver function test value, other laboratory work value, etc.) is on the same scale. Each laboratory value may have a certain range of values. For example, the input data processor 110 regularizes the value of a specific laboratory type within a range so that the data for that laboratory can be compared across patients. The input data processor 110 can also organize the data into a time series. The input data processor 110 can be based on anchor points (e.g., common or reference events such as hospitalization, symptoms, birth, diagnosis date, first ICI application (administration) date, etc., data and associated (one or more) features can be aligned around the common or reference events for model development and prediction) to regularize and / or otherwise adjust the time series data.

[0086] For example, the received patient data may be asynchronous. Selecting anchor points / inflection points / influence points allows data to be aligned for the same, similar, and / or different patients. For example, in a patient population with diabetes, ten years of patient history are reviewed while excluding data occurring after diagnosis. The diagnosis point is selected as an anchor point for retrospectively reviewing patient history data (e.g., when blood glucose levels were not obtained or insulin was not taken, etc.). As another example, data from the first immunotherapy treatment may be used to align patient data. For example, aligning multiple patients relative to the start of their immunotherapy allows correlation between subsequent events for patients in multiple patients.

[0087] For example, date of birth, date of first immunotherapy, date of last immunotherapy, and date of death (or knowledge that the patient is still alive 5 years after starting immunotherapy) can be used to align all patients with the start of their immunotherapy as an anchor, regardless of the patient's age at that point in time. Thus, for example, data is organized for identifying relationships and comparisons. For example, patterns can be identified in the data.

[0088] In some examples, the time series data is formed into a plurality of features by the input data processor 110. These features may be a set of candidate features from which features are selected to train and validate a model. For example, patient demographic data, biometric data, examination data, laboratory data, and other patient health care data may be extracted from patient records to form an input data set that may be associated with a plurality of features, such as red blood cell values, white blood cell values, drug counts of one or more medications, blood albumin, lymphatic system, alkaline phosphate, radiology, etc.

[0089] For example, feature engineering performed by the input data processor 110 can form multiple features based on codes (e.g., ICD-10 codes, etc.). For example, the ICD-10 codes of patients in a given year can be processed to identify codes related to lung and / or respiratory function (C34, C78, ​​etc.), and the time series of those codes can form a function for calculating the relative likelihood of lung disease in the patient. Similarly, multiple features can be formed to predict the development of pneumonia, hepatitis, colitis, and / or other toxicities from immunotherapy treatment.

[0090] Features of the classified or identified data based on lung function, liver biomarkers, blood work (e.g., concentrations in blood, plasma, etc.), etc., can form the basis for the creation of labels and / or other statistical descriptors for model training. For example, the collected data can be compared to a benchmark set of true values ​​for the identified labels to generate labels for the data. Resampling and validation (e.g., leave-one-out cross-validation, other cross-validation, other validation, etc.) can be performed to generate a robust model.

[0091] The example model trainer circuit system 120 uses features and / or other data to train one or more AI models, such as toxicity prediction models (e.g., hepatitis prediction models, colitis prediction models, pneumonia prediction models, etc.), efficacy prediction models (e.g., immunotherapy efficacy models, etc.), and the like. The model may be a high recall with low precision, a low recall with high precision, a harmonic mean (Fl) maximization, a majority of claims undersampling, and the like. A determination of the model focus may be made by and / or for the example model trainer circuit system 120 (e.g., based on settings, modes, types, other model selection criteria, etc.). For example, the model trainer circuit system 120 trains one or more models to maximize the Fl score or maximize the recall rate. When focusing on Fl, resampling may not be necessary. When focusing on recall, resampling (e.g., using a training data set and a test data set, multiple training data sets, and / or multiple test data sets, etc.) may be desirable to improve the model.

[0092] For example, when configuring participation in a drug trial, the focus is on maximizing the Fl score to identify patients with the highest chance of responding well to a given immunotherapy drug. However, when determining clinical treatment for a patient, the goal is to eliminate as many toxicities as possible from the patient. Therefore, models developed with high recall are more important, while low precision is acceptable as cheaper treatment options for hepatitis, colitis, pneumonia, etc.

[0093] Model comparator circuitry 130 may compare and / or otherwise assess multiple training models to validate the models and assess the likelihood of predictions generated by each model. Such assessment / assessment may be used to select one or more models to be deployed. For example, models may be compared and / or otherwise assessed based on average output values, standard deviations of model outputs, etc. One or more models may be selected for storage in model storage circuitry 140 and / or other memory circuitry, deployment in a tool via model deployer circuitry 150, output to one or more external systems 160, etc.

[0094] In some examples, external system 160 may utilize one or more deployed models from model deployer circuitry 150 to drive a tool to evaluate efficacy and / or toxicity associated with an immunotherapy for a patient and / or patient population, etc. Using one or more deployed models, a patient may be assessed at the beginning of an immunotherapy treatment plan, during administration of an immunotherapy treatment plan, as a candidate for an immunotherapy clinical trial, etc. In some examples, a patient's prior efficacy and / or toxicity model predictions may be used with the patient's updated efficacy and / or toxicity model predictions (e.g., combined between prior and current results, with prior results as input to generating current results, etc.).

[0095] Figure 2-14 is a representation of an operation that can be stored in memory circuitry and executed by processor circuitry to implement and actuate Figure 1 An example model is a flowchart of an example process of computer readable instructions of the device 100. Figure 2 The example process 200 of begins at block 210, where the input processor circuit system 110 processes data to form (one or more) inputs for the model trainer circuit system 120. For example, the input processor circuit system 110 may align the data relative to an anchor or reference point (e.g., a date, event, milestone or other marker, etc.). The input processor circuit system 110 may form a time series from the data. Alternatively or additionally, for example, the input processor circuit system 110 may label and / or perform feature engineering on the data to form an input feature set to train the (one or more) models.

[0096] At box 220, the model is trained using the input. For example, multiple AI models are trained by the model trainer circuit system 120 using the input feature set and / or other inputs provided from the input processor circuit system 110 and associated data. For example, by identifying patterns and correlations in the feature-based input data, the model trainer circuit system 120 forms and weights nodes, layers, connections, etc. in the model (e.g., RF model, GB model, boosted decision tree model, etc.). For example, the model can be adjusted by iteratively adding and removing features to find a balance between precision and recall in the model.

[0097] At block 230, the trained model is validated by the model trainer circuitry 120. For example, additional inputs (e.g., features, resampled input data, etc.) are used to validate (e.g., test, confirm, etc.) that the trained model works as expected with a threshold level of precision, accuracy, etc.

[0098] At box 240, the model comparator circuit system 130 evaluates the validated models to select one or more models in the model for deployment. For example, the model comparator circuit system 130 can compare and / or otherwise evaluate multiple trained, validated (e.g., tested, etc.) models according to one or more model selection criteria to evaluate the likelihood of predictions generated by each model, the suitability of each model for a specific purpose (e.g., prediction of immunotherapy treatment efficacy, prediction of immunotherapy toxicity, immunotherapy clinical trial selection, immunotherapy treatment plan adjustment, etc.). One or more models can be selected by the model comparator circuit system 130 using such an assessment / evaluation. For example, the model can be compared and / or otherwise evaluated based on the average output value, the standard deviation of the model output, etc. In some examples, the model comparator circuit system 130 selects a model for use / deployment. In other examples, the model comparator circuit system 130 selects multiple models (e.g., toxicity prediction models and efficacy prediction models, etc.) for use / deployment.

[0099] At block 250, the one or more selected models may be stored by the model storage circuitry 140. Thus, the selected model(s) may be saved for later use, deployment, etc. In some examples, the unselected model(s) may also be stored by the model storage circuitry 140, as the unselected model(s) may be suitable for other use / deployment in response to a subsequent request.

[0100] At block 260, one or more selected models are deployed by the model deployer circuitry 150 to an external system 160. The model(s) may be deployed as part of a tool for immunotherapy treatment and / or clinical trial planning to provide toxicity predictions, efficacy predictions, and the like.

[0101] Figure 3 An example implementation of input processing for model development is illustrated (e.g., Figure 2 At box 310, the input data for each of the plurality of patients is arranged in one or more time series (e.g., placed in chronological order to show evolution, progression, correlation, and / or other patterns). At box 320, the time series data for each of the plurality of patients is aligned relative to an anchor point or other reference event. For example, the various sets of time series data may be aligned according to birth, onset of illness, events such as hospitalization, and the like. For example, by aligning the time series, the data in the time series may be more meaningfully evaluated, compared, and / or otherwise analyzed.

[0102] At block 330, the aligned time series data is labeled. For example, a target label and associated rating may be generated for each set of data values ​​(eg, blood test values, liver function values, lung function values, etc.).

[0103] At block 340, feature engineering is performed on the data to prepare the data into a set of features with labels for training, testing, and / or otherwise validating a model. For example, feature engineering can regularize, transform, and extract the data into a set of patient features for application to a model. In some examples, the features form a set of candidate features that are evaluated to select a subset of patient features for model training, validation, etc.

[0104] Figure 4 illustrates an example implementation of model selection for deployment (e.g., Figure 2 At box 410, the goal or purpose is checked. For example, the request for the AI ​​model may be for clinical trial evaluation, prediction of immunotherapy efficacy, prediction of toxicity and / or other side effects associated with immunotherapy, etc. Such a request may drive the selection of models with certain characteristics, such as high Fl score, high recall, etc. Based on the requested goal or purpose, a subset of available models may be selected. In other words, based on the requested goal or purpose, a subset of available models may be excluded, leaving a subset of models that may be used for further evaluation and selection.

[0105] In addition, a goal or objective can be associated with a risk or tolerance. For example, a clinical trial may tolerate low efficacy, but is also interested in low toxicity. As another example, a clinical trial may focus only on the toxicity risk associated with a given immunotherapy as a gatekeeper for including or excluding patients from the clinical trial. Alternatively, for example, selection criteria for treating patients with a given immunotherapy may balance toxicity risk and possible efficacy to determine whether a patient should receive (or continue to receive) treatment.

[0106] At box 420, a test is performed using each remaining model. For example, a permutation test, a binomial test, and / or other comparison operations are performed using each model (e.g., evaluating a lung function model with respect to smoking, non-smoking, other binomial and / or permutation tests, etc.). At box 430, the output of each model for which the test was performed is evaluated based on one or more criteria. For example, each model can be evaluated based on an average, a standard, and / or other comparison criteria to determine which model best meets one or more criteria.

[0107] At box 440, one or more models are selected based on a comparison of the output and / or other model performance. For example, a single model can be selected for deployment based on an assessment of how the model performs and / or outputs relative to one or more criteria. In some examples, multiple models can be selected based on one or more criteria and / or other factors. For example, multiple models can meet (one or more) criteria. As another example, a request can include a request for therapeutic efficacy and the possibility of associated toxicity. In response to such a request, both an efficacy model and a toxicity model can be selected. In some examples, a single model can be trained for both toxicity and efficacy to output a combined prediction.

[0108] Figure 5 An example sequential procedure 500 for model building and evaluation (eg, by developing a feature set, etc.) is illustrated. Figure 5 As shown in the example of , at 1, the initial data set 510 is divided into at least two partitions. Separate data analysis is performed on the first part 520 of the data and the second part 525 of the data. The first data partition 520 is divided into at least two loops, such as an inner loop 522 and an outer loop 524 for data analysis. At 2, a first test such as a binomial test evaluates the data values ​​in the inner loop 522 and the outer loop 524 (e.g., if the data value Ho includes "smoking", then reject, etc.) to train the model. At 3, a permutation test is used instead of using a separate loop to process a separate "non-contact" test data set 525. For example, the data set 525 can be processed within multiple permutations with criteria such as smoking and non-smoking to identify that the possibility of smoking is greater than the possibility of non-smoking. At 4, the model is evaluated to report the average, standard and / or other evaluation / performance statistics about the first and second data partitions 520, 525 and the associated tests.

[0109] In some examples, one or more of loops 522, 524 are used to evaluate the first data partition 520 via exploratory data analysis, which has a K-fold cross validation (CV) comparing the first feature set M1 and the second feature set M2. Through sequential iterations, if the K-fold CV performance of M1 is less than M2, feature F is added to the first feature set M1. Once the features have been evaluated accordingly, the second data partition 525 is used to test the resulting model performance. If both loops 522, 524 are utilized, M1 and M2 can be compared based on the results of the inner loop 522 (e.g., a binomial test). When M2 is selected, a permutation test is performed to assess whether M2 is sufficiently better than M1 on the validation data set 525. Then, for example, model performance can be evaluated on the outer loop 524. Therefore, the initial feature set M1 or the extended feature set M2 can be selected for model training.

[0110] Figure 6 An example process 600 for preparing input data for model training is illustrated. Using the example process 600, data from different observation databases can be transformed into a common format (e.g., a data model) and a common representation (e.g., common terms, vocabulary, coding schemes, etc.) for system analysis according to a library of analysis routines. The example process 600 accepts input of (one or more) concept tables 605 including concept identifiers, concept names, concept descriptions, etc. For example, the concepts in this example relate to liver function tests. The clinical table 615 is also input with database extracts of patient demographics, hospital visits, laboratory measurements, etc. At box 610, patient liver function test values ​​(e.g., represented in patient blood test history) are extracted from electronic medical records, etc., and are cleaned, quality checked, and organized in a time series format for one or more patients. Such extraction and processing can also be applied to other patient information (e.g., lung function, drug efficacy, etc.).

[0111] At box 620, a label definition algorithm is executed to assign adverse event (AE) levels to blood test / liver function value sets (e.g., ALT, AST, TBILIRUBIN, ALKPHOS, etc.) and create binary target labels for the AI ​​model. Such label definitions can also be applied to other patient test values ​​(e.g., lung function, etc.).

[0112] At box 630, feature engineering is performed. For example, blood test values ​​and / or other values ​​can be regularized to an upper limit of "normal" to aid in feature generation, comparison, and other analysis. Such a range of "normal" can be determined for a particular patient based on specific blood work, other lab tests, etc. Feature engineering transforms the regularized values ​​into discrete symbolic representations, such as a modified symbol set approximation, etc. In some examples, motifs can be extracted from a series of symbolic representations as n-grams, and counts from patient history can be used as features.

[0113] At box 640, the AI ​​model can be trained and evaluated based on features, etc., so as to dynamically predict the risk of immune-related adverse events based on patient data. For example, the risk of immune-related adverse events can be predicted by a model from a fixed-length blood test history, etc. The output may include a model performance metric 625, a data set of AE risk prediction 635, etc. For example, the risk prediction data set 635 may include a risk prediction of immune-related hepatitis adverse events for each blood test value history. That is, for each patient, based on their blood test history, etc., the risk of developing AE hepatitis is predicted until the patient's next ICI treatment appointment. Based on the associated patient data training model, the data set 635 may alternatively include risk predictions for pneumonia, colitis, hospitalization, etc.

[0114] Figure 7 An example process 700 for building an example classification model is illustrated. At box 710, data from an input data set 705 is divided into multiple partitions. For example, structured EHR-derived tabular data, unstructured data tables, measurement features aggregated over time (e.g., 30 days, 60 days, 1 year, etc.), managed labels (e.g., pneumonia labels, hepatitis labels, colitis labels, etc.), other features aggregated over time (e.g., 30 days, 60 days, 1 year, etc.) (e.g., condition, smoking status, etc.), etc. can be collected and partitioned. For example, 90% of the available data can be organized in a first partition, while the remaining 10% of the data can be organized in a second partition. Based on the typical, expected, or desired number of features in the model, a model size input 715 can help adjust the partitions, etc.

[0115] At block 720, a sequential forward selection procedure is performed to evaluate multiple models. The example procedure iterates (e.g., from an initial model to a series of adjusted models) to find patterns in the data to form potential predictive features. For example, the first partition can be evaluated to identify candidate features based on associations between labels and features. A model (e.g., an adjusted model) is selected at each iteration, and then evaluation continues.

[0116] At block 730, model performance is evaluated. For example, the model selected at the last iteration of the selection procedure 720 is cross-validated (such as in an outer loop of nested cross-validation using the input data set 705) to be finalized for deployment and use.

[0117] Figure 8 Provided Figure 7 700 of the example process 700. At block 810, a candidate feature F is identified in a first data partition. For example, the candidate feature F can be identified based on an association between an indicator label and an existing / identified feature. At block 820, a candidate feature F is identified for a first partition of the data having two models. The first partition of performs nested cross validation (CV). That is, the model Include Model features plus new candidate features F.

[0118] At block 830, the inner loop result is evaluated to test whether Is it better than For example, a binomial test is performed to evaluate based on the results of the inner loop. Is it better than If not, control returns to block 810 to identify another candidate feature in the first partition. Better than , then at block 840, a permutation test is performed to assess whether the second partition Is it better than For example, the p-value can be used to evaluate and The p-value represents the fraction of permutations of the candidate feature F, where The performance is not less than performance. Performed “significantly better” , (e.g., with a standard significance level defined as 0.05 or 0.1), then, at block 850, select Otherwise, control returns to block 810 to identify another candidate feature in the first partition.

[0119] At block 860, the model size is evaluated (e.g., by comparing the current feature set to the model size input 715, etc.). If the model size has been reached, the process may proceed to block 730. However, if the model size has not yet been met, control returns to block 810 to select new candidate features in the next iteration of the example process.

[0120] Fig. 9 Provided Figure 7The example implementation of robust performance evaluation 730 of the example process 700 is shown below. At block 730, nested cross validation (CV) is used to evaluate the final selected model. or performance. Fig. 9 As shown in the example of , the input data set 705 is used as an external test set to verify the performance of the selected model and its features. Thus, for example, the inner loop of the partitioned data can be used to evaluate various candidate models, while the outer loop of the test data can be used to verify the performance of the selected model.

[0121] Fig.10 An example process 1000 of preparing data from multiple sources for use in machine learning and / or other AI model training is illustrated. For example, database extraction and / or expert supervision can be used to generate data from / in an input data set 1005. Data can be collected from multiple structured and / or unstructured data sources to form the data set 1005.

[0122] At block 1010, one or more input data sources 1005 are preprocessed to prepare the input data (e.g., by cleaning and extracting features from the data, labeling features, etc.). The input data and resulting features may include smoking history, medication administration, (one or more) medical conditions, radiation treatment history, (one or more) laboratory measurements, anthropometric data, etc. For example, at block 1012, structured data (e.g., medication administration, medical conditions, laboratory measurements, clinical procedures, anthropometric data, etc.) from the input data set 1005 is filtered and cleaned in a common data format (e.g., a common data model such as the OMOP data model format, etc.). At block 1014, curated data (e.g., related to downstream tasks, smoking history, specific condition history, specific medical history, etc.) from the input data set 1005 is filtered and cleaned into a common data format.

[0123] The preprocessed data set forms the input to block 1020, where patient vectors are generated from the filtered, cleaned, aggregated data set. For example, feature filtering can be used to provide a model-agnostic set of aggregated patient vectors. Model-agnostic non-aggregate tables with temporal data structures can be used for feature filtering and vector generation. Patient vectors and / or non-aggregated tables can form part of an output data set 1025.

[0124] At block 1030, the interface filters and aggregates the data. For example, the data may be filtered and arranged around anchor points and / or other references for aggregation and forming a patient vector.

[0125] Fig.11An example implementation of generating a patient vector is provided (e.g., box 1020 of example process 1000). At box 1110, the processed (e.g., filtered, cleaned, etc.) data is evaluated to determine whether a feature in the data is to be used? When the feature is not to be used, at box 1120, the feature is omitted. When the feature is to be used, at box 1130, the available data (source) is evaluated to determine whether multiple sources (e.g., structured and / or managed data sources, etc.) are to be used. When multiple data sources are to be used, at box 1140, features from multiple data sources are combined and coordinated. At box 1150, an unrestricted patient vector is formed from an unaggregated table that is agnostic to the downstream model that retains the temporal structure. For example, the intermediate data set output 1105 may include one or more data sets separated by the input domain and retaining the temporal structure.

[0126] Fig.12 An example implementation of a filtering and aggregation interface is provided (e.g., block 1030 of the example process 1000). At 1210, the patient is evaluated to determine whether the patient is a patient for further evaluation and processing. For example, such a determination can be based at least in part on external patient eligibility criteria 1205 related to downstream tasks. For example, whether the patient meets certain health criteria, age criteria, condition criteria, likelihood of success, etc. When the patient is not eligible, at block 1220, the patient is omitted from further processing.

[0127] When the patient qualifies, at block 1230, an anchor is set for further evaluation. For example, the determination of the anchor can be based on downstream, task-related, external patient qualification criteria 1215. Anchors are patient timeline events that align multiple patients for comparison. For example, events in a patient's life and / or medical history, such as diagnosis, symptoms, birth, hospitalization, initial treatment (e.g., date of first immunotherapy treatment), etc., can be driven by the framework to align time series data for relational processing and comparison. For example, an anchor can be a configurable parameter associated with a downstream task 1205.

[0128] At block 1240, an aggregation period is set. For example, the framework may configure and / or otherwise set an aggregation period before a prediction point (e.g., an anchor point and / or other time point and / or patient's data). For example, the aggregation period may include an aggregation of patient data 200 days before the prediction time, and so on.

[0129] At block 1250, an aggregation period is set for the measurement. For example, the interface framework allows different aggregation periods to be constructed for laboratory measurements, other measurements, etc., since such measurements can change more dynamically.

[0130] At block 1260, one or more restricted aggregated patient vectors are formed as outputs of the interface framework. For example, the patient vectors may be customized for multiple downstream tasks. In some examples, for a given set of inputs, the patient vectors may be dynamically generated as downstream tasks change.

[0131] Fig.13 An example process 1300 for constructing a predictive model of efficacy associated with ICI is illustrated. Such a model can assess multiple efficacy surrogate endpoints to predict response to ICI therapy. Models can be created for individual cancer indications and / or combinations thereof.

[0132] At block 1310, input data from multiple sources is prepared for processing. For example, as described in more detail above, data can be cleaned and features extracted from sources such as structured data sets 1305, managed data sets 1315, etc. For example, raw, automatically exported EHR data can be converted into a common format with managed data to form a feature set.

[0133] At box 1320, a baseline true value prediction label is generated. The prediction label may include ICI treatment time (TOT), next treatment time (TNET) (e.g., after ICI cessation), overall survival (OS), etc. For example, the listed baseline true value endpoints can be derived from the input data 1325 and can be generated on a continuous scale, such as expressed in days past the anchor point. For example, the patient timeline can be aligned based on similarities in the ICI treatment process. For example, the default anchor point can be the first date of the start of ICI treatment. The generated baseline true value labels can be used as is and / or with a modified granularity (e.g., past weeks, months, years, etc.) for training regression, survival analysis-based models, etc. Discretization of the baseline true value can be performed for binary and / or multi-class classification (e.g., responders vs. non-responders, 5-year survival rate, etc.).

[0134] At box 1330, model building is performed on the generated feature matrix, where the benchmark truth values ​​are used as independent modules of the framework. In some examples, the efficacy endpoints can be modeled separately (e.g., individuals, etc.). For example, modeling can be performed without assumptions and / or using different machine learning algorithms. Alternatively or additionally, modeling can be completed on a continuous scale using survival analysis methods. In addition, for example, modeling can follow a multi-class classification method for all endpoints. Example model building and selection are further described above.

[0135] Fig.14 An example method 1400 for model selection and generation is illustrated. Fig.14In some examples, model generation is at least partially driven by the intent, purpose, or goal of the model (e.g., referred to herein as a model selection criterion). For example, the purpose can be to generate a model with a high (e.g., maximized) F1 harmonic score (e.g., the harmonic mean of recall and precision) such as a GB model, a model with a high (e.g., maximized) recall (e.g., almost no false negatives) such as an RF model, a model with high precision (e.g., very few false positives), and so on. In some examples, majority class undersampling can be combined with time series data aggregation to obtain a well-balanced and static data set to feed the model. In some examples, the model creates probability estimates for labels, not just the discrete labels themselves.

[0136] At box 1410, input is prepared, such as by extracting data from one or more EHR data tables 1405. Measurements such as liver biomarker concentrations in plasma, other blood concentration values, etc. can be extracted. For example, time series measurements and / or other data can be put into a single complex data structure (e.g., an aggregation). The aggregated data can be processed using feature engineering to describe the time series data in terms of mean, standard deviation, minimum, maximum, etc. For example, lags and features can be created as well as labels. In some examples, dates of ICI treatment 1415 are generated (e.g., as reference points or anchors) to help align and anchor the time series data.

[0137] At block 1420, the goal or objective of the model(s) is evaluated. As described above, the objective of maximizing Fl results in a different training model (e.g., a gradient boosting model, such as a gradient boosting decision tree, etc.) than the objective of maximizing recall (e.g., a random forest model, etc.). Based on this determination, at block 1430, data set resampling occurs to balance the unbalanced data set from block 1410 for the purpose of maximizing recall. For example, when the objective is to maximize recall, random majority class undersampling is performed on the data set. When the objective is to maximize the Fl score, resampling may be omitted.

[0138] At blocks 1440 and / or 1450, the models are trained. For example, at block 1440, the RF model is trained for recall maximization, and at block 1450, the GB model is trained for F1 score maximization. For example, the respective models are validated, such as using leave-one-out cross-validation, where an individual predicts each sample, where the remaining samples are used as a training data set. For example, the model is trained on a historical data set from a plurality of health care records.

[0139] Thus, certain examples generate models that can be deployed, loaded, and utilized to generate predictions to drive patient treatment, adjust patient treatment, configure groups for clinical trials, and the like. As described above, nested cross validation can be used with multiple loops to train and select models. For example, the inner loop can be used for hyperparameter tuning, such as starting with all data points in a patient's history and reducing using feature elimination, or using a baseline feature set (adding features to the baseline feature set and testing the features using statistical analysis) to determine whether a new model performs better than an old model. In the latter case, when the added features improve the performance of the model, that updated model becomes the new baseline, and iterations can continue through the available features to reach a preferred model for a given feature set. While using all available data points from a patient's record can generate tens of thousands of inputs, a baseline evaluation can generate 10-12 beneficial feature inputs for effective model training before seeing diminishing returns from the added inputs.

[0140] Alternatively or additionally, model types can be evaluated. For example, a multiple linear regression model can be generated and compared to a more complex model (such as a random forest model) to determine whether the balance of precision and recall (and / or metrics) in one model is better than the other. One or more models can then be selected for use in a particular scenario (e.g., for clinical trials or patient treatment, for certain toxicities or for efficacy-related measurements, etc.).

[0141] Certain examples generate and deploy specific models to generate specific output predictions based on a set of patient health care data from one or more patient records. Because the model is trained and tested for specific predictions, the model is not applicable to general predictions. For example, the model cannot successfully predict any toxicity (e.g., including any of pneumonia, colitis, hepatitis, etc.). Instead, such a model generates random chance. However, a model designed to predict colitis from immunotherapy, designed to predict pneumonia from immunotherapy, or designed to predict hepatitis from immunotherapy, etc., can generate reliable, actionable predictions to drive group configuration for clinical trials, establish and / or change patient treatment plans, etc., as described above.

[0142] Example model deployment and use systems and associated methods

[0143] In some examples, a clinical system can utilize one or more deployed models to drive tools to evaluate efficacy and / or toxicity associated with immunotherapy for patients and / or patient populations, etc. Using one or more deployed models, patients can be assessed at the beginning of an immunotherapy treatment plan, during administration of an immunotherapy treatment plan, as candidates for immunotherapy clinical trials, etc. In some examples, previous efficacy and / or toxicity model predictions for a patient can be used with updated efficacy and / or toxicity model predictions for the patient (e.g., combining between previous and current results, using previous as input to produce current results, etc.). Thus, a model is trained and tested using a collection of historical health care data across a variety of patients, and then applied to a given patient to evaluate outcome(s) and treatment(s) for that patient.

[0144] Fig.15 An example immunotherapy prediction device 1500 is illustrated, including an example input processor circuit system 1510, an example memory circuit system 1520, an example model processor circuit system 1530, an example output generator circuit system 1540, and an example interface 1550. Fig.15 As shown in the example of , the example input processor circuit system 1510 processes input from a model source 1560 (e.g., a model generator device, a model library, an EHR, an EMR, etc.) and a data source 1565 (e.g., an EHR, an EMR, a laboratory system, a clinical information system, a scheduling system, etc.).

[0145] Inputs may come at different times, e.g., from model source 1560 and data source 1565. For example, one or more models may be periodically obtained from model source 1560 (e.g., via push and / or pull, etc.) and stored in model storage 1522 of memory circuitry 1520. For example, patient data and / or other input data may be periodically obtained from data source 1565, according to a schedule (e.g., according to a scheduled exam or other appointment, etc.), as requested by a user (e.g., clinician, administrator, triggered by an exam record, etc.), as needed, to be stored in data storage 1540 of memory circuitry 1520 for provision to one or more models by model processor circuitry 1530. Input data may include structured data (e.g., admission information, discharge information, prescription information, billing codes, diagnosis codes, labs, etc.) and unstructured manually curated medical record data, images, etc.

[0146] The model(s) stored in the model repository (store) 1522 have been trained and validated by another system, such as the model generation device 100. For example, based on input patient and / or other clinical data, one or more models may be selected by the model processor circuit system 1530 and / or by the input processor circuit system 1510 for use in the example device 1500. Models may be selected according to various model selection criteria. For example, a model may be selected to predict the likelihood of toxicity, such as pneumonia, hepatitis, colitis, etc., occurring due to immunotherapy according to a patient's treatment plan. Alternatively or additionally, a model may be selected to predict the efficacy of a patient's immunotherapy. For example, a model may be selected to preliminarily determine the efficacy of a patient's immunotherapy treatment plan. The same and / or different models may be selected to determine the continued efficacy of a patient's immunotherapy treatment plan. For example, a model may be selected to assess a patient's suitability for an immunotherapy clinical trial.

[0147] For example, the model may include a toxicity-related prediction model (e.g., a hepatitis prediction model, a colitis prediction model, a pneumonia prediction model, etc.), an efficacy-related prediction model (e.g., an immunotherapy efficacy model, a survival model, a treatment time model, etc.), and the like. The model may be a high recall with low precision, a low recall with high precision, a maximized harmonic mean (F1), a majority undersampling, and the like. The selection may be made by and / or for the example model processor circuit system 1530 (e.g., based on settings, modes, types, queries, patient identifiers, other model selection criteria, and the like). For example, when configuring a drug trial, the focus is on maximizing the F1 score to identify patients with the greatest chance of responding well to a given immunotherapy drug. However, when determining the clinical treatment of a patient, the goal is to eliminate as much toxicity as possible for the patient. Therefore, a developed model with a high recall is more important, while a low precision is acceptable because treatment options for hepatitis, colitis, pneumonia, and the like are cheap. For example, the goal may also depend on the patient's preferences (e.g., expectations for active treatment, expectations for quality of life, adjustments based on life events, and the like). Such models can also be evaluated based on specific thresholds or other patient selection criteria.

[0148] Model processor circuitry 1530 processes input data (e.g., from input processor circuitry 1510, data storage 1522, etc.) using one or more selected models (e.g., from input processor circuitry 1510, model storage 1522, etc.). Model processor circuitry 1530 generates one or more predicted outputs based on one or more provided inputs. The outputs and / or other content may be processed by output generator circuitry 1540 for output to an external device or system 1570 via interface 1550. For example, output generator circuitry 1540 may combine predictions of toxicity and / or efficacy together and / or further with images, interpretations, treatment plan information, patient data, clinical trial information, etc.

[0149] In operation, the example input processor circuit system 1510 processes input patient data related to one or more patients, such as laboratory results, diagnosis codes, billing codes, etc. The input can originate from one or more external systems 1565, such as EHR, EMR, etc. For example, the example input processor circuit system 1510 can extract and organize the patient's input in a time series. In some embodiments, the input processor circuit system 1510 aligns the input data relative to an anchor point (such as the date of the first disease symptom, the date of diagnosis, the date of the first immunotherapy treatment, etc.) to organize the input data in a time series.

[0150] In some examples, the time series data is formed into multiple features by the input data processor 1510. These features can form a patient feature set that identifies and / or classifies health care data input to one or more models using the model processor circuit system 1530. For example, the feature engineering of the input data processor 1510 can form multiple features based on codes (e.g., ICD-10 codes, etc.). For example, the ICD-10 codes of patients in a given year can be processed to identify codes related to lung and / or respiratory function (C34, C78, ​​etc.), and the time series of those codes can form a function for calculating the relative likelihood of lung disease in the patient. Similarly, multiple features can be formed to predict the generation of pneumonia, hepatitis, colitis, and / or other toxicities from immunotherapy treatment. Features can be formed based on lung function, liver biomarkers, blood work (e.g., concentrations in blood, plasma, etc.), medications taken, etc.

[0151] Patient features and / or other patient inputs are applied to one or more selected models by the model processor circuitry 1530. In some examples, a single feature input set or string is provided to the model (e.g., a feature set of the patient's ICD-10 code, etc.). In other examples, multiple inputs including multiple features, previous model output predictions (e.g., previous predictions of efficacy and / or toxicity input to a model for updating predictions), different model output predictions (e.g., providing efficacy model prediction output as input to a toxicity model, providing toxicity model prediction output as input to an efficacy model, etc.), etc. are applied to one or more models. The model processor circuitry 1530 generates outputs from the model(s) based on the inputs.

[0152] The output of the model(s) from the model processor circuitry 1530 is provided to the output generator circuitry 1540, which processes the predictions and / or other outputs of the model(s). For example, the outputs of multiple models from the model processor circuitry 1530 may be compared by the output generator circuitry 1540 to form a resulting output that is provided to the interface 1550, another system, etc. The output generator circuitry 1540 may post-process the output to validate the output, compare the current output to previous and / or other current output predictions, provide feedback to the model source 1560, reformat the output, etc. In some examples, the output generator circuitry 1540 may correlate the model outputs with other data, such as image data, to produce qualified or refined outputs and / or other related / validated results.

[0153] In some examples, the output of the model(s) is interpretable, such as by providing an indication of the input feature(s), rule(s), model layers, etc. that led to the output prediction. For example, the output generator circuitry 1540 can utilize explanations accompanying the outputs to drive decision making and actionable outputs related to treatment plans, clinical trials, and / or other next steps for a patient.

[0154] In some examples, patient selection criteria such as thresholds can be applied to model outputs. For example, a model output can represent the probability of a certain outcome based on the data used to generate the model and the associated features and correlations (e.g., the probability or likelihood of a certain toxicity during immunotherapy treatment, the probability of treatment success or efficacy, etc.). Patient selection thresholds can be used to set points in the continuum of model outputs at which triggering or determination of an action or event occurs. That is, if the predicted model output value falls on one side of the threshold, an action occurs (e.g., the patient is included in immunotherapy, etc.), but if the model predicts that the output falls on the other side of the threshold, another action occurs (e.g., the patient is excluded from treatment). Thus, for example, a model can provide a binary prediction (e.g., an input or an output). Alternatively, for example, a model can provide multiple outcome predictions (e.g., no, yes, yes with warnings, etc.).

[0155] For example, the output generator circuit system 1540 can incorporate the output prediction of efficacy and / or toxicity into the patient's immunotherapy treatment plan. Alternatively or additionally, for example, the output generator circuit system 1540 can be used as a trigger to include or exclude a patient from a clinical trial or study based on the output. For example, a clinical trial can be considered to be a treatment plan involving immunotherapy for a group of multiple patients. The output generator circuit system 1540 can use the output as a trigger to modify the patient's existing immunotherapy treatment plan (e.g., continue, stop, increase, reduce, etc., administer (one or more) immunotherapy drugs to the patient, etc.) based on increased toxicity probability, reduced toxicity probability, increased efficacy, reduced efficacy, etc. In some examples, the prediction drives the modification of the treatment plan to address the increased possibility of toxicity, such as prescribing steroids to treat pneumonia in the patient while continuing the immunotherapy treatment process, pre-treating predicted hepatitis attacks based on the determined possibility of liver toxicity, etc. In some examples, the output generator circuit system 1540 generates an alarm and / or otherwise provides decision support to achieve changes to treatment plans, clinical trials, etc. Current and prior predictions as well as old and new data points can drive treatment plans, adjustments, updated models, etc. in a dynamic, cyclical system.

[0156] For example, output generator circuitry 1540 may store the output in data storage 1524. Output generator circuitry 1540 provides output for transmission via interface 1550, such as graphically to external system 1570, as input / commands / settings for configuration of external system 1570 (e.g., to activate a treatment plan, adjust a treatment plan, form a patient group for a clinical trial, initiate a clinical trial, etc.), and / or other actionable output.

[0157] In some examples, the output generator circuit system 1540 triggers an indication (e.g., a visual indication on a display, a text indication in a patient record or report, an audible alarm) of the processing of the predicted output corresponding to the patient. The indication can be different based on whether the patient is recommended / accepted / selected (e.g., included in a clinical trial, selected for a new / ongoing treatment, etc.) versus rejected / removed (e.g., excluded from a clinical trial, rejected a new treatment, removed from a current treatment, etc.). The output can also include associated actions, such as including the patient in a group for a clinical trial, excluding the patient from a clinical trial, generating an order for the patient to start an immunotherapy treatment, cancelling an order for an ongoing immunotherapy treatment, connecting a clinician and / or patient to a scheduling and / or ordering system, updating a patient record, etc.

[0158] Thus, the example device 1500 is a digital tool that can be used to select patients for clinical trials, as well as to develop and deploy therapies and monitor treatment of patients. The example device 1500 enables toxicities potentially associated with immunotherapy (such as hepatitis, pneumonitis, colitis, etc.) to be evaluated through one or more models, while also evaluating the efficacy of immunotherapy for a particular patient (e.g., the likelihood of patient survival (with and / or without immunotherapy), progression-free survival, treatment time, etc.).

[0159] The example device 1500 can provide multiple predictions for a patient over time (e.g., periodically, at certain milestones, as the patient's condition and / or response to treatment develops, etc.). Comparison of multiple predictions made by the device 1500 enables the risk-to-benefit ratio of a patient to be assessed with respect to an immunotherapy treatment plan. For example, based on the ratio, when the benefits outweigh the risks, the treatment can be continued or increased, and when the risks outweigh the benefits, the treatment can be reduced or stopped. In certain examples, multiple model prediction outputs can be compared to identify trends, update models, drive initiation and / or changes to treatment plans, clinical trials, etc.

[0160] For example, an assessment of a cohort configured for a clinical trial may use only a toxicity-related model because the efficacy may not yet be known (or there may not be enough efficacy-related data available to make a reliable prediction) and / or because a pharmaceutical company may be focused on reducing toxicity without regard to predicted efficacy. An assessment of a patient's course of treatment may use a combination of toxicity-related and efficacy-related models, such as weighted or modified by one or more patient selection criteria or factors (e.g., patient acuity, disease severity, patient preferences, lifestyle, overall patient health or comorbidities, etc.), to determine whether the balance of toxicity risk and treatment efficacy supports or opposes a patient starting a treatment regimen, continuing a treatment regimen, joining a clinical trial cohort, etc.

[0161] Fig.16 and17 is a term that can be stored in a memory circuit and executed by a processor circuit system to implement and actuate Fig.15 A flow chart of an example process of computer readable instructions of an example immunotherapy prediction device 1500. Fig.16 The example process 1600 begins at block 1610, where a request for a prediction and / or other processing trigger is received by the example prediction device 1500. For example, a request for a predictive modeling output is received via interface 1550 (e.g., by user selection via a graphical user interface, by initiation of a software program, via input to processor circuitry 1510, in other cases from external system 1570, etc.). The request may include a toxicity prediction associated with an immunotherapy treatment plan (“immunotherapy treatment plan”), an efficacy prediction for the immunotherapy treatment plan, a likelihood of successful inclusion in an immunotherapy clinical trial, etc.

[0162] At block 1620, one or more models are loaded from model storage 1522 and / or external model source 1560, etc. for processing according to the request of model processor circuit system 1530. For example, RF, GB and / or other models can be loaded based on the predictions (e.g., toxicity, efficacy, eligibility, initial and ongoing, etc.) that are expected and / or otherwise triggered by the request. For example, model selection criteria such as model type, goal (e.g., indicators or goals such as inclusive trial, exclusive trial, simple treatment, active treatment, likely success, speculative success, low risk, high risk, etc.), etc., can be used to select (one or more) models for loading and use. At block 1630, patient and / or other health care data (e.g., from data storage 1524, external data source 1565, other patient health care records, etc.) to be input into the model are loaded for model processor circuit system 1530.

[0163] At box 1640, the health care data for a given patient is processed using the selected model(s). For example, the model processor circuit system 1530 inputs data to the selected model(s) that generates outputs. For example, based on codes and / or other patient data associated with lung function, liver function, blood work, etc., the model(s) determines the likelihood of hepatitis, pneumonitis, colitis, and / or other toxicities for the patient during immunotherapy treatment. Alternatively or additionally, the model(s) can process the inputs to determine the likelihood of immunotherapy efficacy, thereby driving the prescription of a treatment plan, adjustment of a treatment plan, selection of a clinical trial, etc.

[0164] At block 1650 , the outputs may be adjusted, such as via post-processing, comparison, appending model outputs, etc. For example, the outputs of one or more models from the model processor circuitry 1530 may be further processed by the model processor circuitry 1530 to apply another model, compare model outputs, scale / improve / otherwise select model outputs, etc., and / or by an output generator circuit to form an actionable output from the model(s) predictions.

[0165] In some examples, the output is compared to a threshold or other patient selection criteria to determine the associated next action. For example, the patient selection criteria (e.g., including a high threshold, including a medium threshold, including a low threshold, etc.) can be determined based on the desired relationship or balance between the precision and recall of the model. Therefore, the model output can be evaluated relative to the patient selection criteria, and different instructions and associated actions (e.g., as actionable results or outputs) can be generated based on the comparison. For example, if the model predicts that the output meets the patient selection criteria (e.g., exceeds the threshold), a first indication including the patient and an associated action (e.g., in a clinical trial, in treatment, continuing treatment, etc.) are generated. However, when the model predicts that the output does not meet the patient selection criteria (e.g., does not meet the threshold), a second indication excluding the patient and an associated action (e.g., from a clinical trial, from treatment, interrupting treatment, etc.) are generated.

[0166] Patient selection criteria may be provided in the form of values ​​that classify the output as greater than, less than, greater than or equal to, less than or equal to, etc. The patient selection criteria may establish a lower boundary value or threshold, an upper boundary value or threshold, a range of acceptable or unacceptable values ​​(e.g., with a maximum threshold and a minimum threshold), etc. For example, the patient selection criteria may be determined based on an analysis of model characteristics, such as precision and recall plots associated with the model output. Thus, the characteristics of a particular model (considered alone or in conjunction with external / additional concerns of patient care, clinical trial development, etc.) may be processed to generate patient selection criteria (e.g., in the form of one or more individual thresholds, associated ranges, etc.) to trigger the next action based on where the model prediction for a given patient falls relative to the patient selection criteria.

[0167] At box 1660, an actionable result is provided. For example, the output generator circuit system 1540 generates a visual output for the interface 1550 based on the processed predictions of the (one or more) models from the model processor circuit system 1530. As another example, instructions and / or prescriptions for new immunotherapy treatment plans and / or modifications of existing immunotherapy treatment plans can be output by the output generator circuit system 1540 to the external system 1570 via the interface 1550. As another example, instructions and / or notifications / alerts / indications for including patients in immunotherapy clinical trials or removing patients from immunotherapy clinical trials can be output by the output generator circuit system 1540 to the external system 1570 via the interface 1550. For example, the display of instructions and instructions for inclusion / exclusion, etc. can form an order generated by the output generator circuit system 1540 and provided via the interface 1550 to drive actions relative to the patient / patient record at another system (e.g., scheduling, management, clinical care system, clinical trial database, etc.). For example, an output or indication may be driven by comparing the prediction to a threshold or other patient selection criteria.

[0168] Fig.17 Further details of an example implementation of processing input data using one or more models are illustrated (e.g., block 1640 of example process 1600). At block 1710, the pre-processed input is applied to one or more selected models to process the input at block 1720. For example, input patient data associated with one or more patients (such as laboratory results, diagnosis codes, billing codes, etc.) can be aligned in time series relative to an anchor point (e.g., a reference event or time point in a patient's health history and record) and applied to the model(s) of the model processor circuitry 1530.

[0169] For example, the model processor circuit system 1530 inputs data to a selected (one or more) model that generates an output. For example, based on codes and / or other patient data associated with lung function, liver function, blood work, etc., the (one or more) models determine the likelihood of hepatitis, pneumonitis, colitis, and / or other toxicities in the patient during immunotherapy treatment. Alternatively or additionally, the (one or more) models can process the inputs to determine the likelihood of immunotherapy efficacy, thereby driving the prescription of a treatment plan, the adjustment of a treatment plan, the selection of a clinical trial, etc.

[0170] In some examples, the time series data is formed into multiple features by the input data processor 1510. These features can form a patient feature set, which is input into one or more models using the model processor circuit system 1530. For example, the feature engineering of the input data processor 1510 can form multiple features based on codes (e.g., ICD-10 codes, etc.). For example, the ICD-10 codes of patients in a given year can be processed to identify codes related to lung and / or respiratory function (C34, C78, ​​etc.), and the time series of those codes can form a function for calculating the relative likelihood of lung disease in the patient. Similarly, multiple features can be formed to predict the generation of pneumonia, hepatitis, colitis and / or other toxicities from immunotherapy treatment. Features can be formed based on lung function, liver biomarkers, blood work (e.g., concentrations in blood, plasma, etc.), etc.

[0171] Patient features and / or other patient inputs are applied to one or more selected models by the model processor circuitry 1530. In some examples, a single feature input set or string is provided to the model (e.g., a feature set of the patient's ICD-10 code, etc.). In other examples, multiple inputs including multiple features, previous model output predictions (e.g., previous predictions of efficacy and / or toxicity input to a model for updating predictions), different model output predictions (e.g., providing efficacy model prediction output as input to a toxicity model, providing toxicity model prediction output as input to an efficacy model, etc.), etc. are applied to one or more models.

[0172] At box 1730, the selected (one or more) models are evaluated to determine whether the (one or more) models include multiple related models. For example, the model processor circuit system 1530 evaluates the selected (one or more) models to determine whether the (one or more) models include related models for immunotherapy efficacy and toxicity. The model processor circuit system 1530 can also evaluate the selected (one or more) models to determine whether the (one or more) models include the current model and the previous model or model output. At box 1740, if there are related models selected for processing, the model processor circuit system 1530 applies one or more outputs between the related models. For example, the model processor circuit system 1530 can apply previous predictions that are compared to the new model prediction output and / or as input to the new model. For example, the model processor circuit system 1530 can compare and / or otherwise process efficacy and toxicity predictions.

[0173] At block 1750, the output from the model processor circuitry 1530 is post-processed. For example, the model processor circuitry 1530 and / or the output generator circuitry 1540 processes the output(s) of the model prediction(s) to develop and / or otherwise adjust a treatment plan, develop instructions associated with a treatment plan for clinical care or a clinical trial, etc. For example, the prediction model output(s) may be associated with images and / or other data. The prediction model output(s) may be compared to a threshold and / or otherwise evaluated relative to other patient selection criteria(s) to convert the prediction into an action to be taken (e.g., automatically and / or by a clinician, etc.). An explanation and / or other actionable information / instructions may be associated with the prediction output(s) so that the output(s) may be actionable by another system, program, device, etc.

[0174] At box 1760, a processed, actionable prediction output is provided. For example, the output generator circuit system 1540 may incorporate the output prediction of efficacy and / or toxicity into the patient's immunotherapy treatment plan. Alternatively or additionally, for example, the output generator circuit system 1540 can be used as a trigger to include or exclude a patient from a clinical trial or study based on the output. The output generator circuit system 1540 can use the output as a trigger to modify the patient's existing immunotherapy treatment plan (e.g., continue, stop, increase, reduce, etc., administer (one or more) immunotherapy drugs to the patient, etc.), such as based on increased toxicity probability, reduced toxicity probability, increased efficacy, reduced efficacy, etc. In some examples, the prediction drives the modification of the treatment plan to address the increased possibility of toxicity, such as prescribing steroids to treat pneumonia in the patient while continuing the immunotherapy treatment process, pre-treating a predicted hepatitis attack based on a determined possibility of liver toxicity, etc. In some examples, the output generator circuit system 1540 provides decision making, decision support, and / or other notifications / alerts to affect changes in treatment plans, clinical trials, etc. Current and previous predictions as well as old and new data points can drive treatment plans, adjustments, updated models, etc. in a dynamic, cyclical system. For example, output generator circuitry 1540 provides an operable output to interface 1550 for display and / or other distribution to external system 1570 and / or other connected devices.

[0175] Thus, one or more models can be generated, deployed, and configured for predicting toxicity caused by immunotherapy treatment, efficacy of immunotherapy treatment, combinations thereof, and the like. In some examples, the models (e.g., toxicity-related models, efficacy-related models, and the like) are deployed as part of a program or platform that loads the model and provides health care data from a given patient's health care record to the model. The output of the model, such as evaluated relative to a single threshold, range, or other patient selection criteria, is used to configure a treatment plan for a given patient. For example, the treatment plan includes an immunotherapy treatment and can include a group of multiple patients that form a clinical trial.

[0176] Fig.18 Another example process 1800 for configuring a clinical trial or treatment plan associated with a patient is illustrated. At box 1810, a model is loaded based on a model selection criterion. For example, models such as toxicity-related models (e.g., pneumonia toxicity models, colitis toxicity models, hepatitis toxicity models, etc.), efficacy-related models (e.g., survival models, treatment time models, etc.), etc., can be loaded based on model selection criteria, such as permitted or included model objectives (e.g., error in including patients), excluded model objectives (e.g., error in excluding patients), etc. Therefore, depending on the purpose or goal, such as including as many patients as possible in the study, limiting the number of patients in the study, actively treating patients with immunotherapy, conservatively treating patients with immunotherapy, etc., a model trained to emphasize recall over precision, precision over recall, balanced precision and recall, etc. can be selected.

[0177] In some examples, the goals or motivations may change over time (e.g., treatment of a patient starts cautiously and becomes more aggressive, the patient's condition changes to warrant a treatment with less toxicity and / or greater efficacy, a pharmaceutical company does an initial screen and then wants to switch to more extensive confirmation, etc.) Thus, over time, model loading may be repeated with different model selection criteria.

[0178] At block 1820, patient selection criteria are determined. For example, the patient selection criteria can be based on or associated with the model selection criteria (e.g., both are restrictive, both are permissive, etc.). However, the patient selection criteria can also deviate from or otherwise differ from the model selection criteria. For example, while a model that predicts toxicity in patients treated with immunotherapy may be permissible for a clinical trial, the patient selection criteria can be adjusted to seek a different balance of true positives, true negatives, false positives, and false negatives to select patients.

[0179] Threshold or other patient selection criteria are used to define the balance between the precision and recall associated with the confidence that the prediction provided by the model is correct. Thresholds can be set based on one or more factors, such as the confidence that a person will not be poisoned and the confidence that another person will be poisoned. For example, a treatment plan can give priority to the correctness that someone will be poisoned, and another treatment plan can give priority to the correctness that someone will not experience toxicity. Similarly, a treatment plan can give priority to correctly predicting that someone will survive, and another treatment plan can give priority to correctly predicting that someone will not survive. For example, some treatment plans (whether for single patient treatment or for clinical trials) can give priority to a certain minimum precision, a certain minimum recall or a certain balance between precision and recall, to advance the action triggered by prediction.

[0180] In some examples, the patient selection criterion is a threshold value set at a numerical value between 0.0 and 1.0. The threshold can be used to determine the next action regarding the patient based on the predicted output of the model. For example, the patient selection criterion can be set to a threshold of 0.5. Then, the first patient with a toxicity prediction less than 0.5 can be marked as included in the clinical trial, but the second patient with a toxicity prediction greater than 0.5 can be marked as excluded from the clinical trial. As another example, the threshold of the patient selection criterion can be set to 0.3, so that the first patient with an efficacy-related prediction of 0.3 or higher is marked as treated with an immunotherapy drug, while the second patient with an efficacy-related prediction less than 0.3 is not included.

[0181] In some examples, the goals or motivations can change over time (e.g., treatment of a patient begins cautiously and becomes more aggressive, the patient's condition changes to warrant a treatment with less toxicity and / or greater efficacy, a pharmaceutical company performs an initial screening and then wants to switch to a more extensive confirmation, etc.). Thus, the patient's healthcare data can be re-evaluated with the same and / or different models, where the output is transformed over time according to different patient selection criteria and / or different model selection criteria. For example, a patient may not be very ill and is simply looking for an immunotherapy treatment with a low risk of toxicity. If the patient's condition worsens, the goal can shift to finding a treatment with a high likelihood of efficacy for the patient, regardless of toxicity. For example, a different model can be selected and the updated patient data processed with the updated model.

[0182] At box 1830, the patient health care data is processed using the selected model. For example, health care data from a patient's health care record (e.g., an EMR, EHR, other clinical or health care database, etc.) is provided to a selected model (e.g., a toxicity-related model, an efficacy-related model, etc.). The patient data (e.g., blood tests, prescription history, other bioassays and / or laboratory data, etc.) is processed by the model, which associates and weights features associated with the patient data to generate predictions (e.g., predicting the likelihood of toxicity from immunotherapy treatment, predicting the likely efficacy of immunotherapy treatment, etc.). As described above, the model is trained, tested, and deployed for specific predictions, such as the likelihood of a specific toxicity, the efficacy of a specific immunotherapy drug, etc., rather than, for example, providing a general indication of any toxicity that may result from taking the drug to a patient or the general efficacy of any drug in treating a specific type of cancer, etc. Therefore, the model provides a specific predictive output using data from a specific patient's medical record.

[0183] At box 1840, the model output is evaluated. For example, the model output can be a prediction on a scale from 0.0 to 1.0. For example, the model prediction output can be evaluated relative to the patient selection criteria. As described above, the patient selection criteria or thresholds can be set to allow low precision / high recall (low threshold, such as 0.0-0.3), a balance between precision and recall (intermediate threshold, such as 0.4-0.6), high precision / low recall (high threshold, such as 0.7-1.0), etc. Such thresholds or other patient selection criteria can be selected based on tolerance or priority for correctly predicting positive results, incorrectly predicting positive results, correctly predicting negative results, incorrectly predicting negative results, etc.

[0184] Based on the evaluation of the model output according to the patient selection criteria, the next action is triggered. When the model output of the patient meets the patient selection criteria, then at box 1850, a first indication is generated to include the patient in a clinical trial, treatment plan, etc., and an associated action is triggered (e.g., an order for prescribing a prescription for the patient is generated, the patient is included in a trial or research group, the patient's treatment plan is modified, etc.). When the model output of the patient does not meet the patient selection criteria, then at box 1860, a second indication is generated to exclude the patient from the clinical trial, treatment plan, etc., and an associated action is triggered (e.g., a note in the patient record that the patient is excluded from the trial or treatment, the patient's treatment plan is modified, etc.). At box 1870, the indication and the associated action are output. For example, the output can be displayed on a screen, a message is delivered to the responsible clinician, a message is delivered to the patient, generated in an order for an immunotherapy drug or other therapy, routed to a scheduling system for an appointment, used to update the patient's medical record or clinical trial group, etc.

[0185] In some examples, the decision may not be a binary decision of treat / not treat. Instead, the decision may be to decide to treat the patient according to the immunotherapy for a certain period of time and / or to perform additional testing, monitoring, etc. to help ensure that the patient does not begin to experience side effects from the treatment (and / or responds well to the treatment). Thus, previous decisions / outputs can be fed back into the model to be periodically re-evaluated based on updated patient information, etc.

[0186] As described above, the balance between precision and recall can be determined for a single model and / or for a set of related models. In order to make the model output (e.g., prediction) operable, a patient selection criterion (e.g., a threshold or other selection criterion) is set, a first action is taken above the criterion, and a second action is taken below the criterion. A low threshold or selection criterion provides low precision (e.g., many false positives) in the model, but provides a high recall rate (e.g., almost no false negatives) in the model. An intermediate threshold or other selection criterion provides a good balance between false positives and false negatives. The option of the intermediate threshold depends on the characteristics and performance of a specific model, so that it can vary from model to model, rather than being uniform. A high threshold or other selection criterion provides high precision (e.g., very few false positives) but low recall rate (e.g., many false negatives) in the model.

[0187] For example, the predictive output of a single toxicity-related model can be evaluated based on a patient selection criterion / threshold set at a low threshold to accept more false positives (e.g., more toxicity predictions than actually occur) in the interest of enrolling more people in a clinical trial. As another example, an efficacy-related model such as an overall survival model can be configured or set according to a patient selection criterion set at a high threshold to allow for few false positives.

[0188] Thus, as explained and described above, for example, in Fig.19 In the representation 1900, a large amount of aggregated patient-related health care data can be processed according to a subset of features associated with the data 1910. When the first immunotherapy checkpoint inhibitor (CPI) is administered at 1920, the date 1920 can be used as an anchor to align the aggregated patient-related data sets for model training and testing to form multiple models 1930M1, M2, M3, etc. One of the models M1, M2, Me can be selected based on model selection criteria, such as emphasizing precision (e.g., how many retrieved items are relevant), emphasizing recall (e.g., how many relevant items are retrieved), a balance between precision and recall, etc. and deployed for applications, platforms, services, systems, etc.

[0189] Example model features and associated outputs

[0190] As described above, models are built to answer specific questions, such as what is the patient's risk for a particular toxicity, what is the likelihood that this immunotherapy will be effective for this patient. Such questions, also called metrics or goals, allow the available data to be organized and processed to form connections and correlations to predict potential (e.g., possible) answers to these questions. The model takes data, including outcomes observed in a pool of patient data (e.g., this person survived the treatment, this person suffered colitis while on this treatment, etc.) and forms a network of connections, weights, and other correlations to generate a probabilistic output. Thus, the model can transform patient information into probabilities based on the collection of other data and outcomes. For example, such probabilities can be expressed in terms of precision and recall.

[0191] In some examples, each model can be characterized by a relationship or combination of precision and recall associated with its output. Based on the characteristics of the model, as illustrated by the precision and recall of the associated model output, a threshold or other patient selection criteria can be set to process or interpret the output of the model. For example, the model characteristics or behavior as reflected in the precision and recall curve associated with a particular model can be evaluated to set patient selection criteria. If the goal or purpose is to reduce or minimize false positives, the threshold can be set accordingly based on the evaluation of the precision curve associated with the model. If the goal or purpose is to reduce or minimize false negatives, the threshold can be set accordingly based on the evaluation of the recall curve associated with the model. In many instances, the balance of false positives and false negatives results in an analysis of both the precision curve and the recall curve to set the patient selection criteria threshold for the use of the model.

[0192] Setting the threshold as a patient selection criterion indicates that some balance (or dichotomy) between precision and recall is satisfactory for taking action based on a particular model. For example, a clinician or researcher may be satisfied with a certain percentage of error or inaccuracy in the predictions. Alternatively or additionally, a clinician or researcher may require a certain reliability in the predictions. Such risk tolerance may depend on the patient, immunotherapy, health issues, liability issues, etc.

[0193] Figure 20-24An example precision / recall graph related to the reliability of the model output is illustrated. As shown in the following example, the model characteristics are reflected in the graph, and different thresholds can be set as patient selection criteria based on the specific behavior or characteristics of the model. For example, data including patient results or consequences are used to generate or develop a model (e.g., train and test the model so that the model can then be deployed for use). Therefore, the results are used to generate the model, and then the model is used to generate probabilities associated with a specific indicator, focus or problem. Thresholds are determined based on one or more focuses, requirements, boundaries, preferences, etc. to specify a certain combination of precision (e.g., positive predictive value) and recall (e.g., true positive rate or sensitivity) of the model.

[0194] By specifying a certain threshold, the model's positive prediction percentage and its true positive rate are acceptable for the purpose of the model in this situation. The threshold can be based on the situation (e.g., individual patient treatment, group treatment in clinical trials, patient life stage, disease severity, patient preferences, etc.) and / or change over time (e.g., as patients change, diseases / symptoms change, preferences change, etc.). Therefore, based on the threshold selected for the output of this particular patient in this particular situation, the model can have applicability to various situations. As the patient and / or the situation of a given patient changes over time, the model can also change.

[0195] In certain examples, a model may be generated and set to a certain state or mode (e.g., "clinical trial mode," where the model identifies patients who are appropriate for this clinical trial, and "patient treatment mode," where the model determines whether this particular treatment will likely benefit this particular patient, etc.) Thus, a model may be loaded and set for a particular intent, and that intent may be further refined by setting specific patient selection criteria / thresholds.

[0196] Fig. 20 An example prediction output of an overall survival model is illustrated. The x-axis 2010 of the chart corresponds to a threshold value (e.g., a patient selection criterion), while the y-axis 2020 of the chart corresponds to a probability of survival (e.g., defined by a precision curve 2030 or a recall curve 2040). The threshold value 2010 selected for the patient selection criterion sets a point, e.g., at which, when below the threshold, one action is taken, and when above the threshold, another action is taken.

[0197] exist Fig. 20In the example of , the patient results are reflected in the model. The precision curve 2030 and the recall curve 2040 indicate the possibility of true positive, false positive or false negative results. For example, the survival rate of the patient is a data point classified as a negative result. The patient who did not survive is a data point classified as a positive result. The associated probability is indicated on the y-axis 2020. For example, from 0.0 to 1.0 (indicating a 0% possibility to a 100% possibility), a probability greater than 0.5 indicates that the patient is more likely to not survive than to survive. The probability less than 0.5 indicates that the patient is more likely to survive than not survive. The probability of 0.5 indicates that there is a 50% possibility for any result. The precision 2030 and the recall curve 2040 indicate that the sampling of these data points may be true positive, false positive or false negative. The precision curve 2030 and the recall curve 2040 drive the determination of the corresponding patient selection criterion threshold, by which the model output is made operational.

[0198] For example, Fig. 20 As shown in , if a low threshold is chosen (e.g., 0.0 to 0.3), the model results have low precision (e.g., many false positives) and high recall (e.g., almost no false negatives). Therefore, with a low threshold, there is a lower probability of incorrectly identifying someone as alive, but there is a higher probability of incorrectly identifying some people as dead. As a result, almost all patients predicted to survive after immunotherapy treatment will survive. However, the model incorrectly predicts many deaths that did not actually occur. In the drug clinical trial use case, the model will screen out or exclude too many patients from the trial due to the exaggerated number of deaths. In the clinical practice use case of patient treatment, the model will potentially prevent too many patients from receiving immunotherapy due to the exaggerated probability of death.

[0199] For example, if in Fig. 20 In the example of , if a threshold of 0.1 is chosen, no false survivals are predicted (no false negatives) because the recall 2040 is 1.0. However, because the precision 2030 is approximately 0.55, there is a 50% chance that the predicted death is incorrect (false positive).

[0200] If an intermediate threshold is chosen (e.g., 0.4 to 0.6), there is more of a balance between false positives and false negatives. In the pharmaceutical use case, the model will screen out or exclude some patients who may have survived, and screen out or include some patients who will not survive. In a clinical practice context, the model will make some inappropriate predictions, but most patients who are recommended for immunotherapy will survive. In the case of applying this threshold to the model, most patients who are not recommended for treatment will not survive.

[0201] For example, if in Fig. 20In the example of , a threshold of 0.5 is chosen, then relatively few false survivals are predicted, because the recall 2040 is about 0.75. In addition, the model also provides relatively few incorrect predictions about lack of survival, because the precision is about 0.65. Therefore, a good balance of false positives and false negatives is provided at this threshold. Then, for example, patients with a survival probability of at least 0.5 will be selected for treatment.

[0202] If a high threshold is chosen (e.g., 0.7 to 1.0), the model output has high precision (e.g., very few false positives) and low recall (e.g., many false negatives). As a result, almost all patients predicted to die will actually die, and the model incorrectly predicts many survivals that did not actually occur. In a pharmaceutical context, the model will include too many patients who will not survive in clinical trials. In a clinical patient care context, the model will incorrectly prescribe immunotherapy too often.

[0203] For example, if in Fig. 20 In the example above, if we choose a threshold of 0.9, then almost all predicted survivals are false, because the recall 2040 is 0.0. However, all predicted deaths are correct, because the precision 2030 is 1.0.

[0204] use Fig. 20 Based on the precision 2030 and recall 2040 characteristics of the model represented in , a patient selection criteria threshold 2010 can be selected to drive clinical trial group selection, patient treatment determination, etc. A model trained on a broader population of prior health care data outputs a probability of survival for a given patient by processing the health care data for that particular patient to generate a predicted output representing the efficacy of that patient (using overall survivability as a surrogate or replacement for the efficacy of immunotherapy treatment). The model output for that patient is evaluated according to the determined threshold 2010 to determine whether the patient is an acceptable candidate for a clinical trial group, immunotherapy treatment regimen, etc. Based on the motivations and constraints of the clinical trial, the patient's personal circumstances and expectations, etc., the patient selection threshold 2010 can be adjusted to be more or less inclusive / exclusive to include or exclude specific patients from clinical trial groups, treatment plans, etc. As the threshold moves according to the precision 2030 and recall 2040 curves of the model, the associated risk of inaccuracy / accuracy is assumed.

[0205] like Fig. 20As shown in the example of , choosing an intermediate threshold of 0.55 means that the intersection of the precision curve 2030 and the recall curve 2040 is about 0.7. Such an intersection provides good precision and good recall, with less than one-third of false positives and less than one-third of false negatives, resulting in the model making some inaccurate predictions, but most patients who are recommended for immunotherapy will survive, and most patients who are not recommended for immunotherapy will not survive. Therefore, due to the characteristics of the model, the overall patient survival rate is improved.

[0206] To generate Fig. 20 The model represented in uses data and associated features. The features used to train and test this model include features representing a set of laboratory measurements over a period of time (e.g., 120 days, 100 days, 30 days, etc.) and a set of patient conditions, medications, etc. over a period of time (e.g., 1 year, 9 months, 6 months, etc.), which can be generated using one hot encoding and methods such as end value, median time-weighted average (twa), mean, median, etc. Features include lymph node percentage, hemoalbumin (e.g., end value, twa, median, etc.), dexamethasone drug count, count of chemotherapy-related procedures (e.g., number of chemotherapy performed during the aggregation period), red blood cell distribution width (RDW) (e.g., twa, end value, median, etc.), red blood cell count (RBC), hemoglobin (twa, median, end value, etc.), renal function, lung function, cancer stage, cancer recurrence, etc. As described above, the features are combined, weighted, and iterated to generate the model, which is reflected in Fig. 20 This model can be deployed and used to predict patient outcomes.

[0207] Fig.21 An example prediction output of a pneumotoxicity model is illustrated. The x-axis 2110 of the graph corresponds to a threshold value (e.g., a patient selection criterion), and the y-axis 2120 of the graph corresponds to the probability of producing pneumotoxicity (e.g., on a precision curve 2130 or a recall curve 2140). The threshold value 2110 selected for the patient selection criterion sets a point at which, below the threshold, one action is taken, and above the threshold, another action is taken.

[0208] exist Fig.21In the example of , a negative result indicates that the patient did not suffer from pneumotoxicity, while a positive result indicates that the patient did suffer from pneumotoxicity. If a low threshold is chosen (e.g., 0.0 to 0.25), the model results have low precision (e.g., many false positives) and high recall (e.g., almost no false negatives). Therefore, almost all patients who are predicted not to suffer from pneumotoxicity due to taking immunotherapy will not have pneumonia. However, the model incorrectly predicts pneumotoxicity in many patients that did not actually occur. In the drug clinical trial use case, the model will screen out or exclude too many patients from the trial due to exaggerated pneumotoxicity numbers. In the clinical practice use case of patient treatment, the model will potentially prevent immunotherapy from being given to too many patients due to exaggerated toxicity probabilities.

[0209] For example, if in Fig.21 In the example above, if we choose a threshold of 0.1, then almost all predictions of no toxicity are true, since recall 2140 is almost 1.0. However, most predictions of toxicity are incorrect, since precision 2130 is slightly above 0.1.

[0210] However, if an intermediate or high threshold is selected, both precision 2130 and recall 2140 are low, providing a large number of false positives and false negatives. As a result, a large number of patients will be excluded from clinical trials or treatments.

[0211] For example, if in Fig.21 In the example of , if a threshold of 0.3 is chosen, then almost all predictions of no toxicity are false because recall 2140 is 0.3. In addition, most predictions of toxicity are incorrect because precision 2130 is slightly above 0.2. Similarly, if a threshold of 0.45 is chosen, then all predictions of no toxicity and toxicity are false because both precision 2130 and recall 2140 are 0.0.

[0212] use Fig.21Based on the precision 2130 and recall 2140 characteristics of the model represented in, a patient selection criteria threshold 2110 can be selected to drive clinical trial cohort selection, patient treatment determination, etc. By processing the health care data of a particular patient to generate a predicted output representing pneumotoxicity for that patient, a model trained on a broader population of prior health care data outputs the probability that a given patient will develop pneumotoxicity from immunotherapy. The model output for that patient is evaluated according to the determined threshold 2110 to determine whether the patient is an acceptable candidate for a clinical trial cohort, immunotherapy treatment regimen, etc. Based on the motivations and constraints of the clinical trial, the patient's personal circumstances and expectations, etc., the patient selection threshold 2110 can be adjusted to be more or less inclusive / exclusive to include or exclude specific patients from clinical trials, treatment plans, etc., with the associated risk of inaccuracy / accuracy as the threshold moves according to the precision curve 2130 and recall curve 2140 of the model.

[0213] exist Fig.21 In the example of , the intersection of the precision curve 2130 and the recall curve 2140 occurs at about 0.34. However, at this intersection, both the precision 2130 and the recall 2140 are very poor, each less than 0.3, indicating more than 70% false positives and more than 70% false negatives. Therefore, the intersection is not conducive to either precision or recall. Fig.21 In the case of the pneumonia toxicity model, precision 2130 and recall 2140 are not well aligned, and precision 2130 is poor throughout the model. Therefore, a threshold 2110 may be selected based on, for example, a recall 2140 of 0.7 with precision 2130 around 0.18, as precision hovers around 0.2 until both precision 2130 and recall 2140 drop from 0.35 to 0.45.

[0214] Therefore, the desired predictability of the results must be balanced with the reality of the characteristics of the model based on the data and correlations used to generate the model. Fig.21 As shown in , there is not always a perfect balance between precision and recall as the basis for evaluating the model's predictive output. As described above, the characteristics of a model can change over time as available data changes, associated features change, feedback from old models is provided, etc.

[0215] To generate Fig.21The model represented in uses features and associated data. The features used to train and test this model include a collection of laboratory measurements over a period of time (e.g., 120 days, 100 days, 30 days, etc.) and a collection of patient conditions, medications, etc. over a period of time (e.g., 1 year, 9 months, 6 months, etc.), which can be generated using one hot encoding and methods such as end value, median time-weighted average (TWA), mean, median, etc. For example, the data inputs used to train the model can include all ICD-10 codes for each patient within one year before the first immunotherapy treatment, oxygen saturation values ​​measured by the laboratory within 120 days before the first treatment, the date of the first immunotherapy treatment, etc. Those data inputs are transformed into a set of features, including the frequency of lung-related ICD-10 codes (e.g., C34, C78, ​​R91, J, R05, R06, R07, R09, etc.), the frequency of ICD-10 code C34 (malignant neoplasms of the bronchus and lungs), the frequency of ICD-10 code C78 (secondary malignancies of the respiratory and digestive organs), the median oxygen saturation measured by blood labs within 120 days before the first ICI treatment, smoking, etc. The feature training model (e.g., as a binary classifier using random forests, etc.) generates a predicted probability of developing pneumotoxicity at a certain time after immunotherapy treatment. As described above, the features are combined, weighted, and iterated to generate the model, which is reflected in Fig.21 This model can be deployed and used to predict patient outcomes.

[0216] Fig. 22 An example prediction output of a colitis toxicity model is illustrated. The x-axis 2210 of the graph corresponds to a threshold value (e.g., a patient selection criterion), and the y-axis 2220 of the graph corresponds to the probability of colitis toxicity occurring (e.g., on a precision curve 2230 or a recall curve 2240). The threshold value 2210 selected for the patient selection criterion sets a point at which, below the threshold, one action is taken, and above the threshold, another action is taken.

[0217] exist Fig. 22In the example of , a negative result indicates that the patient did not suffer colitis toxicity, while a positive result indicates that the patient did suffer colitis toxicity. If a low threshold is chosen (e.g., 0.0 to 0.2), the model results have low precision (e.g., many false positives) and high recall (e.g., almost no false negatives). As a result, almost all patients who were predicted not to suffer colitis toxicity from receiving immunotherapy will not suffer from colitis. However, the model incorrectly predicts colitis toxicity in many patients that did not actually occur. In a drug clinical trial use case, the model would screen out or exclude too many patients from the trial due to the exaggerated colitis toxicity numbers. In a clinical practice use case of patient treatment, the model would potentially prevent too many patients from receiving immunotherapy due to the exaggerated probability of colitis toxicity.

[0218] For example, if in Fig. 22 In the example above, if we choose a threshold of 0.1, then almost all predictions of no toxicity are true, since the recall 2240 is 0.8. However, most predictions of toxicity are incorrect, since the precision 2230 is 0.2.

[0219] If an intermediate threshold is chosen (e.g., 0.2 to 0.33), there is more of a balance between false positives and false negatives. In the pharmaceutical use case, the model will screen out or exclude some patients who will not experience colitis toxicity, and will screen in or include some patients who will experience colitis toxicity.

[0220] For example, if in Fig. 22 In the example above, if we choose a threshold of 0.3, then less than half of the predictions of no toxicity are true, since the recall 2240 is 0.35. However, about half of the predicted toxicity are correct / incorrect, since the precision 2230 is about 0.5.

[0221] If a high threshold is chosen (e.g., 0.33 to 0.6), the model output has high precision (e.g., very few false positives) and low recall (e.g., many false negatives). As a result, almost all patients predicted to suffer colitis toxicity will truly have colitis, and the model fails to predict many of the colitis toxicities that actually do occur. In a pharmaceutical context, the model will include too many patients who will experience colitis toxicity in clinical trials. In a clinical patient care context, the model will incorrectly underestimate the occurrence of colitis toxicity.

[0222] For example, if in Fig. 22 In the example above, if we choose a threshold of 0.5, then none of the predictions of no toxicity are true because the recall 2240 is 0.0. However, all of the predictions of toxicity are true because the precision 2230 is 1.0.

[0223] use Fig. 22Based on the precision 2230 and recall 2240 characteristics of the model represented in, a patient selection criteria threshold 2210 can be selected to drive clinical trial cohort selection, patient treatment determination, etc. The model trained on a broader population of health care data outputs the probability of a given patient developing colitis toxicity from immunotherapy by processing the health care data of that particular patient to generate a predicted output representing colitis toxicity for that patient. The model output for that patient is evaluated according to the determined threshold 2210 to determine whether the patient is an acceptable candidate for a clinical trial cohort, immunotherapy regimen, etc. Based on the motivations and constraints of the clinical trial, the patient's personal circumstances and expectations, etc., the patient selection threshold 2210 can be adjusted to be more or less inclusive / exclusive to include or exclude specific patients from clinical trials, treatment plans, etc., with the associated risk of inaccuracy / accuracy as the threshold 2210 moves according to the precision curve 2230 and recall curve 2240 of the model.

[0224] like Fig. 22 As shown in the example of , selecting a threshold of 0.25 means that the intersection of the precision curve 2230 and the recall curve 2240 is about 0.4. Such an intersection provides a reasonable precision and a good recall rate. However, both the false positives and false negatives are still above 50%. Therefore, different thresholds 2210 can be selected to prioritize recall rate 2240 (e.g., 0.15 to reduce false negatives) or precision 2230 (e.g., 0.35 to reduce false positives).

[0225] To generate Fig. 22 The model represented in uses data and associated features. Features used to train and test this model include features representing a set of laboratory measurements over a period of time (e.g., 120 days, 100 days, 30 days, etc.) and a set of patient conditions, medications, etc. over a period of time (e.g., 1 year, 9 months, 6 months, etc.), which can be generated using one hot encoding and methods such as end value, median time weighted average (twa), average, median, etc. For example, features formed to train the model may include class of immunotherapy drugs, condition (e.g., one hot encoding, etc.), blood frequency of hemoglobin below normal, number of chemotherapy-related procedures within the time frame, number of lymphocytes below / above normal, number of unique immunotherapy drugs, albumin below / above normal, white blood cell count (WBC) below / above normal, RBC below / above normal, hemoglobin above / below normal, immunotherapy drug class, etc. The features train the model (e.g., as a binary classifier using random forests, etc.) to generate a predicted probability of colitis toxicity at a certain time after immunotherapy treatment. As described above, features are combined, weighted, and iterated to generate a model, which is reflected in Fig. 22This model can be deployed and used to predict patient outcomes.

[0226] Fig.23 An example prediction output of a hepatitis toxicity model is illustrated. The x-axis 2310 of the graph corresponds to a threshold value (e.g., a patient selection criterion), and the y-axis 2320 of the graph corresponds to the probability of producing hepatitis toxicity (e.g., on a precision curve 2330 or a recall curve 2340). The threshold value 2310 selected for the patient selection criterion sets a point at which, below the threshold, one action is taken, and above the threshold, another action is taken.

[0227] exist Fig.23 In the example of , a negative result indicates that the patient did not suffer from hepatitis toxicity, while a positive result indicates that the patient did suffer from hepatitis toxicity. If a low threshold is selected (e.g., 0.0 to 0.2), the model results have low precision (e.g., many false positives) and high recall (e.g., almost no false negatives). As a result, the model incorrectly predicts hepatitis toxicity in many patients that do not actually develop, but the few patients who are predicted not to suffer from hepatitis toxicity due to receiving immunotherapy will actually have hepatitis.

[0228] For example, if in Fig.23 In the example above, if we choose a threshold of 0.1, then all predictions of no toxicity are true because the recall 2340 is 1.0. However, most predictions of toxicity are incorrect because the precision 2330 is slightly less than 0.2.

[0229] If an intermediate threshold is chosen (e.g., 0.2 to 0.6), there is more of a balance between false positives and false negatives. In the pharmaceutical use case, the model will screen out or exclude some patients who will not experience hepatitis toxicity, and will screen in or include some patients who will experience hepatitis toxicity.

[0230] For example, if in Fig.23 In the example above, if we choose a threshold of 0.5, then most predictions of no toxicity are incorrect because the recall 2340 is 0.25. However, about half of the predictions of toxicity are correct because the precision 2330 is 0.55.

[0231] If a high threshold is chosen (e.g., 0.6 to 1.0), the model output has high precision (e.g., few false positives) and low recall (e.g., many false negatives). As a result, many patients predicted to suffer from hepatitis toxicity will actually have hepatitis, but patients predicted not to suffer from hepatitis toxicity will have hepatitis. In a pharmaceutical context, the model will include too many patients who will experience hepatitis toxicity in clinical trials. In a clinical patient care context, the model will incorrectly underestimate the occurrence of hepatitis toxicity.

[0232] For example, if in Fig.23 In the example above, if we choose a threshold of 0.9, then no predictions of no toxicity are true, because recall 2340 is 0.0. However, all of the predictions of toxicity are correct, because precision 2330 is 1.0.

[0233] use Fig.23 Based on the precision 2330 and recall 2340 characteristics of the model represented in, a patient selection criteria threshold 2310 can be selected to drive clinical trial group selection, patient treatment determination, etc. The model trained on a broader population of prior health care data outputs the probability of a given patient developing hepatitis toxicity from immunotherapy by processing the health care data of that particular patient to generate a predicted output representing the patient's hepatitis toxicity. The model output for that patient is evaluated according to the determined threshold 2310 to determine whether the patient is an acceptable candidate for a clinical trial group, immunotherapy treatment regimen, etc. Based on the motivations and constraints of the clinical trial, the patient's personal circumstances and expectations, etc., the patient selection threshold 2310 can be adjusted to be more or less inclusive / exclusive to include or exclude specific patients from clinical trials, treatment plans, etc., with the associated risk of inaccuracy / accuracy as the threshold 2310 moves according to the precision curve 2330 and recall curve 2340 of the model.

[0234] like Fig.23 As shown in the example of , the intersection of the precision curve 2330 and the recall curve 2340 occurs at about 0.28. However, at this intersection, both the precision 2330 and the recall 2340 are slightly below the average, each slightly above 0.4, indicating about 60% false positives and about 60% false negatives. Therefore, the intersection is not very large for either precision or recall. Therefore, the threshold 2310 can be selected based on, for example, a recall 2340 of 1.0 and a precision 2330 of about 0.19, or based on, for example, a recall 2340 of 0.58 and a precision 2330 of 0.25, depending on the selection criteria of the model, patient, etc.

[0235] To generate Fig.23The model represented in uses data and associated features. Features used to train and test this model include features representing a set of laboratory measurements over a period of time (e.g., 120 days, 100 days, 30 days, etc.) and a set of patient conditions, medications, etc. over a period of time (e.g., 1 year, 9 months, 6 months, etc.), which can be generated using one hot encoding and methods such as end value, median time weighted average (TWA), mean, median, etc. For example, features formed to train the model may include blood alkaline phosphatase levels (e.g., end value, TWA, maximum value, minimum value, etc.), blood aspartate aminotransferase (e.g., end value, TWA, maximum value, minimum value, etc.), blood alanine aminotransferase levels (e.g., end value, TWA, maximum value, minimum value, etc.), blood bilirubin (e.g., maximum value, TWA, end value, minimum value, etc.), etc. The features train the model (e.g., using a binary classifier with a random forest) to generate a predicted probability of hepatitis toxicity at a certain time after immunotherapy treatment. As described above, features are combined, weighted, and iterated to generate a model, which is reflected in Fig.23 This model can be deployed and used to predict patient outcomes.

[0236] Fig.24 shows the characteristics of the treatment time model, which can be used to replace Fig. 20 The overall survivability model reflected in or as another surrogate for treatment efficacy as a complement thereof. The x-axis 2410 of the chart corresponds to a threshold value (e.g., a patient selection criterion), and the y-axis 2420 of the chart corresponds to the probability of retaining the immunotherapy treatment (e.g., on a precision curve 2430 or a recall curve 2440). The threshold 2410 selected for the patient selection criterion sets a point at which, below the threshold, one action is taken and above the threshold, another action is taken.

[0237] exist Fig.24 In the example of , a negative result indicates that the patient continued treatment, while a positive result indicates that the patient had stopped treatment. If a low threshold is chosen (e.g., 0.0 to 0.5), the model results have high precision (e.g., very few false positives) and low recall (e.g., many false negatives). Therefore, there is a high probability of incorrectly identifying people who have stopped treatment, but there is a low probability of incorrectly identifying people who are still on treatment. As a result, almost all patients who are predicted to still be receiving immunotherapy will actually receive immunotherapy. However, the model incorrectly predicts many treatment discontinuations that did not actually occur. In the drug clinical trial use case, the model will screen out or exclude too many patients from the trial due to the exaggerated treatment discontinuation numbers. In the clinical practice use case of patient treatment, the model will potentially prevent immunotherapy from being given to too many patients due to the exaggerated probability that immunotherapy will have to be stopped prematurely.

[0238] For example, if in Fig.24 In the example above, if we choose a threshold of 0.3, then all predictions to stop treatment are true because recall 2440 is almost 1.0. However, most predictions to continue treatment are incorrect because precision 2430 is slightly less than 0.2.

[0239] If an intermediate threshold is chosen (e.g., 0.5 to 0.6), there is more of a balance between false positives and false negatives. In the pharmaceutical use case, the model will screen out or exclude some patients who may have completed treatment, and screen in or include some patients who will not continue treatment. However, due to the characteristics of the model, overall survival is improved. In a clinical practice context, the model will make some inappropriate predictions, but most patients who are recommended immunotherapy will complete treatment. In the case of applying this threshold to the model, most patients who are not recommended for treatment will not complete treatment.

[0240] For example, if in Fig.24 In the example above, if we choose a threshold of 0.5, more than half of the predictions to stop treatment are true because the recall 2440 is 0.6. However, most predictions to continue treatment are incorrect because the precision 2430 is 0.2.

[0241] If a high threshold is chosen (e.g., 0.6 to 0.7), the model output has low precision (e.g., many false positives) and high recall (e.g., few false negatives). As a result, almost all patients predicted not to complete treatment will actually not complete treatment, and the model incorrectly predicts many discontinued treatment regimens that actually still occur. In a pharmaceutical context, the model will exclude too many patients who will complete the treatment regimen in a clinical trial. In a clinical patient care context, the model will rarely incorrectly indicate immunotherapy.

[0242] For example, if in Fig.24 In the example above, if we choose a threshold of 0.65, then almost all predictions to stop treatment are false, since recall 2440 is slightly above 0.0. In addition, most predictions to continue treatment are incorrect, since precision 2430 is slightly above 0.4.

[0243] use Fig.24Based on the precision 2430 and recall 2440 characteristics of the model represented in , a patient selection criteria threshold 2410 can be selected to drive clinical trial group selection, patient treatment determination, etc. The model trained on a broader population of prior health care data outputs the probability of successful treatment for a given patient by processing the health care data of that particular patient to generate a predicted output representing the efficacy of that patient (using treatment time as a surrogate or replacement for immunotherapy treatment efficacy). The model output for that patient is evaluated according to the determined threshold 2410 to determine whether the patient is an acceptable candidate for a clinical trial group, immunotherapy treatment regimen, etc. Based on the motivations and constraints of the clinical trial, the patient's personal circumstances and expectations, etc., the patient selection threshold 2410 can be adjusted to be more or less inclusive / exclusive to include or exclude specific patients from clinical trials, treatment plans, etc., with the associated risk of inaccuracy / accuracy as the threshold 2410 moves according to the precision curve 2430 and recall curve 2440 of the model.

[0244] like Fig.24 As shown in the example of , the intersection of the precision 2430 and recall 2440 curves occurs at approximately 0.57. However, at this intersection, both precision 2430 and recall 2440 are poor, each less than 0.3, indicating more than 70% false positives and more than 70% false negatives. Therefore, the intersection is not good for either precision or recall. Fig.24 In the case of the treatment time model, precision 2430 and recall 2440 are not well aligned, and precision 2430 is poor throughout the model. Therefore, threshold 2410 can be selected to maximize recall 2440 to 1.0, for example, because unless recall is 0.0, precision hovers around 20%.

[0245] To generate Fig.24The model represented in uses data and associated features. Features used to train and test this model include features representing a set of laboratory measurements over a period of time (e.g., 120 days, 100 days, 30 days, etc.) and a set of patient conditions, medications, etc. over a period of time (e.g., 1 year, 9 months, 6 months, etc.), which can be generated using one hot encoding and methods such as end value, median time weighted average (twa), mean, median, etc. Features can include RBC (e.g., median, etc.), promethazine drug count, immune granulocytes (e.g., median, etc.), lymphocytes (e.g., median, etc.), blood albumin (e.g., end value, twa, median, etc.), RDW (e.g., median, etc.), neutrophils (e.g., median), alkane phosphatases (e.g., median, etc.), dexamethasone drug count, neutabs (e.g., median, etc.), radioactivity counts, one hot encoded conditions (e.g., codes C77, R05, G89, etc.), chemotherapy within a time window, drug counts, etc. As described above, features are combined, weighted, and iterated to generate a model, which is reflected in Fig.24 This model can be deployed and used to predict patient outcomes.

[0246] Thus, precision and recall may vary from model to model, and a given model may change over time as more data is collected, the patient population changes, a particular patient changes, feedback is provided to regenerate or otherwise update the model, etc. However, the model framework enables the generation of a variety of models and the dynamic setting of (one or more) thresholds and other patient and / or model selection criteria to drive productive, reliable results for patient treatment by a patient's physician, inclusion as part of a clinical trial, etc.

[0247] As described herein, configuring a group for a clinical trial and configuring a treatment plan are two examples of how the developed model can be used. However, a treatment plan for a patient can be or include placing the patient in a clinical trial (e.g., to see if an immunotherapy in a clinical trial helps the patient). Thus, configuring a group for a clinical trial can be a specific instance or context of configuring a treatment plan for a patient involving an immunotherapy.

[0248] In some examples, the decision can be driven by a toxicity-related model alone. For example, toxicity risk (such as pneumonia, hepatitis, or colitis) can be a driving factor for whether a patient is given immunotherapy (e.g., either as part of a clinical trial or as part of an individual treatment plan). For example, a patient may want to be treated at all costs. In other examples, the possibility of therapeutic efficacy can be a driving factor for whether immunotherapy is administered to a patient, regardless of toxicity risk. For example, a patient may prioritize quality of life rather than more aggressive treatment. In such instances, additional tests and monitoring (e.g., registration, model rerun, etc.) can be ordered to help ensure the health of the patient, whether toxicity should begin to occur. In some examples, the outputs of efficacy-related models and one or more toxicity-related models can be combined to balance the possibility of success of immunotherapy and the possibility of producing one or more toxicities. Therefore, for example, a model can provide a binary output (e.g., include / exclude, treat / not treat, etc.) and / or convey multiple aspects or nuances of yes or no answers.

[0249] Thus, the model framework can generate multiple models for use in various ways, based on various thresholds, based on clinical needs, research needs, patient expectations, etc. Such qualitative indicators can be transformed into quantitative thresholds for patient selection criteria that enable the same model to drive different decisions and behaviors depending on the threshold. Patient selection criteria can change over time, as a patient may initially want to be aggressive, then back off to allow them to go on vacation, then recover more aggressively, and may back off again when making a quality of life end decision. For example, by setting patient selection criteria thresholds (and by driving model selection criteria for model generation for a specific purpose or focus), the model can adapt to all of these situations.

[0250] Fig.25 An example infrastructure and associated process 2500 is illustrated by which patient health care data from one or more patient health care records 2510 is organized / managed in one or more data repositories 2520, 2525 for intermediate processing using one or more alignment, mapping, and / or cleansing actions 2530 to form time-related values ​​2540, 2545. As described above, the time-related values ​​from the intermediate tables form features 2550 that can be used for model development. Thus, health care data from existing records 2510 is pulled in and organized 2520 for processing 2530 according to varying levels of complexity. The intermediate tables 2540, 2545 form time series data values ​​or variables associated with multiple features 2550. The features can be used to train and test multiple models, such as described above.

[0251] Although example implementations are described in the present application and associated appendices, one or more of the elements, processes and / or devices described may be combined, divided, rearranged, omitted, eliminated and / or implemented in any other manner. In addition, the example elements may be implemented only by hardware, or by hardware in combination with software and / or firmware. Therefore, for example, any of the example elements may be implemented by processor circuitry, (one or more) analog circuits, (one or more) digital circuits, (one or more) logic circuits, (one or more) programmable processors, (one or more) programmable microcontrollers, (one or more) graphics processing units ((one or more) GPUs), (one or more) digital signal processors ((one or more) DSPs), (one or more) application specific integrated circuits ((one or more) ASICs), (one or more) programmable logic devices ((one or more) PLDs) and / or (one or more) field programmable logic devices ((one or more) FPLDs) such as field programmable gate arrays (FPGAs). Further, in addition to or in place of those described, the example elements may include one or more elements, processes and / or devices, and / or may include more than one of any or all of the elements, processes and devices described.

[0252] In some examples, hardware logic circuit systems, machine-readable instructions, hardware-implemented state machines, and / or any combination thereof may implement (one or more) systems and / or perform the methods disclosed herein. Machine-readable instructions may be one or more executable programs or (one or more) parts of executable programs for execution by a processor circuit system. The program may be embodied in software stored on one or more non-transitory computer-readable storage media, such as a compact disk (CD), a floppy disk, a hard disk drive (HDD), a solid-state drive (SSD), a digital versatile disk (DVD), a Blu-ray disc, a volatile memory (e.g., any type of random access memory (RAM), etc.), or a non-volatile memory (e.g., an electrically erasable programmable read-only memory (EEPROM), a flash memory, a HDD, a SSD, etc.) associated with a processor circuit system located in one or more hardware devices, but the entire program and / or its parts may alternatively be executed by one or more hardware devices other than the processor circuit system and / or embodied in firmware or dedicated hardware. Machine-readable instructions may be distributed across multiple hardware devices and / or executed by two or more hardware devices (e.g., a server and a client hardware device). For example, a client hardware device may be implemented by an endpoint client hardware device (e.g., a hardware device associated with a user) or an intermediate client hardware device (e.g., a radio access network (RAN) gateway that may facilitate communications between a server and an endpoint client hardware device). Similarly, a non-transitory computer-readable storage medium may include one or more media located in one or more hardware devices. In addition, the order of execution may be changed, and / or some of the boxes described may be changed, deleted, or combined. Additionally or alternatively, any or all code boxes may be implemented by one or more hardware circuits (e.g., processor circuit systems, discrete and / or integrated analog and / or digital circuit systems, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.) that are configured to perform corresponding operations without executing software or firmware. The processor circuit system can be distributed in different network locations and / or local to the following devices: one or more hardware devices (e.g., a single-core processor (e.g., a single-core central processing unit (CPU)), a multi-core processor in a single machine (e.g., a multi-core CPU, XPU, etc.), multiple processors distributed across multiple servers in a server rack, multiple processors distributed across one or more server racks, a CPU and / or FPGA located in the same package (e.g., the same integrated circuit (IC) package or two or more separate housings, etc.), and the like.

[0253] The machine-readable instructions described herein may be stored in one or more of a compressed format, an encrypted format, a segmented format, a compiled format, an executable format, a packaged format, etc. The machine-readable instructions as described herein may be stored as data or data structures (e.g., as parts of instructions, codes, code representations, etc.) that can be used to create, manufacture, and / or generate machine-executable instructions. For example, the machine-readable instructions may be segmented and stored on one or more storage devices and / or computing devices (e.g., servers) located at the same or different locations of a network or a collection of networks (e.g., in the cloud, in an edge device, etc.). The machine-readable instructions may need to be installed, modified, adapted, updated, combined, supplemented, configured, decrypted, decompressed, unpacked, distributed, reassigned, compiled, etc., so that they can be directly read, interpreted, and / or executed by a computing device and / or other machine. For example, the machine-readable instructions may be stored in multiple parts that are individually compressed, encrypted, and / or stored on separate computing devices, wherein the parts, when decrypted, decompressed, and / or combined, form a set of machine-executable instructions that implement one or more operations, which together may form a program such as the program described herein.

[0254] In another example, the machine-readable instructions may be stored in a state where they are readable by the processor circuitry, but require the addition of a library (e.g., a dynamic link library (DLL)), a software development kit (SDK), an application programming interface (API), etc., in order to execute the machine-readable instructions on a particular computing device or other device. In another example, the machine-readable instructions may need to be configured (e.g., stored settings, data inputs, recorded network addresses, etc.) before the machine-readable instructions and / or corresponding program(s) can be executed in whole or in part. Thus, as used herein, a machine-readable medium may include machine-readable instructions and / or program(s) regardless of the particular format or state of the machine-readable instructions and / or program(s) when stored or otherwise at rest or in transport.

[0255] The machine-readable instructions described herein may be represented by any past, present, or future instruction language, scripting language, programming language, etc. For example, the machine-readable instructions may be represented using any of the following languages: C, C++, Java, C#, Perl, Python, JavaScript, Hypertext Markup Language (HTML), Structured Query Language (SQL), Swift, etc.

[0256] As mentioned above, the example operations disclosed herein may be implemented using executable instructions (e.g., computer and / or machine readable instructions) stored on one or more non-transitory computer and / or machine readable media (such as optical storage devices, magnetic storage devices, HDDs, flash memory, read-only memory (ROM), CDs, DVDs, caches, any type of RAM, registers, and / or any other storage devices or storage disks where information is stored for any duration (e.g., for an extended period of time, permanently, for a brief moment, for temporary buffering, and / or for caching of information)). As used herein, the terms non-transitory computer-readable medium, non-transitory computer-readable storage medium, non-transitory machine-readable medium, and non-transitory machine-readable storage medium are expressly defined to include any type of computer-readable storage device and / or storage disk, and exclude propagating signals and exclude transmission media. As used herein, the terms "computer-readable storage device" and "machine-readable storage device" are defined to include any physical (mechanical and / or electrical) structure for storing information, but exclude propagating signals and exclude transmission media. Examples of computer-readable storage devices and machine-readable storage devices include any type of random access memory, any type of read-only memory, solid-state memory, flash memory, optical disks, magnetic disks, disk drives, and / or redundant arrays of independent disks (RAID) systems. As used herein, the term "device" refers to a physical structure, such as a mechanical and / or electrical device, hardware, and / or circuit system, which may or may not be configured by computer-readable instructions, machine-readable instructions, etc., and / or is manufactured to execute computer-readable instructions, machine-readable instructions, etc.

[0257] Fig.26 26 is a block diagram of an example processor platform 2600 that is configured to execute and / or instantiate the machine-readable instructions and / or operations disclosed and described herein. The processor platform 2600 may be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cellular phone, a smart phone, a tablet such as an iPad™), a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a game console, a personal video recorder, a set-top box, a headset (e.g., an augmented reality (AR) headset, a virtual reality (VR) headset, etc.) or other wearable device, or any other type of computing device.

[0258] The processor platform 2600 of the illustrated example includes a processor circuit system 2612. The processor circuit system 2612 of the illustrated example is hardware. For example, the processor circuit system 2612 can be implemented by one or more integrated circuits, logic circuits, FPGAs, microprocessors, CPUs, GPUs, DSPs, and / or microcontrollers from any desired series or manufacturer. The processor circuit system 2612 can be implemented by one or more semiconductor-based (e.g., silicon-based) devices.

[0259] The processor circuit system 2612 of the illustrated example includes a local memory 2613 (e.g., cache, registers, etc.). The processor circuit system 2612 of the illustrated example communicates with a main memory (including a volatile memory 2614 and a non-volatile memory 2616) via a bus 2618. The volatile memory 2614 may be a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), Dynamic Random Access Memory The non-volatile memory 2616 may be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 2614, 2616 of the illustrated example is controlled by a memory controller 2617.

[0260] The processor platform 2600 of the illustrated example also includes an interface circuit system 2620. The interface circuit system 2620 may be implemented by hardware according to any type of interface standard, such as an Ethernet interface, a universal serial bus (USB) interface, interface, a near field communication (NFC) interface, a peripheral component interconnect (PCI) interface, and / or a peripheral component interconnect express (PCIe) interface.

[0261] In the illustrated example, one or more input devices 2622 are connected to the interface circuitry 2620. The input device(s) 2622 permit a user to enter data and / or commands into the processor circuitry 2612. The input device(s) 2622 can be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touch screen, a trackpad, a trackball, an isochronous device, and / or a speech recognition system.

[0262] One or more output devices 2624 are also connected to the interface circuit system 2620 of the illustrated example. The output device(s) 2624 can be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube (CRT) display, an in-place switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. Thus, the interface circuit system 2620 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics processor circuit (such as a GPU).

[0263] The interface circuitry 2620 of the illustrated example also includes communication devices (such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces) to facilitate the exchange of data with external machines (e.g., any kind of computing device) over a network 2626. Communications can occur over, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a direct-line wireless system, a cellular telephone system, an optical connection, etc.

[0264] The processor platform 2600 of the illustrated example also includes one or more mass storage devices 2628 that store software and / or data. Examples of such mass storage devices 2628 include magnetic storage devices, optical storage devices, floppy disk drives, HDDs, CDs, Blu-ray disk drives, redundant array of independent disks (RAID) systems, solid-state storage devices (e.g., flash memory devices and / or SSDs), and DVD drives.

[0265] The example computer-readable instructions 2632 may be stored in the mass storage device 1528, in the volatile memory 2614, in the non-volatile memory 2616, and / or on a removable, non-transitory computer-readable storage medium such as a CD or DVD.

[0266] Fig. 27 yes Fig.26 2612. In this example, Fig.26The processor circuit system 2612 is implemented by the microprocessor 2700. For example, the microprocessor 2700 can be a general-purpose microprocessor (e.g., a general-purpose microprocessor circuit). The microprocessor 2700 executes some or all of the machine-readable instructions to effectively instantiate the circuit system described herein as a logic circuit so as to perform operations corresponding to those machine-readable instructions. In some such examples, the circuit system is instantiated by the hardware circuit of the microprocessor 2700 in combination with the instruction. For example, the microprocessor 2700 can be implemented by a multi-core hardware circuit system (such as a CPU, DSP, GPU, XPU, etc.). Although it can include any number of example cores 2702 (e.g., 1 core), the microprocessor 2700 of this example is a multi-core semiconductor device including N cores. The cores 2702 of the microprocessor 2700 can operate independently, or can cooperate to execute machine-readable instructions. For example, the machine code corresponding to the firmware program, embedded software program, or software program can be executed by one of the cores 2702, or can be executed by multiple cores in the core 2702 at the same or different times. In some examples, machine code corresponding to a firmware program, embedded software program, or software program is split into threads and executed in parallel by two or more of the cores 2702. The software program may correspond to a portion or all of the machine-readable instructions and / or operations disclosed herein.

[0267] The core 2702 can communicate via a first example bus 2704. In some examples, the first bus 2704 can be implemented by a communication bus to implement communications associated with (one or more) cores in the core 2702. For example, the first bus 2704 can be implemented by at least one of an inter-integrated circuit (I2C) bus, a serial peripheral interface (SPI) bus, a PCI bus, or a PCIe bus. Additionally or alternatively, the first bus 2704 can be implemented by any other type of computing or electrical bus. The core 2702 can obtain data, instructions, and / or signals from one or more external devices via the example interface circuit system 2706. The core 2702 can output data, instructions, and / or signals to one or more external devices via the example interface circuit system 2706. Although the cores 2702 of this example include example local memory 2720 (e.g., a level 1 (L1) cache that can be split into a level 1 data cache and a level 1 instruction cache), the microprocessor 2700 also includes example shared memory 2710 (e.g., a level 2 (L2 cache)) that can be shared by the cores for high-speed access to data and / or instructions. Data and / or instructions can be transferred (e.g., shared) by writing to and / or reading from the shared memory 2710. The local memory 2720 and the shared memory 2710 of each core in the cores 2702 can be part of a hierarchy of storage devices, which include cache memory and main memory (e.g., Fig.26 2614, 2616). Typically, higher levels of memory in the hierarchy exhibit lower access times and have smaller storage capacities than lower levels of memory. Changes to the various levels of the cache hierarchy are managed (e.g., coordinated) by a cache coherence policy.

[0268] Each core 2702 may be referred to as a CPU, DSP, GPU, etc., or any other type of hardware circuit system. Each core 2702 includes a control unit circuit system 2714, an arithmetic and logic (AL) circuit system (sometimes referred to as an ALU) 2716, a plurality of registers 2718, a local memory 2720, and a second example bus 2722. Other structures may exist. For example, each core 2702 may include a vector unit circuit system, a single instruction multiple data (SIMD) unit circuit system, a load / store unit (LSU) circuit system, a branch / jump unit circuit system, a floating point unit (FPU) circuit system, etc. The control unit circuit system 2714 includes a semiconductor-based circuit configured to control (e.g., coordinate) data movement within the corresponding core 2702. The AL circuit system 2716 includes a semiconductor-based circuit configured to perform one or more mathematical and / or logical operations on the data within the corresponding core 2702. The AL circuit system 2716 of some examples performs integer-based operations. In other examples, the AL circuit system 2716 also performs floating-point operations. In still other examples, the AL circuit system 2716 may include a first AL circuit system that performs integer-based operations and a second AL circuit system that performs floating-point operations. In some examples, the AL circuit system 2716 may be referred to as an arithmetic logic unit (ALU). The register 2718 is a semiconductor-based structure for storing data and / or instructions (such as the results of one or more of the operations performed by the AL circuit system 2716 of the corresponding core 2702). For example, the register 2718 may include (one or more) vector registers, (one or more) SIMD registers, (one or more) general registers, (one or more) flag registers, (one or more) segment registers, (one or more) machine-specific registers, (one or more) instruction pointer registers, (one or more) control registers, (one or more) debug registers, (one or more) memory management registers, (one or more) machine check registers, etc. The register 2718 may be arranged in a manner such as Fig. 27 27. Alternatively, registers 2718 may be organized in any other arrangement, format, or structure, including being distributed throughout core 2702 to reduce access time. Second bus 2722 may be implemented by at least one of an I2C bus, an SPI bus, a PCI bus, or a PCIe bus.

[0269] Each core 2702 and / or more generally the microprocessor 2700 may include additional and / or alternative structures to those shown and described above. For example, one or more clock circuits, one or more power supply devices, one or more power gates, one or more cache home agents (CHA), one or more aggregation / common grid stops (CMS), one or more shifters (e.g., (one or more) barrel shifters) and / or other circuit systems may exist. The microprocessor 2700 is a semiconductor device made to include many transistors, which are interconnected to implement the structure described above in one or more integrated circuits (ICs) contained in one or more packages. The processor circuit system may include one or more accelerators and / or cooperate with one or more accelerators. In some examples, the accelerator is implemented by a logic circuit system to perform certain tasks faster and / or more efficiently than what can be done by a general-purpose processor. Examples of accelerators include ASICs and FPGAs, such as those discussed herein. GPUs or other programmable devices may also be accelerators. The accelerator may be on a board of the processor circuit system, in a chip package identical to the processor circuit system, and / or in one or more packages separated from the processor circuit system.

[0270] Fig.28 yes Fig.26 2800. In this example, the processor circuit system 2612 is implemented by an FPGA circuit system 2800. For example, the FPGA circuit system 2800 can be implemented by an FPGA. The FPGA circuit system 2800 can be used, for example, to perform operations that could otherwise be performed by a processor executing corresponding machine-readable instructions. Fig. 27 However, once configured, FPGA circuitry 2800 instantiates machine-readable instructions in hardware and can therefore often perform operations faster than if the operations could be performed by a general-purpose microprocessor executing corresponding software.

[0271] More specifically, as described above Fig. 27 Compared to the microprocessor 2700 of FIG. 1 (which may be programmed to execute some or all of the machine-readable instructions disclosed herein but whose interconnections and logic circuits are fixed once fabricated), Fig.28The example FPGA circuit system 2800 includes interconnections and logic circuit systems that can be configured and / or interconnected in different ways after fabrication to instantiate, for example, some or all of the machine-readable instructions disclosed herein. In particular, the FPGA circuit system 2800 can be considered as an array of logic gates, interconnections, and switches. The switches can be programmed to change how the logic gates are interconnected by interconnections, thereby effectively forming one or more dedicated logic circuits (unless and until the FPGA circuit system 2800 is reprogrammed). The configured logic circuits enable the logic gates to cooperate in different ways to perform different operations on the data received by the input circuit system. Those operations may correspond to some or all of the software disclosed herein. Therefore, the FPGA circuit system 2800 can be configured to effectively instantiate some or all of the machine-readable instructions as dedicated logic circuits so as to perform operations corresponding to those software instructions in a dedicated manner similar to an ASIC. Therefore, the FPGA circuit system 2800 can perform operations corresponding to some or all of the machine-readable instructions faster than when a general-purpose microprocessor can perform operations corresponding to some or all of the machine-readable instructions.

[0272] exist Fig.28 In the example of FPGA circuit system 2800, FPGA circuit system 2800 is configured to be programmed (and / or reprogrammed one or more times) by an end user via a hardware description language (HDL) such as Verilog. Fig.28 The FPGA circuit system 2800 includes an example input / output (I / O) circuit system 2802, which is used to obtain data from and / or output data to the example configuration circuit system 2804 and / or external hardware 2806. For example, the configuration circuit system 2804 can be implemented by an interface circuit system, which can obtain machine-readable instructions to configure the FPGA circuit system 2800 or (one or more) parts thereof. In some such examples, the configuration circuit system 2804 can obtain machine-readable instructions from a user, a machine (e.g., a hardware circuit system (e.g., a programming or dedicated circuit system) that can implement an artificial intelligence / machine learning (AI / ML) model to generate instructions), etc. In some examples, the external hardware 2806 can be implemented by an external hardware circuit system. For example, the external hardware 2806 can be implemented by Fig. 27 FPGA circuit system 2800 also includes an array of example logic gate circuit system 2808, multiple example configurable interconnects 2810, and example storage circuit system 2812. Logic gate circuit system 2808 and interconnect 2810 are configurable to instantiate one or more operations that may correspond to at least some of the machine-readable instructions and / or other desired operations. Fig.17The logic gate circuit system 2808 shown in is made in groups or blocks. Each block includes a semiconductor-based electrical structure that can be configured as a logic circuit. In some examples, the electrical structure includes a logic gate (e.g., an AND gate, an OR gate, a NOR gate, etc.) that provides a basic building block for the logic circuit. An electrically controllable switch (e.g., a transistor) is present in each of the logic gate circuit systems 2808 to enable the configuration of the logic gates and / or electrical structures to form a circuit that performs the desired operation. The logic gate circuit system 2808 may include other electrical structures, such as a lookup table (LUT), a register (e.g., a flip-flop or a latch), a multiplexer, etc.

[0273] The illustrated example configurable interconnect 2810 is a conductive path, trace, via, or the like, which may include an electrically controllable switch (e.g., a transistor) whose state can be changed by programming (e.g., using an HDL instruction language) to activate or deactivate one or more connections between one or more of the logic gate circuit systems 2808 in order to program the desired logic circuit.

[0274] The storage circuit system 2812 of the illustrated example is configured to store (one or more) results of one or more of the operations performed by the corresponding logic gates. The storage circuit system 2812 can be implemented by registers, etc. In the illustrated example, the storage circuit system 2812 is distributed among the logic gate circuit system 2808 to facilitate access and increase execution speed.

[0275] Fig.28 The example FPGA circuit system 2800 also includes an example special operation circuit system 2814. In this example, the special operation circuit system 2814 includes a special circuit system 2816, which can be called to implement common functions to avoid the need to program those functions on site. Examples of such special circuit systems 2816 include memory (e.g., DRAM) controller circuit systems, PCIe controller circuit systems, clock circuit systems, transceiver circuit systems, memory, and multiplier-accumulator circuit systems. Other types of special circuit systems may exist. In some examples, the FPGA circuit system 2800 may also include an example general programmable circuit system 2818, such as an example CPU 2820 and / or an example DSP 2822. Additionally or alternatively, other general programmable circuit systems 2818 such as GPUs, XPUs, etc. may exist, which can be programmed to perform other operations.

[0276] Although Fig. 27 and Fig.28 Pictured Fig.26 These are two example implementations of the processor circuit system 2612, but many other approaches are contemplated. For example, as mentioned above, a modern FPGA circuit system may include an onboard CPU, such as Fig.28 One or more of the example CPUs 2820. Thus, Fig.26 The processor circuit system 2612 can additionally be combined by Fig. 27 An example microprocessor 2700 and Fig.28 In some such hybrid examples, the first portion of the machine-readable instructions may be implemented by Fig. 27 The second portion of the machine-readable instructions may be executed by one or more of the cores 2702 of Fig.28 The FPGA circuit system 2800 of the present invention may be executed by the FPGA circuit system 2800, and / or the third part of the machine-readable instructions may be executed by the ASIC. It should be understood that, therefore, some or all of the circuit system may be instantiated at the same or different times. For example, some or all of the circuit system may be instantiated in one or more threads executed concurrently and / or serially. In addition, in some examples, some or all of the circuit system may be implemented in one or more virtual machines and / or containers executed on a microprocessor.

[0277] In some examples, Fig.26 The processor circuit system 2612 may be in one or more packages. For example, Fig. 27 The processor circuit system 2700 and / or Fig.28 The FPGA circuit system 2800 may be in one or more packages. In some examples, the XPU may be composed of a Fig.26 The XPU may be implemented by processor circuitry 2612. For example, the XPU may include a CPU in one package, a DSP in another package, a GPU in yet another package, and an FPGA in yet another package.

[0278] The diagram is used to Fig.26 A block diagram of an example software distribution platform 2905 for distributing software such as example machine readable instructions 2632 to a hardware device owned and / or operated by a third party is provided in Fig.29 The example software distribution platform 2905 may be implemented by any computer server, data facility, cloud service, etc. that is capable of storing software and delivering software to other computing devices. A third party may be a customer of the entity that owns and / or operates the software distribution platform 2905. For example, the entity that owns and / or operates the software distribution platform 2905 may be a Fig.26The third party may be a consumer, user, retailer, OEM, etc. who purchases the software and / or licenses the software for use and / or resale and / or sublicense. In the illustrated example, the software distribution platform 2905 includes one or more servers and one or more storage devices. As described above, the storage device stores the machine-readable instructions 2632, which may correspond to Fig.26 The example machine-readable instructions 2632 of the example software distribution platform 2905 are provided. One or more servers of the example software distribution platform 2905 communicate with the example network 2910, which may correspond to any one or more of the Internet and / or any of the example networks described above. In some examples, as part of a business transaction, one or more servers respond to a request to transfer software to a requesting party. Payment for the delivery, sale, and / or license of the software may be processed by one or more servers of the software distribution platform and / or by a third-party payment entity. The server enables purchasers and / or licensees to download the machine-readable instructions 2632 from the software distribution platform 2905. For example, it may correspond to Fig.26 The software of the example machine readable instructions 2632 can be downloaded to the example processor platform 2600, which is to execute the machine readable instructions 2632 to implement the systems and methods described herein. In some examples, one or more servers of the software distribution platform 2905 periodically provide, transmit and / or force updates to the software (e.g., the example machine readable instructions 2632) to ensure that improvements, patches, updates, etc. are distributed and applied to the software at the end-user devices.

[0279] In some examples, deployed models and / or patient data may be uploaded via a cloud-based platform 2905 for remote execution, rather than downloading machine-readable instructions 2932 to a local processor platform 2900. In some examples, the example platform 2905 may host one or more models accessible by a network 2910, and the processor platform 2600 may provide input to the models and receive results, predictions, and / or other outputs.

[0280] From the foregoing, it will be appreciated that example systems, methods, devices, and articles of manufacture that implement model generation and deployment to drive processes for therapy prediction and treatment execution have been disclosed. The disclosed systems, methods, devices, and articles of manufacture improve the efficiency of using computing devices by implementing model generation and deployment to drive processes for therapy prediction and treatment execution. Thus, the disclosed systems, methods, devices, and articles of manufacture are directed to one or more improvements in the operation of machines such as computers or other electronic and / or mechanical devices.

[0281] Certain examples provide systems and methods for training and evaluating AI models to predict adverse risk events based on fixed-length patient history data. The example system and associated methods include submodules for extracting patient blood test history from electronic medical records, cleaning history, and performing data quality checks. The example system and associated methods include a label definition submodule, which instantiates an algorithm for assigning hepatitis adverse event levels to blood test value sets (e.g., ALT, AST, TBILIRUBIN, ALKPHOS) and creating binary target labels for the AI ​​model. The example system and method include a feature engineering submodule, which is used to regularize blood test values ​​to the upper limit of normal values ​​(e.g., specific to patients, laboratories, and / or blood tests, etc.). The example feature engineering submodule is to transform the normalized value into a discrete symbolic representation, such as a modified version of a symbol set approximation, etc. The example feature engineering submodule is to extract motifs in the form of n-grams from a symbol sequence, and use counts in recent patient history as features. The example system and method include submodules for training and evaluating an AI model to dynamically predict immune-related hepatitis adverse event risks from a fixed-length blood test history.

[0282] Certain examples provide systems and methods for building classification models (e.g., pneumonia classification models, etc.) using sequential procedures. Example systems and methods include preprocessing structured EHRs and unstructured data tables. For example, at the time of the first ICI administration, patient timelines are aligned. Statistics are used to aggregate laboratory measurements within a time window (e.g., a 60-day time window, etc.) before the first ICI. For example, other features (e.g., conditions, smoking status, etc.) can use different time windows (e.g., a 1-year time window, etc.).

[0283] Example systems and methods include finding patterns in data to identify potential predictive features associated with the development of ICI-related toxicities such as pneumonia, colitis, and hepatitis. In the case of a small and noisy data set, many data points are utilized. The data is segmented with a first partition (e.g., 90% partition, 80% partition, 95% partition, etc.) to identify candidate features based on the association between pneumonia labels and features.

[0284] Example systems and methods include deciding between two model versions at each iteration of the procedure, one with the original feature set (M1) and one expanded with candidate features (M2). Nested cross validation is performed on the first (e.g., 90%, etc.) partitions, and the inner loop results are used to compare M1 and M2. A binomial test is performed to assess whether M2 is significantly better than M1.

[0285] Example systems and methods include assessing whether M2 has better performance on a second partition (e.g., 10% partition, 5% partition, 20% partition, etc.) that is retained when M2 is significantly better in step 3). A permutation test is performed to estimate the probability of observing better performance by random chance alone. This step serves as a safety measure to avoid overfitting to the first (e.g., 90%, etc.) data partition. If, based on step 4), M2 has significantly better performance, then M2 is selected.

[0286] The example system and method includes continuing to test new candidate features until the desired model size is reached. The performance of the final model is assessed on an outer loop. In this way, a performance estimator with a smaller variance is obtained, and the variability of the test predictions and the instability of the model can be assessed. If the final model has promising performance, the model is evaluated on a sufficiently large external test set.

[0287] Certain examples provide systems and methods that form an automated framework to prepare multi-source electronic health record data for use in machine learning model training. Example methods and associated systems include preparing input data from multiple sources: This portion of the framework focuses on cleaning and extracting features from multiple data sources. This step receives raw, automatically exported EHR data in a data model format (e.g., OMOP data model format, etc.) and multiple expert-curated data sources for additional features and labels. This step is open to extension and includes, but is not limited to, modules for preparing smoking history, medication administration, medical conditions, radiation therapy history, laboratory measurements, and anthropometric data.

[0288] Example methods and associated systems include generating a combined time-correlated unrestricted intermediate output. This step compresses the input data in a uniform format, preserving the timestamps of the individual data items for each patient. This intermediate step provides plug-in possibilities for modules that prepare time-correlated input data for sequence modeling algorithms (not implemented) and provides flexible inputs for the aggregation step of the framework.

[0289] Example methods and associated systems include Aggregate Features for Time-Independent Models. This step is a target-agnostic, highly configurable plug-in module for creating time-independent, aggregated inputs for machine learning models. This step is open to extension, and configurable parameters include, but are not limited to, prediction time points, aggregation length, data sources involved, and feature types involved.

[0290] Certain examples provide systems and methods for building predictive models for efficacy surrogate endpoints associated with immune checkpoint inhibitor therapy. Example methods and associated systems include preparing input data from multiple sources. This part of the framework involves cleaning and extracting features from multiple data sources. For example, this step receives raw, automatically exported EHR data in the OMOP data model format, as well as multiple expert-managed data sources for additional features. Example methods and associated systems include generating benchmark true value prediction labels, such as ICI treatment time (TOT), next treatment time (after ICI interruption) (TNET), overall survival (OS), etc. The listed benchmark true value endpoints are generated on a continuous scale, expressed in days past the anchor point (patient timelines are aligned based on similarities during ICI treatment). The default anchor point is the first date of ICI treatment initiation. The generated benchmark true values ​​can be used as is, or used to train regression or survival analysis-based models at a modified granularity (past weeks, months, years, etc.). Discretization of the ground truth may be performed for binary or multi-class classifications (eg, responders vs. non-responders, 5-year survival rate, etc.).

[0291] Example methods and associated systems include model building based on generated feature matrices and ground truth values ​​as independent modules of the framework. Endpoints can be modeled individually. For example, modeling can be performed without assumptions and with different machine learning algorithms. Model building and selection workflows can be used to generate predictive models for immune checkpoint inhibitor-associated pneumonia and sequential procedures for model building.

[0292] Certain examples provide systems and methods for hepatitis prediction. The example system and associated methods include preparing inputs by extracting relevant portions of blood features from received electronic health record (EHR) data tables. These are measurements of liver biomarker concentrations in plasma (such as ALT, AST, alkaline phosphatase, and bilirubin) and other concentration values ​​in blood. This step is followed by placing the time series data into a single complex data structure, which is an efficient option to then continue to aggregate this information into a final data table, now ready for preprocessing and transformation steps. The aggregation step of feature engineering includes describing the time series data of blood particles with their mean, standard deviation, minimum, and maximum values. Lag features can also be created by taking the last available liver biomarker measurements before treatment. Use definitions obtained from medical experts to create labels, for example, when the level of at least one liver biomarker exceeds 3 times the upper limit of normal within a predetermined window, someone is classified as positive, otherwise negative. The date of ICI treatment used in this scheme can be output from different workflows.

[0293] Example systems and methods include dataset resampling. For example, the dataset obtained from step 1 is unbalanced, so if the goal is to maximize the recall value, a random majority class us is performed on the dataset. If the Fl score is the maximized theme, the resampling step is ignored.

[0294] The example system and method include training and prediction. For example, on the final data obtained from step 2, the model is trained, the model is RF for maximizing the recall rate, and GB for maximizing the Fl score. Validation is performed using leave-one-out cross-validation, where each sample is individually predicted, and the remaining samples are used as training data sets.

[0295] A brief list of some examples

[0296] Further aspects of the disclosure are provided by the subject matter of the following clauses:

[0297] Example 1 is a method for configuring a treatment plan for a patient, the method comprising: a) loading a toxicity-related model for a selected toxicity associated with an immunotherapy, the toxicity-related model being selected from a plurality of candidate models using a model selection criterion; b) determining a patient selection criterion in response to an output of the toxicity-related model to set a balance between precision and recall for an action, the output comprising a toxicity prediction; c) processing health care data from a health care record for a given patient using the toxicity-related model to generate the toxicity prediction; d) evaluating the toxicity prediction relative to the patient and the patient selection criterion; e) generating a first indication and configuring the treatment plan to include the immunotherapy when the toxicity prediction satisfies the patient selection criterion; f) generating a second indication and excluding the immunotherapy from the treatment plan when the toxicity prediction does not satisfy the patient selection criterion; and g) outputting a medical order configuring the treatment plan for the patient based on the first indication or the second indication.

[0298] Example 2 is a method for configuring a group for a clinical trial, the method comprising: a) loading a toxicity-related model for a selected toxicity associated with an immunotherapy, the toxicity-related model being selected from a plurality of candidate models using a model selection criterion; b) determining a patient selection criterion in response to an output of the toxicity-related model to set a balance between precision and recall for an action, the output comprising a toxicity prediction; c) processing health care data from a health care record for a given patient using the toxicity-related model to generate the toxicity prediction; d) evaluating the toxicity prediction relative to the patient and the patient selection criterion; e) generating a first indication and configuring the treatment plan to include the immunotherapy when the toxicity prediction satisfies the patient selection criterion; f) generating a second indication and excluding the immunotherapy from the treatment plan when the toxicity prediction does not satisfy the patient selection criterion; and g) outputting a medical order configuring the treatment plan for the patient based on the first indication or the second indication.

[0299] Example 3 includes the method of any preceding clause, further comprising generating the toxicity-related model by forming features from an input data set obtained from health care records, the features being iteratively compared by association results to form the model, the model being adjusted based on an analysis of precision and recall.

[0300] Example 4 includes the method of any of the preceding clauses, wherein the patient selection criterion is a first patient selection criterion, the balance is a first balance, and the output is a first output, and further comprising: loading an efficacy-related model associated with the efficacy of the immunotherapy; determining a second patient selection criterion to set a second balance between precision and recall for the action in response to a second output of the efficacy-related model, the second output comprising an efficacy prediction; using the efficacy-related model to process health care data from a health care record for a given patient to generate the efficacy prediction; and evaluating the efficacy prediction relative to the patient and the second patient selection criterion.

[0301] Example 5 includes the method described in any of the preceding clauses, further comprising: generating the first indication and configuring the treatment plan to include the immunotherapy when the toxicity prediction satisfies the first patient selection criteria and the efficacy prediction satisfies the second patient selection criteria; and otherwise, generating the second indication and excluding the immunotherapy from the treatment plan.

[0302] Example 6 includes the method of any of the preceding clauses, further comprising evaluating the toxicity prediction in comparison with the efficacy prediction for the given patient to generate the corresponding first or second indication, and configuring the treatment plan to include or exclude the immunotherapy, respectively.

[0303] Example 7 includes the method of any preceding clause, wherein the therapeutic efficacy is measured by patient survival rate or treatment duration.

[0304] Example 8 includes the method of any preceding clause, wherein configuring the treatment plan comprises configuring the treatment plan for a plurality of selected patients to establish a group for a clinical trial.

[0305] Example 9 includes the method of any preceding clause, wherein configuring the treatment plan further comprises determining a schedule for applying the immunotherapy to the patient.

[0306] Example 10 includes the method of any preceding clause, wherein configuring the treatment plan further comprises providing patient monitoring to assess the patient during execution of the treatment plan.

[0307] Example 11 includes the method of any preceding clause, wherein the patient selection criteria comprises a threshold value defining a balance between false positives and false negatives of the toxicity-related model.

[0308] Example 12 includes the method of any preceding clause, wherein the selected toxicity is pneumonitis, colitis, or hepatitis, and wherein the toxicity prediction is i) the probability that the patient will suffer from the selected toxicity or ii) the probability that the patient will not suffer from the selected toxicity.

[0309] Example 13 includes the method of any preceding clause, wherein the healthcare data comprises at least one of a laboratory test result, a diagnosis code, or a billing code.

[0310] Example 14 includes the method of any preceding clause, further comprising aligning the healthcare data relative to a reference point, the reference point corresponding to an event in the medical record for the patient.

[0311] Example 15 includes the method of any preceding clause, further comprising re-evaluating the patient using updated healthcare data and an updated toxicity-related model for the patient.

[0312] Example 16 includes the method of any preceding clause, wherein the model selection criteria comprises at least one of a high F1 score, a high recall, or a high precision.

[0313] Example 17 is a method for configuring a treatment plan for a patient, the method comprising: a) loading an efficacy-related model associated with the efficacy of immunotherapy, the efficacy-related model being selected from a plurality of candidate models using a model selection criterion; b) determining a patient selection criterion in response to an output of the efficacy-related model to set a balance between precision and recall for an action, the output comprising an efficacy prediction; c) using the efficacy-related model to process health care data from a health care record for a given patient to generate the efficacy prediction; d) evaluating the efficacy prediction relative to the patient and the patient selection criterion; e) when the efficacy prediction satisfies the patient selection criterion, generating a first indication and configuring the treatment plan to include the immunotherapy; f) when the efficacy prediction does not satisfy the patient selection criterion, generating a second indication and excluding the immunotherapy from the treatment plan; and g) outputting a medical order configuring the treatment plan for the patient based on the first indication or the second indication.

[0314] Example 18 is a method for configuring a group for a clinical trial, the method comprising: a) loading an efficacy-related model associated with the efficacy of an immunotherapy, the efficacy-related model being selected from a plurality of candidate models using a model selection criterion; b) determining a patient selection criterion in response to an output of the efficacy-related model to set a balance between precision and recall for an action, the output comprising an efficacy prediction; c) using the efficacy-related model to process health care data from a health care record for a given patient to generate the efficacy prediction; d) evaluating the efficacy prediction relative to the patient and the patient selection criterion; e) when the efficacy prediction satisfies the patient selection criterion, generating a first indication and configuring the treatment plan to include the immunotherapy; f) when the efficacy prediction does not satisfy the patient selection criterion, generating a second indication and excluding the immunotherapy from the treatment plan; and g) outputting a medical order configuring the treatment plan for the patient based on the first indication or the second indication.

[0315] Example 19 includes the method of any of the preceding clauses, further comprising generating the efficacy-related model by forming features from an input data set obtained from health care records, the features being iteratively compared through associated results to form the model, the model being adjusted based on an analysis of precision and recall.

[0316] Example 20 includes the method of any preceding clause, wherein the patient selection criterion is a first patient selection criterion, the balance is a first balance, and the output is a first output, and further comprising: loading a toxicity-related model for a selected toxicity associated with the immunotherapy; determining a second patient selection criterion to set a second balance between precision and recall for an action in response to a second output of the toxicity-related model, the second output comprising a toxicity prediction; processing health care data from a health care record for a given patient using the toxicity-related model to generate the toxicity prediction; and evaluating the toxicity prediction relative to the patient and the second patient selection criterion.

[0317] Example 21 includes the method described in any of the preceding clauses, further comprising: generating the first indication and configuring the treatment plan to include the immunotherapy when the efficacy prediction satisfies the first patient selection criteria and the toxicity prediction satisfies the second patient selection criteria; and otherwise, generating the second indication and excluding the immunotherapy from the treatment plan.

[0318] Example 22 includes the method of any of the preceding clauses, further comprising evaluating the toxicity prediction in comparison with the efficacy prediction for the given patient to generate the corresponding first or second indication, and configuring the treatment plan to include or exclude the immunotherapy, respectively.

[0319] Example 23 includes the method of any preceding clause, wherein the selected toxicity is pneumonitis, colitis, or hepatitis, and wherein the toxicity prediction is i) the probability that the patient will suffer from the selected toxicity or ii) the probability that the patient will not suffer from the selected toxicity.

[0320] Example 24 includes the method of any preceding clause, wherein configuring the treatment plan comprises configuring the treatment plan for a plurality of selected patients to construct a group for a clinical trial.

[0321] Example 25 includes the method of any preceding clause, wherein configuring the treatment plan further comprises determining a schedule for applying the immunotherapy to the patient.

[0322] Example 26 includes the method of any preceding clause, wherein configuring the treatment plan further comprises providing patient monitoring to evaluate the patient during execution of the treatment plan.

[0323] Example 27 includes the method of any preceding clause, wherein the patient selection criteria comprises a threshold that defines a balance between false positives and false negatives for the efficacy-related model.

[0324] Example 28 includes the method of any preceding clause, wherein the healthcare data comprises at least one of a laboratory test result, a diagnosis code, or a billing code.

[0325] Example 29 includes the method of any preceding clause, further comprising aligning the healthcare data relative to a reference point, the reference point corresponding to an event in the medical record of the patient.

[0326] Example 30 includes the method of any preceding clause, further comprising re-evaluating the patient using updated healthcare data and an updated outcome-related model for the patient.

[0327] Example 31 includes the method of any preceding clause, wherein the model selection criteria comprises at least one of a high F1 score, a high recall, or a high precision.

[0328] Example 32 includes the method of any preceding clause, wherein the therapeutic efficacy is measured by patient survival rate or treatment duration.

[0329] Example 33 is a method of configuring a treatment plan for a patient, the method comprising: a) generating a toxicity-related model for a selected toxicity associated with an immunotherapy by at least the following steps: i) training an initial model on a plurality of features generated from a plurality of prior health care data, the prior health care data being arranged in a time series and aligned relative to a reference point; ii) iteratively adding and removing each of a set of additional features to adjust the initial model into an adjusted model, and evaluating the resulting precision and recall associated with an output of the adjusted model relative to a model selection criterion; and iii) deploying the adjusted model as the toxicity-related model when the model selection criterion is met; b) in response to the toxicity-related output of a model, determining patient selection criteria to set a balance between precision and recall for an action, the output comprising a toxicity prediction; c) processing health care data from a health care record for a given patient using the toxicity-related model to generate the toxicity prediction; d) evaluating the toxicity prediction relative to the patient and the patient selection criteria; e) when the toxicity prediction satisfies the patient selection criteria, generating a first indication and configuring the treatment plan to include the immunotherapy; f) when the toxicity prediction does not satisfy the patient selection criteria, generating a second indication and excluding the immunotherapy from the treatment plan; and g) outputting a medical order configuring the treatment plan for the patient based on the first indication or the second indication.

[0330] Example 34 is a method of configuring a cohort for a clinical trial, the method comprising: a) generating a toxicity-related model for a selected toxicity associated with an immunotherapy by at least the following steps: i) training an initial model on a plurality of features generated from a plurality of prior health care data, the prior health care data being arranged in a time series and aligned relative to a reference point; ii) iteratively adding and removing each of a set of additional features to adjust the initial model into an adjusted model, and evaluating the resulting precision and recall associated with an output of the adjusted model relative to a model selection criterion; and iii) deploying the adjusted model as the toxicity-related model when the model selection criterion is met; b) in response to the toxicity-related output of a model, determining patient selection criteria to set a balance between precision and recall for an action, the output comprising a toxicity prediction; c) processing health care data from a health care record for a given patient using the toxicity-related model to generate the toxicity prediction; d) evaluating the toxicity prediction relative to the patient and the patient selection criteria; e) when the toxicity prediction satisfies the patient selection criteria, generating a first indication and configuring the treatment plan to include the immunotherapy; f) when the toxicity prediction does not satisfy the patient selection criteria, generating a second indication and excluding the immunotherapy from the treatment plan; and g) outputting a medical order configuring the treatment plan for the patient based on the first indication or the second indication.

[0331] Example 35 includes the method of any preceding clause, further comprising iteratively evaluating the model by dividing the feature into inner and outer loops and comparing the loops according to the model selection criteria.

[0332] Example 36 includes the method of any of the preceding clauses, wherein the patient selection criterion is a first patient selection criterion, the balance is a first balance, and the output is a first output, and further comprising: loading an efficacy-related model associated with the efficacy of the immunotherapy; determining a second patient selection criterion to set a second balance between precision and recall of the action in response to a second output of the efficacy-related model, the second output comprising an efficacy prediction; using the efficacy-related model to process health care data from a health care record for a given patient to generate the efficacy prediction; and evaluating the efficacy prediction relative to the patient and the second patient selection criterion.

[0333] Example 37 includes the method described in any of the preceding clauses, further comprising: generating the first indication and configuring the treatment plan to include the immunotherapy when the toxicity prediction satisfies the first patient selection criteria and the efficacy prediction satisfies the second patient selection criteria; and otherwise, generating the second indication and excluding the immunotherapy from the treatment plan.

[0334] Example 38 includes the method of any of the preceding clauses, further comprising evaluating the toxicity prediction in comparison with the efficacy prediction for the given patient to generate the corresponding first or second indication, and configuring the treatment plan to include or exclude the immunotherapy, respectively.

[0335] Example 39 includes the method of any preceding clause, wherein the therapeutic efficacy is measured by patient survival rate or treatment duration.

[0336] Example 40 includes the method of any preceding clause, wherein configuring the treatment plan comprises configuring the treatment plan for a plurality of selected patients to construct a group for a clinical trial.

[0337] Example 41 includes the method of any preceding clause, wherein configuring the treatment plan further comprises scheduling application of the immunotherapy to the patient.

[0338] Example 42 includes the method of any preceding clause, wherein configuring the treatment plan further comprises providing patient monitoring to evaluate the patient during execution of the treatment plan.

[0339] Example 43 includes the method of any preceding clause, wherein the patient selection criteria comprises a threshold value defining a balance between false positives and false negatives for the toxicity-related model.

[0340] Example 44 includes the method of any preceding clause, wherein the selected toxicity is pneumonitis, colitis, or hepatitis, and wherein the toxicity prediction is i) the probability that the patient will suffer from the selected toxicity or ii) the probability that the patient will not suffer from the selected toxicity.

[0341] Example 45 includes the method of any preceding clause, wherein the healthcare data comprises at least one of a laboratory test result, a diagnosis code, or a billing code.

[0342] Example 46 includes the method of any preceding clause, further comprising aligning the healthcare data relative to a reference point, the reference point corresponding to an event in the healthcare record of the patient.

[0343] Example 47 includes the method of any preceding clause, further comprising re-evaluating the patient using updated healthcare data and an updated toxicity-related model for the patient.

[0344] Example 48 includes the method of any preceding clause, wherein the model selection criteria comprises at least one of a high F1 score, a high recall, or a high precision.

[0345] Example 49 includes the method of any preceding clause, further comprising deploying the toxicity-related model in a tool having an interface to facilitate collection of patient healthcare data and interaction with the toxicity-related model.

[0346] Example 50 is a method for configuring a treatment plan for a patient, the method comprising: a) generating an efficacy-related model for predicting the efficacy of an immune therapy by at least the following steps: i) training an initial model on a plurality of features generated from a plurality of prior health care data, the prior health care data being arranged in a time series and aligned relative to a reference point; ii) iteratively adding and removing each of a set of additional features to adjust the initial model into an adjusted model, and evaluating the resulting precision and recall associated with the output of the adjusted model relative to a model selection criterion; and iii) deploying the adjusted model as the efficacy-related model when the model selection criterion is met; b) deploying the adjusted model in response to the output of the efficacy-related model; out, determining patient selection criteria to set a balance between precision and recall for an action, the output comprising an efficacy prediction; c) using the efficacy-related model to process health care data from a health care record for a given patient to generate the efficacy prediction; d) evaluating the efficacy prediction relative to the patient and the patient selection criteria; e) when the efficacy prediction meets the patient selection criteria, generating a first indication and configuring the treatment plan to include the immunotherapy; f) when the efficacy prediction does not meet the patient selection criteria, generating a second indication and excluding the immunotherapy from the treatment plan; and g) outputting a medical order configuring the treatment plan for the patient based on the first indication or the second indication.

[0347] Example 51 is a method of configuring a group for a clinical trial, the method comprising: a) generating an efficacy-related model for predicting the efficacy of an immunotherapy by at least the following steps: i) training an initial model on a plurality of features generated from a plurality of prior health care data, the prior health care data being arranged in a time series and aligned relative to a reference point; ii) iteratively adding and removing each of a set of additional features to adjust the initial model into an adjusted model, and evaluating the resulting precision and recall associated with the output of the adjusted model relative to a model selection criterion; and iii) deploying the adjusted model as the efficacy-related model when the model selection criterion is met; b) deploying the adjusted model in response to the output of the efficacy-related model; out, determining patient selection criteria to set a balance between precision and recall for an action, the output comprising an efficacy prediction; c) using the efficacy-related model to process health care data from a health care record for a given patient to generate the efficacy prediction; d) evaluating the efficacy prediction relative to the patient and the patient selection criteria; e) when the efficacy prediction meets the patient selection criteria, generating a first indication and configuring the treatment plan to include the immunotherapy; f) when the efficacy prediction does not meet the patient selection criteria, generating a second indication and excluding the immunotherapy from the treatment plan; and g) outputting a medical order configuring the treatment plan for the patient based on the first indication or the second indication.

[0348] Example 52 includes the method of any preceding clause, further comprising iteratively evaluating the model by dividing the feature into inner and outer loops and comparing the loops according to the model selection criteria.

[0349] Example 53 includes the method of any preceding clause, wherein the patient selection criterion is a first patient selection criterion, the balance is a first balance, and the output is a first output, and further comprising: loading a toxicity-related model for a selected toxicity associated with the immunotherapy; determining a second patient selection criterion to set a second balance between precision and recall for an action in response to a second output of the toxicity-related model, the second output comprising a toxicity prediction; processing health care data from a health care record for a given patient using the toxicity-related model to generate the toxicity prediction; and evaluating the toxicity prediction relative to the patient and the second patient selection criterion.

[0350] Example 54 includes the method described in any of the preceding clauses, further comprising: generating the first indication and configuring the treatment plan to include the immunotherapy when the efficacy prediction satisfies the first patient selection criteria and the toxicity prediction satisfies the second patient selection criteria; and otherwise, generating the second indication and excluding the immunotherapy from the treatment plan.

[0351] Example 55 includes the method of any of the preceding clauses, further comprising evaluating the toxicity prediction in comparison with the efficacy prediction for the given patient to generate the corresponding first or second indication, and configuring the treatment plan to include or exclude the immunotherapy, respectively.

[0352] Example 56 includes the method of any preceding clause, wherein the selected toxicity is pneumonitis, colitis, or hepatitis, and wherein the toxicity prediction is i) the probability that the patient will suffer from the selected toxicity or ii) the probability that the patient will not suffer from the selected toxicity.

[0353] Example 57 includes the method of any preceding clause, wherein configuring the treatment plan comprises configuring the treatment plan for a plurality of selected patients to construct a group for a clinical trial.

[0354] Example 58 includes the method of any preceding clause, wherein configuring the treatment plan further comprises scheduling application of the immunotherapy to the patient.

[0355] Example 59 includes the method of any preceding clause, wherein configuring the treatment plan further comprises providing patient monitoring to evaluate the patient during execution of the treatment plan.

[0356] Example 60 includes the method of any preceding clause, wherein the patient selection criteria comprises a threshold value defining a balance between false positives and false negatives for the efficacy-related model.

[0357] Example 61 includes the method of any preceding clause, wherein the healthcare data comprises at least one of a laboratory test result, a diagnosis code, or a billing code.

[0358] Example 62 includes the method of any preceding clause, comprising aligning the healthcare data relative to a reference point, the reference point corresponding to an event in the healthcare record of the patient.

[0359] Example 63 includes the method of any preceding clause, further comprising re-evaluating the patient using updated healthcare data and updated outcome-related models for the patient.

[0360] Example 64 includes the method of any preceding clause, wherein the model selection criteria comprises at least one of a high F1 score, a high recall, or a high precision.

[0361] Example 65 includes the method of any of the preceding clauses, wherein the therapeutic efficacy is measured by patient survival rate or treatment duration.

[0362] Example 66 includes the method of any preceding clause, further comprising deploying the outcome-related model in a tool having an interface to facilitate collection of patient healthcare data and interaction with the outcome-related model.

[0363] Example 67 is a computer-readable medium comprising instructions that, when executed, cause a processor circuit system to implement the method of any preceding clause.

[0364] Example 68 is a device comprising: a memory; instructions; and a processor circuit system for implementing the method of any preceding clause.

[0365] As disclosed and described herein, the processor circuit system provides components for processing (e.g., including components for processing input data, components for training models, components for selection, components for deployment, components for generation, components for evaluation, components for triggering, components for placing medical orders, components for display, etc.) and the memory circuit system provides components for storage. As described above, various circuit systems can be implemented by the processor circuit system and by the memory circuit system.

[0366] The accompanying claims are hereby incorporated into this detailed description by this reference. Although certain example systems, methods, devices, and articles have been disclosed herein, the scope of coverage of this patent is not limited thereto. On the contrary, this patent covers all systems, methods, devices, and articles that fully fall within the scope of the claims of this patent.

Claims

1. A method of configuring a treatment plan for a patient, the method comprising: a) loading a toxicity-related model for a selected toxicity associated with an immunotherapy, the toxicity-related model being selected from a plurality of candidate models using a model selection criterion; b) determining a patient selection criterion to set a balance between precision and recall for an action in response to an output of the toxicity-related model, the output comprising a toxicity prediction; c) processing health care data from a health care record for a given patient using the toxicity-related model to generate the toxicity prediction; d) evaluating the toxicity prediction relative to the patient and the patient selection criteria; e) generating a first indication and configuring the treatment plan to include the immunotherapy when the toxicity prediction satisfies the patient selection criteria; f) generating a second indication and excluding the immunotherapy from the treatment plan when the toxicity prediction does not meet the patient selection criteria; as well as g) outputting a medical order configuring the treatment plan for the patient based on the first indication or the second indication.

2. The method of claim 1 , further comprising generating the toxicity-related model by forming features from an input data set obtained from health care records, the features being iteratively compared by correlation results to form the model, the model being adjusted based on an analysis of precision and recall.

3. A method according to any preceding claim, wherein: The patient selection criterion is a first patient selection criterion, the balance is a first balance, and the output is a first output, and further comprising: loading an efficacy-related model associated with the efficacy of the immunotherapy; determining a second patient selection criterion to set a second balance between precision and recall for the action in response to a second output of the efficacy-related model, the second output comprising a prediction of efficacy; processing health care data from a health care record for a given patient using the outcome-related model to generate the outcome prediction; and The prediction of efficacy is evaluated relative to the patient and the second patient selection criteria.

4. The method according to claim 3, further comprising: generating the first indication and configuring the treatment plan to include the immunotherapy when the toxicity prediction satisfies the first patient selection criterion and the efficacy prediction satisfies the second patient selection criterion; as well as Otherwise, the second indication is generated and the immunotherapy is excluded from the treatment plan.

5. The method of claim 4, further comprising evaluating the toxicity prediction in comparison with the efficacy prediction for the given patient to generate a corresponding first or second indication and configuring the treatment plan to include or exclude the immunotherapy, respectively.

6. The method according to claim 3, wherein: The efficacy is measured by patient survival rate or treatment duration.

7. A method according to any preceding claim, wherein: Configuring the treatment plan includes configuring the treatment plan for a plurality of selected patients to establish a cohort for a clinical trial.

8. A method according to any preceding claim, wherein: Configuring the treatment plan also includes determining a schedule for applying the immunotherapy to the patient.

9. A method according to any preceding claim, wherein: Configuring the treatment plan also includes providing patient monitoring to evaluate the patient during execution of the treatment plan.

10. A method according to any preceding claim, wherein: The patient selection criteria include a threshold that defines a balance between false positives and false negatives for the toxicity correlation model.

11. A method according to any preceding claim, wherein: The selected toxicity is pneumonitis, colitis, or hepatitis, and wherein the toxicity prediction is i) the probability that the patient will suffer from the selected toxicity or ii) the probability that the patient will not suffer from the selected toxicity.

12. A method according to any preceding claim, wherein: The healthcare data includes at least one of a laboratory test result, a diagnosis code, or a billing code.

13. The method of claim 10, further comprising aligning the healthcare data relative to a reference point corresponding to an event in the medical record for the patient.

14. The method of any preceding claim, further comprising re-evaluating the patient using updated healthcare data and an updated toxicity-related model for the patient.

15. A method according to any preceding claim, wherein: The model selection criteria includes at least one of a high F1 score, a high recall, or a high precision.

16. A computer readable medium comprising instructions which, when executed, cause processor circuitry to implement the method of any preceding claim.

17. A device comprising: Memory; instruction; as well as Processor circuitry for implementing the method of any of claims 1-15.

18. A method of configuring a treatment plan for a patient, the method comprising: a) loading a efficacy-related model associated with the efficacy of immunotherapy, wherein the efficacy-related model is selected from a plurality of candidate models using a model selection criterion; b) determining a patient selection criterion to set a balance between precision and recall for an action in response to an output of the efficacy-related model, the output comprising an efficacy prediction; c) processing health care data from a health care record for a given patient using the outcome-related model to generate the outcome prediction; d) evaluating the prediction of efficacy relative to the patient and the patient selection criteria; e) generating a first indication and configuring the treatment plan to include the immunotherapy when the efficacy prediction satisfies the patient selection criteria; f) when the efficacy prediction does not meet the patient selection criteria, generating a second indication and excluding the immunotherapy from the treatment plan; as well as g) outputting a medical order configuring the treatment plan for the patient based on the first indication or the second indication.

19. The method of claim 18, further comprising generating the efficacy-related model by forming features from an input data set obtained from health care records, the features being iteratively compared through associated results to form the model, the model being adjusted based on an analysis of precision and recall.

20. The method according to any of claims 18-19, wherein: The patient selection criterion is a first patient selection criterion, the balance is a first balance, and the output is a first output, and further comprising: loading a toxicity-related model for a selected toxicity associated with the immunotherapy; determining a second patient selection criterion to set a second balance between precision and recall for the action in response to a second output of the toxicity-related model, the second output comprising a toxicity prediction; processing health care data from a health care record for a given patient using the toxicity-related model to generate the toxicity prediction; and The toxicity prediction is evaluated relative to the patient and the second patient selection criteria.

21. The method according to claim 20, further comprising: generating the first indication and configuring the treatment plan to include the immunotherapy when the efficacy prediction satisfies the first patient selection criterion and the toxicity prediction satisfies the second patient selection criterion; as well as Otherwise, the second indication is generated and the immunotherapy is excluded from the treatment plan.

22. The method of claim 21, further comprising evaluating the toxicity prediction in comparison to the efficacy prediction for the given patient to generate a corresponding first or second indication and configuring the treatment plan to include or exclude the immunotherapy, respectively.

23. The method according to claim 21, wherein: The selected toxicity is pneumonitis, colitis, or hepatitis, and wherein the toxicity prediction is i) the probability that the patient will suffer from the selected toxicity or ii) the probability that the patient will not suffer from the selected toxicity.

24. The method according to any of claims 18 to 23, wherein: Configuring the treatment plan includes configuring the treatment plan for a plurality of selected patients to construct a cohort for a clinical trial.

25. The method according to any of claims 18 to 24, wherein: Configuring the treatment plan also includes determining a schedule for applying the immunotherapy to the patient.

26. The method according to any of claims 18 to 25, wherein: Configuring the treatment plan also includes providing patient monitoring to evaluate the patient during execution of the treatment plan.

27. The method according to any of claims 18 to 26, wherein: The patient selection criteria include a threshold value that defines a balance between false positives and false negatives for the efficacy correlation model.

28. The method according to any of claims 18-27, wherein: The healthcare data includes at least one of a laboratory test result, a diagnosis code, or a billing code.

29. The method of claim 28, further comprising aligning the healthcare data relative to a reference point corresponding to an event in the medical record of the patient.

30. The method of any of claims 18-29, further comprising re-evaluating the patient using updated healthcare data and updated outcome-related models for the patient.

31. The method according to any of claims 18-20, wherein: The model selection criteria includes at least one of a high F1 score, a high recall, or a high precision.

32. The method according to any of claims 18 to 31, wherein: The efficacy is measured by patient survival rate or treatment duration.

33. A computer readable medium comprising instructions which, when executed, cause processor circuitry to implement the method of any of claims 18-32.

34. An apparatus comprising: Memory; instruction; as well as Processor circuitry for implementing the method of any of claims 18-32.

35. A method of configuring a treatment plan for a patient, the method comprising: a) generating a toxicity-related model for a selected toxicity associated with the immunotherapy by at least the following steps: i) training an initial model on a plurality of features generated from a plurality of previous health care data, the previous health care data being arranged in a time series and aligned with respect to a reference point; ii) iteratively adding and removing each of a set of additional features to tune the initial model into an adjusted model, and evaluating a resulting precision and recall associated with an output of the adjusted model relative to a model selection criterion; and iii) deploying the adjusted model as the toxicity-related model when the model selection criteria are met; b) determining a patient selection criterion to set a balance between precision and recall for an action in response to an output of the toxicity-related model, the output comprising a toxicity prediction; c) processing health care data from a health care record for a given patient using the toxicity-related model to generate the toxicity prediction; d) evaluating the toxicity prediction relative to the patient and the patient selection criteria; e) generating a first indication and configuring the treatment plan to include the immunotherapy when the toxicity prediction satisfies the patient selection criteria; f) generating a second indication and excluding the immunotherapy from the treatment plan when the toxicity prediction does not meet the patient selection criteria; as well as g) outputting a medical order configuring the treatment plan for the patient based on the first indication or the second indication.

36. The method of claim 35, further comprising iteratively evaluating the model by dividing the feature into inner and outer loops and comparing the loops according to the model selection criteria.

37. The method according to any of claims 35-36, wherein: The patient selection criterion is a first patient selection criterion, the balance is a first balance, and the output is a first output, and further comprising: loading an efficacy-related model associated with the efficacy of the immunotherapy; determining a second patient selection criterion to set a second balance between precision and recall for the action in response to a second output of the efficacy-related model, the second output comprising a prediction of efficacy; processing health care data from a health care record for a given patient using the outcome-related model to generate the outcome prediction; and The prediction of efficacy is evaluated relative to the patient and the second patient selection criteria.

38. The method of claim 37, further comprising: generating the first indication and configuring the treatment plan to include the immunotherapy when the toxicity prediction satisfies the first patient selection criterion and the efficacy prediction satisfies the second patient selection criterion; as well as Otherwise, the second indication is generated and the immunotherapy is excluded from the treatment plan.

39. The method of claim 38, further comprising evaluating the toxicity prediction in comparison to the efficacy prediction for the given patient to generate a corresponding first or second indication and configuring the treatment plan to include or exclude the immunotherapy, respectively.

40. The method according to any of claims 37-39, wherein: The efficacy is measured by patient survival rate or treatment duration.

41. The method according to any of claims 35-40, wherein: Configuring the treatment plan includes configuring the treatment plan for a plurality of selected patients to construct a cohort for a clinical trial.

42. The method according to any of claims 35-41, wherein: Configuring the treatment plan also includes scheduling application of the immunotherapy to the patient.

43. The method according to any of claims 35-42, wherein: Configuring the treatment plan also includes providing patient monitoring to evaluate the patient during execution of the treatment plan.

44. The method according to any of claims 35-43, wherein: The patient selection criteria include a threshold that defines a balance between false positives and false negatives for the toxicity correlation model.

45. The method according to any of claims 35-44, wherein: The selected toxicity is pneumonitis, colitis, or hepatitis, and wherein the toxicity prediction is i) the probability that the patient will suffer from the selected toxicity or ii) the probability that the patient will not suffer from the selected toxicity.

46. ​​The method according to any of claims 35-45, wherein: The healthcare data includes at least one of a laboratory test result, a diagnosis code, or a billing code.

47. The method of any of claims 35-46, further comprising aligning the health care data relative to a reference point corresponding to an event in the health care record of the patient.

48. The method of any of claims 35-47, further comprising re-evaluating the patient using updated healthcare data and an updated toxicity-related model for the patient.

49. The method according to any of claims 35-48, wherein: The model selection criteria includes at least one of a high F1 score, a high recall, or a high precision.

50. The method of any of claims 35-49, further comprising deploying the toxicity-related model in a tool having an interface to facilitate collection of patient healthcare data and interaction with the toxicity-related model.

51. A computer readable medium comprising instructions which, when executed, cause a processor circuit to implement the method of any of claims 35-50.

52. An apparatus comprising: Memory; instruction; as well as Processor circuitry for implementing the method of any of claims 35-50.

53. A method of configuring a treatment plan for a patient, the method comprising: a) generating an efficacy-related model for predicting the efficacy of immunotherapy by at least the following steps: i) training an initial model on a plurality of features generated from a plurality of previous health care data, the previous health care data being arranged in a time series and aligned with respect to a reference point; ii) iteratively adding and removing each of a set of additional features to tune the initial model into a tuned model, and evaluating a resulting precision and recall associated with an output of the tuned model relative to a model selection criterion; as well as iii) deploying the adjusted model as the efficacy-related model when the model selection criteria are met; b) determining a patient selection criterion to set a balance between precision and recall for an action in response to an output of the efficacy-related model, the output comprising an efficacy prediction; c) processing health care data from a health care record for a given patient using the outcome-related model to generate the outcome prediction; d) evaluating the prediction of efficacy relative to the patient and the patient selection criteria; e) generating a first indication and configuring the treatment plan to include the immunotherapy when the efficacy prediction satisfies the patient selection criteria; f) when the efficacy prediction does not meet the patient selection criteria, generating a second indication and excluding the immunotherapy from the treatment plan; as well as g) outputting a medical order for configuring the treatment plan for the patient based on the first indication or the second indication.

54. The method of claim 53, further comprising iteratively evaluating the model by dividing the feature into inner and outer loops and comparing the loops according to the model selection criteria.

55. The method according to any of claims 53-54, wherein: The patient selection criterion is a first patient selection criterion, the balance is a first balance, and the output is a first output, and further comprising: loading a toxicity-related model for a selected toxicity associated with the immunotherapy; determining a second patient selection criterion to set a second balance between precision and recall for the action in response to a second output of the toxicity-related model, the second output comprising a toxicity prediction; processing health care data from a health care record for a given patient using the toxicity-related model to generate the toxicity prediction; and The toxicity prediction is evaluated relative to the patient and the second patient selection criteria.

56. The method of claim 55, further comprising: generating the first indication and configuring the treatment plan to include the immunotherapy when the efficacy prediction satisfies the first patient selection criterion and the toxicity prediction satisfies the second patient selection criterion; as well as Otherwise, the second indication is generated and the immunotherapy is excluded from the treatment plan.

57. The method of claim 56, further comprising evaluating the toxicity prediction in comparison with the efficacy prediction for the given patient to generate a corresponding first or second indication and respectively configuring the treatment plan to include or exclude the immunotherapy.

58. The method of claim 57, wherein: The selected toxicity is pneumonitis, colitis, or hepatitis, and wherein the toxicity prediction is i) the probability that the patient will suffer from the selected toxicity or ii) the probability that the patient will not suffer from the selected toxicity.

59. The method according to any of claims 53-58, wherein: Configuring the treatment plan includes configuring the treatment plan for a plurality of selected patients to construct a cohort for a clinical trial.

60. The method according to any of claims 53-59, wherein: Configuring the treatment plan also includes scheduling application of the immunotherapy to the patient.

61. The method according to any of claims 53-60, wherein: Configuring the treatment plan also includes providing patient monitoring to evaluate the patient during execution of the treatment plan.

62. The method according to any of claims 53-61, wherein: The patient selection criteria include a threshold value that defines a balance between false positives and false negatives for the efficacy correlation model.

63. The method according to any of claims 53-62, wherein: The healthcare data includes at least one of a laboratory test result, a diagnosis code, or a billing code.

64. The method of claim 63, further comprising aligning the health care data relative to a reference point corresponding to an event in the health care record of the patient.

65. The method according to any of claims 53-64, further comprising re-evaluating the patient using updated medical care data and updated outcome-related models for the patient.

66. The method according to any of claims 53-65, wherein: The model selection criteria includes at least one of a high F1 score, a high recall, or a high precision.

67. The method according to any of claims 53-66, wherein: The efficacy is measured by patient survival rate or treatment duration.

68. The method of any of claims 53-67, further comprising deploying the efficacy-related model in a tool having an interface to facilitate collection of patient healthcare data and interaction with the efficacy-related model.

69. A computer readable medium comprising instructions which, when executed, cause processor circuitry to implement the method of any of claims 53-68.

70. An apparatus comprising: Memory; instruction; as well as Processor circuitry for implementing the method of any of claims 53-68.

Images