Model generation device for treatment prediction, related methods, and models

Abstract

Methods, devices, systems, and articles of manufacture are disclosed that aim to generate and apply models for treatment prediction and treatment. Precision and recall can be balanced for at least one of toxicity-related or efficacy-related models to configure immunotherapy treatment for patients and / or patient cohorts. Model outputs can be evaluated in different ways and trigger different actions depending on determined patient selection criteria.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This patent claims the benefit of priority to U.S. Provisional Application No. 63 / 335,215, filed April 26, 2022, International Application No. PCT / US2022 / 032075, filed June 3, 2022, and International Application No. PCT / US2022 / 032084, filed June 3, 2022, each of which is incorporated by reference in its entirety for all purposes.

[0002] The present disclosure relates generally to model generation and processing, and more particularly to the generation and application of models for treatment prediction and processing. [Background technology]

[0003] The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.

[0004] Immunotherapy may be used to provide effective treatment of cancer for some patients. For those patients, immunotherapy may provide higher efficacy and lower toxicity than other therapies. Immunotherapy includes targeted antibodies and immune checkpoint inhibitors (ICIs), cell-based immunotherapies, immunomodulators, vaccines, and viral therapies, and can help the patient's immune system target and destroy malignant tumors. However, for some patients, immunotherapy may cause toxicity and / or other side effects. Side effects of immunotherapy may differ from those associated with other cancer treatments because side effects are the result of an overstimulated or misdirected immune response, rather than the direct action of chemo- or radiotherapy on cancer and healthy tissues. Toxicity of immunotherapy may include illnesses such as colitis, hepatitis, pneumonia, and / or other inflammation that may pose a risk to the patient. Immunotherapy also causes different (heterogeneous) efficacy responses in different patients. Even patients with effective responses may be affected by toxicity. As such, the evaluation of immunotherapy may be highly variable between patients and remains unpredictable. Summary of the Invention [Means for solving the problem]

[0005] An embodiment of the present disclosure provides a method for formulating a treatment plan for a patient, the method comprising: a) loading a toxicity-related model for a selected toxicity associated with the immunotherapy, the toxicity-related model being selected from a plurality of candidate models using model selection criteria; b) determining patient selection criteria to set a balance between precision and recall for actions in response to an output of a toxicity-related model, the output including a toxicity prediction; c) processing healthcare data from a healthcare record for a given patient using the toxicity-associated model to generate a toxicity prediction; d) evaluating the toxicity predictions with respect to patients and patient selection criteria; e) generating a first prescription and configuring a treatment plan to include immunotherapy when the toxicity prediction meets the patient selection criteria; f) generating a second instruction to exclude immunotherapy from the treatment plan when the toxicity prediction does not meet the patient selection criteria; and g) outputting an order constituting a treatment plan for the patient based on the first instruction or the second instruction.

[0006] Another embodiment of the present disclosure provides a method for formulating a treatment plan for a patient, the method comprising: a) loading an efficacy-related model associated with efficacy of an immunotherapy, the efficacy-related model being selected from a plurality of candidate models using a model selection criterion; b) determining patient selection criteria to set a balance between precision and recall for actions responsive to outputs of efficacy-related models, the outputs including efficacy predictions; c) processing health care data from a health care record for a given patient using the efficacy-related model to generate an efficacy prediction; d) evaluating the efficacy predictions with respect to patients and patient selection criteria; e) generating a first prescription and configuring a treatment plan to include immunotherapy when the efficacy prediction meets the patient selection criteria; f) generating a second instruction to exclude immunotherapy from the treatment plan when the efficacy prediction does not meet the patient selection criteria; and g) outputting an order constituting a treatment plan for the patient based on the first instruction or the second instruction.

[0007] Another embodiment of the present disclosure provides a method for formulating a treatment plan for a patient, the method comprising: a) generating a toxicity-related model for a selected toxicity associated with an immunotherapy, comprising at least i) training an initial model on a plurality of features generated from a plurality of prior healthcare data, the prior healthcare data being arranged in a time series and aligned with respect to a reference point; ii) iteratively adding and removing each of the set of additional features to refine the initial model into a refined model, and evaluating the resulting precision and recall associated with the output of the refined model with respect to the model selection criteria; iii) generating the adjusted model by deploying it as a toxicity-relevant model when the model selection criteria are met; b) determining patient selection criteria to set a balance between precision and recall for actions in response to an output of a toxicity-related model, the output including a toxicity prediction; c) processing healthcare data from a healthcare record for a given patient using the toxicity-associated model to generate a toxicity prediction; d) evaluating the toxicity predictions with respect to patients and patient selection criteria; e) generating a first prescription and configuring a treatment plan to include immunotherapy when the toxicity prediction meets the patient selection criteria; f) generating a second instruction to exclude immunotherapy from the treatment plan when the toxicity prediction does not meet the patient selection criteria; and g) outputting an order constituting a treatment plan for the patient based on the first instruction or the second instruction.

[0008] Another embodiment of the present disclosure provides a method for formulating a treatment plan for a patient, the method comprising: a) generating an efficacy-related model predicting efficacy of an immunotherapy, comprising at least i) training an initial model on a plurality of features generated from a plurality of prior healthcare data, the prior healthcare data being arranged in a time series and aligned with respect to a reference point; ii) iteratively adding and removing each of the set of additional features to refine the initial model into a refined model, and evaluating the resulting precision and recall associated with the output of the refined model with respect to the model selection criteria; iii) generating the adjusted model by deploying it as an efficacy-related model when the model selection criteria are met; b) determining patient selection criteria to set a balance between precision and recall for actions responsive to outputs of efficacy-related models, the outputs including efficacy predictions; c) processing health care data from a health care record for a given patient using the efficacy-related model to generate an efficacy prediction; d) evaluating the efficacy predictions with respect to patients and patient selection criteria; e) generating a first prescription and configuring a treatment plan to include immunotherapy when the efficacy prediction meets the patient selection criteria; f) generating a second instruction to exclude immunotherapy from the treatment plan when the efficacy prediction does not meet the patient selection criteria; and g) outputting an order constituting a treatment plan for the patient based on the first instruction or the second instruction. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an exemplary model generator. [Diagram 2] 2 is a flow diagram illustrating the execution of instructions for driving operations using the example model generator of FIG. 1. [Diagram 3] 2 is a flow diagram illustrating the execution of instructions for driving operations using the example model generator of FIG. 1. [Figure 4] 2 is a flow diagram illustrating the execution of instructions for driving operations using the example model generator of FIG. 1. [Diagram 5] 2 is a flow diagram illustrating the execution of instructions for driving operations using the example model generator of FIG. 1. [Figure 6] 2 is a flow diagram illustrating the execution of instructions for driving operations using the example model generator of FIG. 1. [Figure 7] 2 is a flow diagram illustrating the execution of instructions for driving operations using the example model generator of FIG. 1. [Figure 8] 2 is a flow diagram illustrating the execution of instructions for driving operations using the example model generator of FIG. 1. [Figure 9] 2 is a flow diagram illustrating the execution of instructions for driving operations using the example model generator of FIG. 1. [Figure 10] 2 is a flow diagram illustrating the execution of instructions for driving operations using the example model generator of FIG. 1. [Figure 11] 2 is a flow diagram illustrating the execution of instructions for driving operations using the example model generator of FIG. 1. [Figure 12] 2 is a flow diagram illustrating the execution of instructions for driving operations using the example model generator of FIG. 1. [Figure 13] 2 is a flow diagram illustrating the execution of instructions for driving operations using the example model generator of FIG. 1. [Figure 14] 2 is a flow diagram illustrating the execution of instructions for driving operations using the example model generator of FIG. 1. [Figure 15] FIG. 1 illustrates an exemplary immunotherapy prediction device. [Figure 16] 16 is a flow diagram illustrating an example method for processing data using one or more models in accordance with the example immunotherapy prediction device of FIG. 15. [Figure 17] 16 is a flow diagram illustrating an example method for processing data using one or more models in accordance with the example immunotherapy prediction device of FIG. 15. [Figure 18] 16 is a flow diagram illustrating an example method for processing data using one or more models in accordance with the example immunotherapy prediction device of FIG. 15. [Figure 19]FIG. 1 illustrates an exemplary timeline from data aggregation and associated feature generation to model development for the first infusion of immunotherapy treatment. [Figure 20] FIG. 13 shows an exemplary precision / recall graph illustrating the performance of an exemplary model. [Figure 21] FIG. 13 shows an exemplary precision / recall graph illustrating the performance of an exemplary model. [Figure 22] FIG. 13 shows an exemplary precision / recall graph illustrating the performance of an exemplary model. [Diagram 23] FIG. 13 shows an exemplary precision / recall graph illustrating the performance of an exemplary model. [Figure 24] FIG. 13 shows an exemplary precision / recall graph illustrating the performance of an exemplary model. [Diagram 25] FIG. 1 illustrates an example infrastructure and organization of patient health data from patient health records into features that can be used to train, test, and validate models. [Figure 26] 1 is a block diagram of an example processing platform including processor circuitry structured to execute example machine-readable instructions and / or example operations. [Figure 27] FIG. 27 is a block diagram of an exemplary implementation of the processor circuit of FIG. 26. [Figure 28] FIG. 27 is a block diagram of another exemplary implementation of the processor circuit of FIG. 26. [Figure 29] FIG. 1 is a block diagram of an example software distribution platform (e.g., one or more servers) for distributing software (e.g., software corresponding to example machine-readable instructions) to client devices associated with end users and / or consumers (e.g., for license, sale, and / or use), retailers (e.g., for sale, resale, license, and / or sublicense), and / or original equipment manufacturers (OEMs) (e.g., for inclusion in products distributed to retailers and / or other end users, such as direct purchase customers). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] In the following detailed description, reference is made to the accompanying drawings, which form a part hereof, in which are shown by way of illustration specific examples that may be implemented. These examples are described in sufficient detail to enable one skilled in the art to implement the subject matter, but it will be understood that other examples may be utilized, and that logical, mechanical, electrical, and other changes may be made without departing from the scope of the subject matter of this disclosure. Thus, the detailed description set forth below is provided to illustrate exemplary implementations and should not be considered as limiting the scope of the subject matter described in this disclosure. Some features from different aspects of the following description may be combined to form further novel aspects of the subject matter described below.

[0011] Generally, the same reference numbers are used throughout the drawings and the accompanying written specification to refer to the same or like parts. The figures are not to scale.

[0012] definition Throughout this specification and the claims, the following terms have the meanings expressly associated therewith in this specification, unless the context clearly dictates otherwise.

[0013] As used in this patent, connection references (e.g., attached, coupled, connected, and coupled) may include intermediate members between the elements referenced by the connection reference, and / or relative movement between those elements, unless otherwise specified. As such, a connection reference does not necessarily infer that the two elements are directly connected and / or in fixed relationship to one another. As used herein, describing any part as "in contact with" another part is defined to mean that there are no intermediate parts between the two parts.

[0014] When introducing elements of various embodiments of the disclosure, the words "a" (or "an"), "the" (as used in the claims) (or "the"), and "said" are intended to mean that there are one or more of those elements. As used herein, singular references (e.g., "a", "an", "first", "second", etc.) do not exclude a plurality. As used herein, "a" or "an" object refers to one or more of that object. The phrases "a" or "an", "one or more", and "at least one" are used interchangeably herein. Furthermore, although individually listed, multiple means, elements or method actions may be performed by, for example, the same entity or object. In addition, although individual features may be included in different examples or claims, these may in some cases be combined, and inclusion in different examples or claims does not imply that a combination of features is not feasible and / or advantageous.

[0015] The terms "comprising," "including," and "having" are intended to be inclusive, meaning that there may be additional elements other than the recited elements. That is, "comprising" and "comprising" (and all forms and tenses thereof) are used herein as open-ended terms. Thus, whenever a claim employs any form of "comprising" or "comprising" (e.g., comprising, including, comprising, including, having, etc.) as a preamble or within any type of claim recitation, it is to be understood that there may be additional elements, terms, etc. without departing from the scope of the corresponding claim or recitation. In this specification, when the term "at least" is used as a transition term, for example in the preamble of a claim, it is open-ended in the same way that the terms "comprising" and "comprising" are open-ended. The term "and / or," when used in the form, for example, A, B, and / or C, refers to any combination or subset of A, B, C, such as: (1) A alone, (2) B alone, (3) C alone, (4) A in combination with B, (5) A in combination with C, (6) B in combination with C, or (7) A in combination with B and C. As used herein in the context of describing structures, components, items, objects, and / or things, the phrase "at least one of A and B" is intended to refer to implementations that include either (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in the context of describing structures, components, items, objects, and / or things, the phrase “at least one of A or B” is intended to refer to an implementation that includes either (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.As used herein in the context of describing the implementation or performance of a process, instruction, operation, activity, and / or step, the phrase "at least one of A and B" is intended to refer to an implementation that includes either (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in the context of describing the implementation or performance of a process, instruction, operation, activity, and / or step, the phrase "at least one of A and B" is intended to refer to an implementation that includes either (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.

[0016] Unless otherwise noted, descriptors such as "first," "second," "third," and the like are used herein in no way to imply or otherwise indicate priority, physical order, arrangement within a list, and / or ordering, but are used merely as labels and / or arbitrary names to distinguish elements to facilitate understanding of the disclosed examples. In some instances, the descriptor "first" may be used to refer to an element in the detailed description, while in the claims the same element may be referred to using a different descriptor, such as "second" or "third." In such cases, it should be understood that such descriptors are used only to distinguish and identify elements that may otherwise share the same name, for example.

[0017] As used herein, "nearly" and "about" modify their subject / values ​​to recognize the potential existence of variations that occur in real-world applications. For example, "nearly" and "about" may modify dimensions that may not be exact due to manufacturing tolerances and / or other real-world imperfections, as will be understood by those skilled in the art. For example, "nearly" and "about" may indicate that such dimensions may be within a tolerance range of ±10% unless otherwise specified in the following description. "Substantially real-time" as used herein refers to occurring in an almost instantaneous manner recognizing that there may be real-world delays due to computation time, transmission, and the like. Thus, unless otherwise specified, "substantially real-time" refers to real time ±1 second.

[0018] As used herein, the phrase "communicating," including variations of that phrase, encompasses direct communication and / or indirect communication through one or more intermediary components and does not require direct physical (e.g., wired) communication and / or constant communication, but rather, in addition, includes selective communication at regular intervals, scheduled intervals, non-regular intervals, and / or one-time events.

[0019] As used herein, terms such as "system," "unit," "module," "engine," and the like may include hardware and / or software systems that operate to perform one or more functions. For example, a module, unit, or system may include a computer processor, controller, and / or other logic-based device that performs operations based on instructions stored in a tangible, non-transitory computer-readable storage medium, such as a computer memory. Alternatively, a module, unit, engine, or system may include a hardwired device that performs operations based on the device's hardwired logic. The various modules, units, engines, and / or systems illustrated in the accompanying figures may represent hardware that operates based on software or hardwired instructions, software that instructs hardware to perform operations, or a combination thereof.

[0020] As used herein, a "processor circuit" is defined to include (i) one or more special-purpose electrical circuits that are structured to perform specific operations and include one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors), and / or (ii) one or more general-purpose semiconductor-based electrical circuits that are programmable with instructions to perform specific operations and include one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors). Examples of processor circuits include integrated circuits such as programmable microprocessors, field programmable gate arrays (FPGAs) that may instantiate instructions, central processing units (CPUs), graphics processor units (GPUs), digital signal processors (DSPs), XPUs, or microcontrollers, and application specific integrated circuits (ASICs). For example, an XPU may be implemented by a heterogeneous computing system that includes multiple types of processor circuitry (e.g., one or more FPGAs, one or more CPUs, one or more GPUs, one or more DSPs, etc., and / or combinations thereof) and an application programming interface (API) that may assign computing tasks to a processor circuit of the multiple types of processor circuitry that is best suited to perform the computing task.

[0021] Additionally, it should be understood that references to "one embodiment" or "embodiments" in the present disclosure are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features.

[0022] As used herein, a "toxicity-related model" (also referred to herein as a toxicity model) is a machine learning and / or other artificial intelligence model that processes input data to make a prediction of toxicity in a patient and / or model another surrogate or proxy characteristic that correlates with toxicity associated with immunotherapy (e.g., risk of suffering from a side effect or illness associated with immunotherapy treatment, such as pneumonia, colitis, hepatitis, etc.). An "efficacy-related model" (also referred to herein as an efficacy model) is a machine learning and / or other artificial intelligence model that processes input data to make a prediction of efficacy in a patient and / or model another surrogate or proxy characteristic that correlates with efficacy or effectiveness of immunotherapy (e.g., likelihood that immunotherapy will be effective, such as duration of treatment, overall survival, etc.).

[0023] A "feature" or feature variable represents an attribute or type of data. Features can be expressed in a variety of forms, such as categorical, numerical, date, percentage, etc.

[0024] "Precision" refers to the proportion or percentage of results in which the output of an artificial intelligence model was actually correct. For example, a model may output predictions and a group of model output predictions may be evaluated to determine the precision of the model. Precision may be evaluated, for example, on a scale of 0.0 to 1.0, where 0.0 indicates that the model's prediction was never a correct prediction and 1.0 indicates that the model's prediction was always correct. Precision is defined in terms of true positives (TP) and false positives (FP). A true positive is a prediction that was a correct prediction and a false positive is a prediction that was incorrect. The precision of the model may then be determined by the formula Precision = TP / (TP + FP).

[0025] "Recall" refers to the proportion of true positives (actual positives) that are correctly identified. Recall may also be evaluated, for example, on a scale from 0.0 to 1.0, where 1.0 indicates that the model never produced a false negative and 0.0 indicates that the model always produced a false negative. Recall is defined in terms of true positives and false negatives (FN) as Recall = TP / (TP + FN).

[0026] The term "deep learning" is a machine learning technique that utilizes multiple data processing layers to recognize various structures in a dataset and classify the dataset with high accuracy. A deep learning network (DLN), also referred to as a deep neural network (DNN), can be a trained network (e.g., a trained network model or device) that learns patterns based on multiple inputs and outputs. A deep learning network / deep neural network can be a deployed network (e.g., a deployed network model or device) that is generated from a training network and provides outputs in response to inputs.

[0027] The term "supervised learning" refers to machine learning training methods where the machine is provided with data that has already been labeled from a human source. The term "unsupervised learning" refers to machine learning training methods (e.g., random forests, gradient boosting, etc.) where the machine is not given data that has already been labeled, making the machine useful for anomaly detection. The term "semi-supervised learning" refers to machine learning training methods where the machine is provided with a small amount of labeled data from a human source compared to the large amount of unlabeled data available to the machine.

[0028] The term "convolutional neural network" or "CNN" is a biologically inspired network of interconnected data used in deep learning for the detection, segmentation, and recognition of suitable objects and regions in a dataset. CNNs evaluate raw data in the form of multiple sequences, split the data into a series of stages, and examine the data for learned features. Hepatitis and / or toxicity may be predicted using CNNs, for example.

[0029] The term "recurrent neural network" or "RNN" refers to a network in which the connections between nodes form a directed or undirected graph over time. Hepatitis and / or toxicity can be predicted, for example, using an RNN.

[0030] The term "transfer learning" refers to the process in which a machine remembers information used in solving one problem, either properly or improperly, to solve another problem that has the same or similar properties as the first problem. Transfer learning may also be referred to as "inductive learning." Transfer learning can, for example, utilize data from a previous task.

[0031] The term "active learning" refers to a process in machine learning in which a machine selects a set of examples from which to receive training data, rather than passively receiving examples selected by an external entity. For example, as the machine learns, it may be enabled to select the examples that it determines will be most useful for learning, rather than relying solely on an external human expert or external system to identify and provide the examples.

[0032] The terms "computer-aided detection" or "computer-aided diagnosis" refer to a computer analyzing medical data to suggest possible diagnoses.

[0033] Exemplary Model Generation Systems and Related Methods Large amounts of health-related data may be collected using various media and mechanisms on patients. However, processing and interpreting the data to derive actionable results may be difficult. For example, understanding and correlating various forms and sources of data through standardization / normalization, aggregation, and analysis may be difficult, if not impossible, given the magnitude of the data and the various disparate systems, formats, etc. As such, some examples feature apparatus, systems, associated models, and methods for processing and correlating health-related data to predict patient outcomes and drive patient diagnosis and treatment. Some examples provide systems and methods for health data predictive model building. Some examples provide frameworks and machine learning workflows for treatment prediction.

[0034] For example, immune checkpoints regulate the human immune system. Immune checkpoints are pathways that allow the body to become self-tolerant by preventing the immune system from attacking cells indiscriminately. However, some cancers can defend themselves against attacks by stimulating immune checkpoints (e.g., proteins on immune cells). To target cancer cells in the body, immune checkpoint inhibitors (ICIs) can be used to target these immune checkpoint proteins to better identify and attack cancer cells rather than hiding them.

[0035] Despite the great success of cancer treatment with ICIs, such treatments can pose a major threat to human health due to side effects, a type of immune-related adverse events (irAEs) caused by these treatment options. One of these toxicities is hepatitis, which occurs when the liver is affected by an autoimmune-like inflammatory pathological process induced by ICIs. Some examples predict the development of irAE hepatitis before the initiation of the first ICI treatment. More precisely, some examples predict whether irAE hepatitis will occur within a given time frame after the initiation of the first treatment. Other toxicities, such as pneumonia, colitis, etc., can be predicted as well.

[0036] For example, majority class undersampling may be combined with time series data aggregation to obtain a balanced static dataset that can be fed to the model. Exemplary models include gradient boosting (GB) and random forest (RF), and / or other models that can accommodate the size and statistical characteristics of the data. The model may be selected based on a model selection criterion, such as F1 score, which is a measure of the accuracy of the model on the dataset. For example, a GB model without undersampling may maximize the F1 score (e.g., the harmonic mean of recall and precision), and an RF model with undersampling may provide a high recall (e.g., the ratio of true positives found) with a relatively low precision (e.g., the ratio of true positives among estimates), which is acceptable due to the cost-effectiveness of the additional testing required based on the model's decision. These models may also produce probability estimates for the labels, rather than just the discrete labels themselves.

[0037] The input data is prepared to develop and / or feed models that drive predictions, treatments, etc. In some examples, the input is prepared by extracting blood feature information (e.g., relevant sections of blood features, etc.) from electronic health record (EHR) data tables (e.g., received at a processor / processing circuit from an EHR, etc.), electronic medical record (EMR) information, etc. The blood features are measurements of liver biomarker concentrations in plasma (such as ALT, AST, alkaline phosphatase, and bilirubin) as well as other concentration values ​​in blood. The blood features may be represented, for example, as time series data. After the blood features are extracted, the time series data is formed into a single composite data structure. This data structure is used to aggregate the time series blood feature data into a data table, which may be used with preprocessing and transformation.

[0038] For example, feature engineering aggregates blood feature data by describing the time series data of blood particles with associated mean, standard deviation, minimum, and maximum values. Liver features may also be created by taking the last liver biomarker measurement available before treatment. A label may be created (e.g., using definitions obtained from medical experts, etc.) to classify someone as positive when the level of at least one liver biomarker exceeds a threshold (e.g., 3x the upper limit of normal) within a predefined window. Otherwise, the label may classify the patient as negative. Dates of immune checkpoint inhibitor (ICI) treatment may be determined and / or provided in some other form for use with the label and / or time series.

[0039] Other features can include frequency of pulmonary codes (e.g., ICD-9, ICD-10, etc.), frequency of C34 codes, frequency of C78 codes, smoking, blood oxygen saturation, frequency of lower than normal hemoglobin, frequency of higher than normal hemoglobin, frequency of lower than normal albumin, frequency of higher than normal albumin, frequency of lower than normal white blood cell count, frequency of higher than normal white blood cell count, frequency of lower than normal red blood cell count, frequency of higher than normal red blood cell count, frequency of higher than normal lymph node readings, frequency of lower than normal lymph, liver enzymes, drug class, interactions, maximum, minimum, time weighted average, final value, days before first treatment, etc.

[0040] After the input data is prepared (e.g., using feature engineering), the dataset is resampled. That is, the dataset resulting from the input preparation is unbalanced. As such, the dataset may be processed to infer, validate, estimate, and / or otherwise resample the prepared feature data in the dataset. For example, random majority class undersampling is performed on a dataset when the goal is to maximize the recall value. When the F1 score is the target of maximization, resampling may be skipped or ignored.

[0041] A model may be trained and tested to generate predictive values ​​using a dataset (e.g., resampled or otherwise). For example, when recall maximization is desired as the model selection criterion, the dataset may be used to train an RF model. When F1 score maximization is desired as the model selection criterion, the dataset may be used to train a GB model, for example. As such, the model selection criterion may be used to determine a desired goal, target, or focus, such as recall maximization, F1 score maximization, precision maximization, etc. For example, a model generated to compose a cohort of patients for a clinical trial may be selected or otherwise configured to enroll many patients in the clinical trial. As another example, a model generated to determine a treatment plan for a patient may be selected to cautiously (or alternatively, aggressively) proceed with treatment and associated risks based on the likelihood of toxicity balanced against the likelihood of efficacy. For example, a high risk of toxicity combined with a low likelihood of efficacy may avoid a patient receiving immunotherapy, whereas a low risk of toxicity combined with a likelihood of efficacy may encourage a patient to begin (or continue) immunotherapy. A relatively even balance between toxicity risk and efficacy probability can be left to the discretion of the patient and / or their physician, for example, depending on their interests, preferences, and / or goals. Additionally, safety monitoring, blood tests, and / or other measures (e.g., periodic reassessments, check-ins, etc.) may be indicated depending on the likelihood of toxicity to allow for early detection and management of toxicity as treatment continues. In some examples, the trained model may be validated, such as by leave-one-out cross-validation, where each sample is predicted individually with the rest as the training set.

[0042] As such, various "artificial intelligence" (AI) models may be developed and deployed for use in various health predictive applications. For example, models may be used to predict static and / or dynamic prognostic factors for hepatitis using AI models and patient (e.g., EHR, etc.) data.

[0043] Alternatively or additionally, a predictive model for ICI-associated pneumonia may be developed using a small, noisy dataset. Using input data from structured (e.g., EHR, EMR, laboratory data system, etc.) and / or unstructured (e.g., curated from laboratory data system, EHR, EMR, etc.) data, input features may be evaluated to build a model and output a predicted probability of pneumonia development. In some examples, multiple models may be developed and the system and associated processes may iteratively decide between two or more model versions. For example, available data may be split into two partitions in a sequential forward selection process, and robust performance evaluation may be used to validate and compare two developed models to select one model for deployment.

[0044] For example, the available data may be divided into two partitions and a sequential forward selection process may be used to build a model. At each iteration, two model versions (e.g., two refined models refined from an initial model) are compared and the one with the higher performance for both partitions is selected. The second model version (e.g., the refined model) is created by adding candidate features to the first model version (e.g., the initial model or the pre-tuned model). In the larger partition, the comparison is based on the performance results in the inner loop of nested cross-validation (CV). In the smaller partition, permutation testing is performed to test whether the candidate features have predictive power (e.g., the second model version has higher performance). Finally, the outer loop of nested CV is used for robust performance evaluation of the final model.

[0045] Some examples provide an automated framework for preparing EHR and / or other health data for use in machine learning based model training. For example, the framework prepares data from multiple sources and generates a combined time-dependent unbounded intermediate output that is used to aggregate features for training and deploying time-dependent models. For example, input data sources are processed and the data is used to generate a patient vector. The patient vector can be used to filter and aggregate, which forms an interface definition. As such, the model-independent workflow creates an input dataset for multiple model training. The intermediate output preserves the temporal structure for sequential modeling tasks, forming a maintainable, sustainable framework with interfaces.

[0046] In some examples, a predictive model building method related to ICI is provided, in which input data from multiple sources is prepared. Ground truth prediction labels are generated from the prepared data, and / or the labels can be crafted. One or more models are then built on the feature matrix generated using the labels and data, using the ground truth prediction labels as a standalone module of the framework. The framework can then drive a workflow that evaluates multiple efficacy surrogate endpoints, for example, to predict response to ICI treatment.

[0047] In some examples, patient health data is prepared and used to train a model using a system with multiple submodules. For example, the system includes a data extraction and processing submodule for extracting a patient's blood test history from an EMR / EHR, cleaning the blood history data, and performing data quality checks. A label definition submodule defines one or more feature labels related to the blood history data, and a feature engineering submodule can form blood features by aggregating and processing the blood history data with respect to the labels. A model submodule trains and evaluates an AI model that dynamically predicts immune-related hepatitis adverse event risk from a fixed-length blood test history. Alternatively or in addition, liver function test values ​​can be extracted, cleaned, and organized in a time series. A label definition algorithm can be executed to generate an AI model and target labels for each set of blood and / or liver test values, while feature engineering (e.g., normalization, symbolic transformation, and motif extraction) can be used, for example, to train and evaluate an AI risk prediction model. Similarly, drug history information, medical condition history, anthropometric features, etc. can be used for labeling and feature formation.

[0048] Because features can vary widely across toxicities, toxicity-specific feature formation and model training are critical to providing meaningful features and accurate prediction results. Otherwise, the lack of meaningful features impairs the performance of the resulting model. Thus, in some instances, models focused on specific toxicities can be derived, and their outputs can be combined (together and / or with efficacy predictions) to form recommendations, such as based on a risk-benefit analysis of toxicity versus efficacy for a given immunotherapeutic agent.

[0049] As such, some examples drive treatment based on predictions of likely complications from hepatitis, pneumonia, etc. Patients may be selected for immunotherapy treatment, removed from immunotherapy treatment, and / or otherwise have their treatment plan adjusted, selected for immunotherapy clinical trials, etc. based on the predictions and / or other outputs of one or more AI models. The models used to predict can evolve as data continues to be collected from one or more patients, and the associated predictions may likewise change based on the collected data. The models and / or associated predictions may, for example, be tailored to individuals and / or deployed to groups / types / etc. of patients, or groups or individual ICI drugs.

[0050] In some examples, data values ​​are normalized to the upper limit of a "normal" range (e.g., for blood tests, liver tests, etc.) so that values ​​from different sources can be compared on the same scale. Data values ​​and associated normalization / other processing can be specific to a laboratory, patient, type of patient (e.g., male / female, etc.), etc. For example, each laboratory measurement may have a specific normal range that is used to evaluate its values ​​across multiple patients.

[0051] With time series data, a value depends on previous and / or other values ​​such that the data values ​​have a relationship rather than being independent. This dependency can be identified and accounted for to identify patients in the data that change over time. For example, if a data value in a time series of patient blood tests exceeds twice the normal limit the first time, falls within the normal limit the second time, and exceeds twice the normal limit again the third time, this pattern can be identified as significant (e.g., worthy of further analysis). In data processing, for example, this pattern can be flagged or labeled appropriately. As such, clinical data from patient records can be used over time to identify and form features, anomalies, other patterns, etc. The data is mixed into a common model for comparison. Models trained and tested on such data can be robust to outliers, scaling, etc. Features can be created for better (e.g., more efficient, more accurate, more robust, etc.) modeling such as pneumonia modeling, colitis modeling, hepatitis modeling, etc.

[0052] Thus, data processing creates features that can be used to develop models that can be deployed to predict outcomes for patients. For example, a data processing pipeline creates tens of thousands of features (e.g., frequencies of ICD-10 codes such as pneumonia, frequencies of C34 codes, etc.).

[0053] For example, the data values ​​may include ICD-10 codes for a given patient for a one-year period. In some examples, codes may span multiple years (e.g., 10 years, etc.) and be harmonized for processing. ICD-10 codes may be processed to identify codes related to lung or respiratory function, and such codes may then be used to calculate the relative frequency of lung disease for a patient. As another example, a patient history may be analyzed to determine the relative frequency of C34 codes in the patient history, thereby indicating lung cancer. Smoking status may be, for example, a binary flag that is set or unset from a data processing pipeline. In some examples, codes may be converted between code systems (e.g., ICD-9, ICD-10 (C34, C78, ​​etc.), etc.). Codes may be reverse engineered, for example, without all of the keys.

[0054] Using the codes and other data, multiple (e.g., 5, 6, 10, etc.) features can be created and used in a modeling framework to predict the onset of pneumonia in patients. The models are built in an incremental, progressive manner. Labels for the pneumonia models are not inherently included in the dataset, so a ground truth for model training is created based on, for example, expert judgment identifying labels from patient history. Codebooks and quality control can be used, for example, to ensure correct labeling.

[0055] In some examples, historical data received from patients is asynchronous. The system and method then aligns patient (and / or inter-patient) data, allowing for aggregation and analysis of data with respect to a common baseline or benchmark. In some examples, an impact point, anchor point, or other reference point may be selected / determined, and the patient data chronology / timeline is aligned around that determined or selected point (e.g., an event occurring to the patient, such as a check-up, injury, symptom onset, examination, birthday, milestone, etc.). For example, dates of first chemotherapy, ICI therapy, first symptoms / responses (e.g., in lung function, etc.), etc. may be used to align patient data.

[0056] The processed data may then be used to predict static labels in a predefined or otherwise determined time frame, etc. Models may be trained, validated, and deployed for hepatitis, pneumonia, drug efficacy, etc. As such, data from EHRs, EMRs, laboratory systems, and / or other data sources may be pre-processed and provided to models to generate predictions, which may be post-processed and output to users and / or other systems for alerts, follow-up, treatment protocols, etc. In some examples, the predicted values ​​are routed to another system (e.g., a scheduling, laboratory, etc. system) for further processing.

[0057] One or more AI models may be used to facilitate processing, correlation, and prediction based on available patient health data such as blood test results, liver test results, other test results, other patient physiological data, etc. Models may include high recall low precision models, low recall high precision models, harmonic mean maximization (convergence) models, etc. Boosted decision tree models or variants such as random forest (RF), gradient boosting (GB) may be used. For example, majority class undersampling, random forest models may be used to maximize recall with relatively low precision. However, in hepatitis, prevention is cheap and easy, and thus false negatives may be tolerated. As such, gradient boosting models have been developed that can maximize the F1 score without resampling being applied.

[0058] Machine learning techniques, whether deep learning networks or other experiential / observational learning systems, may be used, for example, to characterize and otherwise interpret, extrapolate, conclude, and / or perfect medical data obtained from patients. Deep learning is a subset of machine learning that uses a set of algorithms to model high-level abstractions in data using deep graphs with multiple processing layers, including linear and nonlinear transformations. While many machine learning systems are seeded with initial features and / or network weights that are to be modified through learning and updating of the machine learning network, deep learning networks train themselves to identify "good" (e.g., useful, etc.) features for analysis. By using multi-layer architectures, machines employing deep learning techniques can process raw data better than machines using traditional machine learning techniques. Examining the data for groups of highly correlated values ​​or distinctive themes is facilitated by using different layers of evaluation or abstraction.

[0059] Deep learning is a class of machine learning techniques that employs representation learning methods that allow a machine to be given raw data and determine the representations required for data classification. Deep learning identifies structure within a dataset using a backpropagation algorithm that is used to modify the internal parameters (e.g., node weights) of the deep learning machine. Deep learning machines can utilize a variety of multi-layer architectures and algorithms. While machine learning, for example, involves identifying features to be used in training the network, deep learning processes raw data to identify features of interest without external identification.

[0060] Deep learning in a neural network environment includes a large number of interconnected nodes called neurons. Input neurons activated from an external source activate other neurons based on their connections governed by machine parameters. The neural network exhibits a particular style of behavior based on the respective parameters. Learning refines the machine parameters, and by extension, the connections between the neurons in the network, so that the neural network exhibits a desired style of behavior.

[0061] Various artificial intelligence networks can be deployed to process input data. For example, deep learning, which utilizes convolutional neural networks, uses convolutional filters to segment data and locate and identify learned observable features in the data. Each filter or layer of the CNN architecture transforms the input data to increase the selectivity and invariance of the data. This abstraction of the data allows the machine to focus on the features in the data it is trying to classify and ignore irrelevant background information.

[0062] Deep learning operates on the understanding that many data sets contain high-level features that contain low-level features. For example, while searching an image, it is more efficient to look for edges that form motifs that form parts of the object being searched for, rather than looking for objects. These hierarchies of features can be found in many different forms of data, such as audio and text.

[0063] Learned observable features include the objects and quantifiable regularities learned by the machine during supervised learning. A machine provided with a large set of well-classified data is well equipped to distinguish and extract features associated with successful classification of new data.

[0064] A deep learning machine utilizing transfer learning may appropriately connect features of data to a particular classification confirmed by a human expert. Conversely, the same machine may update parameters for a classification when notified of an incorrect classification by a human expert. Settings and / or other configuration information may be guided, for example, by learned use of settings and / or other configuration information, and as the system is used more (e.g., repeatedly and / or by multiple users), the number of variations and / or other possibilities for settings and / or other configuration information may be reduced for a given situation.

[0065] An exemplary deep learning neural network can be trained, for example, with a set of expert classification data. This set of data establishes initial parameters for the neural network, which is the supervised learning stage, during which the neural network can be tested to determine whether the desired behavior has been achieved.

[0066] After the desired neural network behavior is achieved (e.g., the machine is trained to operate according to specified thresholds, etc.), the machine may be deployed for use (e.g., calibrate the machine on new / updated data, etc.). During operation, the neural network classifications may be confirmed or rejected (e.g., by an expert user, an expert system, a reference database, etc.) to continue to improve the neural network behavior. The exemplary neural network is then placed in a state of transfer learning, as parameters for the classifications that determine the neural network behavior are updated based on ongoing interactions. In some examples, the neural network may provide direct feedback to another process. In some examples, the neural network outputs data that is buffered and verified (e.g., via the cloud, etc.) before being provided to another process.

[0067] Deep learning machines can utilize transfer learning when interacting with physicians to work with the small datasets available in supervised training. These deep learning machines can improve computer-aided diagnosis over time through training and transfer learning. However, larger datasets result in more accurate and more robust deployed deep neural network models that can be applied to transform heterogeneous medical data into actionable results (e.g., system configurations / settings, computer-aided diagnosis results, image correction, etc.).

[0068] One or more such models / machines may be developed and / or deployed on prepared and / or curated data. For example, features may be extracted from EHR / EMR data tables (e.g., liver biomarkers, blood / plasma concentrations, etc.). Curated data extracts structured data from unstructured sources (e.g., diagnosis date from medical notes, etc.). The extracted features form the basis for label creation. Time series measurements are extracted and aggregation generates statistical descriptors for the time series data. Such tables of data are used to train models to make predictions. In some examples, data may be resampled if necessary, desired, or determined. Models are validated using robust cross-validation, such as leave-one-out cross-validation.

[0069] For example, the selection, input, or target can determine whether to maximize the F1 score or recall with a given model. When the F1 score is maximized by the model, no data resampling is performed. When the recall is maximized by the model, data resampling can be performed. This decision can be driven, for example, by the model's deployment environment. When the model is to be deployed in a system that facilitates clinical trials, the F1 score is maximized because pharmaceutical companies want patients with the highest probability of responding well to a given drug. High recall is more important in a clinical treatment setup where the system wants to eliminate as much toxicity as possible. For example, in a treatment setup for hepatitis, a low model precision is acceptable due to the nature of the treatment for hepatitis being cheap.

[0070] Related systems and methods can be used to assess the probability or reliability of predictions generated by the model. The predictions and associated confidence levels or other reliability can be output, for example, to another processing system. The likely response to immunotherapy treatment can be modeled based on available patient data collected in clinical practice.

[0071] FIG. 1 illustrates an exemplary model generator 100 including an exemplary input processor circuit 110, an exemplary model trainer circuit 120, an exemplary model comparator circuit 130, an exemplary model storage circuit 140, and an exemplary model deployer circuit 150. As shown in the example of FIG. 1, the exemplary input processor circuit 110 processes input data pulled from a record to form a set of candidate features. The exemplary model trainer circuit 120 uses the set of candidate features to train at least a first model and a second model. The exemplary model comparator circuit 130 tests at least the first model and the second model and compares the performance of the first model and the second model. Based on the comparison result, the exemplary model comparator circuit 130 selects at least one of the first model or the second model. The exemplary model storage circuit 140 provides a memory circuit for storing the selected first and / or second models. The example model deployer circuit 150 deploys the selected first model and / or second model to predict the likelihood of toxicity, such as pneumonia, hepatitis, colitis, etc., resulting from immunotherapy, according to a treatment plan for the patient.

[0072] In operation, the exemplary input processor circuit 110 processes input data related to one or more patients, such as lab results, diagnostic codes, billing codes, etc. The input may be from one or more external systems 160, such as an EHR, EMR, etc. The exemplary input processor circuit 110 may extract and organize the input, for example, in a chronological order for each patient. In some examples, the input processor circuit 110 aligns the input data with respect to an anchor point to organize the input data in a chronological order. The anchor point is a reference or alignment time point for the input data, such as, for example, an immunotherapy treatment initiation patient, first patient symptom, etc., that allows a group of patients to be tracked, modeled, and compared with respect to their reference time point. In some examples, the input processor circuit 110 generates labels for the input data to form a set of candidate features.

[0073] In some examples, the input processor circuit 110 performs feature engineering on the set of candidate features and / or the underlying data. For example, the input processor circuit performs feature engineering on the set of candidate features, for example, by normalizing, transforming, and / or extracting from the set of candidate features. In some examples, the input processor circuit 110 selects from the set of candidate features to form a set of patient features, thereby training and / or testing at least a first model and a second model based on the feature engineering. In some examples, the input processor circuit 110 generates a feature matrix for training and / or testing / validating the first model and / or the second model based on the feature engineering.

[0074] In some examples, the model deployer circuit 150 deploys the selected first and / or second models as executable tools having interfaces to facilitate collection of patient data and interaction with the selected first and / or second models. The deployed models may be used with tools to select patients for clinical trials, initiate a course of immunotherapy treatment according to a treatment plan or protocol, adjust a current immunotherapy treatment plan, etc. In some examples, the deployed models may be stored and / or utilized in one or more external systems 160, such as an EHR, EMR, scheduling system, clinical information system, etc. The tool, implemented as a portal, online platform, computer application, etc., can be used to load one or more stored models (e.g., toxicity-related models, efficacy-related models, etc.), determine one or more thresholds or other patient selection criteria based on which to evaluate the output of the model, process healthcare data for a given patient using the model, evaluate the predictive output of the model with respect to the thresholds, and provide treatment for the patient based on the thresholds (e.g., construct a treatment plan for the patient to include or exclude immunotherapy, which treatment plan can be a treatment plan for a cohort of patients in a clinical trial, etc.).

[0075] As such, the exemplary model generator 100 can pre-process data from one or more external systems 160 via the exemplary input processor circuit 110. The exemplary model generator 100 then trains and validates multiple models using the model trainer circuit 120 and the model comparator circuit 130. The exemplary model generator 100 post-processes, stores, and deploys one or more of the trained and validated models using the model storage circuit 140 and the model deployer circuit 150. Input data from the EHR, EMR, and / or other external systems 160 is converted into multiple models that can be stored and deployed to predict treatment efficacy, risk of toxicity and / or other side effects, etc., from a particular patient's immunotherapy treatment. Data from multiple patients can be leveraged through processing and modeling, for example, to generate targeted predictions for a particular patient.

[0076] In some examples, the model trainer circuit 120 trains and validates boosted decision tree models, other ensembles and / or regression-based learning models, and the like. For example, boosted decision tree models utilize multiple weakly learning decision trees to create a strongly learning AI model. Trees in the model can, for example, correct errors in other trees in the model, and the entire ensemble of trees in the model works together to generate a predictive output. Models can also be formed as random forest (RF) models, gradient boosting (GB) models, and the like. Data is fitted by the model trainer circuit 120 to a common model for comparison that is, for example, robust to outliers and scalable.

[0077] In some examples, the input data processor 110 normalizes the input data values ​​to the upper limit of normal so that values ​​(e.g., blood test values, liver function test values, other laboratory values, etc.) are on the same scale. Each laboratory value may have a certain range of values. The input data processor 110 normalizes, for example, the values ​​for a particular laboratory type over a range so that the data for that laboratory is comparable between patients. The input data processor 110 may also organize the data into a time series. The time series data may be normalized and / or otherwise adjusted by the input data processor 110 based on an anchor point (e.g., a common or reference event such as hospitalization, symptoms, birth, date of diagnosis, date of first ICI administration, etc., to which the data and associated features may be aligned for model development and prediction).

[0078] For example, the received patient data may be asynchronous. Selecting anchor / inflection / influence points allows data to be aligned for the same, similar, and / or different patients. For example, in a group of patients with diabetes, 10 years of patient history is reviewed while excluding data that occurred after diagnosis. The time of diagnosis is selected as an anchor point for looking back at the patient history data (e.g., when blood glucose values ​​are not taken or insulin is taken). As another example, initial immunotherapy data may be used to align patient data. Aligning multiple patients with respect to the start of immunotherapy allows, for example, to determine correlations between subsequent events for a patient in multiple patients.

[0079] For example, date of birth, date of first immunotherapy administration, date of last immunotherapy administration, and date of death (or information indicating that the patient was still alive 5 years after immunotherapy initiation) may be used to align all patients to the start of immunotherapy as an anchor point, regardless of how old the patient was at that time. As such, the data is organized, for example, to allow for the identification of relationships and comparisons. For example, patterns may be identified from the data.

[0080] In some examples, the time series data is formed into a plurality of features by the input data processor 110. These features may be a set of candidate features from which the features are selected to train and validate the model. For example, patient demographic data, biometric data, survey data, laboratory data, and other patient health care data may be extracted from patient records to form an input data set that may be associated with a plurality of features, such as red blood cell values, white blood cell values, drug counts for one or more medications, blood albumin, lymphatic system, alkaline phosphate, radiology, etc.

[0081] Feature engineering by the input data processor 110 can, for example, form features based on codes (e.g., ICD-10 codes, etc.). For example, ICD-10 codes for a patient in a given year can be processed to identify codes related to lung and / or respiratory function (e.g., C34, C78, ​​etc.), and a time series of those codes can form a function that is used to calculate the relative likelihood of lung disease in the patient. Similarly, features can be formed to predict the development of pneumonia, hepatitis, colitis, and / or other toxicities from immunotherapy treatment.

[0082] Features that classify or identify data based on lung function, liver biomarkers, blood tests (e.g., blood levels, plasma levels, etc.), etc., can form the basis for creating labels and / or other statistical descriptors for model training. For example, collected data can be compared to a ground truth set of identified labels to generate labels for the data. Resampling and even validation (e.g., leave-one-out cross-validation, leave-other cross-validation, leave-other validation, etc.) can be performed to generate robust models.

[0083] The exemplary model trainer circuit 120 uses the features and / or other data to train one or more AI models, such as toxicity prediction models (e.g., hepatitis prediction model, colitis prediction model, pneumonia prediction model, etc.), efficacy prediction models (e.g., immunotherapy efficacy model, etc.). The models may be low precision and high recall, high precision and low recall, harmonic mean (F1) maximized, majority class undersampled, etc. A model focus decision may be made by and / or for the exemplary model trainer circuit 120 (e.g., based on settings, mode, type, other model selection criteria, etc.). For example, the model trainer circuit 120 trains one or more models to maximize F1 score or maximize recall. When focusing on F1, resampling may not be necessary. When focusing on recall, resampling may be desirable to improve the model (e.g., with training and test data sets, multiple training data sets and / or multiple test data sets, etc.).

[0084] For example, when configuring participation in a clinical trial, the focus is on maximizing the F1 score, thereby identifying patients with the highest likelihood of responding well to a given immunotherapy drug. However, when determining clinical treatment for a patient, the goal is to eliminate toxicity to the patient as much as possible. As such, models developed with high recall are more important, and low precision is tolerated due to the availability of inexpensive treatment options for hepatitis, colitis, pneumonia, etc.

[0085] The model comparator circuit 130 can compare and / or otherwise evaluate multiple trained models to validate the models and evaluate the likelihood of the predictions generated by each model. Such assessment / evaluation can be used to select one or more models to be deployed. For example, models can be compared and / or otherwise evaluated based on average output values, standard deviation of model outputs, etc. One or more models can be selected for storage in the model storage circuit 140 and / or other memory circuits, deployment in a tool via the model deployer circuit 150, output to one or more external systems 160, etc.

[0086] In some examples, the external system 160 can utilize one or more deployed models from the model deployer circuit 150 to drive tools, etc., for evaluating efficacy and / or toxicity associated with an immunotherapy for a patient and / or patient population. Using one or more deployed models, a patient can be evaluated at the initiation of an immunotherapy treatment regimen, during administration of an immunotherapy treatment regimen, as a candidate for an immunotherapy clinical trial, etc. In some examples, a previous efficacy and / or toxicity model prediction for a patient can be used along with an updated efficacy and / or toxicity model prediction for the patient (e.g., a combination of previous results with current results, using previous results as input to generate current results, etc.).

[0087] 2-14 are flow charts of example processes representing computer-readable instructions that can be stored in a memory circuit and executed by a processor circuit to implement and operate the example model generator 100 of FIG. 1. The example process 200 of FIG. 2 begins at block 210, where the input processor circuit 110 processes data to form input for the model trainer circuit 120. For example, the input processor circuit 110 can align the data with respect to anchors or reference points (e.g., dates, events, milestones or other markers, etc.). The input processor circuit 110 can form a time series from the data. Alternatively or in addition, the input processor circuit 110 can, for example, label and / or perform feature engineering on the data to form a set of input features for training the model.

[0088] At block 220, a model is trained using the input. For example, multiple AI models are trained by the model trainer circuit 120 using the input feature set and / or other inputs provided from the input processor circuit 110 along with associated data. By identifying patterns and correlations in the input data based on the features, the model trainer circuit 120 forms and weights, for example, the nodes, layers, connections, etc. of a model (e.g., an RF model, a GB model, a boosted decision tree model, etc.). The model may be tuned, for example, by iteratively adding and removing features to find a balance between the model's precision and recall.

[0089] At block 230, the trained model is validated by the model trainer circuit 120. For example, additional inputs (e.g., features, resampled input data, etc.) are used to validate (e.g., test, validate, etc.) that the trained model performs as intended at a threshold level of precision, accuracy, etc.

[0090] At block 240, the model comparator circuit 130 evaluates the validated models and selects one or more models for deployment. For example, the model comparator circuit 130 may compare and / or otherwise evaluate multiple trained and validated (e.g., validated, etc.) models according to one or more model selection criteria, and evaluate the probability of predictions generated by each model, the suitability of each model for a particular purpose (e.g., predicting immunotherapy treatment efficacy, predicting immunotherapy toxicity, immunotherapy clinical trial selection, immunotherapy treatment plan adjustment, etc.). Such assessment / evaluation may be used by the model comparator circuit 130 to select one or more models. For example, the models may be compared and / or otherwise evaluated based on average output values, standard deviation of model outputs, etc. In some examples, the model comparator circuit 130 selects one model for use / deployment. In other examples, the model comparator circuit 130 selects multiple models for use / deployment (e.g., toxicity prediction model and efficacy prediction model, etc.).

[0091] At block 250, the one or more selected models may be stored by the model storage circuitry 140. As such, the selected models may be stored for later use, deployment, etc. In some examples, unselected models may also be stored by the model storage circuitry 140, as the unselected models may be suitable for other uses / deployment upon subsequent requests.

[0092] At block 260, the selected model or models are deployed to an external system 160 by the model deployer circuit 150. The models may be deployed as part of a tool for immunotherapy treatment and / or clinical trial planning, thereby providing toxicity predictions, efficacy predictions, and the like.

[0093] 3 illustrates an example implementation of input processing for model development (e.g., block 210 of the example of FIG. 2). In block 310, input data for each of a plurality of patients is arranged in one or more time series (e.g., arranged in chronological order and showing evolution, progression, dependency, and / or other patterns). In block 320, the time series data for each of the plurality of patients is aligned with respect to an anchor point or other reference event. For example, various sets of time series data may be aligned according to events such as birth, onset of illness, hospitalization, etc. By aligning the time series, the data in the time series may, for example, be more meaningfully evaluated, compared, and / or otherwise analyzed.

[0094] The aligned time series data is labeled at block 330. For example, a target label and associated grade may be generated for each set of data values ​​(e.g., blood test values, liver function values, lung function values, etc.).

[0095] At block 340, feature engineering is performed on the data to prepare the data as a set of features with labels to be used to train, test, and / or otherwise validate a model. For example, feature engineering may normalize, transform, and extract the data into a set of patient features to apply to the model. In some examples, these features form a set of candidate features that are evaluated to select a particular subset of patient features for training, validating, etc. a model.

[0096] FIG. 4 illustrates an exemplary implementation of model selection for deployment (e.g., block 240 of the example of FIG. 2). In block 410, a target or goal is examined. For example, the request for an AI model may be a request for clinical trial evaluation, immunotherapy efficacy prediction, immunotherapy-related toxicity, and / or other side effect prediction, etc. Such a request may prompt the selection of a model with certain characteristics, such as a high F1 score, high recall, etc. Based on the requested target or goal, a particular subset of available models may be selected. In other words, based on the requested target or goal, a particular subset of available models may be filtered out, leaving a subset of available models for further evaluation and selection.

[0097] Furthermore, targets or goals can be associated with risks or tolerances. For example, a clinical trial may be tolerant of low efficacy, but also be interested in low toxicity. As another example, a clinical trial may only be interested in the risk of toxicity associated with a given immunotherapy, as a gatekeeper for deciding whether to include or exclude patients from the clinical trial. Alternatively, the selection criteria for patient treatment using a given immunotherapy can, for example, balance the risk of toxicity with likely efficacy to determine whether a patient should receive (or continue to receive) the treatment.

[0098] At block 420, testing is performed with each remaining model. For example, permutation testing, binomial testing, and / or other comparison operations are performed with each model (e.g., evaluating the lung function model with respect to smoking, non-smoking, other binomial testing, and / or permutation testing, etc.). At block 430, the output of each model performing the testing is evaluated based on one or more criteria. For example, each model may be evaluated based on the mean, standard, and / or other comparison criteria to determine which model best meets one or more criteria.

[0099] At block 440, one or more models are selected based on the comparison of the outputs and / or other model performance. For example, a single model may be selected for deployment based on an evaluation of how the model performed and / or output with respect to one or more criteria. In some examples, multiple models may be selected based on one or more criteria and / or other factors. For example, multiple models may meet the criteria. As another example, a request may include a request for treatment efficacy as well as associated toxicity likelihood. In response to such a request, both an efficacy model and a toxicity model may be selected. In some examples, a single model may be trained on both toxicity and efficacy, thereby outputting a combined prediction.

[0100] FIG. 5 illustrates an exemplary sequential procedure 500 for building and evaluating a model (e.g., through developing a feature set, etc.). As shown in the example of FIG. 5, at 1, an initial data set 510 is divided into at least two partitions. Separate data analyses are performed on a first partition of data 520 and a second partition of data 525. The first data partition 520 is divided into at least two loops, such as an inner loop 522 and an outer loop 524, for data analysis. At 2, a first test, such as a binomial test, evaluates the data values ​​of the inner loop 522 and the outer loop 524 (e.g., the data value H 0 contains "smoking", and so on), and train the model. At 3, a separate "no touch" test dataset 525 is processed using permutation testing, rather than using a separate loop. The dataset 525 can be processed across multiple permutations, for example, using a criterion such as smoking versus no smoking, to identify where smoking is more likely than no smoking. At 4, the model is evaluated, reporting means, standards, and / or other evaluation / performance statistics for the first and second data partitions 520, 525 and associated tests.

[0101] In some examples, one or more of the loops 522, 524 are used to evaluate the first data partition 520 via exploratory data analysis using K-fold cross-validation (CV) to compare the first feature set M1 and the second feature set M2. Through successive iterations, a feature F is added to the first feature set M1 if the K-fold CV performance of M1 is less than M2. After the features are evaluated accordingly, the resulting model performance is tested using the second data partition 525. If both loops 522, 524 are utilized, M1 and M2 may be compared based on the results of the inner loop 522 (e.g., binomial test). When M2 is selected, a permutation test is performed to evaluate whether M2 is sufficiently better than M1 on the validation dataset 525. Then, for example, the model performance may be evaluated on the outer loop 524. As such, the initial feature set M1, or the extended feature set M2 may be selected for model training.

[0102] FIG. 6 illustrates an exemplary process 600 for preparing input data for model training. Using the exemplary process 600, data from disparate observational databases can be converted into a common format (e.g., data model) and common representation (e.g., common terminology, vocabulary, coding scheme, etc.) for systematic analysis according to a library of analysis routines. The exemplary process 600 accepts input of a concept table 605 including concept identifiers, concept names, concept descriptions, etc. For example, the concepts in this example relate to liver function tests. A clinical table 615 is also input with database extracted values ​​for patient demographics, clinic visits, laboratory measurements, etc. In block 610, patient liver function test values ​​(e.g., represented by patient blood test history) are extracted from electronic medical records, etc., cleaned, quality checked, and organized in a time series format for one or more patients. Such extraction and processing can also be applied to other patient information (e.g., lung function, drug effects, etc.).

[0103] At block 620, a label definition algorithm is executed to assign adverse event (AE) grades to a set of blood test / liver function values ​​(e.g., ALT, AST, TBILIRUBIN, ALKPHOS, etc.) and create binary target labels for the AI ​​model. Such label definitions can also be applied to other patient test values ​​(e.g., lung function, etc.).

[0104] At block 630, feature engineering is performed. For example, blood test values ​​and / or other values ​​may be normalized to an upper limit of "normal" to aid in feature generation, comparison, and other analysis. Such "normal" ranges may be determined for a particular patient based on specific blood tests, other laboratory tests, etc. Feature engineering converts the normalized values ​​into a discretized symbolic representation, such as a modified symbolic aggregation approximation. In some examples, motifs may be extracted from a set of symbolic representations as n-grams, and counts from the patient history may be used as features.

[0105] In block 640, an AI model can then be trained and evaluated based on the features, etc., to dynamically predict immune-related adverse event risk from patient data. For example, immune-related adverse event risk can be predicted by the model from fixed-length blood test history, etc. Outputs can include model performance metrics 625, a dataset of AE risk predictions 635, etc. For example, the risk prediction dataset 635 can include immune-related hepatitis adverse event risk predictions per historical blood test value. That is, for each patient, the risk of developing a hepatitis AE by the next ICI treatment appointment for that patient is predicted based on that patient's historical blood test values, etc. The dataset 635 can alternatively include risk predictions for pneumonia, colitis, hospitalization, etc., based on the relevant patient data for training the model.

[0106] FIG. 7 illustrates an example process 700 for building an example classification model. At block 710, data from an input dataset 705 is split into multiple partitions. For example, structured EHR-derived tabular data, unstructured data tables, measurement features aggregated over time (e.g., over 30 days, 60 days, a year, etc.), curated labels (e.g., pneumonia labels, hepatitis labels, colitis labels, etc.), other features aggregated over time (e.g., over 30 days, 60 days, a year, etc.) (e.g., condition, smoking status, etc.), etc. may be collected and partitioned. For example, 90% of the available data may be organized into a first partition and the remaining 10% of the data may be organized into a second partition. A model size input 715 may help, for example, to adjust the partitions based on a typical, expected, or desired number of features in the model.

[0107] At block 720, a sequential forward selection procedure is performed to evaluate multiple models. An exemplary procedure iterates (e.g., from an initial model to a series of trained models) to find patterns in the data to form potential predictive features. For example, a first partition may be evaluated to identify candidate features based on associations between labels and features. At each iteration, a model is selected (e.g., a trained model) and evaluation continues.

[0108] Model performance is evaluated in block 730. For example, the model selected in the final iteration of the selection procedure 720 is cross-validated, such as in an outer loop of nested cross-validation using the input dataset 705, and finalized for deployment and use.

[0109] 8 provides an example implementation of the sequential forward selection procedure 720 of the example process 700 of FIG. 7. At block 810, candidate features F are identified in a first data partition. For example, the candidate features F may be identified based on associations between a target label and existing / identified features. At block 820, nested cross-validation (CV) is performed to validate two models.

number

number

number

[0110] At block 830,

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0111] At block 860, the model size is evaluated (e.g., by comparing the current feature set to the model size input 715, etc.). If the model size has been reached, the process may proceed to block 730. However, if the model size condition has not yet been met, control returns to block 810 to select new candidate features in the next iteration of the exemplary process.

[0112] 9 provides an example implementation of the robust performance evaluation 730 of the example process 700 of FIG. 7. In block 730, the final selected model

number

number

[0113] 10 illustrates an example process 1000 for preparing data from multiple sources for use in training machine learning and / or other AI models. For example, database extraction and / or expert curation can be used to generate data from / in an input dataset 1005. To form the dataset 1005, data can be collected from multiple structured and / or unstructured data sources.

[0114] In block 1010, one or more input data sources 1005 are pre-processed to prepare input data (e.g., by cleaning and extracting features from the data, labeling features, etc.). The input data and resulting features may include smoking history, drug administration, medical condition, radiation therapy history, laboratory measurements, anthropometric data, etc. For example, in block 1012, structured data (e.g., drug administration, medical condition, laboratory measurements, clinical treatment, anthropometric data, etc.) from the input dataset 1005 is filtered and cleaned in a common data format (e.g., a common data model such as the OMOP data model format, etc.). In block 1014, curated data (e.g., related to downstream tasks, smoking history, specific condition history, specific medical history, etc.) from the input dataset 1005 is filtered and cleaned into a common data format.

[0115] The pre-processed dataset forms an input to block 1020, where patient vectors are generated from the filtered and cleaned aggregated dataset. For example, a model-independent set of aggregated patient vectors may be provided along with feature filtering. A model-independent non-aggregated table with a temporal data structure may be used for feature filtering and vector generation. The patient vectors and / or the non-aggregated table may form part of the output dataset 1025.

[0116] The data is filtered and aggregated at the interface at block 1030. For example, the data may be filtered and arranged around anchor points and / or other references for aggregation and formation into a patient vector.

[0117] FIG. 11 provides an example implementation for generating a patient vector (e.g., block 1020 of example process 1000). In block 1110, the processed (e.g., filtered, cleaned, etc.) data is evaluated to determine whether features in the data should be used. When a feature should not be used, the feature is omitted in block 1120. When a feature should be used, in block 1130, the available data (sources) are evaluated to determine whether multiple sources (e.g., structured and / or curated data sources, etc.) should be used. When multiple sources should be used, in block 1140, features from multiple data sources are combined and harmonized. In block 1150, an unrestricted patient vector is formed from the downstream model independent non-aggregated table that preserves the temporal structure. The intermediate dataset output 1105 can include, for example, one or more datasets separated by input domain and that preserve the temporal structure.

[0118] 12 provides an example implementation of a filtering and aggregation interface (e.g., block 1030 of example process 1000). At 1210, a patient is evaluated to determine whether the patient is a candidate for further evaluation and processing. Such a determination may be based, at least in part, on downstream task-dependent external patient eligibility criteria 1205, for example, whether the patient meets certain health criteria, age criteria, condition criteria, likelihood of success, etc. When the patient is not eligible, at block 1220, the patient is omitted from undergoing further processing.

[0119] When a patient is eligible, an anchor point is set for further evaluation in block 1230. The determination of the anchor point may be based, for example, on downstream task-dependent external patient eligibility criteria 1215. The anchor point is a patient timeline event that aligns multiple patients for comparison. For example, events in a patient's life and / or medical history, such as diagnosis, symptoms, birth, hospitalization, initial treatment (e.g., first immunotherapy treatment date), etc., can drive anchor point selection by a framework that aligns time series data for relational processing and comparison. The anchor point may be, for example, a configurable parameter that depends on the downstream task 1205.

[0120] At block 1240, an aggregation period is set. For example, the framework may configure and / or otherwise set the aggregation period prior to the prediction point (e.g., anchor point and / or other time points and / or data for the patient). For example, the aggregation period may include an aggregation of patient data 200 days prior to the prediction time, etc.

[0121] At block 1250, an aggregation period for the measurements is set. For example, the interface framework allows for the establishment of different aggregation periods for laboratory measurements, other measurements, etc., since such measurements may change more dynamically.

[0122] At block 1260, one or more restricted, aggregated patient vectors are formed as an output of the interface framework. The patient vectors may be tailored, for example, to multiple downstream tasks. In some examples, for a given set of inputs, a patient vector may be dynamically generated as downstream tasks change.

[0123] 13 illustrates an exemplary process 1300 for building a predictive model for efficacy related to ICIs. Such models can evaluate multiple efficacy surrogate endpoints to predict response to ICI treatment. Models can be created for separate cancer indications and / or combinations thereof.

[0124] At block 1310, input data from multiple sources is prepared for processing. For example, as described in more detail above, data may be cleaned and features may be extracted from sources such as structured dataset 1305, curated dataset 1315, etc. For example, raw, automatically derived EHR data may be converted into a common format along with curated data to form a set of features.

[0125] At block 1320, ground truth prediction labels are generated. The prediction labels may include ICI treatment duration (TOT), time to next treatment (TNET) (e.g., after ICI discontinuation), overall survival (OS), etc. The enumerated ground truth endpoints may be generated on a continuous scale, e.g., derived from the input data 1325 and expressed as days elapsed from an anchor point. For example, patient timelines may be aligned based on similarities in ICI treatment progress. A predefined anchor point may be, for example, the date of the first day of ICI treatment initiation. The generated ground truth labels may be used as is and / or with modified granularity (e.g., weeks, months, years, etc. elapsed) to train regression models, survival analysis-based models, etc. Ground truth discretization may be performed for two-class classification and / or multi-class classification (e.g., responders vs. non-responders, 5-year survival rate, etc.).

[0126] At block 1330, model building is performed on the generated feature matrix, where the ground truth serves as a standalone module of the framework. In some examples, efficacy endpoints may be modeled separately (e.g., individually, etc.). Modeling may be performed, for example, without hypotheses and / or with different machine learning algorithms. Alternatively, or in addition, modeling may be performed on a continuous scale using survival analysis methods. Furthermore, modeling may follow, for example, a multi-class classification method for all endpoints. Exemplary model building and selection are further described above.

[0127] FIG. 14 illustrates an exemplary method 1400 for model selection and generation. In the example of FIG. 14, model generation is driven at least in part by the model's objectives, goals, or targets (e.g., referred to herein as model selection criteria). For example, the goal may be to generate models with high (e.g., maximized) F1 harmonic scores (e.g., harmonic mean of recall and precision), such as the GB model, models with high (e.g., maximized) recall (e.g., few false negatives), such as the RF model, models with high precision (e.g., very few false positives), etc. In some examples, majority class undersampling may be combined with time series data aggregation to obtain a balanced static data set to be fed to the model. In some examples, the model produces probability estimates for the labels, rather than just the discrete labels themselves.

[0128] At block 1410, inputs are prepared, such as by extracting data from one or more EHR data tables 1405. Measurements, such as liver biomarker concentrations in plasma, other blood concentration values, etc., may be extracted. The time series measurements and / or other data may be, for example, made into a single complex data structure (e.g., aggregated). The aggregated data may be processed using feature engineering to describe the time series data according to mean, standard deviation, minimum, maximum, etc. For example, lags and features and labels may be created. In some examples, dates of ICI treatment 1415 are generated (e.g., as reference or anchor points) to help align and anchor the time series data.

[0129] At block 1420, a target or goal for the model is evaluated. As described above, the goal of F1 maximization is to produce a different trained model (e.g., a gradient boosted model such as a gradient boosted decision tree) than the goal of recall maximization (e.g., a random forest model). Based on this determination, for recall maximization, at block 1430, dataset resampling is performed to balance the unbalanced dataset from block 1410. For example, random majority class undersampling is performed on the dataset when the goal is to maximize recall. When the goal is to maximize the F1 score, resampling may be ignored.

[0130] At block 1440 and / or 1450, the models are trained. For example, at block 1440, the RF model is trained for recall maximization, and at block 1450, the GB model is trained for F1 score maximization. Each model is validated, for example, by leave-one-out cross-validation, where each sample is predicted individually and the remainder serves as the training data set. The models are trained on a collection of historical data, for example, from multiple health care records.

[0131] As such, in some instances, it produces a model that can be deployed, loaded, and utilized to generate predictions for driving patient care, tailoring patient care, configuring cohorts for clinical trials, etc. As described above, nested cross-validation can be used with multiple loops to train and select models. For example, the inner loop can be used for tuning hyperparameters, e.g., starting with all data points in the patient's history and paring down with feature removal, or using a baseline set of features to which features can be added and tested with statistical analysis to determine if the new model performs better than the old model. In the latter case, when the added features improve the model's performance, that updated model becomes the new baseline, and iterations can continue as long as features are available until a preferred model is reached for a given feature set. While using all available data points from the patient record can result in tens of thousands of inputs, the baseline evaluation can yield 10-12 useful feature inputs for effective model training before adding more inputs can result in diminishing returns.

[0132] Alternatively or additionally, model types can be evaluated. For example, a multiple regression model can be generated and compared to a more complex model, such as a random forest model, to determine whether it provides a better balance of precision and recall (and / or metrics) than the other models. One or more models can then be selected for use in a particular scenario (e.g., for a clinical trial or patient treatment, for a particular toxicity, or for an efficacy-related measure, etc.).

[0133] In some examples, a particular model is created and deployed to generate a particular output prediction from a set of patient healthcare data from one or more patient records. Because the model is trained and validated for a particular prediction, the model does not work for general predictions. For example, the model may not be able to successfully predict any toxicity (e.g., including any of pneumonia, colitis, hepatitis, etc.). Such a model would instead produce chance. However, models directed to predicting colitis, predicting pneumonia, predicting hepatitis, etc. from immunotherapy may generate reliable and actionable predictions to drive cohort composition for clinical trials, establish and / or modify patient treatment plans, etc., as described above.

[0134] Exemplary Model Deployment and Usage Systems and Related Methods In some examples, the clinical system can utilize one or more deployed models to drive tools, etc., for evaluating efficacy and / or toxicity associated with immunotherapy for patients and / or patient populations. Using one or more deployed models, patients can be evaluated at the initiation of an immunotherapy treatment regimen, during administration of an immunotherapy treatment regimen, as candidates for immunotherapy clinical trials, etc. In some examples, previous efficacy and / or toxicity model predictions for a patient can be used along with updated efficacy and / or toxicity model predictions for the patient (e.g., a combination of previous and current results, using previous results as inputs to generate current results, etc.). As such, models are trained and validated using a collection of historical healthcare data for various patients, and then applied to a given patient to evaluate outcomes and treatments for that patient.

[0135] 15 illustrates an exemplary immunotherapy prediction device 1500 comprising an exemplary input processor circuit 1510, an exemplary memory circuit 1520, an exemplary model processor circuit 1530, an exemplary output generation circuit 1540, and an exemplary interface 1550. As shown in the example of FIG. 15, the exemplary input processor circuit 1510 processes inputs from model sources 1560 (e.g., model generator devices, model repositories, EHRs, EMRs, etc.) as well as data sources 1565 (e.g., EHRs, EMRs, laboratory systems, clinical information systems, scheduling systems, etc.).

[0136] Inputs may occur at different times, e.g., from model source 1560 and data source 1565. For example, one or more models may be retrieved periodically (e.g., via push and / or pull, etc.) from model source 1560 and stored in model storage 1522 of memory circuit 1520. Patient data and / or other input data may be retrieved periodically, according to a schedule (e.g., according to a scheduled exam or other appointment, etc.), on-demand when requested by a user (e.g., by a clinician, administrator, triggered by a research record, etc.), etc., from data source 1565 and stored, e.g., in data storage 1524 of memory circuit 1520, and provided to one or more models by model processor circuit 1530. Input data may include structured data (e.g., admission information, discharge information, prescription information, billing codes, diagnosis codes, labs, etc.), as well as unstructured manually curated medical record data, images, etc.

[0137] The models stored in the model storage 1522 have been trained and validated by another system, such as the model generation device 100. For example, one or more models may be selected for use in the exemplary device 1500 by the model processor circuit 1530 based on input patient data and / or other clinical data and / or by the input processor circuit 1510. The models may be selected according to various model selection criteria. For example, a model may be selected to predict the likelihood that toxicity, such as pneumonia, hepatitis, colitis, etc., will occur due to immunotherapy following a patient's treatment plan. Alternatively, or in addition, a model may be selected to predict the efficacy of an immunotherapy treatment for a patient. For example, a model may be selected to initially determine the efficacy of an immunotherapy treatment plan for a patient. The same and / or a different model may be selected to determine the ongoing efficacy of an immunotherapy treatment plan for a patient. For example, a model may be selected to assess a patient's suitability for an immunotherapy clinical trial.

[0138] For example, the models may include toxicity-related predictive models (e.g., hepatitis predictive models, colitis predictive models, pneumonia predictive models, etc.), efficacy-related predictive models (e.g., immunotherapy efficacy models, survival models, duration of treatment models, etc.), etc. The models may be low precision and high recall, high precision and low recall, harmonic mean (F1) maximized, majority vote undersampled, etc. Selection may be made by and / or for the exemplary model processor circuit 1530 (e.g., based on settings, mode, type, query, patient identifier, other model selection criteria, etc.). For example, when configuring participation in a clinical trial, the focus is on F1 score maximization, thereby identifying patients with the highest likelihood of responding well to a given immunotherapy drug. However, when determining clinical treatment for a patient, the goal is to eliminate toxicity to the patient as much as possible. As such, models developed with high recall are more important, and low precision is acceptable due to the availability of inexpensive treatment options for hepatitis, colitis, pneumonia, etc. Goals may also depend, for example, on patient preferences (e.g., preference for aggressive treatment, emphasis on quality of life, adjustments for life events, etc.) Such models may also be evaluated according to certain thresholds or other patient selection criteria.

[0139] The model processor circuit 1530 processes input data (e.g., from the input processor circuit 1510, data store 1524, etc.) using one or more selected models (e.g., from the input processor circuit 1510, model store 1522, etc.). The model processor circuit 1530 generates one or more predictive outputs based on the one or more provided inputs. The outputs and / or other content may be processed by the output generation circuit 1540 for output to an external device or system 1570 via an interface 1550. For example, the output generation circuit 1540 may combine toxicity and / or efficacy predictions together and / or further with images, descriptions, treatment plan information, patient data, clinical trial information, etc.

[0140] In operation, the example input processor circuit 1510 processes input patient data related to one or more patients, such as lab results, diagnosis codes, billing codes, etc. The input may be from one or more external systems 1565, such as an EHR, EMR, etc. The example input processor circuit 1510 can extract and organize the input in a chronological order, for example, for a patient. In some examples, the input processor circuit 1510 aligns the input data with respect to an anchor point (date of first disease symptom, date of diagnosis, date of first immunotherapy treatment, etc.) to organize the input data in chronological order.

[0141] In some examples, the time series data is formed into a plurality of features by the input data processor 1510. These features can form a set of patient features that identify and / or classify healthcare data that is input into one or more models using the model processor circuitry 1530. Feature engineering by the input data processor 1510 can form a plurality of features based on codes (e.g., ICD-10 codes, etc.). For example, ICD-10 codes for a patient in a given year can be processed to identify codes related to lung and / or respiratory function (e.g., C34, C78, ​​etc.), and a time series of those codes can form a function that is used to calculate the relative likelihood of lung disease in the patient. Similarly, a plurality of features can be formed to predict the development of pneumonia, hepatitis, colitis, and / or other toxicity from an immunotherapy treatment. Features can be formed based on lung function, liver biomarkers, blood tests (e.g., blood, plasma concentrations, etc.), medications taken, etc.

[0142] The patient features and / or other patient inputs are applied to one or more selected models by the model processor circuit 1530. In some examples, a single feature input set or string is provided to the model (e.g., a feature set of ICD-10 codes for the patient, etc.). In other examples, multiple inputs are applied to one or more models, including multiple features, previous model output predictions (e.g., previous predictions of efficacy and / or toxicity input to the model for updated predictions), different model output predictions (e.g., providing an efficacy model prediction output as an input to a toxicity model, providing a toxicity model prediction output as an input to an efficacy model, etc.). The model processor circuit 1530 generates outputs from the models based on the inputs.

[0143] Outputs from models in the model processor circuit 1530 are provided to an output generation circuit 1540, which processes predictions and / or other outputs of the models. For example, outputs from multiple models in the model processor circuit 1530 can be compared by the output generator circuit 1540 to form a resultant output that is provided to an interface 1550, another system, etc. The output generation circuit 1540 can post-process the output to validate the output, compare the current output to previous and / or other current output predictions, provide feedback to the model source 1560, reformat the output, etc. In some examples, the output generation circuit 1540 can correlate model outputs with other data, such as image data, to generate qualified or refined outputs and / or other correlated / validated results.

[0144] In some examples, the output of the model is explainable, such as by providing an indication of the input features, rules, model layers, etc. that resulted in the output prediction. The output generation circuit 1540 can utilize the explanation accompanying the output to drive decision-making and actionable output related to, for example, treatment plans for patients, clinical trials, and / or other next steps.

[0145] In some examples, patient selection criteria, such as thresholds, may be applied to the model output. For example, the model output may represent the probability of a particular outcome (e.g., the probability or likelihood of a particular experiencing a particular toxicity during immunotherapy treatment, the probability that the treatment will be successful or efficacious, etc.) based on the data and associated features and correlations used to generate the model. The patient selection threshold may be used to set a point in the chain of model outputs at which an action or event trigger or decision occurs. That is, if the predicted model output value falls on one side of the threshold, one action occurs (e.g., the patient enters immunotherapy, etc.), but if the model predicted output falls on the other side of the threshold, the other action occurs (e.g., the patient is removed from treatment, etc.). As such, the model may provide, for example, a binary prediction (e.g., in or out). Alternatively, the model may provide, for example, a multiple outcome prediction (e.g., no, yes, yes with a caveat, etc.).

[0146] The output generating circuit 1540 can incorporate, for example, the output prediction of efficacy and / or toxicity into an immunotherapy treatment plan for the patient. Alternatively or in addition, the output generating circuit 1540 can act as a trigger for, for example, including or excluding a patient from a clinical trial or study based on the output. A clinical trial can be considered, for example, a treatment plan involving immunotherapy for a cohort of patients. The output generating circuit 1540 can utilize the output as a trigger for modifying an existing immunotherapy treatment plan for a patient (e.g., continuing, ceasing, increasing, decreasing administration of an immunotherapy agent to the patient, etc.) based on an increasing probability of toxicity, a decreasing probability of toxicity, increasing efficacy, decreasing efficacy, etc. In some examples, the prediction drives modification of the treatment plan to address an increasing likelihood of toxicity, such as prescribing steroids to treat a patient's pneumonia while continuing a course of immunotherapy treatment, pre-treating a predicted onset of hepatitis based on a determined likelihood of liver toxicity. In some examples, the output generation circuitry 1540 generates alerts and / or otherwise provides decision support for making changes to treatment plans, clinical trials, etc. Current and previous predictions, along with old and new data points, can drive treatment plans, adjustments, updated models, etc. in a dynamic looping system.

[0147] The output generation circuit 1540 can, for example, store the output in the data store 1524. The output generation circuit 1540 provides output for transmission, such as graphically, via the interface 1550 to the external system 1570 as inputs / commands / settings for configuration of the external system 1570 (e.g., activate a treatment plan, adjust a treatment plan, form a cohort of patients for a clinical trial, initiate a clinical trial, etc.) and / or other actionable output.

[0148] In some examples, the output generation circuit 1540 triggers instructions (e.g., visual instructions on a display, text instructions in a patient record or report, audible alert) corresponding to the processing of the predicted output for the patient. The instructions may differ based on whether the patient receives / accepts / selects in the recommendation (e.g., included in a clinical trial, selected for a new / ongoing treatment, etc.) or is declined / removed (e.g., removed from a clinical trial, declined a new treatment, removed from a current treatment, etc.). The output may also include associated actions such as including the patient in a cohort for a clinical trial, excluding the patient from a clinical trial, generating an order for a patient to start an immunotherapy treatment, canceling an order for an ongoing immunotherapy treatment, connecting the clinician and / or patient to a scheduling and / or ordering system, updating the patient record, etc.

[0149] As such, the exemplary device 1500 is a digital tool that may be used to select patients for clinical trials, as well as to develop and deploy therapeutics and monitor patient treatment. The exemplary device 1500 allows potentially associated toxicities with immunotherapy, such as hepatitis, pneumonitis, colitis, etc., to be evaluated by one or more models, while assessing the efficacy of the immunotherapy for a particular patient (e.g., the patient's chances of survival (with and / or without immunotherapy), progression-free survival, duration of treatment, etc.).

[0150] The exemplary device 1500 can provide multiple predictions to a patient over time (e.g., periodically, at specific milestones, as the patient's condition and / or response to treatment progresses, etc.). Comparison of multiple predictions by device 1500 allows for an assessment of the risk-to-benefit ratio for the patient of an immunotherapy treatment plan. Based on this ratio, for example, treatment may be continued or increased when the benefits outweigh the risks, or reduced or discontinued when the risks outweigh the benefits. In some examples, multiple model prediction outputs may be compared to determine trends, update models, drive treatment plans, initiation and / or changes to clinical trials, etc.

[0151] For example, an evaluation for constructing a cohort for a clinical trial may use only a toxicity-related model since efficacy is not yet known (or sufficient efficacy-related data is not available to make a reliable prediction) and / or a pharmaceutical company may be focused on reducing toxicity regardless of predicted efficacy. An evaluation for a patient's course of treatment may use a combination of toxicity-related and efficacy-related models, as weighted or modified by one or more patient selection criteria or factors (e.g., patient severity, disease severity, patient motivation, lifestyle, overall patient health, or comorbidities, etc.), to determine whether the balance of toxicity risk and treatment efficacy favors or disfavors initiating treatment of the patient on a treatment regime, continuing the treatment regime, joining a clinical trial cohort, etc.

[0152] 16 and 17 are flow charts of an exemplary process representing computer-readable instructions storable in a memory circuit and executable by a processor circuit to implement and operate the exemplary immunotherapy prediction device 1500 of FIG. 15. The exemplary process 1600 of FIG. 16 begins at block 1610, where a request for a prediction and / or other processing trigger is received by the exemplary prediction device 1500. For example, a request for a predictive modeling output is received via the interface 1550 (e.g., by user selection via a graphical user interface, by initiation of a software program, via the input processor circuit 1510, in some other manner from an external system 1570, etc.). The request may include a toxicity prediction associated with a regimen for immunotherapy treatment (an "immunotherapy treatment regimen"), an efficacy prediction for the immunotherapy treatment regimen, the likelihood of successful incorporation into an immunotherapy clinical trial, etc.

[0153] At block 1620, one or more models are loaded, such as from the model store 1522 and / or external model source 1560, for processing as requested by the model processor circuit 1530. For example, RF, GB, and / or other models may be loaded based on the predictions (e.g., toxicity, efficacy, eligibility, initial vs. ongoing, etc.) desired and / or otherwise triggered by the request. Model selection criteria, such as model type, purpose (e.g., target or goal, such as inclusive testing, exclusive testing, easy treatment, aggressive treatment, likely success, probable success, low risk, high risk, etc.), may be used, for example, to select the model to load and use. At block 1630, patient and / or other healthcare data to be input into the model is loaded for the model processor circuit 1530 (e.g., from the data store 1524, external data source 1565, other patient healthcare records, etc.).

[0154] At block 1640, the health care data for a given patient is processed using the selected model. For example, the model processor circuit 1530 inputs data into the selected model, which generates an output. For example, based on codes and / or other patient data related to lung function, liver function, blood tests, etc., the model determines the likelihood of hepatitis, pneumonia, colitis, and / or other toxicity for the patient over the course of immunotherapy treatment. Alternatively, or in addition, the model can process the inputs to determine the likelihood that efficacy of the immunotherapy will drive prescribing a treatment plan, adjusting the treatment plan, selecting for a clinical trial, etc.

[0155] At block 1650, the output may be adjusted via post-processing, comparison, additional model outputs, etc. For example, the output of one or more models from the model processor circuit 1530 is further processed by the model processor circuit 1530 applying another model, comparing model outputs, scaling / refine / otherwise selecting model outputs, etc., and / or by an output generation circuit forming actionable outputs from the model predictions.

[0156] In some examples, the output is compared to a threshold or other patient selection criteria to determine an associated next action. For example, the patient selection criteria may be determined based on a desired relationship or balance between precision and recall for the model (e.g., a high threshold to include, a middle threshold to include, a low threshold to include, etc.). As such, the model output may be evaluated with respect to the patient selection criteria, and different instructions and associated actions may be generated (e.g., as actionable results or outputs) based on this comparison. For example, if the model prediction output meets the patient selection criteria (e.g., exceeds a threshold), a first instruction and associated action to include the patient (e.g., in a clinical trial, in a treatment, to continue treatment, etc.) is generated. However, when the model prediction output does not meet the patient selection criteria (e.g., does not meet a threshold condition), a second instruction and associated action to exclude the patient (e.g., from a clinical trial, from a treatment, to discontinue treatment, etc.) is generated.

[0157] The patient selection criteria may be provided in the form of values ​​that divide the output into greater than, less than, greater than or equal to, less than or equal to, etc. The patient selection criteria may establish a lower bound or threshold, an upper bound or threshold, a range of acceptable or unacceptable values ​​(e.g., having a maximum and minimum threshold), etc. The patient selection criteria may be determined based on an analysis of model characteristics, such as, for example, precision and recall graphs associated with the model's output. As such, the characteristics of a particular model, alone or in combination with external / additional concerns for patient care, clinical trial development, etc., may be processed to generate patient selection criteria (e.g., in the form of one or more single thresholds, associated ranges, etc.) and trigger subsequent actions based on where the model prediction for a given patient falls with respect to the patient selection criteria.

[0158] At block 1660, actionable results are provided. For example, the output generation circuit 1540 generates a visual output for the interface 1550 from the processed predictions of the model from the model processor circuit 1530. As another example, instructions and / or prescriptions for new immunotherapy treatment plans and / or for modifications to existing immunotherapy treatment plans may be output by the output generation circuit 1540 to the external system 1570 via the interface 1550. As another example, instructions and / or notifications / alerts / instructions to include or remove a patient from an immunotherapy clinical trial may be output by the output generation circuit 1540 to the external system 1570 via the interface 1550. Display of the inclusion / exclusion, etc. instructions and instructions may form an order generated by the output generation circuit 1540 and provided via the interface 1550 to drive actions in another system (e.g., scheduling, management, clinical care systems, clinical trial databases, etc.) related to the patient / patient record. The output or instructions may be driven, for example, by comparing the predictions to thresholds or other patient selection criteria.

[0159] 17 illustrates further details for an example implementation of processing input data using one or more models (e.g., block 1640 of example process 1600). At block 1710, the preprocessed input is applied to one or more selected models to process the input at block 1720. For example, input patient data relating to one or more patients, such as lab results, diagnostic codes, billing codes, etc., may be aligned in time series with respect to an anchor point (e.g., a reference event or time point in the patient's health history and records) and applied to the models in the model processor circuit 1530.

[0160] For example, the model processor circuit 1530 inputs data into a selected model, and the model processor circuit 1530 generates an output. For example, based on codes and / or other patient data related to lung function, liver function, blood tests, etc., the model determines the likelihood of hepatitis, pneumonia, colitis, and / or other toxicity to the patient over the course of immunotherapy treatment. Alternatively, or in addition, the model can process the inputs to determine the likelihood that efficacy of the immunotherapy will drive prescribing a treatment plan, adjusting the treatment plan, selecting for a clinical trial, etc.

[0161] In some examples, the time series data is formed into a plurality of features by the input data processor 1510. These features can form a set of patient features that are input into one or more models using the model processor circuit 1530. Feature engineering by the input data processor 1510 can form a plurality of features, for example, based on codes (e.g., ICD-10 codes, etc.). For example, ICD-10 codes for a patient in a given year can be processed to identify codes related to pulmonary and / or respiratory function (e.g., C34, C78, ​​etc.), and a time series of those codes can form a function that is used to calculate the relative likelihood of pulmonary disease in the patient. Similarly, a plurality of features can be formed to predict the development of pneumonia, hepatitis, colitis, and / or other toxicity from an immunotherapy treatment. Features can be formed based on pulmonary function, liver biomarkers, blood tests (e.g., blood, plasma concentrations, etc.), etc.

[0162] The patient features and / or other patient inputs are applied to one or more selected models by the model processor circuit 1530. In some examples, a single feature input set or string is provided to the model (e.g., a feature set of ICD-10 codes for the patient, etc.). In other examples, multiple inputs are applied to one or more models, including multiple features, previous model output predictions (e.g., previous predictions of efficacy and / or toxicity input to the model for the updated predictions), different model output predictions (e.g., providing an efficacy model prediction output as an input to a toxicity model, providing a toxicity model prediction output as an input to an efficacy model, etc.).

[0163] At block 1730, the selected model is evaluated to determine whether the model includes multiple related models. For example, the model processor circuit 1530 evaluates the selected model to determine whether the model includes related models for efficacy and toxicity of the immunotherapy. The model processor circuit 1530 may also evaluate the selected model to determine whether the model includes a current model and a previous model or model output. At block 1740, if there are selected related models to process, the model processor circuit 1530 applies one or more outputs between the related models. For example, the model processor circuit 1530 may apply previous predictions in comparison to the new model prediction output and / or as input to the new model. The model processor circuit 1530 may compare and / or otherwise process the predictions of efficacy and toxicity, for example.

[0164] At block 1750, the output from the model processor circuit 1530 is post-processed. For example, the model processor circuit 1530 and / or the output generation circuit 1540 process the output of the model predictions to develop and / or otherwise adjust a treatment plan, develop instructions related to a treatment plan for clinical care or a clinical trial, etc. The predictive model output can be correlated, for example, with image data and / or other data. The predictive model output can be compared to thresholds and / or evaluated with respect to other patient selection criteria to translate the prediction into an action to be taken (e.g., automatically and / or by a clinician, etc.). Explanations and / or other actionable information / instructions can be associated with the predictive output to make the output actionable by another system, program, device, etc.

[0165] At block 1760, a processed actionable predictive output is provided. For example, the output generating circuit 1540 can incorporate the output predictions, for example, of efficacy and / or toxicity, into an immunotherapy treatment plan for the patient. Alternatively or in addition, the output generating circuit 1540 can act as a trigger, for example, to include or exclude the patient from a clinical trial or study based on the output. The output generating circuit 1540 can utilize the output as a trigger to modify an existing immunotherapy treatment plan for the patient (e.g., continue, stop, increase, decrease, etc., administration of an immunotherapy agent to the patient) based on an increased probability of toxicity, a decreased probability of toxicity, increased efficacy, decreased efficacy, etc. In some examples, the prediction drives modification of the treatment plan to address an increased likelihood of toxicity, such as prescribing steroids to treat the patient's pneumonia while continuing the course of immunotherapy treatment, pre-treating a predicted onset of hepatitis based on a determined likelihood of liver toxicity. In some examples, the output generating circuit 1540 provides decision making, decision support, and / or other notifications / alerts to affect changes to a treatment plan, clinical trial, etc. Current and previous predictions, along with old and new data points, can drive treatment plans, adjustments, updated models, etc. in a dynamic looping system. The output generation circuit 1540 provides actionable output to the interface 1550 for display and / or other distribution, for example, to an external system 1570 and / or other connected devices.

[0166] As such, one or more models may be generated, deployed, and configured for use in predicting toxicity resulting from an immunotherapy treatment, immunotherapy treatment efficacy, combinations thereof, and the like. In some examples, a model (e.g., a toxicity-related model, an efficacy-related model, etc.) is deployed as part of a program or platform that loads the model and provides healthcare data from a healthcare record for a given patient to the model. The output of the model, as evaluated against a single threshold, range, or other patient selection criteria, is used to construct a treatment plan for the given patient. The treatment plan includes the immunotherapy treatment and may include a cohort of patients forming, for example, a clinical trial.

[0167] FIG. 18 illustrates another exemplary process 1800 for configuring a clinical trial or treatment plan associated with a patient. At block 1810, models are loaded based on model selection criteria. For example, models such as toxicity-related models (e.g., pneumonia toxicity model, colitis toxicity model, hepatitis toxicity model, etc.), efficacy-related models (e.g., survival model, treatment duration model, etc.), etc., may be loaded based on model selection criteria such as permissive or inclusive model targets (e.g., include patients even if they fail), exclusive model targets (e.g., exclude patients even if they fail), etc. As such, depending on the purpose or goal of including as many patients as possible in the study, limiting the number of patients in the study, treating patients aggressively with immunotherapy, treating patients conservatively with immunotherapy, etc., a model trained to favor recall over precision, favor precision over recall, balance precision and recall, etc. may be selected.

[0168] In some instances, the objectives or motivations may change over time (e.g., a patient begins treatment cautiously and then becomes more aggressive, a patient's situation changes, warranting less toxic and / or more efficacious treatment, a pharmaceutical company desires to perform an initial screen and then switch to more extensive validation, etc.) As such, model loading may be repeated with different model selection criteria over time.

[0169] At block 1820, patient selection criteria are determined. For example, the patient selection criteria may be based on or correlated with the model selection criteria (e.g., both restrictive, both permissive, etc.). However, the patient selection criteria may deviate from or otherwise differ from the model selection criteria. For example, a model predicting patient toxicity from immunotherapy treatment may be permissive for clinical trials, but the patient selection criteria may be adjusted to seek a different balance of true positives, true negatives, false positives, and false negatives in selecting patients.

[0170] A threshold or other patient selection criterion is used to define a balance between precision and recall associated with a confidence level that the prediction provided by the model is correct. The threshold may be set based on one or more factors, such as confidence in predicting that one person will not develop toxicity versus confidence in predicting that another person will develop toxicity. One treatment plan may prioritize being correct that someone will develop toxicity, while the other treatment plan may prioritize being correct that someone will not develop toxicity, for example. Similarly, one treatment plan may prioritize being correct that someone will survive, while the other treatment plan may prioritize being correct that someone will not survive. Some treatment plans, whether for individual patient treatment or for clinical trials, may prioritize, for example, a particular minimum precision, a particular minimum recall, or a particular balance between precision and recall in order to proceed with an action triggered by a prediction.

[0171] In some examples, the patient selection criteria is a threshold value set to a value between 0.0 and 1.0. The threshold value can be used to determine the next course of action for the patient based on the model's predictive output. For example, the patient selection criteria can be set to a threshold value of 0.5. A first patient with a toxicity prediction of less than 0.5 can then be flagged for inclusion in the clinical trial, while a second patient with a toxicity prediction of more than 0.5 can be flagged for exclusion from the clinical trial. As another example, a threshold value is set to 0.3 for the patient selection criteria, and a first patient with an efficacy-related prediction of 0.3 or greater is flagged for treatment with an immunotherapeutic drug, while a second patient with an efficacy-related prediction of less than 0.3 is not included.

[0172] In some instances, the objectives or motivations may change over time (e.g., a patient begins treatment cautiously and becomes increasingly aggressive, the patient's situation changes, warranting a less toxic and / or more efficacious treatment, a pharmaceutical company may want to perform an initial screening and then switch to more extensive confirmation, etc.). As such, a patient's healthcare data may be re-evaluated in the same and / or different models with outputs transformed according to different patient and / or model selection criteria over time. For example, a patient may only be looking for an immunotherapy treatment that is less ill and has a low risk of toxicity. If the patient's condition worsens, the goal may switch to looking for a treatment that is likely to be efficacious for that patient, regardless of toxicity. For example, a different model may be selected and updated patient data may be processed with the updated model.

[0173] At block 1830, the patient healthcare data is processed using the selected model. For example, healthcare data from a patient's healthcare record (e.g., EMR, EHR, other clinical or healthcare database, etc.) is provided to the selected model (e.g., toxicity-related model, efficacy-related model, etc.). The patient data (e.g., blood tests, prescription history, other biometric and / or laboratory data, etc.) is processed by the model, which correlates features associated with the patient data, weights the features, and generates predictions (e.g., predicting the likelihood of toxicity from an immunotherapy treatment, predicting the likelihood of efficacy of an immunotherapy treatment, etc.). As noted above, the models are trained, tested, and deployed to aim for specific predictions, such as the likelihood of a particular toxicity, the efficacy of a particular immunotherapy drug, etc., rather than providing a general indication, such as, for example, the likely toxicity of a patient taking a drug, or the general efficacy of a drug to treat a particular type of cancer. As such, the model provides a specific predictive output using data from a particular patient's medical record.

[0174] At block 1840, the model output is evaluated. For example, the model output can be a predicted value in the range of 0.0 to 1.0. The model predicted output can be evaluated, for example, with respect to patient selection criteria. As described above, patient selection criteria or thresholds can be set to allow for low precision / high recall (low threshold, such as 0.0-0.3), a balance between precision and recall (medium threshold, such as 0.4-0.6), high precision / low recall (high threshold, such as 0.7-1.0), etc. Such thresholds or other patient selection criteria can be selected based on tolerance or preference for correctly predicting positive results, incorrectly predicting positive results, correctly predicting negative results, incorrectly predicting negative results, etc.

[0175] Based on the evaluation of the model output according to the patient selection criteria, the following actions are triggered: When the patient's model output meets the patient selection criteria, in block 1850, an initial instruction is generated to include the patient in a clinical trial, treatment plan, etc., and an associated action is triggered (e.g., generate a prescription order for the patient, include the patient in a clinical trial or research cohort, modify the patient's treatment plan, etc.). When the patient's model output does not meet the patient selection criteria, in block 1860, a second instruction is generated to exclude the patient from the clinical trial, treatment plan, etc., and an associated action is triggered (e.g., note in the patient record that the patient has been excluded from the clinical trial or treatment, modify the patient's treatment plan, etc.). In block 1870, the instruction and associated action are output. For example, the output may be displayed on a screen, sent as a message to the responsible clinician, sent as a message to the patient, generated into an order for an immunotherapy or other regimen, routed to a scheduling system for an appointment, used to update the patient's medical record or clinical trial cohort, etc.

[0176] In some instances, the decision may not be a binary treat / no treat decision. Instead, the decision may be to treat the patient according to the immunotherapy for a period of time and / or with additional testing, monitoring, etc. to help ensure that the patient does not begin to experience side effects from the treatment (and / or is responding well to the treatment). As such, previous decisions / outputs can be fed back into the model, which can then be periodically re-evaluated based on updated patient information, etc.

[0177] As described above, the balance between precision and recall can be determined for a single model and / or for a set of related models. To make the model output (e.g., prediction) actionable, a patient selection criterion (e.g., a threshold or other selection criterion) is set, above which a first action is performed, and below which a second action is performed. A low threshold or selection criterion results in a low precision rate in the model (e.g., many false positives), but a high recall rate in the model (e.g., few false negatives). An intermediate threshold or other selection criterion results in a good balance between false positives and false negatives. The selection of the intermediate threshold depends on the characteristics and performance of a particular model, and thus it may vary from model to model rather than uniform. A high threshold or other selection criterion results in a high precision rate in the model (e.g., very few false positives), but a low recall rate (e.g., many false negatives).

[0178] For example, the predictive output of a single toxicity-related model may be evaluated based on patient selection criteria / thresholds that are set at a low threshold to accept more false positives (e.g., more toxicity predictions than actually occur) with the goal of enrolling more people in clinical trials. As another example, an efficacy-related model, such as an overall survival model, may be configured or configured according to patient selection criteria that are set at a high threshold to only tolerate a small number of false positives.

[0179] Thus, as discussed above, a large amount of aggregated patient-related healthcare data may be processed according to a subset of features associated with the data 1910, for example as depicted in representation 1900 of FIG. 19. The date 1920 when a first immunotherapy checkpoint inhibitor (CPI) was administered may be used as an anchor point in aligning the aggregated patient-related dataset for use in training and testing the models to form a plurality of models 1930M1, M2, M3, etc. One of the models M1, M2, Me may be selected based on model selection criteria such as emphasis on precision (e.g., how many items retrieved are relevant), emphasis on recall (e.g., how many related items are retrieved), a balance between precision and recall, etc., and deployed for use in an application, platform, service, system, etc.

[0180] Example Model Characteristics and Associated Outputs As described above, models are constructed to answer specific questions, such as: what is the patient's risk of a particular toxicity? What is the likelihood that this immunotherapy will be effective for this patient? Such questions, also referred to as targets or objectives, allow available data to be organized and processed to form connections and correlations to predict potential (e.g., likely) answers to these questions. The model takes data, including outcomes observed in a patient data pool (e.g., this person survived the treatment, this person suffered from colitis during this treatment, etc.), forms a network of connections, weights, and other correlations, and generates a probabilistic output. As such, the model can turn patient information into a probability based on a set of other data and outcomes. Such probabilities can be expressed, for example, in precision and recall.

[0181] In some examples, each model may be characterized by a relationship or combination of precision and recall associated with its output. Based on the characteristics of the model, as exemplified by the precision and recall of the associated model output, a threshold or other patient selection criteria may be set to process or interpret the model output. For example, the characteristics or behavior of the model, as reflected in the precision and recall curves associated with a particular model, may be evaluated to set the patient selection criteria accordingly. If the goal or objective is to reduce or minimize false positives, the threshold may be set accordingly based on an evaluation of the precision curve associated with the model. If the goal or objective is to reduce or minimize false negatives, the threshold may be set accordingly based on an evaluation of the recall curve associated with the model. In many cases, if the false positives and false negatives are balanced, an analysis of both the precision curve and the recall curve is performed to set the threshold of the patient selection criteria for use of the model.

[0182] Setting a threshold as a patient selection criterion indicates that a particular balance of precision and recall (or a dichotomy between them) is satisfactory in order to take action based on a particular model. For example, a clinician or researcher may be satisfied with a particular percentage of error or inaccuracy in the prediction. Alternatively, or in addition, a clinician or researcher may require a particular confidence in the prediction. Such risk tolerance may depend on the patient, the immunotherapy, health concerns, liability concerns, etc.

[0183] 20-24 illustrate exemplary precision / recall graphs that relate to the reliability of model output. As shown in the following examples, model characteristics are reflected in the graphs, and different thresholds can be set as patient selection criteria based on specific behaviors or characteristics of the model. For example, data including patient outcomes or results are used to generate or develop a model (e.g., train and validate a model, which can then be deployed for use). As such, the results are used to generate a model, which is then used to generate probabilities associated with a particular target, focus, or question. The thresholds are determined based on one or more concerns, requirements, boundaries, preferences, etc., and specify a particular combination of precision (e.g., positive predictive value) and recall (e.g., true positive rate or sensitivity) of the model.

[0184] By specifying a particular threshold, the model's percentage of positive predictions and its true positive rate are acceptable for the model's purpose in that context. The threshold may change based on the context (e.g., individualized patient treatment, cohort treatment in a clinical trial, patient life stage, disease severity, patient preferences, etc.) and / or over time (e.g., as patients change, diseases / symptoms change, preferences change, etc.). As such, the model may have applicability to a variety of contexts based on the threshold selected for output for that particular patient in that particular context. As patients and / or a given patient's context change over time, the model may change as well.

[0185] In some examples, a model may be generated and configured into a particular state or mode (e.g., a "clinical trial mode" where the model identifies patients suitable for this clinical trial, a "patient treatment mode" where the model determines whether this particular treatment is likely to benefit this particular patient, etc.). As such, the model may be loaded and configured for a particular objective, which may be further refined by setting particular patient selection criteria / thresholds.

[0186] 20 illustrates an exemplary predictive output of an overall survival model. The x-axis 2010 of the graph corresponds to a threshold (e.g., patient selection criterion) and the y-axis 2020 of the graph corresponds to the survival probability (e.g., as modified by a precision curve 2030 or a recall curve 2040). The threshold 2010 selected for the patient selection criterion sets, for example, a point below which one action is taken and above which another action is taken.

[0187] In the example of FIG. 20, patient outcomes are fed into the model. The precision 2030 and recall 2040 curves indicate the likelihood of a true positive, false positive, or false negative outcome. For example, patient survival is a data point that would be classified as a negative outcome. A patient who did not survive is a data point that would be classified as a positive outcome. The associated probability is shown on the y-axis 2020. For example, between 0.0 and 1.0 (representing 0% chance to 100% chance), a probability greater than 0.5 indicates that the patient is more likely to not survive than to survive. A probability less than 0.5 indicates that the patient is more likely to survive than not to survive. A probability of 0.5 indicates a 50 / 50 chance of either outcome. The precision 2030 and recall 2040 curves indicate whether a sampling of these data points is likely to be a true positive, false positive, or false negative. The precision 2030 and recall 2040 curves facilitate the determination of corresponding patient selection criteria thresholds to use in making the model output actionable.

[0188] For example, as shown in FIG. 20, if a low threshold (e.g., 0.0 to 0.3) is selected, the model results in a low precision (e.g., many false positives) and a high recall (e.g., few false negatives). As such, a low threshold is less likely to falsely identify someone as alive, but more likely to falsely identify someone as dead. As a result, almost all patients predicted to survive after immunotherapy treatment will survive. However, the model will incorrectly predict many deaths that do not actually occur. In a pharmaceutical clinical trial use case, the inflated death counts will cause the model to sift or exclude too many patients from the trial. Also, in a clinical practice use case for patient treatment, the inflated death probabilities will cause the model to potentially withhold immunotherapy from too many patients.

[0189] For example, if a threshold of 0.1 is chosen in the example of Figure 20, then no incorrect survival rates will be predicted (no false negatives) since the recall 2040 is 1.0. However, there is approximately a 50 / 50 chance that the predicted mortality rate will be incorrect (false positive) since the precision 2030 is approximately 0.55.

[0190] If an intermediate threshold (e.g., 0.4 to 0.6) is chosen, there is a better balance between false positives and false negatives. In a pharmaceutical use case, the model will filter out or exclude patients who may have survived and filter in or include patients who will not survive. In a clinical practice situation, the model will make some inappropriate predictions, but most patients for whom immunotherapy is recommended will survive. If this threshold were applied to the model, most patients for whom treatment is not recommended would not have survived.

[0191] For example, if a threshold of 0.5 is selected in the example of FIG. 20, a relatively small number of false survival predictions are made, since the recall 2040 is about 0.75. In addition, the model provides a relatively small number of false predictions of lack of survival, since the precision is about 0.65. As such, a good balance between false positives and false negatives is provided at this threshold. Patients with a survival probability of at least 0.5, for example, would then be selected for treatment.

[0192] If a high threshold (e.g., 0.7 to 1.0) is chosen, the model output will have high precision (e.g., very few false positives) and low recall (e.g., many false negatives). As a result, almost all patients predicted to die will truly die, and the model will erroneously predict many survivals that do not actually occur. In the pharmaceutical context, the model will include too many patients in clinical trials who will not survive. In the clinical patient care context, the model will too often misdirect immunotherapy.

[0193] For example, if a threshold of 0.9 is chosen in the example of Figure 20, almost all predicted survival counts will be incorrect, since recall 2040 is 0.0. However, all predicted death counts will be correct, since precision 2030 is 1.0.

[0194] Using the precision 2030 and recall 2040 properties of the model depicted in FIG. 20, a patient selection criteria threshold 2010 may be selected to drive clinical trial cohort selection, patient treatment decisions, and the like. A model trained on a broader collection of prior healthcare data outputs a survival probability for a given patient by processing that particular patient's healthcare data to generate a predicted output representative of efficacy (using overall survival as a surrogate or proxy for efficacy of the immunotherapy treatment) for that patient. The model output for that patient is evaluated according to the determined threshold 2010 to determine whether the patient is an acceptable candidate for the clinical trial cohort, immunotherapy treatment regime, and the like. Based on the motivations and constraints of the clinical trial, the patient's personal circumstances and aspirations, and the like, the patient selection threshold 2010 may be adjusted to be less or more inclusive / exclusive to include or exclude certain patients from the clinical trial cohort, treatment plan, and the like, with the associated risk of inaccuracy / accuracy as the threshold moves according to the precision 2030 and recall 2040 curves for the model.

[0195] As shown in the example of Figure 20, selecting an intermediate threshold of 0.55 represents a crossover point between the precision 2030 and recall 2040 curves at approximately 0.7. Such a crossover point results in good precision and good recall with less than one-third false positives and less than one-third false negatives, such that although the model will make some inaccurate predictions, most patients for whom immunotherapy is recommended will survive and most patients for whom immunotherapy is not recommended will not survive. As such, overall patient survival is improved due to the properties of the model.

[0196] The data and associated features were used to generate the model depicted in FIG. 20. Features used to train and test this model included features representing aggregations of laboratory measurements over a period of time (e.g., 120 days, 100 days, 30 days, etc.) as well as aggregations of patient status, medications, etc. over a period of time (e.g., 1 year, 9 months, 6 months, etc.), which may be generated using one hot encoding and methods such as final, median time-weighted average (twa), mean, median, etc. Features included lymph node percentage, blood albumin (e.g., final, twa, median, etc.), dexamethasone drug count, chemotherapy-related treatment count (e.g., number of chemotherapy administered during the aggregation period), red blood cell distribution width (RDW) (e.g., twa, final, median, etc.), red blood cell count (RBC), hemoglobin (twa, median, final, etc.), renal function, lung function, cancer stage, cancer recurrence, etc. As described above, features are combined, weighted, and iterated to generate a model, reflected in the example in Figure 20. This model can be deployed and used to predict outcomes for patients.

[0197] 21 illustrates an exemplary predictive output of a pneumonia toxicity model. The x-axis 2110 of the graph corresponds to a threshold (e.g., a patient selection criterion) and the y-axis 2120 of the graph corresponds to a pneumonia toxicity development probability (e.g., a precision curve 2130 or a recall curve 2140). The threshold 2110 selected for the patient selection criterion sets a point below which one action is taken and above which another action is taken.

[0198] In the example of FIG. 21, a negative result indicates that the patient does not suffer from pneumotoxicity, and a positive result indicates that the patient does suffer from pneumotoxicity. If a low threshold (e.g., 0.0 to 0.25) is selected, the model results have low precision (e.g., many false positives) and high recall (e.g., few false negatives). As a result, almost all patients predicted not to suffer from pneumotoxicity as a result of receiving immunotherapy do not suffer from pneumotoxicity. However, the model incorrectly predicts many pneumotoxicities in patients that do not actually occur. In a pharmaceutical clinical trial use case, the inflated number of pneumotoxicities would cause the model to sieve or exclude too many patients from the trial. Also, in a clinical practice use case for patient treatment, the inflated probability of toxicity would cause the model to potentially withhold immunotherapy from too many patients.

[0199] For example, if a threshold of 0.1 is chosen in the example of Figure 21, almost all predictions of no toxicity are correct, since the recall 2140 is almost at 1.0. However, most predicted toxicities are incorrect, since the precision 2130 is slightly above 0.1.

[0200] However, if an intermediate or high threshold is selected, precision 2130 and recall 2140 will both be low, resulting in a large number of false positives and false negatives. As such, many will be excluded from clinical trials or treatments.

[0201] For example, if a threshold of 0.3 is selected in the example of Figure 21, almost all predictions of no toxicity are incorrect since the recall 2140 is 0.3. In addition, most predicted toxicity is incorrect since the precision 2130 is just over 0.2. Similarly, if a threshold of 0.45 is selected, all predictions of no toxicity and toxicity are incorrect since both the precision 2130 and recall 2140 are 0.0.

[0202] Using the precision 2130 and recall 2140 characteristics of the model depicted in FIG. 21, a patient selection criteria threshold 2110 may be selected to drive clinical trial cohort selection, patient treatment decisions, and the like. The model, trained on a broader collection of prior healthcare data, outputs the probability of developing pulmonary toxicity from immunotherapy for a given patient by processing that particular patient's healthcare data to generate a predicted output representative of pulmonary toxicity for that patient. The model output for that patient is evaluated according to the determined threshold 2110 to determine whether the patient is an acceptable candidate for the clinical trial cohort, immunotherapy treatment regime, and the like. Based on the motivations and constraints of the clinical trial, the patient's personal circumstances and aspirations, and the like, the patient selection threshold 2110 may be adjusted to be less or more inclusive / exclusive to include or exclude certain patients from clinical trials, treatment plans, and the like, with the associated risk of inaccuracy / accuracy as the threshold moves according to the precision 2130 and recall 2140 curves for the model.

[0203] In the example of FIG. 21, the intersection of the precision 2130 and recall 2140 curves occurs at approximately 0.34. However, at this intersection, both precision 2130 and recall 2140 are poor at less than 0.3, respectively, indicating more than 70% false positives and more than 70% false negatives. As such, this intersection is poor for both precision and recall. In the pneumonia toxicity model of FIG. 21, precision 2130 and recall 2140 are not well aligned, with precision 2130 being poor throughout the model. Thus, the threshold 2110 may be selected, for example, based on a recall 2140 of 0.7 and a precision 2130 around 0.18, as precision 2130 hovers around 0.2 until both precision 2130 and recall 2140 drop from 0.35 to 0.45.

[0204] As such, the desired predictability of outcomes must be balanced with the reality of the model's characteristics based on the data and correlations used to generate that model. As shown in Figure 21, there is not always a perfect balance between precision and recall, which are the basis for evaluating a model's predictive output. As explained above, model characteristics can change over time as available data changes, relevant features change, feedback from older models is provided, etc.

[0205] Features and associated data were used to generate the model depicted in Figure 21. Features used to train and test this model include features representing aggregations of laboratory measurements over a period of time (e.g., 120 days, 100 days, 30 days, etc.) as well as aggregations of patient status, medications, etc. over a period of time (e.g., 1 year, 9 months, 6 months, etc.), which may be generated using one hot encoding and methods such as final value, median time weighted average (twa), mean, median, etc. For example, data inputs for training the model may include, for each patient, all ICD-10 codes within the year prior to the first immunotherapy treatment, oxygen saturation values ​​from laboratory measurements within 120 days prior to the first treatment, the date of first immunotherapy treatment, etc. Those data inputs are transformed into a set of features including the frequency of lung-related ICD-10 codes (e.g., C34, C78, ​​R91, J, R05, R06, R07, R09, etc.), the frequency of ICD-10 code C34 (bronchial and pulmonary malignant neoplasms), the frequency of ICD-10 code C78 (secondary respiratory and digestive malignant neoplasms), median oxygen saturation in blood lab measurements within 120 days prior to initial ICI treatment, smoking, etc. The features train a model (e.g., as a binary classifier using random forest, etc.) to generate a predicted probability of developing pulmonary toxicity at some time point following immunotherapy treatment. As described above, the features are combined, weighted, and iterated to generate a model, reflected in the example of FIG. 21. This model can be deployed and used to predict outcomes for patients.

[0206] 22 illustrates an exemplary predictive output of the colitis toxicity model. The x-axis 2210 of the graph corresponds to a threshold (e.g., patient selection criteria) and the y-axis 2220 of the graph corresponds to the colitis toxicity development probability (e.g., precision curve 2230 or recall curve 2240). The threshold 2210 selected for the patient selection criteria sets a point below which one action is taken and above which another action is taken.

[0207] In the example of FIG. 22, a negative result indicates that the patient does not suffer from colitis toxicity, and a positive result indicates that the patient does suffer from colitis toxicity. If a low threshold (e.g., 0.0 to 0.2) is selected, the model results have low precision (e.g., many false positives) and high recall (e.g., few false negatives). As a result, almost all patients predicted not to suffer from colitis toxicity as a result of receiving immunotherapy do not suffer from colitis. However, the model incorrectly predicts many colitis toxicities in patients that do not actually occur. In a pharmaceutical clinical trial use case, the inflated number of colitis toxicities would cause the model to sieve or exclude too many patients from the trial. Also, in a clinical practice use case for patient treatment, the inflated probability of colitis toxicity would cause the model to potentially withhold immunotherapy from too many patients.

[0208] For example, if a threshold of 0.1 is selected in the example of Figure 22, almost all predictions of no toxicity are true, since the recall 2240 is 0.8. However, most predicted toxicity is incorrect, since the precision 2230 is 0.2.

[0209] If an intermediate threshold is chosen (e.g., 0.2 to 0.33), there is a better balance between false positives and false negatives. In a pharmaceutical use case, the model will filter out or exclude patients who do not experience colitis toxicity and filter in or include patients who do experience colitis toxicity.

[0210] For example, if a threshold of 0.3 is selected in the example of Figure 22, less than half of the predictions of no toxicity are true, since the recall 2240 is 0.35. However, about half of the predicted toxicities are correct / incorrect, since the precision 2230 is about 0.5.

[0211] If a high threshold (e.g., 0.33 to 0.6) is selected, the model output will have high precision (e.g., very few false positives) and low recall (e.g., many false negatives). As a result, almost all patients predicted to suffer from colitis toxicity truly have colitis, and the model fails to predict many colitis toxicities that actually occur. In a pharmaceutical context, the model will over-include patients who experience colitis toxicity in clinical trials. In a clinical patient care context, the model will erroneously underestimate colitis toxicity occurrence.

[0212] For example, if a threshold of 0.5 is chosen in the example of Figure 22, none of the predictions of no toxicity are true, since recall 2240 is 0.0. However, all of the predicted toxicities are correct, since precision 2230 is 1.0.

[0213] Using the precision 2230 and recall 2240 characteristics of the model depicted in FIG. 22, a patient selection criteria threshold 2210 may be selected to drive clinical trial cohort selection, patient treatment decisions, and the like. The model trained on a broader collection of healthcare data outputs the probability of developing colitis toxicity from immunotherapy for a given patient by processing that particular patient's healthcare data to generate a predicted output representative of colitis toxicity for that patient. The model output for that patient is evaluated according to the determined threshold 2210 to determine whether the patient is an acceptable candidate for the clinical trial cohort, immunotherapy treatment regime, and the like. Based on the motivations and constraints of the clinical trial, the patient's personal circumstances and aspirations, and the like, the patient selection threshold 2210 may be adjusted to be less or more inclusive / exclusive to include or exclude certain patients from clinical trials, treatment plans, and the like, with the associated risk of inaccuracy / accuracy as the threshold 2210 moves along the precision 2230 and recall 2240 curves for the model.

[0214] As shown in the example of FIG. 22, selecting a threshold of 0.25 indicates that the precision 2230 and recall 2240 curves form an intersection at approximately 0.4. Such an intersection provides a fair precision and good recall; however, both false positives and false negatives are still above 50%. As such, a different threshold 2210 may be selected to prioritize recall 2240 (e.g., at 0.15 to reduce false negatives) or precision 2230 (e.g., at 0.35 to reduce false positives).

[0215] The data and associated features were used to generate the model depicted in Figure 22. Features used to train and test this model include features representing aggregations of laboratory measurements over a period of time (e.g., 120 days, 100 days, 30 days, etc.) as well as aggregations of patient status, medications, etc. over a period of time (e.g., 1 year, 9 months, 6 months, etc.), which may be generated using one hot encoding and methods such as final value, median time weighted average (twa), mean, median, etc. For example, features formed to train the model may include immunotherapy class, condition (e.g., one hot encoded, etc.), lower than normal hemoglobin blood frequency, number of chemotherapy related procedures in a time window, lower / higher than normal lymphocyte count, number of unique immunotherapy drugs, lower / higher than normal albumin, lower / higher than normal white blood cell count (WBC), lower / higher than normal RBC, higher / lower than normal hemoglobin, immunotherapy class, etc. The features train a model (e.g., as a binary classifier using random forest) to generate a predicted probability of developing colitis toxicity at some time point following immunotherapy treatment. As described above, the features are combined, weighted, and iterated to generate a model, reflected in the example of Figure 22. This model can be deployed and used to predict outcomes for patients.

[0216] 23 illustrates an exemplary predictive output of a hepatitis toxicity model. The x-axis 2310 of the graph corresponds to a threshold (e.g., patient selection criteria) and the y-axis 2320 of the graph corresponds to the hepatitis toxicity development probability (e.g., precision curve 2330 or recall curve 2340). The threshold 2310 selected for the patient selection criteria sets a point below which one action is taken and above which another action is taken.

[0217] In the example of Figure 23, a negative result indicates that the patient does not suffer from hepatitis toxicity, and a positive result indicates that the patient does suffer from hepatitis toxicity. If a low threshold (e.g., 0.0 to 0.2) is selected, the model results have a low precision (e.g., many false positives) and a high recall (e.g., few false negatives). As a result, the model will erroneously predict many hepatitis toxicities in patients that do not actually occur, but a small number of patients who are predicted not to suffer from hepatitis toxicity as a result of receiving immunotherapy will actually have hepatitis.

[0218] For example, if a threshold of 0.1 is chosen in the example of Figure 23, all predictions of no toxicity are true, since the recall 2340 is 1.0. However, most of the predicted toxicity is incorrect, since the precision 2330 is slightly less than 0.2.

[0219] If an intermediate threshold is chosen (e.g., 0.2 to 0.6), there is a better balance between false positives and false negatives. In a pharmaceutical use case, the model will filter out or exclude patients who do not experience hepatitis toxicity and filter in or include patients who do experience hepatitis toxicity.

[0220] For example, if a threshold of 0.5 is chosen in the example of Figure 23, most predictions of no toxicity will be incorrect, since the recall 2340 is 0.25. However, about half of the predicted toxicities will be correct, since the precision 2330 is 0.55.

[0221] If a high threshold (e.g., 0.6 to 1.0) is selected, the model output will have high precision (e.g., few false positives) and low recall (e.g., many false negatives). As a result, many patients predicted to suffer from hepatitis toxicity will actually have hepatitis, while patients predicted not to suffer from hepatitis toxicity will have hepatitis. In a pharmaceutical context, the model will over-include patients who experience hepatitis toxicity in clinical trials. In a clinical patient care context, the model will erroneously underestimate hepatitis toxicity occurrence.

[0222] For example, if a threshold of 0.9 is chosen in the example of Figure 23, none of the predictions of no toxicity are true, since recall 2340 is 0.0. However, all of the predicted toxicities are correct, since precision 2330 is 1.0.

[0223] Using the precision 2330 and recall 2340 characteristics of the model depicted in FIG. 23, a patient selection criteria threshold 2310 may be selected to drive clinical trial cohort selection, patient treatment decisions, and the like. The model, trained on a broader collection of prior healthcare data, outputs the probability of developing hepatitis toxicity from immunotherapy for a given patient by processing that particular patient's healthcare data to generate a predicted output representative of hepatitis toxicity for that patient. The model output for that patient is evaluated according to the determined threshold 2310 to determine whether the patient is an acceptable candidate for the clinical trial cohort, immunotherapy treatment regime, and the like. Based on the motivations and constraints of the clinical trial, the patient's personal circumstances and aspirations, and the like, the patient selection threshold 2310 may be adjusted to be less or more inclusive / exclusive to include or exclude certain patients from clinical trials, treatment plans, and the like, with the associated risk of inaccuracy / accuracy as the threshold 2310 moves along the precision 2330 and recall 2340 curves for the model.

[0224] As shown in the example of FIG. 23, the intersection of the precision 2330 and recall 2340 curves occurs at about 0.28. However, at this intersection, both precision 2330 and recall 2340 are slightly below average at just over 0.4 each, indicating about 60% false positives and about 60% false negatives. As such, the intersection is not large for either precision or recall. Thus, the threshold 2310 may be selected based on, for example, a recall 2340 of 1.0 at a precision 2330 of around 0.19, or based on, for example, a recall 2340 of 0.58 at a precision 2330 of 0.25, depending on the selection criteria for the model, patient, etc.

[0225] The data and associated features were used to generate the model depicted in Figure 23. Features used to train and test the model include features representing aggregations of laboratory measurements over a period of time (e.g., 120 days, 100 days, 30 days, etc.) as well as aggregations of patient status, medications, etc. over a period of time (e.g., 1 year, 9 months, 6 months, etc.), which may be generated using one hot encoding and methods such as final, median time weighted average (twa), mean, median, etc. For example, features formed to train the model may include levels of alkaline phosphatase in blood (e.g., final, twa, max, min, etc.), levels of aspartate aminotransferase in blood (e.g., final, twa, max, min, etc.), levels of alanine transaminase in blood (e.g., final, twa, max, min, etc.), bilirubin in blood (e.g., max, twa, final, min, etc.), etc. The features train a model (e.g., a binary classifier using random forests) to generate a predicted probability of developing hepatitis toxicity at some time point following immunotherapy treatment. As described above, the features are combined, weighted, and iterated to generate a model, reflected in the example of Figure 23. This model can be deployed and used to predict outcomes for patients.

[0226] Figure 24 illustrates the properties of a duration of treatment model, another proxy for treatment efficacy that may be used instead of or in addition to the overall survival model reflected in Figure 20. The x-axis 2410 of the graph corresponds to a threshold (e.g., patient selection criteria) and the y-axis 2420 of the graph corresponds to the probability of remaining on immunotherapy treatment (e.g., precision curve 2430 or recall curve 2440). The threshold 2410 selected for the patient selection criteria sets the point below which one action is taken and above which another action is taken.

[0227] In the example of FIG. 24, a negative result indicates that the patient continues treatment, and a positive result indicates that the patient has discontinued treatment. If a low threshold (e.g., 0.0 to 0.5) is selected, the model results have high precision (e.g., very few false positives) and low recall (e.g., many false negatives). As such, it is likely to misidentify people who have discontinued treatment, but is unlikely to misidentify people who are still on treatment. As a result, almost all patients predicted to still be on immunotherapy treatment are in fact receiving immunotherapy. However, the model incorrectly predicts many treatment discontinuations that do not actually occur. In a pharmaceutical clinical trial use case, the model would sieve or exclude too many patients from the trial due to the inflated number of treatment discontinuations. Also, in a clinical practice use case for patient treatment, the model would potentially withhold immunotherapy due to the inflated probability that immunotherapy would have to be prematurely discontinued.

[0228] For example, if a threshold of 0.3 is chosen in the example of Figure 24, all predictions of treatment discontinuation are true, since recall 2440 is almost 1.0. However, most of the predictions of treatment continuation are incorrect, since precision 2430 is slightly less than 0.2.

[0229] If an intermediate threshold (e.g., 0.5 to 0.6) is chosen, there is a better balance between false positives and false negatives. In the pharmaceutical use case, the model will filter out or exclude patients who may have completed treatment and filter in or include patients who do not continue treatment. However, overall survival is improved due to the model's properties. In the context of clinical practice, the model makes some inappropriate predictions, but most patients for whom immunotherapy is recommended will complete the treatment. If this threshold were applied to the model, most patients for whom treatment is not recommended would not have been able to complete the treatment.

[0230] For example, if a threshold of 0.5 is chosen in the example of Figure 24, then over half of the predictions of treatment discontinuation are true, since recall 2440 is 0.6. However, most of the predictions of treatment continuation are incorrect, since precision 2430 is 0.2.

[0231] If a high threshold (e.g., 0.6 to 0.7) is chosen, the model output will have low precision (e.g., many false positives) and high recall (e.g., few false negatives). As a result, almost all patients predicted not to complete treatment will in fact not complete treatment, and the model will erroneously predict that many treatment regimens that are still being administered have been discontinued. In a pharmaceutical context, the model will exclude too many patients in clinical trials who would complete their treatment regimens. In a clinical patient care context, it is too rare for the model to misprescribe immunotherapy.

[0232] For example, if a threshold of 0.65 is chosen in the example of Figure 24, almost all predictions of treatment discontinuation will be incorrect, since recall 2440 is slightly higher than 0.0. In addition, most of the predictions of treatment continuation will be incorrect, since precision 2430 is slightly higher than 0.4.

[0233] Using the precision 2430 and recall 2440 properties of the model depicted in FIG. 24, a patient selection criteria threshold 2410 may be selected to drive clinical trial cohort selection, patient treatment decisions, and the like. The model, trained on a broader collection of prior healthcare data, outputs the probability of treatment success for a given patient by processing that particular patient's healthcare data to generate a predicted output representative of efficacy (using duration of treatment as a surrogate or proxy for efficacy of the immunotherapy treatment) for that patient. The model output for that patient is evaluated according to the determined threshold 2410 to determine whether the patient is an acceptable candidate for the clinical trial cohort, immunotherapy treatment regime, and the like. Based on the motivations and constraints of the clinical trial, the patient's personal circumstances and aspirations, and the like, the patient selection threshold 2410 may be adjusted to be less or more inclusive / exclusive to include or exclude certain patients from clinical trials, treatment plans, and the like, with the associated risk of inaccuracy / accuracy as the threshold 2410 moves along the precision 2430 and recall 2440 curves for the model.

[0234] As shown in the example of FIG. 24, the intersection of the precision 2430 and recall 2440 curves occurs at approximately 0.57. However, at this intersection, both precision 2430 and recall 2440 are poor at less than 0.3 each, indicating more than 70% false positives and more than 70% false negatives. As such, this intersection is poor for both precision and recall. In the treatment duration model of FIG. 24, precision 2430 and recall 2440 are not well aligned, with precision 2430 being poor throughout the model. Thus, threshold 2410 may be selected to maximize recall 2440 at, for example, 1.0, since precision hovers around 20% unless recall is 0.0.

[0235] The data and associated features were used to generate the model depicted in Figure 24. The features used to train and test this model included features representing aggregations of laboratory measurements over a period of time (e.g., 120 days, 100 days, 30 days, etc.) as well as aggregations of patient status, medications, etc. over a period of time (e.g., 1 year, 9 months, 6 months, etc.), which may be generated using one hot encoding and methods such as final value, median time weighted average (twa), mean, median, etc. The features may include RBC (e.g., median, etc.), number of promethazine doses, immature granulocytes (e.g., median, etc.), lymphocytes (e.g., median, etc.), blood albumin (e.g., final, twa, median, etc.), RDW (e.g., median, etc.), neutrophils (e.g., median, etc.), alkanine phosphatase (e.g., median, etc.), number of dexamethasone doses, neutabs (median, etc.), number of radiology studies, one-hot encoded status (e.g., for codes C77, R05, G89, etc.), chemotherapy within a time frame, number of drug doses, etc. As described above, the features are combined, weighted, and iterated to generate a model, which is reflected in the example of FIG. 24. This model may be deployed and used to predict outcomes for patients.

[0236] As such, precision and recall may vary from model to model, and a given model may change over time as more data is collected, patient populations change, specific patients change, feedback is provided to regenerate or otherwise update the model, etc. However, the model framework enables the generation of a variety of models, and the dynamic setting of thresholds and other patient and / or model selection criteria to promote productive and reliable results for patient care, including by the patient's physician as part of a clinical trial.

[0237] As described herein, constructing a cohort for a clinical trial and constructing a treatment plan are two examples of how the developed model can be used. However, a patient's treatment plan may be or may include placing the patient in a clinical trial (e.g., to see if immunotherapy in a clinical trial helps the patient). As such, constructing a cohort for a clinical trial may be a specific case or environment for constructing a patient's treatment plan to involve immunotherapy.

[0238] In some examples, the decision may be driven solely by the toxicity-related model. For example, the risk of toxicity, such as pneumonia, hepatitis, or colitis, may be the driver for whether a patient is administered immunotherapy (e.g., as part of a clinical trial or as part of an individual's treatment plan). For example, the patient may be eager to receive the treatment. In other examples, the likelihood of treatment efficacy may be the driver for whether to administer immunotherapy to a patient, regardless of the risk of toxicity. For example, the patient may prioritize quality of life over more aggressive treatment. In such cases, additional testing and monitoring (e.g., checking in, rerunning the model, etc.) may be ordered to ensure the patient's health if toxicity begins to develop. In some examples, the output of both the efficacy-related model and one or more toxicity-related models may be combined, thereby balancing the likelihood of immunotherapy success against the likelihood of one or more toxicities developing. As such, the model may provide a binary output (e.g., include / exclude, treat / don't treat, etc.), and / or a multi-faceted or nuanced output that conveys more than just a yes or no answer, for example.

[0239] As such, the model framework can generate multiple models to be used in different ways, according to different thresholds, based on clinical needs, research needs, patient desires, etc. Such qualitative indicators can be translated into quantitative thresholds for patient selection criteria, allowing the same model to drive different decisions and behaviors according to thresholds. Patient selection criteria can change over time, as a patient may initially want aggressive treatment, then withdraw treatment to take a vacation, then resume treatment more aggressively, and possibly withdraw treatment again when end-of-life quality decisions are made. The model can adapt to all of these situations, for example, through the setting of patient selection criteria thresholds (and also through model selection criteria that drive model generation in a targeted or targeted manner).

[0240] FIG. 25 illustrates an example infrastructure and associated processes 2500 whereby patient healthcare data from one or more patient healthcare records 2510 are organized / curated into one or more data stores 2520, 2525 for intermediate processing by using one or more alignment, mapping, and / or cleaning actions 2530 to form time-dependent values ​​2540, 2545. The time-dependent values ​​from the intermediate tables form features 2550 that may be used for model development, as described above. Thus, healthcare data from existing records 2510 are pulled in and organized 2520 for processing 2530 according to various degrees of complexity. The intermediate tables 2540, 2545 form time series data values ​​or variables that correlate to multiple features 2550. The features may be used to train and test multiple models as described above.

[0241] Exemplary implementations are illustrated in this application and the associated appendices, although one or more of the illustrated elements, processes, and / or devices may be combined, divided, rearranged, omitted, eliminated, and / or implemented in any other manner. Furthermore, the exemplary elements may be implemented by hardware alone or in combination with software and / or firmware. Thus, for example, any of the exemplary elements could be implemented by a processor circuit, an analog circuit, a digital circuit, a logic circuit, a programmable processor, a programmable microcontroller, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), and / or a field programmable logic device (FPLD), such as a field programmable gate array (FPGA). Still further, the exemplary elements may include one or more elements, processes, and / or devices in addition to or in place of the illustrated elements, and / or may include a plurality or all of any of the illustrated elements, processes, and devices.

[0242] In some examples, hardware logic circuits, machine-readable instructions, hardware-implemented state machines, and / or any combination thereof can implement the systems disclosed herein and / or execute the methods disclosed herein. The machine-readable instructions may be one or more executable programs or parts of executable programs executed by a processor circuit. The programs may be embodied in software stored on one or more non-transitory computer-readable storage media, such as a compact disc (CD), a floppy disk, a hard disk drive (HDD), a solid-state drive (SSD), a digital versatile disk (DVD), a Blu-ray disk, a volatile memory (e.g., any type of random access memory (RAM)), or a non-volatile memory (e.g., an electrically erasable programmable read-only memory (EEPROM), a FLASH memory, a HDD, an SSD, etc.) associated with a processor circuit located in one or more hardware devices, although the entire program and / or parts thereof may alternatively be executed by one or more hardware devices other than the processor circuit and / or embodied in firmware or dedicated hardware. The machine-readable instructions may be distributed among multiple hardware devices and / or executed by two or more hardware devices (e.g., a server and a client hardware device). For example, a client hardware device may be implemented by an end-point client hardware device (e.g., a hardware device associated with a user) or an intermediate client hardware device (e.g., a Radio Access Network (RAN) gateway that may facilitate communications between a server and an end-point client hardware device). Similarly, a non-transitory computer-readable storage medium may include one or more media located in one or more hardware devices. Additionally, the order of execution may be changed and / or some of the described blocks may be changed, eliminated, or combined.Additionally or alternatively, any or all of the code blocks may be implemented by one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.) structured to perform the corresponding operations without executing software or firmware. The processor circuits may be distributed across different network locations and / or local to one or more hardware devices (e.g., a single core processor (e.g., a single core central processing unit (CPU)), a multi-core processor (e.g., a multi-core CPU, XPU, etc.) within a single machine, multiple processors distributed across multiple servers in a server rack, multiple processors distributed across one or more server racks, CPUs and / or FPGAs located in the same package (e.g., the same integrated circuit (IC) package, or two or more separate housings, etc.).

[0243] The machine-readable instructions described herein may be stored in one or more of a compressed format, an encrypted format, a fragmented format, a compiled format, an executable format, a packaged format, etc. The machine-readable instructions described herein may be stored as data or data structures (e.g., portions of instructions, code, representations of code, etc.) that can be utilized to create, manufacture, and / or generate machine-executable instructions. For example, the machine-readable instructions may be fragmented and stored on one or more storage devices and / or computing devices (e.g., servers) located at the same location or different locations of a network or collection of networks (e.g., in a cloud, in an edge device, etc.). The machine-readable instructions may require one or more of installation, modification, adaptation, updating, combination, supplementation, configuration, decryption, decompression, decompression, distribution, reallocation, compilation, etc., in order to make the instructions directly readable, interpretable, and / or executable by the computing device and / or other machines. For example, machine-readable instructions may be stored in multiple portions that may be individually compressed, encrypted, and / or stored on separate computing devices, which portions, when decoded, decompressed, and / or combined, form a set of machine-executable instructions that implement one or more operations that may together form a program as described herein.

[0244] In another example, machine-readable instructions may be stored in a state that can be read by a processor circuit, but require the addition of a library (e.g., a dynamic link library (DLL)), a software development kit (SDK), an application programming interface (API), etc., to execute the machine-readable instructions on a particular computing device or other device. In another example, the machine-readable instructions may need to be configured (e.g., settings are stored, data is entered, network addresses are recorded, etc.) before the machine-readable instructions and / or corresponding programs can be executed in whole or in part. Thus, as used herein, a machine-readable medium may include machine-readable instructions and / or programs regardless of the particular form or state of the machine-readable instructions and / or programs when stored or otherwise at rest or in communication.

[0245] The machine-readable instructions described herein may be expressed by any past, present, or future instruction language, scripting language, programming language, etc. For example, the machine-readable instructions may be expressed using any of the following languages: C, C++, Java, C#, Perl, Python, JavaScript, Hypertext Markup Language (HTML), Structured Query Language (SQL), Swift, etc.

[0246] As noted above, the example operations disclosed herein may be implemented using executable instructions (e.g., computer and / or machine readable instructions) stored on one or more non-transitory computer media and / or machine readable media, such as optical storage devices, magnetic storage devices, HDDs, flash memory, read-only memory (ROM), CDs, DVDs, caches, any type of RAM, registers, and / or any other storage device or storage disk in which information is stored for any duration (e.g., over an extended period of time, permanently, for short instances, for temporary buffering, and / or for caching of information). As used herein, the terms non-transitory computer readable medium, non-transitory computer readable storage medium, non-transitory machine readable medium, and non-transitory machine readable storage medium are expressly defined to include any type of computer readable storage device and / or storage disk, to exclude propagating signals, and to exclude transmission media. As used herein, the terms "computer-readable storage device" and "machine-readable storage device" are defined to include any physical (mechanical and / or electrical) structure for storing information, but to exclude propagating signals and to exclude transmission media. Examples of computer-readable storage devices and machine-readable storage devices include any type of random access memory, any type of read-only memory, solid-state memory, flash memory, optical disks, magnetic disks, disk drives, and / or redundant array of independent disks (RAID) systems. As used herein, the term "device" refers to a physical structure, such as mechanical and / or electrical equipment, hardware, and / or circuitry, that may or may not be configured with and / or manufactured to execute computer-readable instructions, machine-readable instructions, and the like.

[0247] 26 is a block diagram of an exemplary processor platform 2600 configured to execute and / or instantiate the machine-readable instructions and / or operations disclosed and described herein. The processor platform 2600 may be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., neural networks), a mobile device (e.g., a mobile phone, a smartphone, a tablet such as an iPad), a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a game console, a personal video recorder, a set-top box, a headset (e.g., an augmented reality (AR) headset, a virtual reality (VR) headset, etc.) or other wearable device, or any other type of computing device.

[0248] The processor platform 2600 of the illustrated example includes a processor circuit 2612. The processor circuit 2612 of the illustrated example is hardware. For example, the processor circuit 2612 may be implemented by one or more integrated circuits, logic circuits, FPGAs, microprocessors, CPUs, GPUs, DSPs, and / or microcontrollers from any desired family or manufacturer. The processor circuit 2612 may be implemented by one or more semiconductor-based (e.g., silicon-based) devices.

[0249] The processor circuitry 2612 of the illustrated example includes a local memory 2613 (e.g., cache, registers, etc.). The processor circuitry 2612 of the illustrated example communicates with a main memory including a volatile memory 2614 and a non-volatile memory 2616 by a bus 2618. The volatile memory 2614 may be implemented by synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), RAMBUS® Dynamic Random Access Memory (RDRAM®), and / or any other type of RAM device. The non-volatile memory 2616 may be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 2614, 2616 of the illustrated example is controlled by a memory controller 2617.

[0250] The processor platform 2600 of the illustrated example also includes an interface circuit 2620. The interface circuit 2620 may be implemented by hardware according to any type of interface standard, such as an Ethernet interface, a Universal Serial Bus (USB) interface, a Bluetooth® interface, a Near Field Communication (NFC) interface, a Peripheral Component Interconnect (PCI) interface, and / or a Peripheral Component Interconnect Express (PCIe) interface.

[0251] In the illustrated example, one or more input devices 2622 are connected to the interface circuit 2620. The input devices 2622 allow a user to input data and / or commands into the processor circuit 2612. The input devices 2622 may be implemented by, for example, audio sensors, microphones, cameras (still or video), keyboards, buttons, mice, touch screens, track pads, track balls, isopoint devices, and / or voice recognition systems.

[0252] One or more output devices 2624 are also connected to the interface circuitry 2620 of the illustrated example. The output device(s) 2624 may be implemented by, for example, a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube (CRT) display, an in-place switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. Thus, the interface circuitry 2620 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics processor circuit such as a GPU.

[0253] The interface circuitry 2620 of the illustrated example also includes communications devices such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces to facilitate data exchange with external machines (e.g., any type of computing device) over a network 2626. Communications may be, for example, over an Ethernet connection, a Digital Subscriber Line (DSL) connection, a phone line connection, a coaxial cable system, a satellite system, a line-of-sight wireless system, a cellular telephone system, an optical connection, etc.

[0254] The processor platform 2600 of the illustrated example also includes one or more mass storage devices 2628 for storing software and / or data. Examples of such mass storage devices 2628 include magnetic storage devices, optical storage devices, floppy disk drives, HDDs, CDs, Blu-ray disk drives, redundant array of independent disks (RAID) systems, flash memory devices and / or solid state storage devices such as SSDs, and DVD drives.

[0255] The machine-readable instructions 2632 may be stored on the mass storage device 2628, the volatile memory 2614, the non-volatile memory 2616, and / or a removable non-transitory computer-readable storage medium, such as a CD or DVD.

[0256] FIG. 27 is a block diagram of an exemplary implementation of the processor circuit 2612 of FIG. 26. In this example, the processor circuit 2612 of FIG. 26 is implemented by a microprocessor 2700. For example, the microprocessor 2700 may be a general-purpose microprocessor (e.g., a general-purpose microprocessor circuit). The microprocessor 2700 executes some or all of the machine-readable instructions to effectively instantiate the circuits described herein as logic circuits and perform operations corresponding to those machine-readable instructions. In some such examples, the circuits are instantiated by the hardware circuitry of the microprocessor 2700 in combination with the instructions. For example, the microprocessor 2700 may be implemented by a multi-core hardware circuit such as a CPU, DSP, GPU, XPU, etc. While any number of exemplary cores 2702 (e.g., one core) may be included, the microprocessor 2700 of this example is a multi-core semiconductor device including N cores. The cores 2702 of the microprocessor 2700 may operate independently or cooperate to execute the machine-readable instructions. For example, a firmware program, embedded software program, or machine code corresponding to a software program may be executed by one of the cores 2702, or may be executed at the same or different times by multiple of the cores 2702. In some examples, a firmware program, embedded software program, or machine code corresponding to a software program is divided into threads and executed in parallel by two or more of the cores 2702. A software program may correspond to some or all of the machine-readable instructions and / or operations disclosed herein.

[0257] The cores 2702 may communicate via a first exemplary bus 2704. In some examples, the first bus 2704 may be implemented by a communication bus that effects communication associated with one (or more) of the cores 2702. For example, the first bus 2704 may be implemented by at least one of an Inter-Integrated Circuit (I2C) bus, a Serial Peripheral Interface (SPI) bus, a PCI bus, or a PCIe bus. Additionally or alternatively, the first bus 2704 may be implemented by any other type of computing or electrical bus. The cores 2702 may obtain data, instructions, and / or signals from one or more external devices via the exemplary interface circuitry 2706. The cores 2702 may output data, instructions, and / or signals to one or more external devices via the interface circuitry 2706. The cores 2702 in this example include an exemplary local memory 2720 (e.g., a level 1 (L1) cache that may be divided into an L1 data cache and an L1 instruction cache), but the microprocessor 2700 also includes an exemplary shared memory 2710 (e.g., a level 2 (L2) cache) that may be shared by the cores for fast access to data and / or instructions. Data and / or instructions may be transferred (e.g., shared) by writing to and / or reading from the shared memory 2710. The local memory 2720 and the shared memory 2710 of each of the cores 2702 may be part of a hierarchy of storage devices that includes multiple levels of cache memories and main memories (e.g., main memories 2614, 2616 of FIG. 26). Typically, memories at higher levels in the hierarchy exhibit shorter access times and have smaller storage capacities than memories at lower levels. Changes in the various levels of the cache hierarchy are managed (e.g., coordinated) by cache coherency policies.

[0258] Each core 2702 may be referred to as a CPU, DSP, GPU, etc., or any other type of hardware circuit. Each core 2702 includes a control unit circuit 2714, an arithmetic logic (AL) circuit (sometimes referred to as an ALU) 2716, a number of registers 2718, a local memory 2720, and a second exemplary bus 2722. Other structures may exist. For example, each core 2702 may include a vector operation unit circuit, a single instruction multiple data (SIMD) unit circuit, a load / store unit (LSU) circuit, a branch / jump unit circuit, a floating point unit (FPU) circuit, etc. The control unit circuit 2714 includes semiconductor-based circuitry structured to control (e.g., coordinate) data movement within the corresponding core 2702. The AL circuitry 2716 includes semiconductor-based circuitry structured to perform one or more mathematical and / or logical operations on data within the corresponding core 2702. Some example AL circuits 2716 perform integer-based operations. In other examples, the AL circuitry 2716 also performs floating point operations. In yet other examples, the AL circuitry 2716 may include a first AL circuit that performs integer-based operations and a second AL circuit that performs floating point operations. In some examples, the AL circuitry 2716 may be referred to as an arithmetic logic unit (ALU). The registers 2718 are semiconductor-based structures that store data and / or instructions, such as results of one or more of the operations performed by the AL circuitry 2716 of the corresponding core 2702. For example, the registers 2718 may include vector registers, SIMD registers, general purpose registers, flag registers, segment registers, machine specific registers, instruction pointer registers, control registers, debug registers, memory management registers, machine check registers, etc. The registers 2718 may be arranged in banks as shown in FIG. 27. Alternatively, the registers 2718 may be organized in any other arrangement, format, or structure, including distributed throughout the cores 2702 to reduce access times. The second bus 2722 may be implemented by at least one of an I2C bus, an SPI bus, a PCI bus, or a PCIe bus.

[0259] Each core 2702 and / or more generally, the microprocessor 2700 may include structures in addition to and / or in place of those shown and described above. For example, one or more clock circuits, one or more power supplies, one or more power gates, one or more cache home agents (CHAs), one or more converged / common mesh stops (CMSs), one or more shifters (e.g., barrel shifters), and / or other circuits may be present. The microprocessor 2700 is a semiconductor device fabricated to include a number of transistors interconnected to implement the structures described above in one or more integrated circuits (ICs) housed in one or more packages. The processor circuitry may include and / or cooperate with one or more accelerators. In some examples, the accelerators are implemented by logic circuits to perform some tasks faster and / or more efficiently than would be performed by a general-purpose processor. Examples of accelerators include ASICs and FPGAs as described herein. A GPU or other programmable device may also be an accelerator. The accelerator may be mounted on the processor circuit, on the same chip package as the processor circuit, and / or in one or more packages separate from the processor circuit.

[0260] 28 is a block diagram of another exemplary implementation of the processor circuit 2612 of FIG. 26. In this example, the processor circuit 2612 is implemented by an FPGA circuit 2800. For example, the FPGA circuit 2800 may be implemented by an FPGA. The FPGA circuit 2800 may be used to perform operations that may otherwise be performed by, for example, the exemplary microprocessor 2700 of FIG. 27 executing corresponding machine-readable instructions. However, once configured, the FPGA circuit 2800 instantiates the machine-readable instructions in hardware and thus can perform operations in many cases faster than may be performed by a general-purpose microprocessor executing corresponding software.

[0261] More specifically, in contrast to the microprocessor 2700 of FIG. 27 described above (which is a general-purpose device whose interconnects and logic circuitry are fixed once fabricated, although it may be programmed to execute some or all of the machine-readable instructions disclosed herein), the example FPGA circuit 2800 of FIG. 28 includes interconnects and logic circuitry that may be configured and / or interconnected in different ways after fabrication, e.g., to instantiate some or all of the machine-readable instructions disclosed herein. In particular, the FPGA circuit 2800 may be considered an array of logic gates, interconnects, and switches. The switches are programmed to change the way the logic gates are interconnected by the interconnects, thereby effectively forming one or more dedicated logic circuits (unless the FPGA circuit 2800 is reprogrammed). The configured logic circuits allow the logic gates to cooperate in different ways to perform different operations on data received by the input circuits. Those operations may correspond to some or all of the software disclosed herein. As such, the FPGA circuitry 2800 may be structured to effectively instantiate some or all of the machine-readable instructions as special-purpose logic circuitry and perform operations corresponding to those software instructions in a dedicated manner, similar to an ASIC. Thus, the FPGA circuitry 2800 may perform operations corresponding to some or all of the machine-readable instructions faster than a general-purpose microprocessor would.

[0262] In the example of FIG. 28, the FPGA circuit 2800 is structured to be programmed (and / or reprogrammed one or more times) by an end user with a hardware description language (HDL) such as Verilog. The FPGA circuit 2800 of FIG. 28 includes an example input / output (I / O) circuit 2802 that obtains and / or outputs data from an example configuration circuit 2804 and / or external hardware 2806. For example, the configuration circuit 2804 may be implemented by an interface circuit that may obtain machine-readable instructions for configuring the FPGA circuit 2800, or a portion thereof. In some such examples, the configuration circuit 2804 may obtain the machine-readable instructions from a user, a machine (e.g., a hardware circuit (e.g., a programmed or dedicated circuit) that may implement an artificial intelligence / machine learning (AI / ML) model to generate the instructions), etc. In some examples, the external hardware 2806 may be implemented by an external hardware circuit. For example, the external hardware 2806 may be implemented by the microprocessor 2700 of FIG. 27. The FPGA circuit 2800 also includes an array of exemplary logic gate circuits 2808, a plurality of exemplary configurable interconnects 2810, and an exemplary storage circuit 2812. The logic gate circuits 2808 and the configurable interconnects 2810 are configurable to instantiate one or more operations that may correspond to at least some of the machine-readable instructions and / or other desired operations. The logic gate circuits 2808 shown in FIG. 28 are fabricated in groups or blocks. Each block includes a semiconductor-based electrical structure that can be configured into a logic circuit. In some examples, the electrical structure includes logic gates (e.g., And gates, Or gates, Nor gates, etc.) that provide the basic building blocks for the logic circuit. Electrically controllable switches (e.g., transistors) are present within each of the logic gate circuits 2808 to enable configuration of the electrical structures and / or logic gates to form circuits that perform the desired operations. The logic gate circuits 2808 may include other electrical structures such as look-up tables (LUTs), registers (e.g., flip-flops or latches), multiplexers, etc.

[0263] The configurable interconnects 2810 in the illustrated example are conductive paths, traces, vias, or the like that may include electrically controllable switches (e.g., transistors) whose state can be changed by programming (e.g., using an HDL instruction language) the desired logic circuit to activate or deactivate one or more connections between one or more of the logic gate circuits 2808.

[0264] The storage circuits 2812 in the illustrated example are structured to store the results of one or more operations performed by corresponding logic gates. The storage circuits 2812 may be implemented by registers or the like. In the illustrated example, the storage circuits 2812 are distributed among the logic gate circuits 2808 to facilitate access and increase execution speed.

[0265] The example FPGA circuit 2800 of FIG. 28 also includes an example dedicated computation circuit 2814. In this example, the dedicated computation circuit 2814 includes special purpose circuitry 2816 that can be called upon to implement commonly used functions in the field to avoid the need to program those functions. Examples of such special purpose circuitry 2816 include memory (e.g., DRAM) controller circuitry, PCIe controller circuitry, clock circuitry, transceiver circuitry, memory, and multiplier-accumulator circuitry. Other types of special purpose circuitry may also be present. In some examples, the FPGA circuit 2800 may also include an example general purpose programmable circuitry 2818, such as an example CPU 2820 and / or an example DSP 2822. Other general purpose programmable circuits 2818 may additionally or alternatively be present as GPUs, XPUs, etc. that may be programmed to perform other operations.

[0266] 27 and 28 illustrate two exemplary implementations of the processor circuit 2612 of FIG. 26, many other approaches are contemplated. For example, as noted above, modern FPGA circuits may include an on-board CPU, such as one or more of the exemplary CPUs 2820 of FIG. 28. Thus, the processor circuit 2612 of FIG. 26 may be implemented by combining the exemplary microprocessor 2700 of FIG. 27 and the exemplary FPGA circuit 2800 of FIG. 28 in addition thereto. In some such hybrid examples, a first portion of the machine-readable instructions may be executed by one or more of the cores 2702 of FIG. 27, a second portion of the machine-readable instructions may be executed by the FPGA circuit 2800 of FIG. 28, and / or a third portion of the machine-readable instructions may be executed by an ASIC. Thus, it should be understood that some or all of the circuitry may be instantiated at the same time or at different times. Some or all of the circuitry may be instantiated, for example, in one or more threads executing simultaneously and / or serially. Further, in some examples, some or all of the circuitry may be implemented within one or more virtual machines and / or containers running on a microprocessor.

[0267] In some examples, the processor circuitry 2612 of Figure 26 may be in one or more packages. For example, the microprocessor 2700 of Figure 27 and / or the FPGA circuitry 2800 of Figure 28 may be in one or more packages. In some examples, an XPU may be implemented by the processor circuitry 2612 of Figure 26, which may be in one or more packages. For example, an XPU may have a CPU in one package, a DSP in another package, a GPU in yet another package, and an FPGA in yet another package.

[0268] A block diagram illustrating an exemplary software distribution platform 2905 for distributing software, such as the exemplary machine-readable instructions 2632 of FIG. 26, to hardware devices owned and / or operated by a third party is illustrated in FIG. 29. The exemplary software distribution platform 2905 may be implemented by any computer server, data facility, cloud service, etc. that can store and transmit software to other computing devices. The third party may be a customer of the entity that owns and / or operates the software distribution platform 2905. For example, the entity that owns and / or operates the software distribution platform 2905 may be a developer, seller, and / or licensor of the software, such as the exemplary machine-readable instructions 2632 of FIG. 26. The third party may be a consumer, user, retailer, OEM, etc. that purchases and / or licenses the software for use and / or resale and / or sublicense. In the illustrated example, the software distribution platform 2905 comprises one or more servers and one or more storage devices. The storage device stores machine-readable instructions 2632, which may correspond to the example machine-readable instructions 2632 of FIG. 26, as described above. One or more servers of the example software distribution platform 2905 communicate with the example network 2910, which may correspond to any one or more of the Internet and / or any of the example networks described above. In some examples, the one or more servers respond to requests to transmit the software to a requesting party as part of a business transaction. Payment for the distribution, sale, and / or license of the software may be handled by one or more servers of the software distribution platform and / or by a third party payment entity. The server enables purchasers and / or licensors to download the machine-readable instructions 2632 from the software distribution platform 2905.26 may be downloaded to the exemplary processor platform 2600, which executes the machine-readable instructions 2632 to implement the systems and methods described herein. In some examples, one or more servers of the software distribution platform 2905 periodically provide, transmit, and / or force updates to the software (e.g., the exemplary machine-readable instructions 2632) to ensure that improvements, patches, updates, etc. are distributed and applied to the software on end-user devices.

[0269] In some examples, rather than downloading the machine-readable instructions 2932 to a local processor platform 2900, the deployed models and / or patient data may be uploaded for remote execution via a cloud-based platform 2905. In some examples, the example platform 2905 can host one or more models accessible by a network 2910, and the processor platform 2600 can provide inputs to the models and receive results, predictions, and / or other outputs.

[0270] From the foregoing, it will be appreciated that exemplary systems, methods, apparatus, and articles of manufacture are disclosed that enable model generation and deployment to drive processes for treatment prediction and treatment delivery. The disclosed systems, methods, apparatus, and articles of manufacture improve the efficiency of using computing devices by enabling model generation and deployment to drive processes for treatment prediction and treatment delivery. The disclosed systems, methods, apparatus, and articles of manufacture are accordingly directed to one or more improvements in the operation of machines, such as computers or other electronic and / or mechanical devices.

[0271] Some examples provide systems and methods for training and evaluating AI models to predict adverse risk events from fixed-length patient history data. Exemplary systems and related methods include submodules that extract a patient's blood test history from an electronic medical record, clean the history, and perform data quality checks. Exemplary systems and related methods include a label definition submodule that instantiates an algorithm that assigns a Hepatitis adverse event grade to a set of blood test values ​​(e.g., ALT, AST, TBILIRUBIN, ALKPHOS) and creates a binary target label for the AI ​​model. Exemplary systems and methods include a feature engineering submodule that normalizes the blood test values ​​to an upper limit of normal (e.g., specific to a patient, laboratory, and / or blood test, etc.). An exemplary feature engineering submodule is to convert the normalized values ​​into a discretized symbolic representation, such as a modified version of Symbolic Aggregate ApproXimation. An exemplary feature engineering submodule extracts motifs from the symbolic sequence as n-grams and uses counts in recent patient history as features. The exemplary system and method includes a sub-module for training and evaluating an AI model that dynamically predicts immune-related hepatitis adverse event risk from fixed-length blood test history.

[0272] Some examples provide systems and methods that use a sequential procedure to build a classification model (e.g., a pneumonia classification model, etc.). Exemplary systems and methods include pre-processing of structured EHR and unstructured data tables. Patient timelines are aligned, for example, at the time of first ICI administration. Laboratory measurements are aggregated over a time frame prior to first ICI (e.g., a 60-day time frame, etc.) using statistics. Other features (e.g., condition, smoking status, etc.) can use, for example, different time frames (e.g., a 1-year time frame, etc.).

[0273] The exemplary system and method includes finding patterns in the data to identify potential predictive features associated with the development of ICI-associated toxicities such as pneumonia, colitis, and hepatitis. For small, noisy datasets, a large number of data points are utilized. The data is partitioned to identify candidate features based on associations between pneumonia labels and features in a first partition (e.g., 90% partition, 80% partition, 95% partition, etc.).

[0274] The exemplary system and method involves deciding between two model versions, one with the original feature set (M1) and one augmented with candidate features (M2) at each iteration of the procedure. Nested cross-validation is performed on the first (e.g., 90%) partition, and the results of the inner loop are used to compare M1 and M2. A binomial test is performed to evaluate whether M2 is significantly better than M1.

[0275] Exemplary systems and methods include evaluating whether M2 has better performance on a held-out second partition (e.g., a 10% partition, a 5% partition, a 20% partition, etc.) when M2 is significantly better in step 3). Permutation testing is performed to estimate the probability of observing better performance by chance. This step serves as a safety measure to avoid overfitting to the first (e.g., 90%) data partition. If M2 performs sufficiently well based on step 4), M2 is selected.

[0276] Exemplary systems and methods include continuing to test new candidate features until a desired model size is reached. The performance of the final model on the outer loop is evaluated. In this way, performance estimates with lower variance can be obtained and the variability of the test predictions and model instability can be evaluated. If the final model has promising performance, it is evaluated on a sufficiently large external test set.

[0277] Some examples provide systems and methods forming an automated framework for preparing multiple source electronic health record data for use in machine learning model training. Exemplary methods and associated systems include preparing input data from multiple sources. This portion of the framework is primarily concerned with cleaning and extracting features from multiple data sources. This step takes in raw automatically derived EHR data in a data model format (e.g., OMOP data model format, etc.) and multiple expert-curated data sources for additional features and labels. This step is open to extensions, including, but not limited to, modules for preparation of smoking history, medication history, medical condition, radiation therapy history, laboratory measurements, and anthropometric data.

[0278] This exemplary method and associated system includes generating a combined time-dependent unbounded intermediate output, which condenses the input data in a unified format but preserves the timestamps of the individual data items per patient. This intermediate step provides pluggability for modules that prepare time-dependent input data (not implemented) for sequence modeling algorithms and provides flexible input for the aggregation step of the framework.

[0279] Exemplary methods and associated systems include aggregating features for a time-independent model, which is a target-agnostic, highly configurable plug-in module for creating time-independent aggregate inputs to machine learning models, which can accept extensions and configurable parameters include, but are not limited to, prediction time point, length of aggregation, associated data sources, and associated feature types.

[0280] Some examples provide systems and methods for predictive model building for efficacy surrogate endpoints related to immune checkpoint inhibitor treatment. Exemplary methods and related systems include preparing input data from multiple sources. This part of the framework involves cleaning and extracting features from multiple data sources. This step incorporates raw automatically derived EHR data, for example in the OMOP data model format, and multiple expert-curated data sources for additional features. Exemplary methods and related systems include generating ground truth prediction labels such as time on ICI treatment (TOT), time to next treatment (after ICI cessation) (TNET), overall survival (OS), etc. The listed ground truth endpoints are generated on a continuous scale represented by days elapsed from an anchor point (patient timelines are aligned based on similarities in the ICI treatment course). The default anchor point is the first date of ICI treatment initiation. The generated ground truth can be used to train regression or survival analysis-based models as is or with modified granularity (e.g., elapsed weeks, months, years, etc.). Ground truth discretization can be performed for two-class or multi-class classification (e.g., responders vs. non-responders, 5-year survival rate, etc.).

[0281] Exemplary methods and related systems include model building on the generated feature matrix and ground truth as a standalone module of the framework. Endpoints can be modeled separately. Modeling can be performed, for example, without hypotheses, using different machine learning algorithms. Model building and selection workflows can be used to generate predictive models for immune checkpoint inhibitor-associated pneumonia and sequential procedures for model building.

[0282] Some examples provide systems and methods for hepatitis prediction. The exemplary systems and related methods include input preparation through extraction of relevant sections of blood features from received electronic health record (EHR) data tables. These are measurements of liver biomarker concentrations in plasma (ALT, AST, alkaline phosphatase, bilirubin, etc.) and other concentration values ​​in blood. After this step, the time series data is compiled into a single composite data structure, which is an efficient option to continue thereafter by aggregating this information into a final data table, which is now ready for the pre-processing and transformation steps. The aggregation step of feature engineering consists of describing the time series data of blood particles with mean, standard deviation, minimum, and maximum values. Lag features can also be created by taking the last liver biomarker measurements available before treatment. Labels are created using definitions obtained from medical experts, which, for example, classify as positive when the level of at least one liver biomarker exceeds three times the upper limit of normal within a predefined window, and as negative otherwise. The date of ICI treatment used in this scheme can be output from a different workflow.

[0283] Exemplary systems and methods include resampling of the dataset. For example, the dataset resulting from step 1 is unbalanced, so if the goal is to maximize the recall value, a random majority vote us is run on the dataset. If the F1 score is the object of maximization, the resampling step is ignored.

[0284] The exemplary system and method includes training and prediction. For example, on the final data resulting from step 2, a model is trained that is RF for maximizing recall and GB for maximizing F1 score. Validation is performed using, for example, leave-one-out cross-validation, where each sample is predicted individually with the rest as the training set.

[0285] A brief list of some examples Further aspects of the present disclosure are provided by the subject matter of the following sections:

[0286] Example 1 is a method of configuring a treatment plan for a patient, the method including: a) loading a toxicity-associated model for a selected toxicity associated with an immunotherapy, the toxicity-associated model being selected from a plurality of candidate models using model selection criteria; b) determining patient selection criteria to set a balance between precision and recall for actions responsive to an output of the toxicity-associated model, the output including a toxicity prediction; c) processing healthcare data from a healthcare record for a given patient using the toxicity-associated model to generate a toxicity prediction; d) evaluating the toxicity prediction with respect to the patient and the patient selection criteria; e) generating a first prescription and configuring the treatment plan to include the immunotherapy when the toxicity prediction meets the patient selection criteria; f) generating a second prescription and excluding the immunotherapy from the treatment plan when the toxicity prediction does not meet the patient selection criteria; and g) outputting an order to configure the treatment plan for the patient based on the first prescription or the second prescription.

[0287] Example 2 is a method of configuring a cohort for a clinical trial, the method including: a) loading a toxicity-associated model for a selected toxicity associated with an immunotherapy, where the toxicity-associated model is selected from a plurality of candidate models using a model selection criterion; b) determining patient selection criteria to set a balance between precision and recall for actions responsive to an output of the toxicity-associated model, where the output includes a toxicity prediction; c) processing healthcare data from a healthcare record for a given patient using the toxicity-associated model to generate a toxicity prediction; d) evaluating the toxicity prediction with respect to the patient and the patient selection criterion; e) generating a first prescription and configuring a treatment plan to include the immunotherapy when the toxicity prediction meets the patient selection criterion; f) generating a second prescription and excluding the immunotherapy from the treatment plan when the toxicity prediction does not meet the patient selection criterion; and g) outputting an order to configure a treatment plan for the patient based on the first prescription or the second prescription.

[0288] Example 3 includes the method of any preceding paragraph, further including generating a toxicity-associated model by forming features from an input dataset obtained from health care records, the features being compared by iteratively iterating over associated outcomes to form a model, and the model being tuned according to precision and recall analysis.

[0289] Example 4 includes the method of any preceding paragraph, wherein the patient selection criterion is a first patient selection criterion, the balance is a first balance, and the output is a first output, this example further including loading an efficacy-related model related to efficacy of the immunotherapy; determining a second patient selection criterion for setting a second balance between precision and recall for an action responsive to a second output of the efficacy-related model, the second output including an efficacy prediction; processing healthcare data from the healthcare record for a given patient using the efficacy-related model to generate an efficacy prediction; and evaluating the efficacy prediction with respect to the patient and the second patient selection criterion.

[0290] Example 5 includes the method of any preceding paragraph, further including generating a first prescription and configuring the treatment plan to include immunotherapy when the toxicity prediction meets a first patient selection criterion and the efficacy prediction meets a second patient selection criterion, and otherwise generating a second prescription and excluding immunotherapy from the treatment plan.

[0291] Example 6 includes the method of any preceding paragraph, further including evaluating the toxicity prediction relative to the efficacy prediction for the given patient, and generating respective first or second instructions to configure the treatment plan to include or exclude immunotherapy, respectively.

[0292] Example 7 includes the method of any preceding paragraph, wherein efficacy is measured by patient survival or duration of treatment.

[0293] Example 8 includes the method of any preceding paragraph, wherein constructing the treatment plan includes constructing the treatment plan for a plurality of selected patients to constitute a cohort for a clinical trial.

[0294] Example 9 includes the method of any preceding paragraph, wherein configuring the treatment regimen further includes determining a schedule for administering the immunotherapy to the patient.

[0295] Example 10 includes the method of any preceding paragraph, wherein configuring the treatment plan further includes enabling patient monitoring to evaluate the patient during implementation of the treatment plan.

[0296] Example 11 includes the method of any preceding paragraph, wherein the patient selection criteria includes a threshold, the threshold defining a balance between false positives and false negatives for the toxicity-associated model.

[0297] Example 12 includes the method of any preceding paragraph, wherein the selected toxicity is pneumonia, colitis, or hepatitis, and the toxicity prediction is i) the probability that the patient will suffer from the selected toxicity, or ii) the probability that the patient will not suffer from the selected toxicity.

[0298] Example 13 includes the method of any preceding paragraph, wherein the healthcare data includes at least one of a laboratory test result, a diagnostic code, or a billing code.

[0299] Example 14 includes the method of any preceding paragraph, further including aligning the healthcare data with respect to a reference point corresponding to an event in a healthcare record for the patient.

[0300] Example 15 includes the method of any preceding paragraph, further including re-evaluating the patient using the updated healthcare data for the patient and the updated toxicity-associated model.

[0301] Example 16 includes the method of any preceding section, wherein the model selection criteria includes at least one of a high F1 score, a high recall, or a high precision.

[0302] Example 17 is a method of configuring a treatment plan for a patient, the method including: a) loading an efficacy-related model associated with efficacy of an immunotherapy, where the efficacy-related model is selected from a plurality of candidate models using model selection criteria; b) determining patient selection criteria to set a balance between precision and recall for actions responsive to an output of the efficacy-related model, where the output includes an efficacy prediction; c) processing healthcare data from a healthcare record for a given patient using the efficacy-related model to generate an efficacy prediction; d) evaluating the efficacy prediction with respect to the patient and the patient selection criteria; e) generating a first prescription and configuring the treatment plan to include the immunotherapy when the efficacy prediction meets the patient selection criteria; f) generating a second prescription and excluding the immunotherapy from the treatment plan when the efficacy prediction does not meet the patient selection criteria; and g) outputting an order that configures the treatment plan for the patient based on the first prescription or the second prescription.

[0303] Example 18 is a method of configuring a cohort for a clinical trial, the method including: a) loading an efficacy-related model associated with efficacy of an immunotherapy, where the efficacy-related model is selected from a plurality of candidate models using model selection criteria; b) determining patient selection criteria to set a balance between precision and recall for actions responsive to an output of the efficacy-related model, where the output includes an efficacy prediction; c) processing healthcare data from a healthcare record for a given patient using the efficacy-related model to generate an efficacy prediction; d) evaluating the efficacy prediction with respect to the patient and the patient selection criteria; e) generating a first prescription and configuring a treatment plan to include the immunotherapy when the efficacy prediction meets the patient selection criteria; f) generating a second prescription and excluding the immunotherapy from the treatment plan when the efficacy prediction does not meet the patient selection criteria; and g) outputting an order to configure a treatment plan for the patient based on the first prescription or the second prescription.

[0304] Example 19 includes the method of any preceding paragraph, further including generating an efficacy-related model by forming features from an input dataset obtained from health care records, the features being compared by iteratively comparing associated outcomes to form a model, and the model being tuned according to precision and recall analysis.

[0305] Example 20 includes the method of any preceding clause, wherein the patient selection criterion is a first patient selection criterion, the balance is a first balance, and the output is a first output, this example further including loading a toxicity-associated model for a selected toxicity associated with the immunotherapy; determining a second patient selection criterion for setting a second balance between precision and recall for an action responsive to a second output of the toxicity-associated model, the second output including a toxicity prediction; processing healthcare data from the healthcare record for the given patient using the toxicity-associated model to generate a toxicity prediction; and evaluating the toxicity prediction with respect to the patient and the second patient selection criterion.

[0306] Example 21 includes the method of any preceding paragraph, further including generating a first prescription and configuring the treatment plan to include immunotherapy when the efficacy prediction meets a first patient selection criterion and the toxicity prediction meets a second patient selection criterion, and otherwise generating a second prescription and excluding immunotherapy from the treatment plan.

[0307] Example 22 includes the method of any preceding paragraph, further including evaluating the toxicity prediction relative to the efficacy prediction for the given patient, and generating respective first or second instructions to configure the treatment plan to include or exclude immunotherapy, respectively.

[0308] Example 23 includes the method of any preceding clause, wherein the selected toxicity is pneumonia, colitis, or hepatitis, and the toxicity prediction is i) the probability that the patient will suffer from the selected toxicity, or ii) the probability that the patient will not suffer from the selected toxicity.

[0309] Example 24 includes the method of any preceding section, wherein constructing the treatment plan includes constructing the treatment plan for a plurality of selected patients to constitute a cohort for a clinical trial.

[0310] Example 25 includes the method of any preceding paragraph, wherein formulating the treatment regimen further includes determining a schedule for administering the immunotherapy to the patient.

[0311] Example 26 includes the method of any preceding paragraph, wherein configuring the treatment plan further includes enabling patient monitoring to evaluate the patient during implementation of the treatment plan.

[0312] Example 27 includes the method of any preceding section, wherein the patient selection criteria include a threshold, the threshold defining a balance between false positives and false negatives for the efficacy-related model.

[0313] Example 28 includes the method of any preceding paragraph, wherein the healthcare data includes at least one of a laboratory test result, a diagnostic code, or a billing code.

[0314] Example 29 includes the method of any preceding paragraph, further including aligning the healthcare data with respect to a reference point corresponding to an event in a healthcare record for the patient.

[0315] Example 30 includes the method of any preceding paragraph, further including re-evaluating the patient using the updated health care data for the patient and the updated efficacy-related model.

[0316] Example 31 includes the method of any preceding section, wherein the model selection criteria includes at least one of a high F1 score, a high recall, or a high precision.

[0317] Example 32 includes the method of any preceding paragraph, wherein efficacy is measured by patient survival or duration of treatment.

[0318] Example 33 is a method of configuring a treatment plan for a patient, comprising: a) generating a toxicity-associated model for a selected toxicity associated with an immunotherapy by at least i) training an initial model on a plurality of features generated from a plurality of prior healthcare data, the prior healthcare data being arranged in a time series and aligned with respect to a reference point; ii) iteratively adding and removing each of a set of additional features to tune the initial model into a tuned model; evaluating the resulting precision and recall associated with an output of the tuned model with respect to a model selection criterion; and iii) deploying the tuned model as the toxicity-associated model when the model selection criterion is met; and b) generating a toxicity-associated model output. determining patient selection criteria to set a balance between precision and recall for actions responsive to the toxicity prediction, where the output includes a toxicity prediction; c) processing healthcare data from the healthcare record for a given patient using a toxicity-related model to generate a toxicity prediction; d) evaluating the toxicity prediction with respect to the patient and the patient selection criteria; e) generating a first prescription and configuring a treatment plan to include immunotherapy when the toxicity prediction meets the patient selection criterion; f) generating a second prescription and excluding immunotherapy from the treatment plan when the toxicity prediction does not meet the patient selection criterion; and g) outputting an order to configure a treatment plan for the patient based on the first prescription or the second prescription.

[0319] Example 34 is a method of configuring a cohort for a clinical trial, comprising: a) generating a toxicity-associated model for a selected toxicity associated with an immunotherapy by at least i) training an initial model on a plurality of features generated from a plurality of prior healthcare data, the prior healthcare data being arranged in a time series and aligned with respect to a reference point; ii) iteratively adding and removing each of a set of additional features to tune the initial model into a tuned model; evaluating the resulting precision and recall associated with the output of the tuned model with respect to a model selection criterion; and iii) deploying the tuned model as the toxicity-associated model when the model selection criterion is met; and b) generating a toxicity-associated model for a selected toxicity associated with an immunotherapy by at least i) training an initial model on a plurality of features generated from a plurality of prior healthcare data, the prior healthcare data being arranged in a time series and aligned with respect to a reference point; ii) iteratively adding and removing each of a set of additional features to tune the initial model into a tuned model; determining patient selection criteria to set a balance between precision and recall for force-responsive actions, the output including a toxicity prediction; c) processing healthcare data from a healthcare record for a given patient using a toxicity-relevant model to generate a toxicity prediction; d) evaluating the toxicity prediction with respect to the patient and the patient selection criteria; e) generating a first prescription and configuring a treatment plan to include immunotherapy when the toxicity prediction meets the patient selection criteria; f) generating a second prescription and excluding immunotherapy from the treatment plan when the toxicity prediction does not meet the patient selection criteria; and g) outputting an order to configure a treatment plan for the patient based on the first prescription or the second prescription.

[0320] Example 35 includes the method of any preceding section and further includes partitioning the features into an inner loop and an outer loop and iteratively evaluating the models by comparing the loops according to a model selection criterion.

[0321] Example 36 includes the method of any preceding clause, wherein the patient selection criterion is a first patient selection criterion, the balance is a first balance, and the output is a first output, this example further including loading an efficacy-related model related to efficacy of the immunotherapy; determining a second patient selection criterion for setting a second balance between precision and recall for an action responsive to a second output of the efficacy-related model, the second output including an efficacy prediction; processing healthcare data from the healthcare record for a given patient using the efficacy-related model to generate an efficacy prediction; and evaluating the efficacy prediction with respect to the patient and the second patient selection criterion.

[0322] Example 37 includes the method of any preceding paragraph, further including generating a first prescription and configuring the treatment plan to include immunotherapy when the toxicity prediction meets a first patient selection criterion and the efficacy prediction meets a second patient selection criterion, and otherwise generating a second prescription and excluding immunotherapy from the treatment plan.

[0323] Example 38 includes the method of any preceding paragraph, further including evaluating the toxicity prediction relative to the efficacy prediction for the given patient, and generating respective first or second instructions to configure the treatment plan to include or exclude immunotherapy, respectively.

[0324] Example 39 includes the method of any preceding paragraph, wherein efficacy is measured by patient survival or duration of treatment.

[0325] Example 40 includes the method of any preceding paragraph, wherein constructing the treatment plan includes constructing the treatment plan for a plurality of selected patients to constitute a cohort for a clinical trial.

[0326] Example 41 includes the method of any preceding paragraph, wherein forming the treatment regimen further includes scheduling administration of the immunotherapy to the patient.

[0327] Example 42 includes the method of any preceding paragraph, wherein configuring the treatment plan further includes enabling patient monitoring to evaluate the patient during implementation of the treatment plan.

[0328] Example 43 includes the method of any preceding section, wherein the patient selection criteria include a threshold, the threshold defining a balance between false positives and false negatives for the toxicity association model.

[0329] Example 44 includes the method of any preceding clause, wherein the selected toxicity is pneumonia, colitis, or hepatitis, and the toxicity prediction is i) the probability that the patient will suffer from the selected toxicity, or ii) the probability that the patient will not suffer from the selected toxicity.

[0330] Example 45 includes the method of any preceding clause, wherein the healthcare data includes at least one of a laboratory test result, a diagnostic code, or a billing code.

[0331] Example 46 includes the method of any preceding paragraph, further including aligning the healthcare data with respect to a reference point corresponding to an event in a healthcare record for the patient.

[0332] Example 47 includes the method of any preceding paragraph, further including re-evaluating the patient using the updated healthcare data for the patient and the updated toxicity-associated model.

[0333] Example 48 includes the method of any preceding section, wherein the model selection criteria includes at least one of a high F1 score, high recall, or high precision.

[0334] Example 49 includes the method of any preceding paragraph, further including deploying the toxicity-relevant model in a tool having an interface to facilitate collection of patient healthcare data and interaction with the toxicity-relevant model.

[0335] Example 50 is a method of configuring a treatment plan for a patient, the method comprising: a) generating an efficacy-related model that predicts efficacy of an immunotherapy by at least i) training an initial model on a plurality of features generated from a plurality of prior healthcare data, the prior healthcare data being arranged in a time series and aligned with respect to a reference point; ii) iteratively adding and removing each of a set of additional features to tune the initial model into a tuned model; evaluating the resulting precision and recall associated with an output of the tuned model with respect to a model selection criterion; and iii) deploying the tuned model as an efficacy-related model when the model selection criterion is met; and b) taking an action responsive to the output of the efficacy-related model. c) determining patient selection criteria to set a balance between precision and recall for a given patient, the output including an efficacy prediction; c) processing healthcare data from the healthcare record for the given patient using the efficacy-related model to generate an efficacy prediction; d) evaluating the efficacy prediction with respect to the patient and the patient selection criteria; e) generating a first prescription and configuring a treatment plan to include the immunotherapy when the efficacy prediction meets the patient selection criterion; f) generating a second prescription and excluding the immunotherapy from the treatment plan when the efficacy prediction does not meet the patient selection criterion; and g) outputting an order configuring a treatment plan for the patient based on the first prescription or the second prescription.

[0336] Example 51 is a method of configuring a cohort for a clinical trial, the method comprising: a) generating an efficacy-related model that predicts efficacy of an immunotherapy by at least i) training an initial model on a plurality of features generated from a plurality of prior healthcare data, the prior healthcare data being arranged in a time series and aligned with respect to a reference point; ii) iteratively adding and removing each of a set of additional features to tune the initial model into a tuned model; evaluating the resulting precision and recall associated with the output of the tuned model with respect to a model selection criterion; and iii) deploying the tuned model as an efficacy-related model when the model selection criterion is met; and b) taking an action responsive to the output of the efficacy-related model. c) determining patient selection criteria to set a balance between precision and recall for a given patient, the output including an efficacy prediction; c) processing healthcare data from the healthcare record for a given patient using the efficacy-related model to generate an efficacy prediction; d) evaluating the efficacy prediction with respect to the patient and the patient selection criteria; e) generating a first prescription and configuring a treatment plan to include the immunotherapy when the efficacy prediction meets the patient selection criterion; f) generating a second prescription and excluding the immunotherapy from the treatment plan when the efficacy prediction does not meet the patient selection criterion; and g) outputting an order configuring a treatment plan for the patient based on the first prescription or the second prescription.

[0337] Example 52 includes the method of any preceding section and further includes partitioning the features into an inner loop and an outer loop and iteratively evaluating models by comparing the loops according to a model selection criterion.

[0338] Example 53 includes the method of any preceding clause, wherein the patient selection criterion is a first patient selection criterion, the balance is a first balance, and the output is a first output, this example further including loading a toxicity-associated model for a selected toxicity associated with the immunotherapy; determining a second patient selection criterion for setting a second balance between precision and recall for an action responsive to a second output of the toxicity-associated model, the second output including a toxicity prediction; processing healthcare data from the healthcare record for the given patient using the toxicity-associated model to generate a toxicity prediction; and evaluating the toxicity prediction with respect to the patient and the second patient selection criterion.

[0339] Example 54 includes the method of any preceding paragraph, further including generating a first prescription and configuring the treatment plan to include immunotherapy when the efficacy prediction meets a first patient selection criterion and the toxicity prediction meets a second patient selection criterion, and otherwise generating a second prescription and excluding immunotherapy from the treatment plan.

[0340] Example 55 includes the method of any preceding paragraph, further including evaluating the toxicity prediction relative to the efficacy prediction for the given patient, and generating respective first or second instructions to configure the treatment plan to include or exclude immunotherapy, respectively.

[0341] Example 56 includes the method of any preceding clause, wherein the selected toxicity is pneumonia, colitis, or hepatitis, and the toxicity prediction is i) the probability that the patient will suffer from the selected toxicity, or ii) the probability that the patient will not suffer from the selected toxicity.

[0342] Example 57 includes the method of any preceding paragraph, wherein constructing the treatment plan includes constructing the treatment plan for a plurality of selected patients to constitute a cohort for a clinical trial.

[0343] Example 58 includes the method of any preceding paragraph, wherein forming the treatment regimen further includes scheduling administration of the immunotherapy to the patient.

[0344] Example 59 includes the method of any preceding clause, wherein configuring the treatment plan further includes enabling patient monitoring to evaluate the patient during implementation of the treatment plan.

[0345] Example 60 includes the method of any preceding paragraph, wherein the patient selection criteria includes a threshold, the threshold defining a balance between false positives and false negatives for the efficacy-related model.

[0346] Example 61 includes the method of any preceding paragraph, wherein the healthcare data includes at least one of a laboratory test result, a diagnostic code, or a billing code.

[0347] Example 62 includes the method of any preceding paragraph, including aligning the healthcare data with respect to a reference point corresponding to an event in a healthcare record for the patient.

[0348] Example 63 includes the method of any preceding paragraph, further including re-evaluating the patient using the updated health care data for the patient and the updated efficacy-related model.

[0349] Example 64 includes the method of any preceding section, wherein the model selection criteria includes at least one of a high F1 score, a high recall, or a high precision.

[0350] Example 65 includes the method of any preceding paragraph, wherein efficacy is measured by patient survival or duration of treatment.

[0351] Example 66 includes the method of any preceding paragraph, further including deploying the efficacy-related model in a tool having an interface for facilitating collection of patient healthcare data and interaction with the efficacy-related model.

[0352] Example 67 is a computer-readable medium containing instructions that, when executed, cause a processor circuit to implement the method of any preceding item.

[0353] Example 68 is an apparatus comprising memory, instructions, and processor circuitry for implementing the method of any preceding paragraph.

[0354] As disclosed and described herein, the processor circuitry provides means for processing (including, for example, means for processing input data, means for training models, means for selecting, means for deploying, means for generating, means for evaluating, means for triggering, means for ordering, means for displaying, etc.), and the memory circuitry provides means for storing. As described above, various circuits may be implemented by the processor circuitry and the memory circuitry.

[0355] The following claims are hereby incorporated by reference into the detailed description of this specification. Although certain exemplary systems, methods, apparatus, and articles of manufacture have been disclosed herein, the scope of this patent is not limited thereto. On the contrary, this patent covers all systems, methods, apparatus, and articles of manufacture that fairly fall within the scope of the claims of this patent. [Explanation of symbols]

[0356] 100 Model Generator 110 Input processor circuit, input data processor 120 Model Trainer Circuit 130 Model Comparator Circuit 140 Model Memory Circuit 150 Model Deployer Circuit 160 External Systems 200 processes 500 Sequential Procedures 510 Initial Data Set 520 First partition of data 522 Inner Loop 524 Outer Loop Second partition of 525 data 600 processes 605 Concept Table 615 Clinical Table 625 Model Performance Metrics 635 AE risk prediction dataset 700 processes 705 Input Dataset 715 Model size input 720 Selection Procedure 730 Robust Performance Evaluation 1000 processes 1005 Input Dataset 1025 output datasets 1105 Interim Data Set 1205 Downstream Task-Dependent External Patient Eligibility Criteria 1215 Downstream Task-Dependent External Patient Eligibility Criteria 1300 Processes 1305 Structured Data Sets 1315 Curated Datasets 1325 Input Data 1400 methods 1405 EHR Data Table 1500 Immunotherapy Prediction Device 1510 Input processor circuit, input data processor 1520 Memory Circuit 1522 Model Memory Section 1524 Data storage unit 1530 Model Processor Circuit 1540 output generation circuit 1550 Interface 1560 Model Source 1565 Data Sources 1570 External Device or System 1600 processes 1800 Processes 1900 expression 1910 Data 1920 Date 1930 model 2010 x-axis 2020 y-axis 2030 Precision Curve 2040 recall curve 2110 x-axis 2120 y-axis 2130 Precision Curve 2140 recall curve 2210 x-axis 2220 y-axis 2230 Precision Curve 2240 recall curve 2310 x-axis 2320 y-axis 2330 Precision Curve 2340 recall curve 2410 x-axis 2420 y-axis 2430 Precision Curve 2440 recall curve 2500 Infrastructure and Related Processes 2510 Patient Health Care Records 2520, 2525 Datastore 2530 Cleaning Action 2540, 2545 Time-dependent values 2550 features 2600 Processor Platform 2612 Processor Circuit 2613 Local Memory 2614 Volatile Memory 2616 Non-volatile memory 2617 Memory Controller 2618 Bus 2620 Interface Circuit 2622 Input Devices 2624 Output Device 2626 Network 2628 Mass Storage Device 2632 Machine-Readable Instructions 2700 Microprocessor 2702 Core 2704 First Exemplary Bus 2706 Interface circuit 2710 Shared Memory 2714 Control unit circuit 2716 Arithmetic and Logic (AL) Circuits 2718 Register 2720 ​​Local Memory 2722 Second Exemplary Bus 2800 FPGA circuits 2802 Input / Output (I / O) Circuit 2804 Configuration Circuit 2806 External Hardware 2808 Logic Gate Circuit 2810 Configurable Interconnect 2812 Memory circuit 2814 Dedicated arithmetic circuit 2816 Special Purpose Circuit 2818 General Purpose Programmable Circuit 2820 CPU 2822 DSP 2905 Software Distribution Platform 2910 Network

Claims

1. A method for constructing a treatment plan for a patient, a) A step of loading a model, wherein the model is selected from a plurality of candidate models using model selection criteria, and the model is (i) a toxicity-related model for selected toxicity associated with the immunotherapy, or (ii) a efficacy-related model for the efficacy of the immunotherapy. b) A step of determining patient selection criteria for setting a balance between precision and recall for actions in response to the output of the model, wherein the output includes predictions; c) Using the model, the step of processing health management data from health management records for a given patient to generate the prediction, d) A step of evaluating the prediction with respect to the patient and the patient selection criteria, e) When the prediction satisfies the patient selection criteria, the steps include generating a first instruction and configuring the treatment plan to include the immunotherapy, f) When the prediction does not meet the patient selection criteria, the step of generating a second instruction and excluding the immunotherapy from the treatment plan, g) A method comprising the step of outputting an order that constitutes the treatment plan for the patient based on the first instruction or the second instruction.

2. The method according to claim 1, further comprising the step of generating the model by forming features from an input dataset obtained from health management records, wherein the features are compared by iterating over relevant results to form the model, and the model is tuned according to precision and recall analysis.

3. The patient selection criterion is the first patient selection criterion, the balance is the first balance, the output is the first output, the model is the toxicity-related model for the selected toxicity associated with the immunotherapy, and the prediction is the toxicity prediction. The steps include loading a efficacy-related model related to the efficacy of the immunotherapy, A step of determining a second patient selection criterion for setting a second balance between precision and recall for an action in response to a second output of the efficacy-related model, wherein the second output includes efficacy predictions. Using the efficacy-related model, the process involves processing health management data from health management records for a given patient to generate the efficacy prediction. The method according to claim 1, further comprising the step of evaluating the efficacy prediction with respect to the patient and the second patient selection criterion.

4. The steps include generating a first instruction and configuring the treatment plan to include the immunotherapy when the toxicity prediction satisfies the first patient selection criterion and the efficacy prediction satisfies the second patient selection criterion, The method according to claim 3, further comprising the step of generating the second instruction and excluding the immunotherapy from the treatment plan if otherwise.

5. The method according to claim 3, wherein the efficacy is measured by patient survival rate or duration of treatment.

6. The method according to any one of claims 1 to 5, wherein the step of configuring the treatment plan includes the step of configuring the treatment plan for a number of selected patients to build a cohort for a clinical trial.

7. The method according to any one of claims 1 to 5, further comprising the step of determining a schedule for applying the immunotherapy to the patient, wherein the steps of constituting the treatment plan are further a step of determining a schedule for applying the immunotherapy to the patient.

8. The method according to any one of claims 1 to 5, further comprising the step of configuring the treatment plan, the step of configuring the patient so that patient monitoring can be performed to evaluate the patient during the execution of the treatment plan.

9. The method according to any one of claims 1 to 5, wherein the patient selection criteria include a threshold, and the threshold defines a balance between false positives and false negatives for the model.

10. The method according to any one of claims 1 to 5, wherein the selected toxicity is pneumonia, colitis, or hepatitis, and the prediction is i) the probability that the patient will suffer from the selected toxicity, or ii) the probability that the patient will not suffer from the selected toxicity.

11. The method according to any one of claims 1 to 5, wherein the health management data comprises at least one of laboratory test results, diagnostic codes, or claim codes.

12. The method according to claim 11, further comprising the step of aligning the health management data with respect to a reference point corresponding to an event in the health management record for the patient.

13. The method according to any one of claims 1 to 5, further comprising the step of re-evaluating the patient using updated health management data and an updated model, wherein the updated model is an updated toxicity-related model or an updated efficacy-related model.

14. The method according to any one of claims 1 to 5, wherein the model selection criterion includes at least one of a high F1 score, high recall, or high precision.

15. The patient selection criterion is the first patient selection criterion, the balance is the first balance, the output is the first output, the model is the efficacy-related model related to the immunotherapy, and the prediction is the efficacy prediction. The steps include loading toxicity-related models for selected toxicity associated with the immunotherapy, A step of determining a second patient selection criterion for setting a second balance between precision and recall for an action in response to a second output of the toxicity-related model, wherein the second output includes toxicity prediction. Using the toxicity-related model, the process involves processing health management data from health management records for a given patient to generate the toxicity prediction. The method according to claim 1, further comprising the step of evaluating the toxicity prediction with respect to the patient and the second patient selection criterion.

16. The steps include generating a first instruction and configuring the treatment plan to include the immunotherapy when the efficacy prediction satisfies the first patient selection criterion and the toxicity prediction satisfies the second patient selection criterion, The method according to claim 15, further comprising the step of generating the second instruction and excluding the immunotherapy from the treatment plan if otherwise.

17. The method according to claim 4 or 16, further comprising the steps of evaluating the toxicity prediction in comparison with the efficacy prediction for a given patient, generating the respective first or second instructions, and configuring the treatment plan to include or exclude the immunotherapy, respectively.

18. The step of loading the model is: a) A step of generating the model, wherein at least i) Train an initial model on multiple features generated from multiple previous health management data, wherein the previous health management data is arranged and structured in time series and aligned with respect to a reference point. ii) Repeatedly add and remove each of the additional feature sets to adjust the initial model into an adjusted model, and evaluate the resulting precision and recall associated with the output of the adjusted model with respect to the model selection criteria. iii) A step of generating by deploying the adjusted model as the model when the model selection criteria are met. The method according to any one of claims 1 to 5, including the method described in any one of claims 1 to 5.

19. The method of claim 18, further comprising the step of iteratively evaluating the model by partitioning the feature quantities into an inner loop and an outer loop and comparing the loops according to the model selection criteria.

20. A computer-readable medium containing instructions that, when executed, cause a processor circuit to implement the method described in any one of claims 1 to 5.

21. It is a device, Memory and Commands and, An apparatus comprising a processor circuit for implementing the method described in any one of claims 1 to 5.

Images