Model generation device for treatment prediction, related methods, and models

Abstract

A method, apparatus, system, and article of manufacture are disclosed for generating and applying models for treatment prediction and processing. An exemplary apparatus at least processes input data pulled from records to form a set of candidate features, trains a first model and a second model using the set of candidate features, validates the first model and the second model, compares the performance of the first model and the performance of the second model, selects at least one of the first model or the second model based on this comparison, stores the selected first model and / or second model, and deploys the selected first model and / or second model to predict at least one of a) toxicity resulting from immunotherapy according to a treatment plan, or b) the efficacy of a treatment plan for a patient.

Description

Technical Field

[0001] Cross - reference to Related Applications This patent claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 335,215, filed on April 26, 2022, which is hereby incorporated by reference in its entirety for all purposes.

[0002] The present disclosure generally relates to the generation and processing of models, and more particularly, to the generation and application of models for treatment prediction and treatment.

Background Art

[0003] The description in this section merely presents background information related to the present disclosure and may not constitute prior art.

[0004] Immunotherapy can be used to provide effective treatment for cancer in some patients. For those patients, immunotherapy can provide higher efficacy and lower toxicity than other treatment methods. Immunotherapy includes targeted antibodies and immune checkpoint inhibitors (ICIs), cell - based immunotherapies, immunomodulators, vaccines, and viral therapies, which can help the patient's immune system target and destroy malignant tumors. However, in some patients, immunotherapy can cause toxicity and / or other side effects. The side effects of immunotherapy can be different from those associated with other cancer treatments because they result not from the direct action of chemotherapy or radiation therapy on cancer and healthy tissues, but rather from an over - stimulated or misdirected immune response. The toxicity of immunotherapy can include diseases such as colitis, hepatitis, pneumonia, and / or other inflammations that can pose a risk to the patient. Immunotherapy also elicits different (heterogeneous) efficacy responses in different patients. As such, the evaluation of immunotherapy can vary widely among patients and remains unpredictable.

Brief Description of the Drawings

[0005]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

[0006] Generally, the same reference numbers are used throughout the drawings and the entire written description to refer to the same or similar parts. The figures are not drawn to scale.

[0007] As used in this patent, stating that any part (e.g., a layer, film, area, region, or plate) is in some way on another part (e.g., positioned, disposed, arranged, or formed, etc.) indicates that the part being referred to is in contact with the other part or that the part being referred to is over the other part with one or more intermediate parts disposed therebetween.

[0008] As used herein, connection references (e.g., attached, coupled, connected, and linked) can, unless otherwise specified, include intermediate members between the elements referenced by the connection reference, and / or relative movement between these elements. As such, a connection reference does not necessarily infer that two elements are directly connected and / or in a fixed relationship to each other. As used herein, to describe that any part is "in contact" with another part is defined to mean that there is no intermediate part between the two parts.

[0009] When introducing elements of various embodiments of the present disclosure, the articles "a" (corresponding to the English indefinite articles "a" and "an"), "the" (corresponding to the English definite article "the"), and "said" are intended to mean that one or more of these elements are present.

[0010] The phrases "comprising," "including," and "having" are intended to be inclusive and mean that there may be additional elements other than the recited elements.

[0011] Unless otherwise specified, descriptors such as "first," "second," "third," etc. are used herein never to imply a priority, physical order, arrangement configuration within a list, and / or order, and are used only as labels and / or arbitrary names to distinguish elements to facilitate understanding of the disclosed examples. In some examples, the descriptor "first" may be used to refer to an element in the detailed description, but the same element in the claims may be referred to using a different descriptor such as "second" or "third." In such cases, it should be understood that such descriptors are used only to distinguish and identify elements that may otherwise share the same name.

[0012] As used herein, "substantially" and "about" modify their subjects / values to account for the potential presence of variations that occur in real-world applications. For example, "substantially" and "about" may modify dimensions that may not be exact due to manufacturing tolerances and / or other real-world imperfections, as would be understood by one of ordinary skill in the art. For example, "substantially" and "about" may indicate that such dimensions may be within a tolerance range of ±10% unless otherwise specified in the following description. "Substantially real-time" as used herein refers to occurring in a manner that is almost instantaneous, recognizing that there may be real-world delays due to, for example, computational time, transmission, etc. Thus, unless otherwise specified, "substantially real-time" refers to real time ±1 second.

[0013] As used herein, the phrase "communicating" includes direct communication and / or indirect communication through one or more intermediary components, including variations of that phrase, and does not require direct physical (e.g., wired) communication and / or constant communication, but rather includes selective communication at periodic intervals, scheduled intervals, aperiodic intervals, and / or one-time events in addition to that.

[0014] As used herein, terms such as "system", "unit", "module", "engine", etc. may include a hardware system and / or a software system that operates to perform one or more functions. For example, a module, unit, or system may include a computer processor, a controller, and / or other logic-based devices that execute operations based on instructions stored on a tangible, non-transitory computer-readable storage medium such as a computer memory. Alternatively, a module, unit, engine, or system may include a hardwired device that executes operations based on the hardwired logic of the device. The various modules, units, engines, and / or systems shown in the accompanying figures may be represented as hardware that operates based on software or hardwired instructions, software that commands the hardware to perform operations, or a combination thereof.

[0015] As used herein, a "processor circuit" is defined as including (i) one or more dedicated electrical circuits structured to perform certain operations and including one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors), and / or (ii) one or more general-purpose semiconductor-based electrical circuits that are programmable using instructions to perform certain operations and include one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors). Examples of processor circuits include programmable microprocessors, field programmable gate arrays (FPGAs) that can instantiate instructions, central processing units (CPUs), graphics processing unit units (GPUs), digital signal processors (DSPs), XPUs, or microcontrollers, and integrated circuits such as application specific integrated circuits (ASICs). For example, an XPU can be implemented by a heterogeneous computing system that includes multiple types of processor circuits (e.g., one or more FPGAs, one or more CPUs, one or more GPUs, one or more DSPs, etc., and / or combinations thereof), and an application programming interface (API) that can allocate computing tasks to the processor circuit among the multiple types of processor circuits that is most suitable for executing the computing task.

[0016] In addition, it should be understood that even though the present disclosure may be described with respect to "one embodiment" or "an embodiment", it is not intended to exclude the existence of additional embodiments that incorporate the recited features.

[0017] In the following detailed description, reference is made to the accompanying drawings, which form a part hereof, and in which are shown, by way of illustration, specific examples that may be implemented. These examples are described in sufficient detail to enable one skilled in the art to practice the subject matter, but other examples may be utilized and logical, mechanical, electrical, and other changes may be made without departing from the scope of the subject matter of this disclosure. Accordingly, the following detailed description is provided to illustrate exemplary implementations and should not be considered a limitation on the scope of the subject matter described in this disclosure. Some features from different aspects of the following description may be combined to form further novel aspects of the subject matter described below.

[0018] A large amount of health-related data can be collected using various media and mechanisms related to patients. However, it can be difficult to derive actionable results by processing and interpreting the data. For example, understanding and correlating the various forms and sources of data through standardization / normalization, aggregation, and analysis can be difficult, if not impossible, given the size of the data and the various heterogeneous systems, formats, etc. As such, in some examples, apparatuses, systems, related models, and methods for processing health-related data to find correlations, predict patient outcomes, and facilitate patient diagnosis and treatment are taken up. Some examples provide systems and methods for constructing health data prediction models. Some examples provide frameworks and machine learning workflows for treatment prediction.

[0019] For example, immune checkpoints regulate the human immune system. Immune checkpoints are pathways that allow the body to achieve self-tolerance by preventing the immune system from attacking cells indiscriminately. However, some cancers can protect themselves from attack by stimulating immune checkpoints (e.g., proteins on immune cells). To target cancer cells in the body, immune checkpoint inhibitors (ICIs) can be used to target these immune checkpoint proteins and, rather than hiding cancer cells, better identify and attack them.

[0020] Despite the great success of cancer treatment with ICIs, such treatments can pose a significant threat to human health due to side effects, a type of immune-related adverse event (irAE) caused by these treatment options. One of these toxicities is hepatitis, which occurs when the liver is affected by an autoimmune-like inflammatory pathological process induced by ICIs. In some cases, the onset of irAE hepatitis is predicted before the start of the first ICI treatment. More precisely, some cases predict whether irAE hepatitis will occur within a given time frame after the start of the first treatment. Other toxicities, such as pneumonia and colitis, can be predicted similarly.

[0021] For example, majority vote class undersampling can be combined with time series data aggregation to thereby obtain a balanced static dataset that can be supplied to a model. Exemplary models include gradient boosting (GB) and random forest (RF), and / or other models that can accommodate the size and statistical characteristics of the data. The model can be selected based on the F1 score, which is a measure of the accuracy of the model on the dataset. For example, a GB model without undersampling can maximize the F1 score (e.g., the harmonic mean of recall and precision), and an RF model with undersampling can provide a high recall (e.g., the ratio of true positives discovered) with a relatively low precision (e.g., the ratio of true positives among the predictions), which is acceptable due to the cost-effectiveness of the additional testing required based on the model's decision. These models can create probability estimates for the labels rather than just the discrete labels themselves.

[0022] Input data is prepared to develop and / or supply a model that drives predictions, treatments, etc. In some examples, the input is prepared by extracting blood feature information (e.g., related sections of blood features) from an electronic health record (EHR) data table (e.g., received in a processor / processing circuit from an EHR, etc.), electronic medical record (EMR) information, etc. Blood features are measurements of liver biomarker concentrations (such as ALT, AST, alkaline phosphatase, and bilirubin) in plasma and other concentration values in the blood. Blood features can be represented, for example, as time series data. After the blood features are extracted, the time series data is formed into a single composite data structure. This data structure is used to aggregate the time series blood feature data into a data table and can be used with preprocessing and transformation.

[0023] For example, in feature engineering, blood feature data is aggregated by describing time series data of blood particles along with the associated mean value, standard deviation, minimum value, and maximum value. Liver features can also be created by taking the last liver biomarker measurement available before treatment. Labels can be created to classify someone as positive when the level of at least one liver biomarker exceeds a threshold (e.g., three times the upper limit of normal values) within a pre-defined framework (e.g., using definitions obtained from medical experts, etc.). Otherwise, the label can classify the patient as negative. The date of immune checkpoint inhibitor (ICI) treatment is determined for use along with the label and / or time series and / or provided in some other form.

[0024] After the input data is prepared (e.g., using feature engineering), the dataset is resampled. That is, the dataset resulting from the input preparation is unbalanced. As such, the dataset can be processed to infer, validate, estimate, and / or resample the prepared feature data within the dataset in some other way. For example, random majority vote class undersampling is performed on the dataset when the goal is to maximize the recall value. When the F1 score is the target of maximization, resampling can be skipped or ignored.

[0025] Using the dataset (e.g., resampled or otherwise), a model for generating prediction values can be trained and tested. For example, when maximizing recall is desired, this dataset can be used to train an RF model. When maximizing the F1 score is desired, the dataset can be used to train, for example, a GB model. In some examples, the trained model may be well-validated by leave-one-out cross-validation or the like, where each sample is predicted individually using the rest as the training set.

[0026] As such, various "artificial intelligence" (AI) models can be developed and deployed for use in various health prediction applications. For example, a model can be used to predict static and / or dynamic prognostic factors for hepatitis using AI model and patient (e.g., EHR, etc.) data.

[0027] Alternatively or in addition, a prediction model for ICI-related pneumonia can be developed using small, noisy datasets. Input features can be evaluated using input data from structured (e.g., EHR, EMR, laboratory data systems, etc.) and / or unstructured (e.g., curated from laboratory data systems, EHR, EMR, etc.) data, thereby constructing a model and outputting the predicted probability of pneumonia onset. In some examples, multiple models can be developed and the system and related processes can iteratively determine between versions of two or more models. For example, the available data can be split into two partitions in a sequential forward selection process and robust performance evaluation can be used to select one model to deploy by validating and comparing two developed models.

[0028] For example, the available data can be split into two partitions and a sequential forward selection process can be used to build the model. At each iteration, two model versions are compared and the one with higher performance for both partitions is selected. The second model version is created by adding candidate features to the first model version. In the larger partition, the comparison is based on performance results in the inner loop of nested cross-validation (CV). In the smaller partition, a permutation test is performed to test whether the candidate features have predictive power (e.g., the second model version has high performance). Finally, the outer loop of the nested CV is used for robust performance evaluation of the final model.

[0029] Some examples provide an automated framework for preparing EHR and / or other health data for use in machine learning-based model training. For example, this framework prepares data from multiple sources and generates combined time-dependent unbounded intermediate outputs that are used to aggregate features for training and deployment of time-dependent models. For example, input data sources are processed and the data is used to generate patient vectors. The patient vectors can be used to perform filtering and aggregation, which forms an interface definition. As such, the model-agnostic workflow creates input data sets for multiple model trainings. The intermediate outputs retain the temporal structure for sequential modeling tasks and form a maintainable, sustainable framework with an interface.

[0030] Some examples provide predictive model construction related to ICI where input data from multiple sources is prepared. The ground truth prediction labels are generated from the prepared data and / or the labels can be crafted. One or more models are then constructed on the feature matrix generated using the ground truth prediction labels as a stand-alone module of the framework, using the labels and data. The framework can then drive a workflow that evaluates multiple efficacy surrogate endpoints, for example, to predict response to ICI treatment.

[0031] In some examples, patient health data is prepared and used to train a model using a system with multiple sub-modules. For example, the system includes data extraction and processing sub-modules for extracting a patient's blood test history from an EMR / EHR, cleaning the blood history data, and performing data quality checks. A label definition sub-module defines one or more feature labels related to the blood history data, and a feature engineering sub-module can form blood features by aggregating and processing the blood history data with respect to the labels. A model sub-module trains and evaluates an AI model that dynamically predicts the risk of immune-related hepatitis adverse events from a fixed-length blood test history. Alternatively or in addition, liver function test values can be extracted, cleaned, and organized in a time series. A label definition algorithm is executed, which can generate an AI model and target labels for each set of blood and / or liver test values, while feature engineering (e.g., normalization, symbol conversion, and motif extraction) can be used, for example, in the training and evaluation of an AI risk prediction model. Similarly, drug history information, medical history, anthropometric features, etc. can be used for labeling and feature formation.

[0032] Since features can vary greatly depending on toxicity, the formation of toxicity-specific features and model training are important in providing meaningful features and accurate prediction results. Otherwise, the lack of meaningful features will impair the performance of the resulting model. Thus, in some examples, models focused on specific toxicities are derived, and their outputs can be combined (together and / or further with the prediction of efficacy) to form recommendations based on, for example, a risk-benefit analysis of toxicity versus efficacy of a given immunotherapy drug.

[0033] As such, some examples drive treatment based on predictions of the likelihood of complications from, for example, hepatitis, pneumonia, etc. Patients may be selected for immunotherapy treatment, removed from immunotherapy treatment, and / or have their treatment plan adjusted in some other way, and selected for immunotherapy clinical trials, etc., based on the predictions and / or other outputs of one or more AI models. The models used for prediction can evolve as data continues to be collected from one or more patients, and the associated predictions can likewise change based on the data collected. The model and / or associated predictions can be tailored, for example, to an individual and / or deployed for a group / type of patient, etc., or for a group or individual ICI agent.

[0034] In some examples, data values are normalized to the upper limit of the “normal” range (e.g., for blood tests, liver tests, etc.) so that values from different sources can be compared on the same scale. The data values and associated normalization / other processing can be specific to a laboratory, patient, patient type (e.g., male / female, etc.), etc. For example, each laboratory measurement value can have a specific normal range used to evaluate that value across multiple patients.

[0035] With time series data, a value depends on previous and / or other values such that the data values have a relationship rather than being independent. This dependency can identify patients within the data that changes over time and is identified and taken into account. For example, if the data values in a time series of patient blood tests exceed twice the normal limit value on the first occasion, decrease to within the normal limit value range on the second occasion, and exceed twice the normal limit value again on the third occasion, this pattern can be identified as significant (e.g., worthy of further analysis). In data processing, for example, this pattern can be flagged or labeled appropriately. As such, clinical data from patient records can be used over time to identify and form features, anomalies, other patterns, etc. The data is conflated into a common model for comparison. A model trained and tested with such data can be robust to outliers, scaling, etc. Features can be created for better (e.g., more efficient, more accurate, more robust, etc.) modeling such as pneumonia modeling, colitis modeling, hepatitis modeling, etc.

[0036] Therefore, data processing can be used to create features that can be used to develop a model that can be deployed to predict outcomes regarding patients. For example, a data processing pipeline can create hundreds of thousands of features (e.g., frequency of pneumonia, frequency of ICD-10 codes, frequency of C34 codes, etc.).

[0037] For example, data values can include ICD-10 codes for a given patient over a one-year period. In some examples, the codes can span multiple years (e.g., 10 years, etc.) and be reconciled for processing. The ICD-10 codes are processed to identify codes related to the lungs or respiratory function, and such codes can then be used to calculate the relative frequency of the patient's lung diseases. As another example, a patient history can be analyzed to determine the relative frequency of C34 codes in the patient history that indicate lung cancer. The smoking status can be, for example, a binary flag set or unset from a data processing pipeline. In some examples, the codes can be converted between code systems (e.g., ICD-9, ICD-10 (C34, C78, etc.)). The codes can be reverse engineered, for example, even without all the keys.

[0038] By using the codes and other data, multiple (e.g., 5, 6, 10, etc.) features are created and can be used in a modeling framework to predict the onset of pneumonia in a patient. The model is constructed in a stepwise, forward manner. Labels for the pneumonia model are not inherently included in the dataset, and thus, for example, ground truths for model training are created based on the judgment of an expert who identifies the labels from the patient history. A codebook and quality control can be used, for example, to label correctly.

[0039] In some examples, the historical data received from a patient is asynchronous. The system and method then align the patient (and / or among patients) data and enable aggregation and analysis of data regarding a common baseline or benchmark. In some examples, impact points, or other reference points may be selected / decided upon, and the time series / timeline of patient data is aligned around that decided or selected point (e.g., an event occurring to the patient such as a physical examination, an injury, symptom onset, a test, a birthday, an anniversary, etc.). For example, the date of the first chemotherapy, ICI therapy, first symptom / reaction (e.g., in pulmonary function, etc.) can be used for aligning patient data.

[0040] The processed data can then be used, for example, to predict static labels within a predefined or otherwise determined time frame. The model can be trained, validated, and deployed for hepatitis, pneumonia, drug efficacy, etc. As such, data from an EHR, an EMR, a laboratory system, and / or other data sources can be preprocessed and provided to the model to generate predictions, which can be post-processed and output to a user and / or other systems for warnings, follow-up, treatment protocols, etc. In some examples, the predicted values are routed to another system (e.g., a scheduling, laboratory, etc. system) for further processing.

[0041] One or more AI models can be used to facilitate processing, correlating, and predicting based on available patient health data such as blood test results, liver test results, other test results, physiological data of other patients, etc. The models can include high recall - low precision models, low recall - high precision models, harmonic mean maximization (convergence) models, etc. Boosted decision tree models or variants such as Random Forest (RF), Gradient Boosting (GB) can be used. For example, the majority vote class undersampling, random forest model can be used to maximize recall at a relatively low precision. However, in hepatitis, prevention is inexpensive and easy, and thus false negatives can be tolerated. As such, a gradient boosting model has been developed, by which the F1 score can be maximized without applying resampling.

[0042] Machine learning techniques, whether deep learning networks or other empirical / observational learning systems, can be used to characterize, otherwise interpret, extrapolate, conclude, and / or complete medical data obtained from patients. Deep learning is a subset of machine learning that uses a series of algorithms to model high - level abstractions within data using deep graphs with multiple processing layers including linear and non - linear transformations. Many machine learning systems seed initial features and / or network weights that should be modified through the learning and updating of the machine learning network, while deep learning networks train themselves to identify "good" (e.g., useful, etc.) features for analysis. By using a multi - layer architecture, machines employing deep learning techniques can process raw data better than machines using traditional machine learning techniques. Examining data for groups of highly correlated values or unique themes is facilitated by using different layers of evaluation or abstraction.

[0043] Throughout this specification and the claims, the following terms have the meanings explicitly associated with them in this specification unless the context clearly indicates otherwise. The term "deep learning" is a machine learning technique that utilizes multiple data processing layers to recognize various structures within a dataset and classify the dataset with high accuracy. A deep learning network (DLN), also referred to as a deep neural network (DNN), can be a training network (e.g., a training network model or device) that learns patterns based on multiple inputs and outputs. A deep learning network / deep neural network can be a deployed network (e.g., a deployed network model or device) that is generated from a training network and provides an output in response to an input.

[0044] The term "supervised learning" is a machine learning training method in which a machine is provided with data that has already been classified by a human source. The term "unsupervised learning" is a machine learning training method in which a machine is not given data that has already been classified, and the machine is made useful for anomaly detection (e.g., random forest, gradient boosting, etc.). The term "semi-supervised learning" is a machine learning training method in which a machine is provided with a small amount of classified data from a human source compared to a large amount of unclassified data available to the machine.

[0045] The term "convolutional neural network" or "CNN" is a biologically inspired network of interconnected data used in deep learning for the detection, segmentation, and recognition of appropriate objects and regions within a dataset. A CNN evaluates raw data in the form of multiple arrays, divides the data into a series of stages, and examines the data for learned features. Hepatitis and / or toxicity, for example, can be predicted using a CNN.

[0046] The term "recurrent neural network" or "RNN" relates to a network in which the connections between nodes form a directed or undirected graph along a time series. Hepatitis and / or toxicity can be predicted, for example, using an RNN.

[0047] The term "transfer learning" is a process in which a machine stores information used when solving one problem, whether appropriately or inappropriately, and then uses that information to solve another problem with the same or similar nature as the first problem. Transfer learning can also be referred to as "inductive learning". Transfer learning can utilize data from a previous task, for example.

[0048] The term "active learning" is a machine learning process in which a machine selects a set of examples for receiving training data, rather than passively receiving examples selected by an external entity. For example, when a machine learns, the machine can be enabled to select examples that it determines will be most useful for learning, rather than relying solely on an external human expert or external system to identify and provide examples.

[0049] The term "computer-aided detection" or "computer-aided diagnosis" refers to a computer that analyzes medical data to suggest possible diagnoses.

[0050] Deep learning is a class of machine learning techniques that employ a representation learning method that enables a machine to be given raw data and determine the representations necessary for data classification. Deep learning uses the backpropagation algorithm, which is used to change the internal parameters (e.g., node weights) of a deep learning machine, to identify the structure within a dataset. A deep learning machine can utilize various multi-layer architectures and algorithms. Machine learning involves, for example, the identification of features to be used when training a network, while deep learning processes raw data to identify features of interest without external identification.

[0051] Deep learning in a neural network environment involves a large number of interconnected nodes called neurons. Input neurons activated from an external source activate other neurons based on their connections to other neurons governed by machine parameters. A neural network exhibits a specific pattern of behavior based on each parameter. Learning refines the machine parameters and, by extension, the connections between neurons within the network, such that the neural network exhibits the desired pattern of behavior.

[0052] Various artificial intelligence networks can be deployed to process input data. For example, deep learning utilizing a convolutional neural network segments data using convolutional filters and identifies learned observable features within the data. Each filter or layer of the CNN architecture transforms the input data to enhance the selectivity and invariance of the data. This abstraction of the data enables the machine to focus on the features within the data it is attempting to classify and ignore irrelevant background information.

[0053] Deep learning operates based on the understanding that many datasets contain high-level features that include low-level features. For example, while examining an image, it is often more efficient to look for the edges that form the motifs that form the parts that form the object being searched for rather than searching for the object itself. These hierarchies of features can be found within many different forms of data such as audio and text.

[0054] The learned observable features include the objects and quantifiable regularities learned by the machine during supervised learning. A machine provided with a large set of appropriately classified data is well-equipped to distinguish and extract the features relevant to the successful classification of new data.

[0055] Deep learning machines that utilize transfer learning can appropriately connect the features of data to specific classifications verified by human experts. Conversely, the same machine can update the parameters for classification when notified of an incorrect classification by a human expert. Settings and / or other configuration information can be derived, for example, by the learned use of the settings and / or other configuration information, and when the system is used more (e.g., by repeated and / or multiple users), a number of variations and / or other possibilities for the settings and / or other configuration information can be reduced for a given situation.

[0056] An exemplary deep learning neural network can be trained, for example, with a set of expert classification data. This set of data constructs the initial parameters for the neural network, which becomes the supervised learning stage. In the supervised learning stage, the neural network can be tested to determine whether the desired behavior has been achieved thereby.

[0057] After the desired neural network behavior has been achieved (e.g., the machine has been trained to operate according to a specified threshold, etc.), the machine can be deployed for use (e.g., testing the machine with new / updated data, etc.). During operation, the neural network classification can be verified or denied (e.g., by an expert user, an expert system, a reference database, etc.) to continuously improve the neural network behavior. An exemplary neural network is then placed in a state of transfer learning because the parameters for the classification that determine the neural network behavior are updated based on ongoing interactions. In some examples, the neural network can provide direct feedback to another process. In some examples, the neural network is buffered (e.g., via the cloud, etc.) and outputs verified data before being provided to another process.

[0058] Deep learning machines can utilize transfer learning when interacting with physicians and addressing small datasets available for supervised training. These deep learning machines can improve computer-aided diagnosis over time through training and transfer learning. However, if the dataset is larger, more accurate and robust deployed deep neural network models can be obtained that can be applied to convert heterogeneous medical data into actionable outcomes (such as system configuration / settings, computer-aided diagnosis results, image correction, etc.).

[0059] One or more such models / machines can be developed and / or deployed with respect to prepared data and / or curated data. For example, features can be extracted from EHR / EMR data tables (such as liver biomarkers, blood / plasma concentrations, etc.). Curated data extracts structured data from unstructured sources (such as the diagnosis date from medical notes, etc.). The extracted features form the basis for labeling. Time series measurements are extracted and aggregation generates statistical descriptors for the time series data. Tables of such data are used to train the model to make predictions. In some examples, the data can be resampled if necessary, desirable, or as determined. The model is validated using robust cross-validation such as leave-one-out cross-validation.

[0060] For example, the selection, input, or target can determine whether to maximize the F1 score or recall rate according to a given model. When the F1 score is maximized by the model, data resampling is not performed. When the recall rate is maximized by the model, data resampling can be performed. This decision can be driven, for example, by the model's deployment environment. When the model is to be deployed in a system that facilitates clinical trials, the F1 score is maximized because the pharmaceutical company wants the patients with the highest likelihood of showing a good response to a given drug. A high recall rate is more important in a clinical treatment setup where the system wants to rule out as much toxicity as possible. For example, in a treatment setup for hepatitis, a low model fit rate is tolerated due to the nature of the treatment for hepatitis being inexpensive.

[0061] Related systems and methods can be used to evaluate the probability or reliability of predictions generated by a model. The predicted values and related reliability levels or other reliabilities can be output, for example, to another processing system. Possible responses to immunotherapy treatment can be modeled based on available patient data collected in clinical practice.

[0062] FIG. 1 illustrates an exemplary model generation apparatus 100 that includes an exemplary input processor circuit 110, an exemplary model trainer circuit 120, an exemplary model comparator circuit 130, an exemplary model memory circuit 140, and an exemplary model deployer circuit 150. As shown in the example of FIG. 1, the exemplary input processor circuit 110 processes input data pulled from records to form a set of candidate features. The exemplary model trainer circuit 120 uses the set of candidate features to train at least a first model and a second model. The exemplary model comparator circuit 130 tests at least the first model and the second model and compares the performance of the first model with the performance of the second model. Based on this comparison result, the exemplary model comparator circuit 130 selects at least one of the first model or the second model. The exemplary model memory circuit 140 provides a memory circuit for storing the selected first and / or second models. The exemplary model deployer circuit 150 deploys the selected first model and / or second model to predict the likelihood of toxicity, such as pneumonia, hepatitis, colitis, etc., resulting from immunotherapy, according to a treatment plan for a patient.

[0063] In operation, the exemplary input processor circuit 110 processes input data related to one or more patients, such as laboratory results, diagnostic codes, claim codes, etc. The input may be from one or more external systems 160 such as an EHR, an EMR. The exemplary input processor circuit 110 can, for example, extract and organize the input in a time series for each patient. In some examples, the input processor circuit 110 aligns the input data with respect to an anchor point to organize the input data in a time series. In some examples, the input processor circuit 110 generates labels for the input data to form a set of candidate features.

[0064] In some examples, the input processor circuit 110 performs feature engineering on a set of candidate features and / or underlying data. For example, the input processor circuit performs feature engineering on a set of candidate features by, for example, performing normalization, transformation, and / or extraction from the set of candidate features. In some examples, the input processor circuit 110 makes selections from the set of candidate features to form a set of patient features, thereby training and / or validating at least a first model and a second model based on feature engineering. In some examples, the input processor circuit 110 generates a feature matrix for training and / or validating a first model and / or a second model based on feature engineering.

[0065] In some examples, the model deployer circuit 150 deploys the selected first and / or second model as an executable tool having an interface for facilitating the collection of patient data and interaction with the selected first and / or second model. The deployed model can be used with tools that, for example, select patients for a clinical trial, initiate a course of immunotherapy treatment according to a treatment plan or protocol, and adjust the current immunotherapy treatment plan. In some examples, the deployed model can be stored and / or utilized in one or more external systems 160 such as an EHR, an EMR, a scheduling system, a clinical information system, and the like.

[0066] As such, an exemplary model generation device 100 can preprocess data from one or more external systems 160 via an exemplary input processor circuit 110. The exemplary model generation device 100 then trains and validates multiple models using a model trainer circuit 120 and a model comparator circuit 130. The exemplary model generation device 100 uses a model storage circuit 140 and a model deployer circuit 150 to post-process, store, and deploy one or more of the trained and validated models. Input data from an EHR, EMR, and / or other external system 160 is stored and deployed to be converted into multiple models that can predict, for example, treatment efficacy, toxicity, and / or other side effect risks from a specific patient's immunotherapy treatment. Data from multiple patients can be utilized through processing and modeling to generate, for example, target predictions for a specific patient.

[0067] In some examples, the model trainer circuit 120 trains and validates, for example, a boosted decision tree model, other ensemble, and / or regression-based learning models. For example, a boosted decision tree model utilizes multiple weak learning decision trees to create a strong learning AI model. The trees within the model can, for example, correct errors in other trees of the model, and the ensemble of trees within the model work together to generate a prediction output. The model can also be formed as, for example, a random forest (RF) model, a gradient boost (GB) model, etc. The data is adapted by the model trainer circuit 120 to a common model that is, for example, robust to outliers and scalable for comparison.

[0068] In some examples, the input data processor 110 normalizes the input data values to the upper limit of the normal value so that the values (e.g., blood test values, liver function test values, other laboratory test values, etc.) are on the same scale. Each laboratory value may have a certain range of values. The input data processor 110 normalizes the values for a particular laboratory type, for example, over a certain range so that the data for that laboratory is comparable among patients. The input data processor 110 can also organize the data in a time series. The time series data can be normalized by the input data processor 110 based on an anchor point (e.g., a common or reference event such as admission, symptom, birth, diagnosis date, first ICI administration date, etc.) and / or adjusted in some other way.

[0069] For example, the received patient data can be asynchronous. Selecting an anchor point / kink point / impact point enables the data to be aligned for the same, similar, and / or different patients. For example, in a group of patients with diabetes, a 10-year patient history is reviewed while excluding data that occurred after diagnosis. The diagnosis time is selected as an anchor point for looking back at the patient history data (e.g., when blood glucose values have not been obtained or when insulin has been taken). As another example, the data of the first immunotherapy can be used for aligning the patient data. As such, the data is organized, for example, so that relationships and comparisons can be identified. For example, patterns can be identified from the data.

[0070] In some examples, the time series data is formed into a plurality of features by the input data processor 110. These features can be a set of candidate features from which features are selected to train and validate the model. Feature engineering by the input data processor 110 can form a plurality of features based on, for example, codes (such as ICD-10 codes). For example, the ICD-10 codes for a patient in a given year are processed to identify codes related to the lungs and / or respiratory function (such as C34, C78, etc.), and the time series of those codes can form a function used to calculate the relative likelihood of lung disease in the patient. Similarly, a plurality of features can be formed to predict the onset of pneumonia, hepatitis, colitis, and / or other toxicities from immunotherapy treatment.

[0071] Features based on lung function, liver biomarkers, blood tests (such as blood concentration, plasma concentration, etc.) can form a basis for creating labels and / or other statistical descriptors for model training. For example, the collected data can be compared to a ground truth set of identified labels to generate labels for the data. Resampling, and further validation (such as leave-one-out cross-validation, other cross-validation, other validations, etc.) can be performed, thereby generating a robust model.

[0072] The exemplary model trainer circuit 120 uses features and / or other data to train one or more AI models, such as a toxicity prediction model (e.g., hepatitis prediction model, colitis prediction model, pneumonia prediction model, etc.), an efficacy prediction model (e.g., immunotherapy efficacy model, etc.). The models can be those with a low precision rate and a high recall rate, those with a high precision rate and a low recall rate, those with a maximized harmonic mean (F1), those with majority vote class undersampling, etc. The determination can be made by the exemplary model trainer circuit 120 (e.g., based on settings, modes, types, etc.) and / or for the exemplary model trainer circuit 120. For example, the model trainer circuit 120 trains one or more models to maximize the F1 score or to maximize the recall rate. When focusing on F1, resampling may not be necessary. When focusing on the recall rate, resampling (e.g., by a training dataset and a validation dataset, multiple training datasets and / or multiple validation datasets, etc.) may be desirable to improve the model.

[0073] For example, when constructing participation in a clinical trial, the focus is on maximizing the F1 score, thereby identifying patients with the highest likelihood of responding well to a given immunotherapy drug. However, when determining clinical treatment for a patient, the goal is to eliminate the toxicity to the patient as much as possible. As such, a model developed with a high recall rate is more important, and inexpensive treatment options for hepatitis, colitis, pneumonia, etc. allow for a low precision rate.

[0074] The model comparator circuit 130 can compare a plurality of trained models and / or evaluate them in some other way to verify the models and evaluate the likelihood of predictions generated by each model. Such an assessment / evaluation can be used to select one or more models to be deployed. For example, the models can be compared based on, e.g., the average output value, the standard deviation of the model output, and / or evaluated in some other way. One or more models can be selected for storage in the model storage circuit 140 and / or other memory circuits, deployment in tools via the model deployer circuit 150, output to one or more external systems 160, etc.

[0075] In some examples, the external system 160 can utilize one or more deployed models from the model deployer circuit 150 to drive tools for evaluating, e.g., the efficacy and / or toxicity associated with immunotherapy for a patient and / or patient population. By using one or more deployed models, a patient can be evaluated at the start of an immunotherapy treatment plan, during administration of an immunotherapy treatment plan, as a candidate for an immunotherapy clinical trial, etc. In some examples, previous efficacy and / or toxicity model predictions for a patient can be used along with updated efficacy and / or toxicity model predictions for the patient (e.g., a combination of previous and current results, using previous results as input to generate current results, etc.).

[0076] Figures 2 to 14 are exemplary process flowcharts representing computer-readable instructions that can be stored in a memory circuit and executed by a processor circuit to implement and operate the exemplary model generation device 100 of FIG. 1. The exemplary process 200 of FIG. 2 starts at block 210, where the input processor circuit 110 processes data to form an input for the model trainer circuit 120. For example, the input processor circuit 110 can align data with respect to an anchor or reference point (e.g., a date, event, item, or other marker). The input processor circuit 110 can form a time series from the data. Alternatively, or in addition, the input processor circuit 110 can, for example, label and / or perform feature engineering on the data to form an input feature set for training the model.

[0077] At block 220, the model is trained using the input. For example, multiple AI models are trained by the model trainer circuit 120 using the input feature set and / or other inputs provided by the input processor circuit 110. By identifying patterns and correlations in the input, the model trainer circuit 120 forms and weights, for example, the nodes, layers, connections, etc. of the model (e.g., an RF model, a GB model, a boosted decision tree model, etc.).

[0078] At block 230, the trained model is verified by the model trainer circuit 120. For example, additional inputs (e.g., features, resampled input data, etc.) are used to verify (e.g., test, validate, etc.) that the trained model functions as intended at threshold levels such as precision, accuracy, etc.

[0079] In block 240, the model comparator circuit 130 evaluates the verified models and selects one or more models to be deployed. For example, the model comparator circuit 130 can compare multiple trained and verified (e.g., tested, etc.) models and / or evaluate them in some other way, and evaluate the probability of predictions generated by each model and the suitability of each model for a specific purpose (e.g., prediction of immunotherapy treatment efficacy, prediction of immunotherapy toxicity, immunotherapy clinical trial selection, immunotherapy treatment plan adjustment, etc.). Such assessment / evaluation can be used by the model comparator circuit 130 to select one or more models. For example, the models can be compared based on, e.g., average output values, standard deviation of model outputs, etc., and / or evaluated in some other way. In some examples, the model comparator circuit 130 selects one model to be used / deployed. In other examples, the model comparator circuit 130 selects multiple models (e.g., toxicity prediction model and efficacy prediction model, etc.) to be used / deployed.

[0080] In block 250, one or more selected models can be stored by the model storage circuit 140. As such, the selected models can be stored for later use, deployment, etc. In some examples, unselected models can also be stored by the model storage circuit 140 because the unselected models may be suitable for other uses / deployments in response to subsequent requests.

[0081] In block 260, one or more selected models are deployed by the model deployer circuit 150 to the external system 160. The models are deployed as part of tools for immunotherapy treatment and / or clinical trial planning, thereby enabling toxicity prediction, efficacy prediction, etc.

[0082] Figure 3 illustrates an exemplary implementation of the input processing for model development (e.g., block 210 of the example of FIG. 2). In block 310, the input data for each of a plurality of patients is arranged in one or more time series (e.g., arranged in chronological order, showing evolution, progression, dependencies, and / or other patterns). In block 320, the time series data for each of a plurality of patients is aligned with respect to an anchor point or other reference event. For example, various sets of time series data can be aligned according to events such as birth, onset of illness, hospitalization, etc. By aligning the time series, the data in the time series can be evaluated, compared, and / or analyzed in some other way, for example, to be more meaningful.

[0083] In block 330, the aligned time series data is labeled. For example, target labels and associated grades can be generated for each set of data values (e.g., blood test values, liver function values, lung function values, etc.).

[0084] In block 340, feature engineering is performed on the data to prepare the data as a set of features with labels to be used to train, test, and / or validate the model in some other way. For example, feature engineering can normalize, transform, and extract the data into a set of patient features to apply to the model. In some examples, these features form a set of candidate features that are evaluated to select a particular subset of patient features for model training, validation, etc.

[0085] Figure 4 illustrates an exemplary implementation of model selection for deployment (e.g., block 240 of the example of FIG. 2). At block 410, a target or goal is examined. For example, the requirements for an AI model can be requirements such as clinical trial evaluation, immunotherapy efficacy prediction, immunotherapy-related toxicity, and / or other side effect prediction. Such requirements can prompt the selection of a model having specific characteristics such as a high F1 score, high recall rate, etc. Based on the required target or goal, a specific subset of the available models can be selected. Put another way, based on the required target or goal, a specific subset of the available models can be excluded, leaving a subset of the models available for further evaluation and selection.

[0086] At block 420, a test is performed using each remaining model. For example, a permutation test, a binomial test, and / or other comparison operations are performed using each model (e.g., evaluating a lung function model with respect to smoking, quitting smoking, other binomial tests, and / or permutation tests, etc.). At block 430, the output of each model performing the test is evaluated based on one or more criteria. For example, each model is evaluated based on an average, a standard, and / or other comparison criteria, thereby determining which model best meets one or more criteria.

[0087] In block 440, one or more models are selected based on the output and / or comparison results of other model performance. For example, a single model to be deployed can be selected, and this selection is made based on an evaluation of how the model performed and / or how it output with respect to one or more criteria. In some examples, multiple models can be selected based on one or more criteria and / or other factors. For example, multiple models can meet the criteria. As another example, the requirements can include requirements for therapeutic efficacy and even the potential for related toxicity. In response to such requirements, both an efficacy model and a toxicity model can be selected. In some examples, a single model can be trained for both toxicity and efficacy, thereby outputting combined predictions.

[0088] FIG. 5 illustrates an exemplary sequential procedure 500 for model construction and evaluation (e.g., through the development of a feature set, etc.). As shown in the example of FIG. 5, in 1, an initial dataset 510 is divided into at least two partitions. Separate data analysis is performed on a first partition 520 of the data and a second partition 525 of the data. The first data partition 520 is divided into at least two loops such as an inner loop 522 and an outer loop 524 for data analysis. In 2, a first test such as a binomial test evaluates the data values of the inner loop 522 and the outer loop 524 (e.g., reject if the data value H 0 contains "smoking", etc.) and trains the model. In 3, a separate "no touch" test dataset 525 is processed using a permutation test rather than using a separate loop. The dataset 525 is processed over multiple permutations using criteria such as, for example, a comparison of with-smoking and without-smoking, and can identify that the likelihood of smoking is higher compared to the case of no smoking. In 4, the model is evaluated and reports the mean, standard, and / or other evaluation / performance statistics with respect to the first and second data partitions 520, 525 and the related tests.

[0089] In some examples, one or more of loops 522, 524 are used to evaluate a first data partition 520 via exploratory data analysis using K-fold cross-validation (CV) that compares a first set of features M1 and a second set of features M2. Through sequential iteration, a feature F is added to the first set of features M1 if the K-fold CV performance of M1 is less than that of M2. After the features are appropriately evaluated, the resulting model performance is tested using a second data partition 525. If both loops 522, 524 are utilized, M1 and M2 can be compared based on the results of the inner loop 522 (e.g., a binary test). When M2 is selected, a permutation test is performed to evaluate whether M2 is sufficiently better than M1 on the validation data set 525. Then, for example, the model performance can be evaluated on the outer loop 524. As such, an initial set of features M1, or an extended set of features M2, can be selected for model training.

[0090] FIG. 6 illustrates an exemplary process 600 for preparing input data for model training. By using the exemplary process 600, data from heterogeneous observational databases can be transformed into a common format (e.g., a data model) and a common representation (e.g., common terms, vocabulary, coding systems, etc.) for performing systematic analysis according to a library of analysis routines. The exemplary process 600 receives an input of a concept table 605 that includes concept identifiers, concept names, concept descriptions, and the like. For example, the concepts in this example relate to liver function tests. A clinical table 615 is also input along with database extraction values for patient demographics, hospital visits, laboratory measurements, and the like. In block 610, patient liver function test values (e.g., represented by a patient blood test history) are extracted from an electronic medical record or the like, cleaned, quality checked, and compiled in a time series format for one or more patients. Such extraction and processing can also be applied to other patient information (e.g., lung function, drug efficacy, etc.).

[0091] In block 620, a label definition algorithm is executed, thereby assigning an adverse event (AE) grade to a set of blood test / liver function values (e.g., ALT, AST, TBILIRUBIN, ALKPHOS, etc.) and creating a binary target label for the AI model. Such label definitions can also be applied to other patient test values (e.g., lung function, etc.).

[0092] In block 630, feature engineering is executed. For example, blood test values and / or other values can be normalized to the upper limit of "normal" to assist in the generation, comparison, and other analysis of features. Such a "normal" range can be determined for a particular patient based on specific blood tests, other laboratory tests, etc. Feature engineering converts the normalized values into a discretized symbolic representation such as modified symbol aggregation approximation. In some examples, motifs are extracted as n-grams from a series of symbolic representations, and counts from the patient history can be used as features.

[0093] In block 640, the AI model is then trained and evaluated based on features, etc., to dynamically predict the immune-related adverse event risk from patient data. For example, the immune-related adverse event risk can be predicted by the model from a fixed-length blood test history, etc. The output can include model performance metrics 625, a dataset 635 of AE risk predictions, etc. For example, the risk prediction dataset 635 can include immune-related hepatitis adverse event risk predictions for each blood test value history. That is, for each patient, the risk of developing hepatitis AE by the next ICI treatment reservation for that patient is predicted based on the patient's blood test value history, etc. The dataset 635 can alternatively include risk predictions for pneumonia, colitis, hospitalization, etc., based on the relevant patient data used to train the model.

[0094] Figure 7 illustrates an exemplary process 700 for building an exemplary classification model. At block 710, data from input dataset 705 is split into a plurality of partitions. For example, tabular data from structured EHRs, unstructured data tables, measurement features aggregated over time (e.g., over 30 days, 60 days, 1 year, etc.), curated labels (e.g., pneumonia label, hepatitis label, colitis label, etc.), other features aggregated over time (e.g., status, smoking status, etc.) can be collected and partitioned. For example, 90% of the available data can be compiled into a first partition and the remaining 10% of the data can be compiled into a second partition. Model size input 715 can help, such as by adjusting the partitions, based on the typical, expected, or desired number of features in the model.

[0095] At block 720, a sequential forward selection procedure is executed to evaluate a plurality of models. An exemplary procedure repeatedly looks for patterns in the data to form potential predictive features. For example, the first partition can be evaluated to identify candidate features based on the relevance between the label and the features. A model is selected at each iteration and the evaluation continues.

[0096] At block 730, model performance is evaluated. For example, the model selected in the final iteration of selection procedure 720 is cross-validated, such as in an outer loop of nested cross-validation using input dataset 705.

[0097] Figure 8 provides an exemplary implementation of the sequential forward selection procedure 720 of the exemplary process 700 of Figure 7. At block 810, candidate feature F is identified in a first data partition. For example, candidate feature F can be identified based on an association between a target label and an existing / identified feature. At block 820, nested cross-validation (CV) is performed on two models

Number

Number

Number

[0098] In block 830,

Number

Number

Number

Number

Number

Number

Number

Number

Number

Number

Number

Number

Number

Number

[0099] In block 860, the model size is evaluated (for example, by comparing the current feature set with the model size input 715). If the model size has been reached, the process can proceed to block 730. However, if the model size condition is not yet met, the control returns to block 810 to select a new candidate feature in the next iteration of the exemplary process.

[0100] FIG. 9 provides an exemplary implementation of the robust performance evaluation 730 of the exemplary process 700 of FIG. 7. In block 730, the final selected model

Number

[0101] Figure 10 illustrates an exemplary process 1000 for preparing data from multiple sources for use in training machine learning and / or other AI models. For example, database extraction and / or expert curation can be used to generate data from within / from the input dataset 1005. To form the dataset 1005, data can be collected from multiple structured and / or unstructured data sources.

[0102] In block 1010, one or more input data sources 1005 are preprocessed to prepare the input data (e.g., by cleaning and extracting features from the data, labeling features, etc.). The input data and the resulting features can include smoking history, drug administration, medical conditions, radiation therapy history, laboratory measurements, anthropometric data, etc. For example, in block 1012, structured data from the input dataset 1005 (e.g., drug administration, medical conditions, laboratory measurements, clinical procedures, anthropometric data, etc.) is filtered and cleaned in a common data format (e.g., a common data model such as the OMOP data model format). In block 1014, curated data from the input dataset 1005 (e.g., related to downstream tasks, smoking history, specific condition history, specific medical history, etc.) is filtered and cleaned in a common data format.

[0103] The preprocessed dataset forms the input to block 1020, where patient vectors are generated from the filtered and cleaned aggregated dataset. For example, a model-independent set of aggregated patient vectors can be provided along with feature filtering. A model-independent non-aggregated table with a temporal data structure can be used for feature filtering and vector generation. The patient vectors and / or the non-aggregated table can form part of the output dataset 1025.

[0104] In block 1030, at the interface, the data is filtered and aggregated. For example, the data can be filtered and arranged around anchor points and / or other references for aggregation and formation into patient vectors.

[0105] FIG. 11 provides an exemplary implementation for generating patient vectors (e.g., block 1020 of exemplary process 1000). In block 1110, the processed (e.g., filtered, cleaned, etc.) data is evaluated to determine whether features in the data should be used. When a feature should not be used, in block 1120, that feature is omitted. When a feature should be used, in block 1130, the available data (sources) are evaluated to determine whether multiple sources (e.g., structured and / or curated data sources, etc.) should be used. When multiple sources should be used, in block 1140, features from multiple data sources are combined and reconciled. In block 1150, an unrestricted patient vector is formed from a downstream model-independent non-aggregated table that retains a temporal structure. The intermediate dataset output 1105 can include, for example, one or more datasets separated by input domain and retaining a temporal structure.

[0106] FIG. 12 provides an exemplary implementation of a filtering and aggregation interface (e.g., block 1030 of exemplary process 1000). At 1210, a patient is evaluated to determine whether the patient is a candidate for further evaluation and processing. Such a determination may be based at least in part on, for example, downstream task-dependent external patient eligibility criteria 1205. For example, whether the patient meets certain health criteria, age criteria, status criteria, likelihood of success, etc. When the patient is not eligible, at block 1220, the patient is spared from further processing.

[0107] When the patient is eligible, at block 1230, an anchor point is set for further evaluation. The determination of the anchor point may be based, for example, on downstream task-dependent external patient eligibility criteria 1215. The anchor point is a patient timeline event that aligns multiple patients for comparison. For example, events in the life and / or medical history of a patient, such as diagnosis, symptoms, birth, admission, initial treatment, etc., can drive anchor point selection by a framework that aligns time series data for related processing and comparison. The anchor point can be a configurable parameter that depends, for example, on downstream task 1205.

[0108] At block 1240, an aggregation period is set. For example, the framework can configure the aggregation period before a prediction point (e.g., the anchor point for a patient and / or other time points and / or data) and / or set it in some other way. For example, the aggregation period can include the aggregation of patient data 200 days prior to the prediction time.

[0109] At block 1250, an aggregation period for measurement is set. For example, the interface framework allows for the establishment of different aggregation periods for laboratory measurements, other measurements, etc., because such measurements can change more dynamically.

[0110] In block 1260, one or more restricted, aggregated patient vectors are formed as the output of the interface framework. The patient vectors can be tailored, for example, to a plurality of downstream tasks. In some examples, for a given set of inputs, the patient vectors can be dynamically generated as the downstream tasks change.

[0111] FIG. 13 illustrates an exemplary process 1300 for constructing a predictive model for efficacy related to ICI. Such a model can evaluate a plurality of efficacy surrogate endpoints to predict response to ICI treatment. The model can be created for distinct cancer indications and / or combinations thereof.

[0112] In block 1310, input data from a plurality of sources is prepared for processing. For example, as described in more detail above, data can be cleaned and features can be extracted from sources such as structured data set 1305, curated data set 1315, etc. For example, raw, automatically derived EHR data can be converted to a common format along with curated data to form a set of features.

[0113] In block 1320, ground truth prediction labels are generated. The prediction labels can include, for example, the ICI treatment period (TOT), the time until the next treatment (TNET) (e.g., after interruption of ICI), the overall survival period (OS), etc. The listed ground truth endpoints can be derived, for example, from the input data 1325 and generated on a continuous scale such as represented by the number of days elapsed from an anchor point. For example, patient timelines can be aligned based on similarities in the progress of ICI treatment. A given anchor point can be, for example, the date of the first day of ICI treatment start. The generated ground truth labels can be used as is and / or with modified granularity (e.g., elapsed weeks, months, years, etc.) to train regression models, survival analysis-based models, etc. Discretization of the ground truth can be performed for binary classification and / or multi-class classification (e.g., responder vs. non-responder, 5-year survival rate, etc.).

[0114] In block 1330, model construction is performed on the generated feature matrix, where the ground truth acts as a stand-alone module of the framework. In some examples, the efficacy endpoints can be modeled separately (e.g., individually, etc.). Modeling can be performed, for example, without hypotheses and / or using different machine learning algorithms. Alternatively, or in addition, modeling can be performed on a continuous scale using survival time analysis methods. Further, modeling can, for example, follow a multi-class classification method for all endpoints. Exemplary model construction and selection are further described above.

[0115] FIG. 14 illustrates an exemplary method 1400 for model selection and generation. In the example of FIG. 14, model generation is at least in part driven by the purpose, goal, or target of the model. For example, the goal can be to generate a model with a high (e.g., maximized) F1 harmonic score (e.g., the harmonic mean of recall and precision), such as a GB model, a model with a high (e.g., maximized) recall rate (e.g., the ratio of true positives discovered), such as an RF model, and so on. In some examples, majority vote class undersampling is combined with time series data aggregation, thereby obtaining a balanced static dataset to be supplied to the model. In some examples, the model creates probability estimates for the labels rather than just the discrete labels themselves.

[0116] In block 1410, the input is prepared, such as by extracting data from one or more EHR data tables 1405. Measurements such as liver biomarker concentrations in plasma, other blood concentration values, etc. can be extracted. Time series measurements and / or other data can be made into a single complex data structure (e.g., aggregated), for example. The aggregated data can be processed using feature engineering to describe the time series data according to, for example, mean values, standard deviations, minimum values, maximum values, etc. For example, lags and features and labels can be created. In some examples, the date 1415 of ICI treatment is generated, thereby making it easier to align and fix the time series data.

[0117] In block 1420, a target or goal for the model is evaluated. As explained above, the goal of maximizing F1 produces a trained model (such as a gradient boost model like a gradient boost decision tree) that is different from the goal of maximizing recall (such as a random forest model). Based on this decision, in block 1430, dataset resampling is performed to balance the imbalanced dataset from block 1410 for maximizing recall. For example, random majority vote class undersampling is performed on the dataset when the goal is to maximize recall. When the goal is to maximize the F1 score, resampling may be ignored.

[0118] In block 1440 and / or 1450, the model is trained. For example, in block 1440, an RF model is trained for maximizing recall, and in block 1450, a GB model is trained for maximizing the F1 score. Each model is verified, for example, by leave-one-out cross-validation where each sample is predicted individually and the rest serves as the training dataset.

[0119] Exemplary implementations are illustrated in the present application and the related appendices, but one or more of the illustrated elements, processes, and / or devices may be combined, divided, reconfigured, omitted, excluded, and / or implemented in any other way. Further, the exemplary elements may be implemented by hardware alone, or in combination with software and / or firmware by hardware. Thus, for example, any of the exemplary elements may be implemented by a field programmable logic device (FPLD) such as a processor circuit, analog circuit, digital circuit, logic circuit, programmable processor, programmable microcontroller, graphics processing unit (GPU), digital signal processor (DSP), application specific integrated circuit (ASIC), programmable logic device (PLD), and / or field programmable gate array (FPGA). Still further, the exemplary elements may include one or more elements, processes, and / or devices in addition to, or instead of, the illustrated elements, and / or may include a plurality or all of any of the illustrated elements, processes, and devices.

[0120] In some examples, a hardware logic circuit, machine-readable instructions, a hardware-implemented state machine, and / or any combination thereof can implement the systems disclosed herein and / or execute the methods disclosed herein. The machine-readable instructions may be one or more executable programs or a portion of an executable program executed by a processor circuit. The program may be embodied as software stored on one or more non-transitory computer-readable storage media such as a compact disc (CD), floppy disk, hard disk drive (HDD), solid state drive (SSD), digital versatile disc (DVD), Blu-ray disc, volatile memory (e.g., any type of random access memory (RAM), etc.), or non-volatile memory (e.g., electrically erasable programmable read-only memory (EEPROM), FLASH memory, HDD, SSD, etc.) associated with a processor circuit disposed on one or more hardware devices, but the entire program and / or a portion thereof may alternatively be executed by one or more hardware devices other than the processor circuit and / or embodied as firmware or dedicated hardware. The machine-readable instructions may be distributed among multiple hardware devices and / or executed by two or more hardware devices (e.g., a server and a client hardware device). For example, the client hardware device may be implemented by an endpoint client hardware device (e.g., a hardware device associated with a user) or an intermediate client hardware device (e.g., a radio access network (RAN) gateway that can facilitate communication between a server and an endpoint client hardware device). Similarly, the non-transitory computer-readable storage media may include one or more media disposed on one or more hardware devices. Further, the order of execution may be changed and / or some of the blocks described may be changed, eliminated, or combined.In addition to, or alternatively, any or all code blocks may be implemented by one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.) structured to perform the corresponding operations without executing software or firmware. The processor circuits may be distributed across different network locations and / or may be locally disposed in one or more hardware devices (e.g., a single-core processor (e.g., a single-core central processing unit (CPU)), a multi-core processor within a single machine (e.g., a multi-core CPU, XPU, etc.), multiple processors distributed across multiple servers in a server rack, multiple processors distributed across one or more server racks, a CPU and / or FPGA disposed in the same package (e.g., the same integrated circuit (IC) package, or two or more separate housings, etc.)).

[0121] The machine-readable instructions described herein may be stored in one or more of a compressed form, an encrypted form, a fragmented form, a compiled form, an executable form, a packaged form, and the like. The machine-readable instructions described herein may be stored as data or data structures (e.g., portions of instructions, code, representations of code, etc.) that can be used to create, manufacture, and / or generate machine-executable instructions. For example, the machine-readable instructions may be fragmented and stored on one or more storage device devices and / or computing devices (e.g., servers) located at the same or different locations within a network or collection of networks (e.g., within a cloud, within an edge device, etc.). The machine-readable instructions may require one or more of installation, modification, adaptation, update, combination, supplementation, configuration, decryption, decompression, decompression, distribution, reassignment, compilation, etc. to make the instructions directly readable, interpretable, and / or executable by a computing device and / or other machine. For example, the machine-readable instructions may be stored in multiple parts, which are individually compressed, encrypted, and / or stored on separate computing devices, and these parts, when decrypted, decompressed, and / or combined, form a set of machine-executable instructions that implement one or more operations that together form a program as described herein.

[0122] In another example, the machine-readable instructions can be stored in a state readable by a processor circuit, but in order to execute the machine-readable instructions on a particular computing device or other device, it may be necessary to add libraries (e.g., Dynamic Link Libraries (DLLs)), software development kits (SDKs), application programming interfaces (APIs), etc. In another example, the machine-readable instructions may need to be configured (e.g., settings are stored, data is input, network addresses are recorded, etc.) before all or part of the machine-readable instructions and / or the corresponding program can be executed. Thus, a machine-readable medium, as used herein, can include machine-readable instructions and / or programs regardless of the particular form or state of the machine-readable instructions and / or programs when stored or at any other time when stopped or in communication.

[0123] The machine-readable instructions described herein can be expressed in any past, current, or future instruction language, scripting language, programming language, etc. For example, the machine-readable instructions can be expressed using any of the languages such as C, C++, Java, C#, Perl, Python, JavaScript, Hypertext Markup Language (HTML), Structured Query Language (SQL), Swift, etc.

[0124] As described above, the exemplary operations disclosed herein may be implemented using executable instructions (e.g., computer and / or machine-readable instructions) stored on one or more non-transitory computer media and / or machine-readable media such as optical storage device devices, magnetic storage device devices, HDDs, flash memories, read-only memories (ROMs), CDs, DVDs, caches, any type of RAM, registers, and / or any other storage device device or storage disk where information is stored for any duration (e.g., over a long period of time, permanently, for short instances, for temporary buffering, and / or for caching of information). As used herein, the terms non-transitory computer-readable media, non-transitory computer-readable storage media, non-transitory machine-readable media, and non-transitory machine-readable storage media are explicitly defined to include any type of computer-readable storage device device and / or storage disk and to exclude propagated signals and to exclude transmission media. As used herein, the terms "computer-readable storage device device" and "machine-readable storage device device" are defined to include any physical (mechanical and / or electrical) structure for storing information and to exclude propagated signals and to exclude transmission media. Examples of computer-readable storage device devices and machine-readable storage device devices include any type of random access memory, any type of read-only memory, solid state memory, flash memory, optical disk, magnetic disk, disk drive, and / or redundant array of independent disks (RAID) systems. As used herein, the term "device" refers to a physical structure such as a mechanical and / or electrical apparatus, hardware, and / or circuitry that may or may not be made up of and / or manufactured to execute computer-readable instructions, machine-readable instructions, etc.

[0125] As used herein, "comprising" and "including" (and all forms and tenses thereof) are used as non-limiting terms. Thus, whenever a claim adopts any form of "comprising" or "including" (e.g., includes, comprises, including, comprising, having, etc.) as a preamble or within the recitation of any type of claim, it should be understood that additional elements, terms, etc. may exist without departing from the scope of the corresponding claim or recitation. In this specification, when the phrase "at least" is used as a transitional term, for example, in the preamble of a claim, it is open-ended in the same way as the phrases "comprising" and "including" are open-ended. The term "and / or" when used in forms such as A, B, and / or C, for example, refers to any combination or subset of A, B, C, such as (1) A alone, (2) B alone, (3) C alone, (4) a combination of A and B, (5) a combination of A and C, (6) a combination of B and C, or (7) a combination of A, B, and C. As used in the context of describing a structure, component, item, object, and / or thing herein, the phrase "at least one of A and B" is intended to refer to an implementation form that includes (1) at least one A, (2) at least one B, or (3) either at least one A and at least one B. Similarly, as used in the context of describing a structure, component, item, object, and / or thing herein, the phrase "at least one of A or B" is intended to refer to an implementation form that includes (1) at least one A, (2) at least one B, or (3) either at least one A and at least one B. As used in the context of describing the implementation or execution of a process, instruction, action, activity, and / or step herein, the phrase "at least one of A and B" is intended to refer to an implementation form that includes (1) at least one A, (2) at least one B, or (3) either at least one A and at least one B.Similarly, as used in the context of describing the implementation or execution of a process, instruction, action, activity, and / or step in this specification, the phrase "at least one of A or B" is intended to refer to an implementation form that includes (1) at least one A, (2) at least one B, or (3) either at least one A and at least one B.

[0126] As used in this specification, singular references (e.g., "a", "an" in the English original text, "the first", "the second" in the translated text, etc.) do not exclude a plurality. One (an "a" or "an" in the English original text, which may not be used in the Japanese text) object, as used in this specification, refers to one or more of that object. The phrases "one" (an "a" or "an" in the English original text, which may not be used in the Japanese text), "one or more", and "at least one" are used interchangeably in this specification. Further, although multiple means, elements, or method acts are listed individually, they may be implemented, for example, by the same entity or object. In addition, individual features may be included in different examples or claims, but these may, in some cases, be combined, and inclusion in different examples or claims does not mean that the combination of features is not feasible and / or advantageous.

[0127] FIG. 15 is a block diagram of an exemplary processor platform 1500 structured to execute and / or instantiate the machine-readable instructions and / or operations disclosed and described herein. The processor platform 1500 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a mobile phone, a smartphone, a tablet such as an iPad®), a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a game console, a personal video recorder, a set-top box, a headset (e.g., an augmented reality (AR) headset, a virtual reality (VR) headset, etc.) or other wearable device, or any other type of computing device.

[0128] The illustrated example of the processor platform 1500 includes processor circuitry 1512. The illustrated example of the processor circuitry 1512 is hardware. For example, the processor circuitry 1512 can be implemented by one or more integrated circuits, logic circuits, FPGAs, microprocessors, CPUs, GPUs, DSPs, and / or microcontrollers from any desired family or manufacturer. The processor circuitry 1512 can be implemented by one or more semiconductor-based (e.g., silicon-based) devices.

[0129] The processor circuit 1512 of the illustrated example includes a local memory 1513 (e.g., cache, register, etc.). The processor circuit 1512 of the illustrated example communicates with a main memory including a volatile memory 1514 and a non-volatile memory 1516 via a bus 1518. The volatile memory 1514 can be implemented by a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), a Rambus (registered trademark) dynamic random access memory (RDRAM (registered trademark)), and / or any other type of RAM device. The non-volatile memory 1516 can be implemented by a flash memory and / or any other desired type of memory device. Access to the main memories 1514, 1516 of the illustrated example is controlled by a memory controller 1517.

[0130] The processor platform 1500 of the illustrated example also includes an interface circuit 1520. The interface circuit 1520 can be implemented in hardware according to any type of interface standard, such as an Ethernet interface, a Universal Serial Bus (USB) interface, a Bluetooth (registered trademark) interface, a Near Field Communication (NFC) interface, a Peripheral Component Interconnect (PCI) interface, and / or a Peripheral Component Interconnect Express (PCIe) interface.

[0131] In the illustrated example, one or more input devices 1522 are connected to the interface circuit 1520. The input device 1522 enables a user to input data and / or commands to the processor circuit 1512. The input device 1522 can be implemented by, for example, an audio sensor, a microphone, a camera (still image or video), a keyboard, a button, a mouse, a touch screen, a track pad, a track ball, an isopoint device, and / or a voice recognition system.

[0132] One or more output devices 1524 are also connected to the exemplary interface circuit 1520 illustrated. The output device 1524 can be implemented, for example, by a display device (e.g., a light-emitting diode (LED), an organic light-emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube (CRT) display, an in-plane switching (IPS) display, a touch screen, etc.), a haptic output device, a printer, and / or a speaker. Thus, the exemplary interface circuit 1520 illustrated typically includes a graphics processor circuit such as a graphics driver card, a graphics driver chip, and / or a GPU.

[0133] The exemplary interface circuit 1520 illustrated also includes a communication device such as a network interface for facilitating data exchange with an external machine (e.g., any type of computing device) via a transmitter, a receiver, a transceiver, a modem, a residential gateway, a wireless access point, and / or a network 1526. The communication can be, for example, communication via an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-site wireless system, a cellular telephone system, an optical connection, etc.

[0134] The exemplary processor platform 1500 illustrated also includes one or more mass storage device devices 1528 for storing software and / or data. Examples of such mass storage device devices 1528 include magnetic storage device devices, optical storage device devices, floppy disk drives, HDDs, CDs, Blu-ray disk drives, redundant arrays of independent disks (RAID) systems, solid state storage device devices such as flash memory devices and / or SSDs, and DVD drives.

[0135] The machine-readable instructions 1532 can be stored in a mass storage device 1528, volatile memory 1514, non-volatile memory 1516, and / or a removable non-transitory computer-readable storage medium such as a CD or DVD.

[0136] FIG. 16 is a block diagram of an exemplary implementation of the processor circuit 1512 of FIG. 15. In this example, the processor circuit 1512 of FIG. 15 is implemented by a microprocessor 1600. For example, the microprocessor 1600 may be a general-purpose microprocessor (e.g., a general-purpose microprocessor circuit). The microprocessor 1600 executes some or all of the machine-readable instructions to effectively instantiate the circuits described herein as logic circuits and perform operations corresponding to those machine-readable instructions. In some such examples, the circuits are instantiated by the hardware circuits of the microprocessor 1600 in combination with the instructions. For example, the microprocessor 1600 may be implemented by a multi-core hardware circuit such as a CPU, DSP, GPU, XPU, etc. It may include any number of exemplary cores 1602 (e.g., 1 core), but the microprocessor 1600 in this example is a multi-core semiconductor device including N cores. The cores 1602 of the microprocessor 1600 may operate independently or cooperate to execute machine-readable instructions. For example, a firmware program, an embedded software program, or machine code corresponding to a software program may be executed by one of the cores 1602 or by multiple ones of the cores 1602 at the same time or at different times. In some examples, a firmware program, an embedded software program, or machine code corresponding to a software program is divided into threads and executed in parallel by two or more of the cores 1602. The software program may correspond to some or all of the machine-readable instructions and / or operations disclosed herein.

[0137] Core 1602 may communicate via a first exemplary bus 1604. In some examples, the first bus 1604 may be implemented by a communication bus that achieves communication associated with one or more of the cores 1602. For example, the first bus 1604 may be implemented by at least one of an Inter-Integrated Circuit (I2C) bus, a Serial Peripheral Interface (SPI) bus, a PCI bus, or a PCIe bus. In addition or alternatively, the first bus 1604 may be implemented by any other type of computing or electrical bus. Core 1602 may obtain data, instructions, and / or signals from one or more external devices via an exemplary interface circuit 1606. Core 1602 may output data, instructions, and / or signals to one or more external devices via interface circuit 1606. The core 1602 of this example includes an exemplary local memory 1620 (e.g., a level 1 (L1) cache that may be split into an L1 data cache and an L1 instruction cache), but the microprocessor 1600 also includes an exemplary shared memory 1610 (e.g., a level 2 (L2) cache) that may be shared by the cores for fast access to data and / or instructions. Data and / or instructions may be transferred (e.g., shared) by writing to and / or reading from the shared memory 1610. Each local memory 1620 and shared memory 1610 of the core 1602 may be part of a hierarchy of storage device devices that includes multiple levels of cache memory and main memory (e.g., main memories 1514, 1516 of FIG. 15). Typically, a higher level of memory in the hierarchy exhibits a shorter access time and has a smaller storage capacity than a lower level of memory. Changes in the various levels of the cache hierarchy are managed (e.g., coordinated) by a cache coherence policy.

[0138] Each core 1602 may be referred to as a CPU, DSP, GPU, etc., or any other type of hardware circuit. Each core 1602 includes a control unit circuit 1614, an arithmetic logic (AL) circuit (sometimes referred to as an ALU) 1616, a plurality of registers 1618, a local memory 1620, and a second exemplary bus 1622. Other structures may exist. For example, each core 1602 may include a vector arithmetic unit circuit, a single instruction multiple data (SIMD) unit circuit, a load / store unit (LSU) circuit, a branch / jump unit circuit, a floating point unit (FPU) circuit, etc. The control unit circuit 1614 includes a semiconductor-based circuit structured to control (e.g., coordinate) data movement within the corresponding core 1602. The AL circuit 1616 includes a semiconductor-based circuit structured to perform one or more mathematical and / or logical operations on the data within the corresponding core 1602. Some examples of the AL circuit 1616 perform integer-based operations. In other examples, the AL circuit 1616 also performs floating point operations. In still other examples, the AL circuit 1616 may include a first AL circuit that performs integer-based operations and a second AL circuit that performs floating point operations. In some examples, the AL circuit 1616 may be referred to as an arithmetic logic unit (ALU). The register 1618 is a semiconductor-based structure that stores data and / or instructions such as the result of one or more of the operations performed by the AL circuit 1616 of the corresponding core 1602. For example, the register 1618 may include vector registers, SIMD registers, general-purpose registers, flag registers, segment registers, machine-specific registers, instruction pointer registers, control registers, debug registers, memory management registers, machine check registers, etc. The register 1618 may be arranged and configured within a bank as shown in FIG. 16. Alternatively, the register 1618 may be organized in any other arrangement, form, or structure including a distributed arrangement across the entire core 1602 to reduce access time. The second bus 1622 may be implemented by at least one of an I2C bus, an SPI bus, a PCI bus, or a PCIe bus.

[0139] Each core 1602 and / or more generally, the microprocessor 1600 may include structures that add to and / or are alternative to those shown and described above. For example, one or more clock circuits, one or more power supplies, one or more power gates, one or more cache home agents (CHA), one or more converged / common mesh stop (CMS), one or more shifters (e.g., barrel shifters) and / or other circuits may be present. The microprocessor 1600 is a semiconductor device fabricated to include a number of transistors interconnected to implement the structures described above in one or more integrated circuits (ICs) housed in one or more packages. The processor circuit may include and / or cooperate with one or more accelerators. In some examples, the accelerator is implemented by logic circuitry to perform some tasks faster and / or more efficiently compared to being executed by a general-purpose processor. Examples of accelerators include ASICs and FPGAs as described herein. A GPU or other programmable device may also be an accelerator. The accelerator may be mounted on the processor circuit, on the same chip package as the processor circuit, and / or on one or more packages separate from the processor circuit.

[0140] FIG. 17 is a block diagram of an exemplary alternative implementation of the processor circuit 1512 of FIG. 15. In this example, the processor circuit 1512 is implemented by an FPGA circuit 1700. For example, the FPGA circuit 1700 can be implemented by an FPGA. The FPGA circuit 1700 can be used, for example, to perform operations that can be performed in some other form by the exemplary microprocessor 1600 of FIG. 16 that executes corresponding machine-readable instructions. However, after configured, the FPGA circuit 1700 instantiates the machine-readable instructions in hardware and, thus, can perform computations faster in many cases compared to what can be performed by a general-purpose microprocessor executing the corresponding software.

[0141] More specifically, in contrast to the microprocessor 1600 of FIG. 16 described above (which is a general-purpose device that can be programmed to execute some or all of the machine-readable instructions disclosed herein, but whose interconnections and logic circuitry are fixed once fabricated), the FPGA circuit 1700 of the example of FIG. 17 includes interconnections and logic circuitry that can be configured and / or interconnected in different ways after fabrication, for example, to instantiate some or all of the machine-readable instructions disclosed herein. In particular, the FPGA circuit 1700 can be thought of as an array of logic gates, interconnections, and switches. The switches are programmed to change the way the logic gates are interconnected by the interconnections, thereby effectively forming one or more dedicated logic circuits (unless the FPGA circuit 1700 is reprogrammed). The configured logic circuits enable the logic gates to cooperate in different ways to perform different operations on the data received by the input circuits. Those operations can correspond to some or all of the software disclosed herein. As such, the FPGA circuit 1700 can be structured to effectively instantiate some or all of the machine-readable instructions as dedicated logic circuits and execute operations corresponding to those software instructions in a dedicated manner similar to an ASIC. Thus, the FPGA circuit 1700 can execute operations corresponding to some or all of the machine-readable instructions at a higher speed compared to a general-purpose microprocessor executing the same.

[0142] In the example of FIG. 17, the FPGA circuit 1700 is structured to be programmed (and / or reprogrammed one or more times) by an end user using a hardware description language (HDL) such as Verilog. The FPGA circuit 1700 of FIG. 17 includes an exemplary input / output (I / O) circuit 1702 that acquires and / or outputs data to and / or from an exemplary configuration circuit 1704 and / or external hardware 1706. For example, the configuration circuit 1704 may be implemented by an interface circuit that can acquire machine-readable instructions for configuring the FPGA circuit 1700 or a portion thereof. In some such examples, the configuration circuit 1704 can acquire the machine-readable instructions from a user, a machine (e.g., a hardware circuit that can implement an artificial intelligence / machine learning (AI / ML) model for generating the instructions (e.g., a programmed or dedicated circuit)), etc. In some examples, the external hardware 1706 may be implemented by an external hardware circuit. For example, the external hardware 1706 may be implemented by the microprocessor 1600 of FIG. 16. The FPGA circuit 1700 also includes an array of exemplary logic gate circuits 1708, a plurality of exemplary configurable interconnects 1710, and an exemplary memory circuit 1712. The logic gate circuits 1708 and the configurable interconnects 1710 are configurable to instantiate one or more operations corresponding to at least some of the machine-readable instructions and / or other desired operations. The logic gate circuits 1708 shown in FIG. 17 are processed in groups or blocks. Each block includes a semiconductor-based electrical structure that can be configured as a logic circuit. In some examples, the electrical structure includes logic gates (e.g., And gates, Or gates, Nor gates, etc.) that provide basic building blocks for the logic circuit. Electrically controllable switches (e.g., transistors) are present within each of the logic gate circuits 1708 to enable the electrical structure and / or the configuration of the logic gates to form a circuit that performs the desired operations. The logic gate circuits 1708 may include other electrical structures such as lookup tables (LUTs), registers (e.g., flip-flops or latches), multiplexers, etc.

[0143] The configurable interconnect 1710 of the illustrated example can be a conductive path, trace, via, or the like that includes an electrically controllable switch (e.g., a transistor) whose state can be changed by programming (e.g., using an HDL instruction language) to activate or deactivate one or more connections between one or more of the logic gate circuits 1708 to program a desired logic circuit.

[0144] The memory circuit 1712 of the illustrated example is structured to store the results of one or more operations performed by corresponding logic gates. The memory circuit 1712 can be implemented by a register or the like. In the illustrated example, the memory circuit 1712 is distributed among the logic gate circuits 1708 to facilitate access and increase the execution speed.

[0145] The exemplary FPGA circuit 1700 of FIG. 17 also includes an exemplary dedicated arithmetic circuit 1714. In this example, the dedicated arithmetic circuit 1714 includes special-purpose circuits 1716 that can be called to implement those functions in order to avoid the need to program functions commonly used in the field. Examples of such special-purpose circuits 1716 include memory (e.g., DRAM) controller circuits, PCIe controller circuits, clock circuits, transceiver circuits, memories, and multiplier-accumulator circuits. Other types of special-purpose circuits may also exist. In some examples, the FPGA circuit 1700 may also include exemplary general-purpose programmable circuits 1718 such as an exemplary CPU 1720 and / or an exemplary DSP 1722. Other general-purpose programmable circuits 1718 may also exist as GPUs, XPU, etc. that are programmed to perform other operations in addition to, or instead of, these.

[0146] Figures 16 and 17 illustrate two exemplary implementations of the processor circuit 1512 of FIG. 15, although many other approaches are contemplated. For example, as described above, state-of-the-art FPGA circuits may include an on-board CPU, such as one or more of the exemplary CPUs 1720 of FIG. 17. Thus, the processor circuit 1512 of FIG. 15 may be implemented by combining, in addition thereto, the exemplary microprocessor 1600 of FIG. 16 and the exemplary FPGA circuit 1700 of FIG. 17. In some such hybrid examples, a first portion of the machine-readable instructions may be executed by one or more of the cores 1602 of FIG. 16, a second portion of the machine-readable instructions may be executed by the FPGA circuit 1700 of FIG. 17, and / or a third portion of the machine-readable instructions may be executed by an ASIC. Thus, it should be understood that part or all of the circuit may be instantiated at the same time or at different times. Part or all of the circuit may be instantiated, for example, in one or more threads executing concurrently and / or in series. Further, in some examples, part or all of the circuit may be implemented within one or more virtual machines and / or containers executing on a microprocessor.

[0147] In some examples, the processor circuit 1512 of FIG. 15 may be within one or more packages. For example, the microprocessor 1600 of FIG. 16 and / or the FPGA circuit 1700 of FIG. 17 may be within one or more packages. In some examples, the XPU may be implemented by the processor circuit 1512 of FIG. 15, which may be in one or more packages. For example, the XPU may be such that it houses a CPU in one package, a DSP in another package, a GPU in yet another package, and an FPGA in yet another package.

[0148] FIG. 18 illustrates a block diagram exemplifying an exemplary software distribution platform 1805 for distributing software such as the exemplary machine-readable instructions 1532 of FIG. 15 to a hardware device owned and / or operated by a third party. The exemplary software distribution platform 1805 can be implemented by any computer server, data facility, cloud service, etc. that can store software and transmit it to other computing devices. The third party may be a customer of the entity that owns and / or operates the software distribution platform 1805. For example, the entity that owns and / or operates the software distribution platform 1805 may be a developer, seller, and / or licensor of software such as the exemplary machine-readable instructions 1532 of FIG. 15. The third party can be a consumer, user, retailer, OEM, etc. that purchases and / or licenses the software for use and / or resale and / or sublicense. In the example illustrated, the software distribution platform 1805 includes one or more servers and one or more storage device devices. The storage device device stores the machine-readable instructions 1532, which may correspond to the exemplary machine-readable instructions 1532 of FIG. 15 as described above. One or more servers of the exemplary software distribution platform 1805 communicate with an exemplary network 1810, which may correspond to any one or more of the Internet and / or the exemplary networks described above. In some examples, one or more servers respond to requests to transmit software to a requesting party as part of a commercial transaction. Payment for the distribution, sale, and / or license of the software can be handled by one or more servers of the software distribution platform and / or by a third-party payment entity. The server enables a purchaser and / or licensor to download the machine-readable instructions 1532 from the software distribution platform 1805.For example, software that may correspond to the exemplary machine-readable instructions 1532 of FIG. 15 may be downloaded to the exemplary processor platform 1500, which executes the machine-readable instructions 1532 to implement the systems and methods described herein. In some examples, one or more servers of the software distribution platform 1805 periodically provide, transmit, and / or enforce updates to the software (e.g., the exemplary machine-readable instructions 1532) to ensure that improvements, patches, updates, etc. are distributed and applied to the software on the end-user device.

[0149] In some examples, rather than downloading the machine-readable instructions 1832 to the local processor platform 1800, the deployed model and / or patient data may be uploaded to be executed remotely via the cloud-based platform 1805. In some examples, the exemplary platform 1805 can host one or more models accessible by the network 1810, and the processor platform 1500 can provide inputs to the model and receive results, predictions, and / or other outputs.

[0150] From the foregoing description, it will be understood that exemplary systems, methods, apparatuses, and articles of manufacture are disclosed that enable model generation and deployment to drive processes for treatment prediction and treatment implementation. The disclosed systems, methods, apparatuses, and articles of manufacture improve the efficiency of using a computing device by enabling model generation and deployment to drive processes for treatment prediction and treatment implementation. The disclosed systems, methods, apparatuses, and articles of manufacture are accordingly directed to one or more improvements in the operation of machines such as computers or other electronic and / or mechanical devices.

[0151] Some examples provide systems and methods for training and evaluating an AI model to predict adverse risk events from fixed-length patient history data. Exemplary systems and related methods include submodules that extract a patient's blood test history from an electronic chart, clean the history, and perform data quality checks. Exemplary systems and related methods include a label definition submodule that instantiates an algorithm to assign hepatitis adverse event grades to a set of blood test values (e.g., ALT, AST, TBILIRUBIN, ALKPHOS), creating a binary target label for the AI model. Exemplary systems and methods include a feature engineering submodule that normalizes blood test values to the upper limit of normal values (e.g., specific to a patient, laboratory, and / or blood test, etc.). An exemplary feature engineering submodule is to convert the normalized values into a discretized symbolic representation such as a modified version of Symbolic Aggregate ApproXimation. An exemplary feature engineering submodule extracts motifs as n-grams from the symbol sequence and uses the counts in the recent patient history as features. Exemplary systems and methods include a submodule that trains and evaluates an AI model to dynamically predict immune-related hepatitis adverse event risks from a fixed-length blood test history.

[0152] Some examples provide systems and methods for building a classification model (e.g., a pneumonia classification model, etc.) using sequential procedures. Exemplary systems and methods include preprocessing structured EHR and unstructured data tables. The patient timeline is aligned, for example, at the time of the first ICI administration. Laboratory measurements are aggregated using statistics over a time frame prior to the first ICI (e.g., a 60-day time frame, etc.). Other features (e.g., status, smoking status, etc.) can use different time frames (e.g., a 1-year time frame, etc.).

[0153] This exemplary system and method includes finding patterns in the data to identify potential predictive features associated with the onset of ICI-related toxicities such as pneumonia. For small, noisy datasets, a large number of data points are utilized. The data is split and candidate features are identified based on the association between pneumonia labels and features in a first partition (e.g., 90% partition, 80% partition, 95% partition, etc.).

[0154] This exemplary system and method includes determining, in each iteration of the procedure, between two model versions, one with the original set of features (M1) and one augmented with the candidate features (M2). Nested cross-validation is performed on a first (e.g., 90% etc.) partition, and the results of the inner loop are used to compare M1 and M2. A binomial test is performed to evaluate whether M2 is significantly better than M1.

[0155] The exemplary system and method includes evaluating whether M2 has better performance on a second held-out partition (e.g., 10% partition, 5% partition, 20% partition, etc.) when M2 is significantly better in step 3). A permutation test is performed to estimate the probability of observing better performance by chance. This step functions as a safeguard to avoid overfitting to the first (e.g., 90% etc.) data partition. If M2 has sufficiently good performance based on step 4), M2 is selected.

[0156] The exemplary system and method includes continuing to test new candidate features until the desired model size is reached. The performance of the final model on the outer loop is evaluated. By doing so, performance estimates with less variance can be obtained, and the variability of test predictions and model instability can be evaluated. If the final model has promising performance, the model is evaluated on a sufficiently large external test set.

[0157] Some examples provide systems and methods for forming an automated framework to prepare multiple source electronic health record data for use in machine learning model training. Exemplary methods and related systems include preparing input data from multiple sources. This part of the framework mainly relates to cleaning and extracting features from multiple data sources. In this step, raw automatically derived EHR data in a data model format (such as OMOP data model format) and multiple expert curation data sources for additional features and labels are incorporated. This step can accept extensions and includes, but is not limited to, modules for preparing smoking history, medication history, medical conditions, radiation therapy history, laboratory measurements, and anthropometric data.

[0158] This exemplary method and related system include generating a combined time-dependent unrestricted intermediate output. In this step, the input data is condensed into a unified format while retaining the timestamps of individual data items for each patient. This intermediate step provides the possibility of plugging into modules that prepare time-dependent input data (not implemented) for sequence modeling algorithms and provides flexible input for the aggregation step of the framework.

[0159] Exemplary methods and related systems include aggregating features for a time-independent model. This step is a target-independent, highly configurable plug-in module for creating a time-independent aggregated input for a machine learning model. This step can accept extensions and the configurable parameters include, but are not limited to, prediction time, length of aggregation, accompanying data sources, and accompanying feature types.

[0160] Some examples provide systems and methods for constructing prediction models for efficacy surrogate endpoints related to immune checkpoint inhibitor therapy. Exemplary methods and related systems include preparing input data from multiple sources. This part of the framework involves cleaning and extracting features from multiple data sources. In this step, for example, raw automatically derived EHR data in the OMOP data model format, and multiple expert curation data sources for additional features are incorporated. Exemplary methods and related systems include generating ground truth prediction labels such as immunotherapy treatment duration (TOT), time to next treatment (TNET) after ICI discontinuation, overall survival (OS), etc. The listed ground truth endpoints are generated on a continuous scale represented by the number of days elapsed from an anchor point (patient timelines are aligned based on similarities in the course of ICI treatment). The default anchor point is the first date of ICI treatment start. The generated ground truth can be used as is, or with modified granularity (elapsed weeks, months, years, etc.) to train regression models or survival analysis-based models. Discretization of the ground truth can be performed for binary classification or multi-class classification (e.g., responders vs. non-responders, 5-year survival rate, etc.).

[0161] Exemplary methods and related systems include model construction for the feature matrix and ground truth generated as a stand-alone module of the framework. Endpoints can be modeled separately. Modeling can be performed, for example, without hypotheses, using different machine learning algorithms. The model construction and selection workflow can be used to generate sequential procedures for prediction models and model construction for immune checkpoint inhibitor-related pneumonia.

[0162] Some examples provide systems and methods for hepatitis prediction. Exemplary systems and related methods include input preparation by extracting relevant sections of blood features from a received electronic health record (EHR) data table. These are measurements of liver biomarker concentrations in plasma (such as ALT, AST, alkaline phosphatase, bilirubin, etc.) and other concentration values in the blood. After this step, time series data is combined into a single composite data structure, which is an efficient option to follow by aggregating this information into a final data table, thus preparing for the subsequent preprocessing and conversion steps. The aggregation step of feature engineering consists of describing the time series data of blood particles using the mean value, standard deviation, minimum value, and maximum value. Lag features can also be created by taking the last liver biomarker measurement available before treatment. Labels are created using definitions obtained from medical experts, which are classified as positive, for example, when the level of at least one liver biomarker exceeds three times the upper limit of the normal value within a predefined frame, and negative otherwise. The date of ICI treatment used in this scheme can be output from different workflows.

[0163] Exemplary systems and methods include resampling of the dataset. For example, the dataset obtained as a result from step 1 is unbalanced, and thus, when the goal is to maximize the recall value, the random majority vote class us is executed on the dataset. If the F1 score is the target for maximization, the resampling step is ignored.

[0164] Exemplary systems and methods include training and prediction. For example, on the final data obtained as a result from step 2, models that are RF for maximizing recall and GB for maximizing the F1 score are trained. Validation is performed using, for example, leave-one-out cross-validation, and each sample is individually predicted using the rest as the training set.

[0165] Further aspects of the disclosure are provided by the subject matter of the following clauses.

[0166] Example 1 is an apparatus comprising a memory circuit, instructions, and a processor circuit that processes input data pulled from a record to form a set of candidate features, uses the set of candidate features to train at least a first model and a second model, tests at least the first model and the second model, compares the performance of the first model and the performance of the second model, selects at least one of the first model or the second model based on this comparison, stores at least the selected one of the first model or the second model, deploys at least the selected one of the first model or the second model, and executes instructions to predict at least one of a) toxicity resulting from immunotherapy according to a treatment plan, or b) the efficacy of a treatment plan for a patient.

[0167] Example 2 includes the apparatus of any preceding clause and further includes deploying at least the selected one of the first model or the second model with a tool having an interface to facilitate the collection of patient data and the interaction of the patient data with at least the selected one of the first model or the second model.

[0168] Example 3 includes the apparatus of any preceding clause, wherein the input data includes at least one of laboratory test results, diagnostic codes, or billing codes.

[0169] Example 4 includes the apparatus of any preceding clause, wherein the toxicity includes at least one of pneumonia, colitis, or hepatitis.

[0170] Example 5 includes the apparatus of any preceding clause, wherein the efficacy of a treatment plan for a patient is measured by at least one of patient survival rate or treatment duration.

[0171] Example 6 includes the apparatus of any preceding clause, wherein the processor circuit extracts the input data and arranges it in a time series.

[0172] Example 7 includes the apparatus of any preceding item, and the processor circuit aligns the input data with respect to the anchor points and arranges the input data in time series.

[0173] Example 8 includes the apparatus of any preceding item, and the processor circuit generates labels for the input data to form a set of candidate features.

[0174] Example 9 includes the apparatus of any preceding item, and the processor circuit performs feature engineering on the set of candidate features by at least one of normalization, transformation, or extraction from the set of candidate features.

[0175] Example 10 includes the apparatus of any preceding item, and the processor circuit selects from the set of candidate features to form a set of patient features and performs at least one of training or validating at least a first model and a second model based on feature engineering.

[0176] Example 11 includes the apparatus of any preceding item, and the processor circuit generates a feature matrix and performs at least one of training or validating at least a first model and a second model based on feature engineering.

[0177] Example 12 includes the apparatus of any preceding item, and the processor circuit deploys at least a selected one of the first model or the second model as an executable tool having an interface.

[0178] Example 13 includes at least one computer-readable storage medium including instructions which, when executed by a processor circuit, cause the processor circuit to at least process input data pulled from a record to form a set of candidate features, use the set of candidate features to train at least a first model and a second model, test at least the first model and the second model, compare the performance of the first model and the performance of the second model, select at least one of the first model or the second model based on this comparison, store at least the selected one of the first model or the second model, and deploy at least the selected one of the first model or the second model to predict at least one of a) toxicity resulting from immunotherapy according to a treatment plan, or b) the efficacy of a treatment plan for a patient.

[0179] Example 14 includes at least one computer-readable storage medium of any preceding item, and the instructions, when executed, cause the processor circuit to deploy at least the selected one of the first model or the second model with a tool having an interface for facilitating the collection of patient data and the interaction with at least the selected one of the first model or the second model.

[0180] Example 15 includes at least one computer-readable storage medium of any preceding item, and the instructions, when executed, cause the processor circuit to extract input data and organize it in a time series with respect to anchor points.

[0181] Example 16 includes at least one computer-readable storage medium of any preceding item, and the instructions, when executed, cause the processor circuit to generate labels for the input data to form a set of candidate features.

[0182] Example 17 includes at least one computer-readable storage medium of any preceding item, and the instructions, when executed, cause a processor circuit to perform feature engineering on a set of candidate features by at least one of normalization, transformation, or extraction from the set of candidate features.

[0183] Example 18 is a computer-implemented method, including processing input data pulled from a record to form a set of candidate features, training at least a first model and a second model using the set of candidate features, testing at least the first model and the second model and comparing the performance of the first model and the performance of the second model, selecting at least one of the first model or the second model based on this comparison, storing at least the selected one of the first model or the second model, and deploying at least the selected one of the first model or the second model to predict at least one of a) toxicity resulting from immunotherapy according to a treatment plan, or b) the efficacy of a treatment plan for a patient.

[0184] Example 19 includes the method of any preceding item, and deploying includes deploying at least the selected one of the first model or the second model with a tool having an interface for facilitating the collection of patient data and the interaction with at least the selected one of the first model or the second model.

[0185] Example 20 includes the method of any preceding item, and further includes extracting input data and organizing it in a time series with respect to anchor points.

[0186] Example 21 includes the method of any preceding item, and further includes generating labels for the input data to form the set of candidate features.

[0187] Example 22 includes the method of any preceding item, and further includes performing feature engineering on the set of candidate feature quantities by at least one of normalization, transformation, or extraction from the set of candidate feature quantities.

[0188] Example 23 is an apparatus, including a memory circuit, instructions, and a plurality of models for predicting at least one of a) toxicity in response to immunotherapy or b) the efficacy of immunotherapy, the plurality of models being trained and verified using data from previous patients, and a processor circuit that, via an interface, receives an input of data associated with a first patient and uses at least one of the plurality of models to generate a prediction of at least one of a) toxicity that occurs when performing immunotherapy according to a treatment plan for the first patient, or b) the efficacy of the treatment plan for the first patient, and outputs a recommendation for the first patient regarding the treatment plan.

[0189] Example 24 includes the apparatus of any preceding item, and the toxicity includes at least one of pneumonia, colitis, or hepatitis.

[0190] Example 25 includes the apparatus of any preceding item, and the efficacy is defined with respect to patient survival rate.

[0191] Example 26 includes the apparatus of any preceding item, and the efficacy is measured by either progression-free survival rate or treatment duration.

[0192] Example 27 includes the apparatus of any preceding item, and the predictions of both toxicity and efficacy are generated for the first patient to enable an assessment of the risk-benefit ratio for the first patient regarding the treatment plan.

[0193] Example 28 includes the apparatus of any preceding item, and the processor circuit extracts and arranges in time series the data associated with the patient to be provided to the model.

[0194] Example 29 includes the apparatus of any preceding item, and the processor circuit aligns data associated with a patient with respect to an anchor point and arranges the patient data in a time series.

[0195] Example 30 includes the apparatus of any preceding item, and the treatment plan includes a clinical trial involving a first patient.

[0196] Example 31 includes the apparatus of any preceding item, and the treatment plan is part of the clinical care for a first patient.

[0197] Example 32 includes the apparatus of any preceding item, the input is the first input of the initial data, the prediction is the first prediction, the processor circuit processes the second input of data from the first patient for the second time, generates a second prediction using the model, and the processor circuit compares the second prediction with the first prediction to adjust the recommended output for the patient.

[0198] Example 33 includes the apparatus of any preceding item, and the first prediction is used by at least one of a plurality of models to generate the second prediction.

[0199] Example 34 includes the apparatus of any preceding item, and the processor circuit compares the second prediction, the first prediction, and the image data to adjust the recommendation output for the patient.

[0200] Example 35 includes the apparatus of any preceding item, and further includes an interface circuit connected to an electronic medical record for at least one of retrieving data associated with a first patient or storing a prediction.

[0201] Example 36 includes the apparatus of any preceding item, and the processor circuit obtains feedback regarding a recommendation for adjusting the model.

[0202] Example 37 includes at least one computer-readable storage medium including instructions that, when executed by a processor circuit, cause the processor circuit to at least receive, via an interface, an input of data associated with a first patient, use at least one of a plurality of models to generate a prediction of at least one of a) toxicity that occurs when performing immunotherapy according to a treatment plan for the first patient, or b) the efficacy of the treatment plan for the first patient, where the plurality of models predict at least one of a) toxicity responsive to immunotherapy or b) the efficacy of immunotherapy, and the plurality of models are trained and validated using data from previous patients, and output a recommendation for the first patient regarding the treatment plan.

[0203] Example 38 includes at least one computer-readable storage medium of at least one of any of the preceding items, and the instructions, when executed, cause the processor circuit to extract and organize into a time series patient data to be provided to the model.

[0204] Example 39 includes at least one computer-readable storage medium of at least one of any of the preceding items, and the instructions, when executed, cause the processor circuit to align patient data with respect to an anchor point and organize the patient data into a time series.

[0205] Example 40 includes at least one computer-readable storage medium of at least one of any of the preceding items, the input is a first input of initial data, the prediction is a first prediction, the processor circuit processes a second input of data from the first patient at a second time to generate a second prediction using the model, and the processor circuit compares the second prediction with the first prediction to adjust the recommendation output for the patient.

[0206] Example 41 involves receiving, via an interface, an input of data associated with a first patient, and by using a processor and at least one of a plurality of models to execute instructions, generating a prediction of at least one of a) the toxicity that occurs when performing immunotherapy according to a treatment plan for the first patient, or b) the efficacy of the treatment plan for the first patient, wherein the plurality of models predict at least one of a) the toxicity that responds to immunotherapy or b) the efficacy of immunotherapy, and the plurality of models are trained and validated using data from previous patients, and outputting, by using the processor to execute instructions, a recommendation for the first patient regarding the treatment plan.

[0207] Example 42 includes the method of any preceding item and further includes extracting the patient data to be provided to the model and organizing it in a time series.

[0208] Example 43 includes the method of any preceding item and further includes organizing the patient data in a time series by aligning the patient data with respect to an anchor point.

[0209] Example 44 includes the method of any preceding item, wherein the input is the first input of the initial data, the prediction is the first prediction, further including processing a second input of data from the first patient for the second time to generate a second prediction using the model, and comparing the second prediction with the first prediction to adjust the recommendation output for the patient.

[0210] Example 45 is an apparatus comprising means for processing input data pulled from a record to form a set of candidate features, means for training at least a first model and a second model using the set of candidate features, means for testing at least the first model and the second model and comparing the performance of the first model with the performance of the second model, means for selecting at least one of the first model or the second model based on this comparison, means for storing the selected at least one of the first model or the second model, and means for deploying the selected at least one of the first model or the second model to predict at least one of a) toxicity resulting from immunotherapy according to a treatment plan, or b) the efficacy of a treatment plan for a patient.

[0211] As disclosed and described herein, a processor circuit provides means for processing (including, for example, means for processing input data, means for training a model, means for selecting, means for deploying, etc.), and a memory circuit provides means for storing. As described above, various circuits can be implemented by the processor circuit and the memory circuit.

[0212] The following claims are hereby incorporated by reference into the detailed description herein by this disclosure. Although some exemplary systems, methods, apparatuses, and articles of manufacture are disclosed herein, the scope of this patent is not limited thereto. On the contrary, this patent is directed to all systems, methods, apparatuses, and articles of manufacture that properly fall within the scope of the claims of this patent.

Description of the Reference Signs

[0213] 100 Model generation device 110 Input processor circuit, input data processor 120 Model trainer circuit 130 Model comparator circuit 140 Model storage circuit 150 Model deployer circuit 160 External system 200 Process 500 Sequential procedure 510 Initial data set 520 First partition of data 522 Inner loop 524 Outer loop 525 Second partition of data 600 Process 605 Concept table 615 Clinical table 625 Model performance metrics 635 Data set for AE risk prediction 700 Process 705 Input data set 715 Model size input 720 Selection procedure 730 Robust performance evaluation 1000 Process 1005 Input data set 1025 Output data set 1105 Intermediate data set 1205 Downstream task-dependent external patient eligibility criteria 1215 Downstream task-dependent external patient eligibility criteria 1300 Process 1305 Structured data set 1315 Curated data set 1325 Input data 1400 Method 1405 EHR data table 1500 Processor platform 1512 Processor circuit 1513 Local memory 1514 Volatile memory 1516 Non-volatile memory 1514, 1516 Main memory 1517 Memory controller 1520 Interface circuit 1522 Input Device 1524 Output Device 1526 Network 1528 Mass Storage Device 1532 Machine Readable Instruction 1600 Processor Circuit 1602 Core 1604 Bus 1606 Interface Circuit 1610 Shared Memory 1614 Control Unit Circuit 1616 Arithmetic Logic (AL) Circuit 1618 Register 1620 Local Memory 1700 FPGA Circuit 1702 Input / Output (I / O) Circuit 1704 Configuration Circuit 1706 External Hardware 1708 Logic Gate Circuit 1710 Interconnection (I) 1712 Memory Circuit 1714 Dedicated Arithmetic Circuit 1716 Special Purpose Circuit 1718 General-Purpose Programmable Circuit 1720 CPU 1722 DSP 1805 Software Distribution Platform 1810 Network 1832 Machine Readable Instruction

Claims

1. An apparatus comprising: a memory circuit; instructions; a processor circuit wherein the processor circuit executes the instructions to process input data pulled from a record to form a set of candidate features; train at least a first model and a second model using the set of candidate features; assay at least the first model and the second model and compare the performance of the first model with the performance of the second model; select at least one of the first model or the second model based on the comparison; store at least the selected one of the first model or the second model; deploy at least the selected one of the first model or the second model to predict at least one of a) toxicity resulting from immunotherapy according to a treatment plan, or b) the efficacy of the treatment plan for a patient and perform the above operations.

2. The apparatus of claim 1, further comprising deploying at least the selected one of the first model or the second model with a tool having an interface for facilitating collection of patient data and interaction with at least the selected one of the first model or the second model.

3. The apparatus of claim 1, wherein the input data includes at least one of laboratory test results, diagnostic codes, or billing codes.

4. The apparatus of claim 1, wherein the toxicity includes at least one of pneumonia, colitis, or hepatitis.

5. The apparatus of claim 1, wherein the efficacy of the treatment plan for the patient is measured by at least one of patient survival rate or treatment duration.

6. The apparatus of claim 1, wherein the processor circuit extracts the input data and arranges it in a time series.

7. The apparatus of claim 6, wherein the processor circuit aligns the input data with respect to anchor points and arranges the input data in the time series.

8. The apparatus of claim 1, wherein the processor circuit generates labels for the input data to form the set of candidate features.

9. ​ The apparatus according to claim 1, wherein the processor circuit performs feature engineering on the set of candidate feature quantities by at least one of normalization, transformation, or extraction from the set of candidate feature quantities.

10. The apparatus according to claim 9, wherein the processor circuit forms a set of patient feature quantities by selecting from the set of candidate feature quantities, and performs at least one of training or validating at least the first model and the second model based on the feature engineering.

11. The apparatus according to claim 9, wherein the processor circuit generates a feature matrix and performs at least one of training or validating at least the first model and the second model based on the feature engineering.

12. The apparatus according to claim 1, wherein the processor circuit deploys at least one of the selected first model or the second model as an executable tool having an interface.

13. At least one computer-readable storage medium storing instructions, wherein the instructions, when executed by a processor circuit, cause the processor circuit to at least process input data pulled from a record to form a set of candidate feature quantities; use the set of candidate feature quantities to train at least a first model and a second model; validate at least the first model and the second model and compare the performance of the first model with the performance of the second model; select at least one of the first model or the second model based on the comparison; store at least one of the selected first model or the second model; and deploy at least one of the selected first model or the second model to predict at least one of a) toxicity resulting from immunotherapy according to a treatment plan, or b) the efficacy of the treatment plan for a patient At least one computer-readable storage medium for causing the above to be performed.

14. The instructions, when executed, cause the processor circuit to Deploying at least one of the selected first or second models with a tool having an interface for facilitating collection of patient data and interaction with at least one of the selected first or second models, the at least one computer-readable storage medium of claim 13.

15. The instructions, when executed, cause the processor circuit to Extract the input data and arrange it in time series with respect to the anchor points, the at least one computer-readable storage medium of claim 13.

16. The instructions, when executed, cause the processor circuit to Generate labels for the input data to form the set of candidate features, the at least one computer-readable storage medium of claim 13.

17. The instructions, when executed, cause the processor circuit to Perform feature engineering on the set of candidate features by at least one of normalization, transformation, or extraction from the set of candidate features, the at least one computer-readable storage medium of claim 13.

18. A method implemented by a computer, comprising: Processing input data pulled from a record to form a set of candidate features; Training at least a first model and a second model using the set of candidate features; Testing at least the first model and the second model and comparing the performance of the first model with the performance of the second model; Selecting at least one of the first model or the second model based on the comparison; Storing at least one of the selected first model or the second model; Deploying at least one of the selected first model or the second model to predict at least one of a) toxicity resulting from immunotherapy according to a treatment plan, or b) the efficacy of the treatment plan for a patient A method comprising.

19. The step of deploying includes deploying at least one of the selected first model or the second model with a tool having an interface for facilitating the collection of patient data and the interaction with at least one of the selected first model or the second model, the method according to claim 18.

20. The method according to claim 18, further comprising the step of extracting the input data and arranging it in a time series with respect to the anchor point.

21. The method according to claim 18, further comprising the step of generating labels for the input data to form the set of candidate features.

22. The method according to claim 18, further comprising the step of performing feature engineering on the set of candidate features by at least one of normalization, transformation, or extraction from the set of candidate features.

Images