Model generation device for therapy prediction and associated method and model

Abstract

Methods, apparatus, systems, and articles of manufacture for generation and application of models for therapy prediction and treatment are disclosed. An example apparatus includes processing circuitry at least to process input data extracted from a record to form a set of candidate features; training a first model and a second model using the candidate feature set; testing the first model and the second model to compare the performance of the first model and the second model; selecting at least one of the first model or the second model based on the comparison; storing the selected first model and / or second model; and deploying the selected first model and / or second model to predict a likelihood of at least one of: a) toxicity due to immunotherapy according to the treatment plan, or b) efficacy of the treatment plan on the patient.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This patent claims the benefit of priority to U.S. Provisional Patent Application Serial No. 63 / 335,215, filed on April 26, 2022, which is hereby incorporated by reference in its entirety for all purposes. Technical Field

[0002] The present disclosure relates generally to model generation and processing, and more particularly to the generation and application of models for therapy prediction and treatment. Background Art

[0003] The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.

[0004] Immunotherapy can be used to provide effective cancer treatment for some patients. For those patients, immunotherapy can provide higher efficacy and lower toxicity than other therapies. Immunotherapy can include targeted antibodies and immune checkpoint inhibitors (ICI), cell-based immunotherapy, immunomodulators, vaccines and oncolytic viruses to help the patient's immune system target and destroy malignant tumors. However, in some patients, immunotherapy can cause toxicity and / or other adverse side effects. The side effects of immunotherapy may be different from those associated with other cancer treatments because the side effects are caused by overstimulated or misguided immune responses, rather than the direct effects of chemotherapy or radiotherapy on cancer and healthy tissues. Immunotherapy toxicity may include conditions such as colitis, hepatitis, pneumonia and / or other inflammations that may be dangerous to patients. Immunotherapy also triggers different (heterogeneous) efficacy responses in different patients. Therefore, the evaluation of immunotherapy remains unpredictable, with the possibility of huge differences between patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 An example model generation apparatus is illustrated.

[0006] Figure 2-Figure 14 The diagram shows the use of Figure 1 An example model generates a flow chart of a device executing instructions to drive operations.

[0007] Fig.15 is a block diagram of an example processing platform including processor circuitry configured to execute example machine-readable instructions and / or example operations.

[0008] Fig.16 yes Fig.15 A block diagram of an example implementation of a processor circuit system.

[0009] Fig.17 yes Fig.15A block diagram of another example implementation of a processor circuit system.

[0010] Fig.18 1 is a block diagram of an example software distribution platform (e.g., one or more servers) for distributing software (e.g., software corresponding to example machine-readable instructions) to client devices associated with end users and / or consumers (e.g., for licensing, sale, and / or use), retailers (e.g., for sale, resale, licensing, and / or sublicensing), and / or original equipment manufacturers (OEMs) (e.g., for inclusion in products to be distributed to, for example, retailers and / or to other end users such as direct purchasing customers).

[0011] In general, the same reference numbers will be used to refer to the same or like parts throughout the drawing(s) and the accompanying written description.The drawings are not to scale.

[0012] As used in this patent, stating that any part (e.g., a layer, film, region, area, or plate) is on another part in any way (e.g., positioned on another part, located on another part, disposed on another part, or formed on another part, etc.), indicates that the referenced part is in contact with the other part, or that the referenced part is above the other part with one or more intermediate parts located therebetween.

[0013] As used herein, connection references (e.g., attached, coupled, connected, and joined) may include intermediate members between elements referenced by the connection reference and / or relative movement between those elements, unless otherwise indicated. Thus, connection references do not necessarily imply that two elements are directly connected and / or are in a fixed relationship to each other. As used herein, stating that any part is "in contact with" another part is defined to mean that there is no intermediate part between the two parts.

[0014] When introducing elements of various embodiments of the present disclosure, the articles "a," "an," "the," and "said" are intended to mean that there are one or more of the elements. The terms "comprising," "including," and "having" are intended to be inclusive and mean that there may be additional elements besides the listed elements.

[0015] Unless otherwise specifically stated, descriptors such as "first," "second," "third," etc., are used herein without importing or otherwise indicating any meaning of priority, physical order, arrangement in a list, and / or ordering in any manner, but are merely used as labels and / or arbitrary names to distinguish elements to facilitate understanding of the disclosed examples. In some examples, the descriptor "first" may be used to refer to an element in the detailed description, while the same element may be referred to in the claims by a different descriptor such as "second" or "third." In such instances, it should be understood that such descriptors are only used to clearly identify those elements that, for example, might otherwise share the same name.

[0016] As used herein, "approximately" and "approximately" modify their subject matter / values ​​to recognize the possible existence of variations that occur in real-world applications. For example, "approximately" and "approximately" may modify dimensions that may not be exact due to manufacturing tolerances and / or other real-world imperfections as would be understood by one of ordinary skill in the art. For example, "approximately" and "approximately" may indicate that such a dimension may be within a tolerance range of + / - 10%, unless otherwise specified in the description below. As used herein, "substantially real time" means occurring in a near instantaneous manner, recognizing that there may be real-world delays in computation time, transmission, etc. Therefore, unless otherwise specified, "substantially real time" means real time + / - 1 second.

[0017] As used herein, the phrase "in communication" (including variations thereof) includes direct communication and / or indirect communication through one or more intermediate components without requiring direct physical (e.g., wired) communication and / or continuous communication, but additionally includes selective communication at periodic intervals, predetermined intervals, non-periodic intervals and / or one-time events.

[0018] As used herein, the terms "system," "unit," "module," "engine," and the like may include hardware and / or software systems that operate to perform one or more functions. For example, a module, unit, or system may include a computer processor, controller, and / or other logic-based device that performs operations based on instructions stored on a tangible and non-transitory computer-readable storage medium (such as a computer memory). Alternatively, a module, unit, engine, or system may include a hard-wired device that performs operations based on the hard-wired logic of the device. The various modules, units, engines, and / or systems shown in the accompanying drawings may represent hardware that operates based on software or hard-wired instructions, software that directs hardware to perform operations, or a combination thereof.

[0019] As used herein, "processor circuitry" is defined as including (i) one or more special-purpose circuits that are configured to perform (one or more) specific operations and include one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors), and / or (ii) one or more general-purpose semiconductor-based circuits that can be programmed with instructions to perform specific operations and include one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors). Examples of processor circuitry include programmable microprocessors, field programmable gate arrays (FPGAs) that can instantiate instructions, central processor units (CPUs), graphics processor units (GPUs), digital signal processors (DSPs), XPUs or microcontrollers, and integrated circuits such as application-specific integrated circuits (ASICs). For example, an XPU can be implemented by a heterogeneous computing system that includes multiple types of processor circuitry (e.g., one or more FPGAs, one or more CPUs, one or more GPUs, one or more DSPs, etc. and / or combinations thereof) and (one or more) application programming interfaces ((one or more) APIs), which can assign (one or more) computing tasks to the one (or more) of the multiple types of processor circuitry that is most suitable for performing the (one or more) computing tasks.

[0020] Additionally, it should be understood that references to "one embodiment" or "an embodiment" of the present disclosure are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features. DETAILED DESCRIPTION

[0021] In the following detailed description, reference is made to the accompanying drawings forming a part thereof, and shown therein by way of illustration of specific examples that may be implemented. These examples are described in sufficient detail to enable those skilled in the art to implement the subject matter, and it is to be understood that other examples may be utilized, and logical, mechanical, electrical and other changes may be made, without departing from the scope of the subject matter of the present disclosure. Therefore, the following detailed description is provided to describe an exemplary implementation, and is not to be regarded as a limitation on the scope of the subject matter described in the present disclosure. Certain features from the different aspects described below may be combined to form yet new aspects of the subject matter discussed below.

[0022] A large amount of health-related data can be collected using various media and mechanisms about patients. However, processing and interpreting the data can be difficult to drive actionable results. For example, given the magnitude of the data and the various different systems, formats, etc., it may be difficult, if not impossible, to understand and correlate data in various forms and sources through standardization / normalization, aggregation, and analysis. Therefore, certain examples provide devices, systems, associated models, and methods for processing and correlating health-related data to predict patient outcomes and drive patient diagnosis and treatment. Certain examples provide systems and methods for health data prediction model construction. Certain examples provide frameworks and machine learning workflows for therapy prediction.

[0023] For example, immune checkpoints regulate the human immune system. Immune checkpoints are pathways that allow the body to tolerate itself by preventing the immune system from indiscriminately attacking cells. However, some cancers can protect themselves from attack by stimulating immune checkpoints (e.g., proteins on immune cells). To target cancer cells in the body, immune checkpoint inhibitors (ICIs) can be used to target these immune checkpoint proteins to better recognize and attack, rather than hide from, cancer cells.

[0024] Despite the great success of ICI cancer treatment, such treatment can pose a great threat to human health due to its side effects, which is an immune-related adverse event (irAE) caused by these treatment options. One of these toxicities is hepatitis, which occurs when the liver is affected by an autoimmune-like inflammatory pathological process triggered by ICI. Some examples predict the onset of irAE hepatitis before the start of the first ICI treatment. More precisely, some examples predict whether irAE hepatitis will occur within a given time window after the start of the first treatment. Other toxicities such as pneumonia, colitis, etc. can be similarly predicted.

[0025] For example, majority class undersampling is combined with time series data aggregation to obtain a well-balanced and static data set that can be fed to the model. Example models include gradient boosting (GB) and random forest (RF), and / or other models that can adapt to the size and statistical properties of the data. The model can be selected based on the Fl score, which is a measure of the accuracy of the model on the data set. For example, a GB model without undersampling can maximize the Fl score (e.g., the harmonic mean of recall and precision), while a RF model with undersampling can provide a high recall (e.g., the true positive ratio found) with a relatively low precision (e.g., the true positive ratio among estimates), which is acceptable due to the cost-effectiveness of the additional testing required for model-based decision making. The model is also able to create probability estimates for labels, not just the discrete labels themselves.

[0026] Prepare input data to develop and / or feed (one or more) models to drive predictions, therapies, etc. In some examples, the input is prepared by extracting blood feature information (e.g., relevant portions of blood features, etc.) from an electronic health record (EHR) data table (e.g., received from the EHR, etc. at a processor / processing circuit system), electronic medical record (EMR) information, etc. Blood features are measurements of liver biomarker concentrations in plasma (such as ALT, AST, alkaline phosphatase, and bilirubin, etc.) and other concentration values ​​in the blood. For example, blood features can be represented as time series data. After the blood features are extracted, the time series data is formed into a single complex data structure. The data structure is used to aggregate the time series blood feature data into a data table, which can be used with preprocessing and transformation.

[0027] For example, feature engineering aggregates blood feature data by describing the time series data of blood particles with associated mean, standard deviation, minimum and maximum values. Liver features can also be created by taking the last liver biomarker measurement available before treatment. When the level of at least one liver biomarker exceeds a threshold value (e.g., 3 times the upper limit of normal, etc.) within a predefined window, a label (e.g., using a definition obtained from a medical expert, etc.) can be created to classify someone as positive. Otherwise, the label can classify the patient as negative. The date of immune checkpoint inhibitor (ICI) treatment can be determined and / or otherwise provided for label and / or time series use.

[0028] After preparing the input data (e.g., using feature engineering), the dataset is resampled. That is, the dataset resulting from the input preparation is unbalanced. Therefore, the dataset can be processed to infer, verify, estimate, and / or otherwise resample the prepared feature data in the dataset. For example, when the goal is to maximize the recall value, random majority class undersampling is performed on the dataset. When the Fl score is the subject of maximization, resampling can be skipped or ignored.

[0029] Using the data set (e.g., resampled or otherwise), the model can be trained and tested to generate predictions. For example, when the recall is expected to be maximized, the data set can be used to train the RF model. For example, when the Fl score is expected to be maximized, the data set can be used to train the GB model. In some examples, the trained model can be validated (such as using leave-one-out cross-validation), where each sample is predicted by an individual and the rest are used as a training set.

[0030] Therefore, various "artificial intelligence" (AI) models can be developed and deployed for use in various health prediction applications. For example, the model can be used to predict static and / or dynamic prognostic factors of hepatitis using AI models and patient (e.g., EHR, etc.) data.

[0031] Alternatively or additionally, a predictive model can be developed for ICI-related pneumonia using a small, noisy data set. Using input data from structured (e.g., EHR, EMR, (one or more) laboratory data systems, etc.) and / or unstructured (e.g., from (one or more) laboratory data systems, EHR, EMR, etc. managed (curate)) data, the input features can be evaluated to build a model and output the predicted probability of developing pneumonia. In some examples, multiple models can be developed, and the system and associated processes can iterate and decide between two or more model versions. For example, the available data can be divided into two partitions using a sequential forward selection process, and a robust performance evaluation can be used to verify and compare the two developed models to select one for deployment.

[0032] For example, the available data can be divided into two partitions, and a sequential forward selection process can be used to build a model. In each iteration, the two model versions are compared, and the version with higher performance on both partitions is selected. The second model version is created by adding candidate features to the first model version. On the larger partition, the performance results in the inner loop based on nested cross validation (CV) are compared. On the smaller partition, a permutation test is performed to test whether the candidate features have predictive power (e.g., the second model version has higher performance). Finally, the outer loop of nested CV is used for robust performance evaluation of the final model.

[0033] Certain examples provide an automated framework for preparing EHR and / or other health data for use in machine learning-based model training. For example, the framework prepares data from multiple sources and generates a combined time-dependent, unrestricted intermediate output that is used to aggregate features for training and deploying time-dependent models. For example, input data sources can be processed and patient vectors can be generated using the data. The patient vectors can be used for filtering and aggregation, forming an interface definition. Thus, a model-independent workflow creates input data sets for multiple model training. The intermediate outputs preserve temporal structure for sequential modeling tasks and form a maintainable, sustainable framework with an interface.

[0034] Certain examples provide prediction model construction related to ICI, wherein input data from multiple sources are prepared. A ground truth prediction label can be generated from the prepared data and / or a label can be professionally created. Then, one or more models are built on a feature matrix generated using the labels and data, wherein the ground truth prediction label is an independent module of the framework. For example, the framework can then drive a workflow to assess multiple efficacy surrogate endpoints to predict (one or more) responses to ICI therapy.

[0035] In some examples, a system involving multiple submodules is used to prepare and use patient health data to train a model. For example, the system includes a data extraction and processing submodule that is used to extract patient blood test history from EMR / EHR, clean blood history data, and perform (one or more) data quality checks. The label definition submodule defines one or more feature labels related to blood history data, and the feature engineering submodule can form blood features by aggregating and processing blood history data about labels. The model submodule trains and evaluates AI models to dynamically predict the risk of immune-related hepatitis adverse events from a fixed length of blood test history. Alternatively or additionally, liver function test values ​​can be extracted, cleaned and organized in time series. For example, a label definition algorithm can be executed to generate AI models and indicator labels for each set of blood and / or liver test values, while feature engineering (e.g., normalization, symbol transformation and motif extraction) can be used to train and evaluate (one or more) AI risk prediction models. Similarly, drug history information, medical condition history, anthropometric features, etc. can be used for labeling and feature formation.

[0036] Because features can vary significantly between toxicities, toxicity-specific feature formation and model training are important to provide meaningful features and accurate predictive results. Otherwise, the lack of meaningful features undermines the performance of the resulting model. Therefore, certain examples derive models that focus on specific toxicities, and the outputs can be combined (together and / or further with efficacy predictions) to form recommendations, such as a risk-to-benefit analysis based on toxicity versus efficacy of a given immunotherapy drug.

[0037] Thus, certain examples drive treatment based on predictions of the likelihood of complications of hepatitis, pneumonia, etc. Based on the predictions and / or other outputs of one or more AI models, patients may be selected for immunotherapy treatment, removed from immunotherapy treatment and / or otherwise have their treatment plan adjusted, selected for immunotherapy clinical trials, etc. As data continues to be collected from one or more patients, the (one or more) models used in the predictions may evolve, and the associated (one or more) predictions may also change based on the collected data. For example, the (one or more) models and / or associated (one or more) predictions may be customized for individuals and / or deployed for groups / types / etc. of patients, or for a group or individual ICI drugs, etc.

[0038] In some examples, data values ​​are normalized to the upper limit of a "normal" range (e.g., for blood tests, liver tests, etc.) so that values ​​from different sources can be compared on the same scale. Data values ​​and associated normalization / other processing can be specific to a laboratory, patient, patient type (e.g., male / female, etc.), etc. For example, each laboratory measurement may have a specific normal range for evaluating its values ​​across multiple patients.

[0039] For time series data, one value depends on previous values ​​and / or other values, so that the data values ​​have a relationship, rather than being independent. Correlations can be identified and taken into account to identify patients in the data over time. For example, if the data values ​​in the time series of a patient's blood work exceed twice the normal limit at the first time, decrease within the normal limit at the second time, and exceed twice the normal limit again at the third time, this pattern can be identified as important (e.g., worthy of further analysis). For example, data processing can annotate or mark such a pattern accordingly. Therefore, clinical data from the patient's records can be used to identify and form features, anomalies, other patterns, etc. over time. The data is confused into a general model for comparison. The resulting model trained and tested on such data can be robust to outliers, scaling, etc. Features can be created for better (e.g., more efficient, more accurate, more robust, etc.) modeling, such as pneumonia modeling, colitis modeling, hepatitis modeling, etc.

[0040] Thus, data processing creates feature(s) that can be used to develop model(s) that can be deployed to predict outcome(s) for patient(s). For example, a data processing pipeline creates thousands of features (e.g., pneumonia, frequency of ICD-10 codes, frequency of C34 codes, etc.).

[0041] For example, the data value may include the ICD-10 code for a given patient within a one-year period. In some examples, the code may span multiple years (e.g., ten years, etc.) and be coordinated for processing. The ICD-10 code is processed to identify the code associated with the lung or respiratory function, and then such a code can be used to calculate the relative frequency of lung disease in the patient. As another example, the patient history can be analyzed to determine the relative frequency of the C34 code in the patient history, which indicates lung cancer. For example, smoking status can also be a binary flag that is set or not set from the data processing pipeline. In some examples, the code can be converted between code systems (e.g., ICD-9, ICD-10 (such as C34, C78, ​​etc.), etc.). For example, the code can be reverse engineered without all the keys in the secret key.

[0042] Using the codes and other data, multiple (e.g., 5, 6, 10, etc.) features can be created and used in a modeling framework to predict the development of pneumonia in a patient. The model is built in a step-by-step forward manner. Labels for the pneumonia model are not inherently present in the dataset, so ground truth is created for model training based on expert judgment to identify labels, for example, from (one or more) patient histories. For example, a code book and quality control can be used for correct labeling.

[0043] In some examples, historical data received from patients is asynchronous. Systems and methods then align data for patients (and / or between patients) to enable aggregation and analysis of data relative to a common baseline or benchmark. In some examples, an impact point or other reference can be selected / determined, and patient data time series / timelines can be aligned around the determined or selected point (e.g., an event that occurred for (one or more) patients, such as a physical examination, injury, attack, test, birthday, milestone, etc.). For example, the date of the first chemotherapy, ICI therapy, the first symptom / reaction (e.g., lung function, etc.), etc. can be used to align patient data.

[0044] The processed data can then be used to predict static labels in predefined or otherwise determined time windows, etc. Models can be trained, validated, and deployed for hepatitis, pneumonia, drug efficacy, etc. Thus, data from EHRs, EMRs, laboratory systems, and / or other data sources can be pre-processed and provided to the model to generate predictions, which can be post-processed and output to users and / or other systems for alerts, follow-up, treatment plans, etc. In some examples, the predicted values ​​are routed to another system (e.g., scheduling, laboratory, etc.) for further processing.

[0045] One or more AI models can be used to facilitate processing, association, and prediction of available patient health data based on, for example, blood test results, liver test results, other test results, other patient physiological data, and the like. Models can include high recall low precision models, low recall high precision models, harmonic mean maximization (convergence) models, and the like. Enhanced decision tree models or variants such as random forests (RF), gradient boosting (GB), and the like can be used. For example, a majority class undersampled random forest model can be used to maximize recall with relatively low precision. However, for hepatitis, prevention is cheap and easy, so false negatives are affordable. Therefore, a gradient boosting model can be developed to maximize the Fl score without applying resampling.

[0046] For example, machine learning techniques (whether deep learning networks or other empirical / observational learning systems) can be used to characterize and otherwise interpret, extrapolate, infer, and / or complete medical data obtained from patients. Deep learning is a subset of machine learning that uses a collection of algorithms to model high-level abstractions in the data using deep graphs with multiple processing layers (including linear and nonlinear transformations). While many machine learning systems are seeded with initial features and / or network weights to be modified by learning and updating of the machine learning network, deep learning networks train themselves to identify "good" (e.g., useful, etc.) features for analysis. Using a multi-layer architecture, machines using deep learning techniques can process raw data better than machines using traditional machine learning techniques. Examining data for groups of highly correlated values ​​or unique topics is facilitated using different evaluation or abstraction layers.

[0047] Throughout the specification and claims, the following terms take the meanings explicitly associated herein unless the context clearly dictates otherwise. The term "deep learning" is a machine learning technique that utilizes multiple data processing layers to identify various structures in a data set and classify the data set with high accuracy. A deep learning network (DLN) (also known as a deep neural network (DNN)) can be a trained network (e.g., a trained network model or device) based on multiple input and output learning patterns. A deep learning network / deep neural network can be a deployed network (e.g., a deployed network model or device) that generates and provides outputs in response to inputs from a trained network.

[0048] The term "supervised learning" is a machine learning training method in which the machine is provided with already classified data from a human source. The term "unsupervised learning" is a machine learning training method in which the machine is not given already classified data but is made available for anomaly detection (e.g., random forests, gradient boosting, etc.). The term "semi-supervised learning" is a machine learning training method in which the machine is provided with a small amount of classified data from a human source compared to the large amount of unclassified data available to the machine.

[0049] The term "convolutional neural network" or "CNN" is a biologically inspired network of interconnected data used in deep learning to detect, segment, and identify related objects and regions in a data set. CNN evaluates raw data in the form of multiple arrays, breaking down the data in a series of stages, examining the data for learned features. For example, CNN can be used to predict hepatitis and / or toxicity.

[0050] The term "recurrent neural network" or "RNN" refers to a network in which the connections between nodes form a directed or undirected graph along a time sequence. For example, an RNN can be used to predict hepatitis and / or toxicity.

[0051] The term "transfer learning" is the process by which a machine stores information used to correctly or incorrectly solve one problem in order to solve another problem of the same or similar nature as the first problem. Transfer learning can also be called "inductive learning." For example, transfer learning can leverage data from previous tasks.

[0052] The term "active learning" is the process of machine learning in which a machine selects the set of examples for which it receives training data, rather than passively receiving examples selected by an external entity. For example, when learning, the machine can be allowed to select examples that the machine determines will be most helpful for learning, rather than relying solely on external human experts or external systems to identify and provide examples.

[0053] The term "computer-aided detection" or "computer-aided diagnosis" refers to a computer that analyzes medical data to come up with a possible diagnosis.

[0054] Deep learning is a class of machine learning techniques that employs a representation learning approach, which allows a machine to be given raw data and determine the representations needed to classify the data. Deep learning uses a back-propagation algorithm to determine the structure in a data set, which is used to change the internal parameters of the deep learning machine (e.g., node weights). Deep learning machines can utilize a variety of multi-layer architectures and algorithms. For example, while machine learning involves identifying features to be used to train a network, deep learning processes raw data to identify features of interest without external identification.

[0055] Deep learning in the context of a neural network consists of numerous interconnected nodes called neurons. Input neurons activated from an external source activate other neurons based on connections to those other neurons controlled by the machine parameters. A neural network behaves in a certain way based on its own parameters. Learning refines the machine parameters and, by extension, the connections between neurons in the network so that the neural network behaves in the desired manner.

[0056] Various artificial intelligence networks can be deployed to process input data. For example, deep learning using convolutional neural networks segments data using convolutional filters to locate and identify learned observable features in the data. Each filter or layer of the CNN architecture transforms the input data to increase the selectivity and invariance of the data. This abstraction of the data allows the machine to focus on the features in the data it is trying to classify and ignore irrelevant background information.

[0057] Deep learning operates on the understanding that many data sets consist of high-level features that are comprised of lower-level features. For example, when examining an image, rather than looking for objects, it is more efficient to look for edges that form patterns that form components that form the objects being looked for. These hierarchies of features can be found in many different forms of data, such as speech and text.

[0058] Learned observable features include objects and quantifiable rules learned by the machine during supervised learning. Machines provided with large collections of well-classified data are better equipped to discern and extract features that are relevant to the successful classification of new data.

[0059] A deep learning machine utilizing transfer learning may correctly connect data features to certain classifications confirmed by a human expert. Conversely, the same machine may update parameters used for classification when notified of an incorrect classification by a human expert. For example, settings and / or other configuration information may be guided by learned use of settings and / or other configuration information, and as the system is used more (e.g., repeatedly and / or by multiple users), the number of variations and / or other possibilities for settings and / or other configuration information may be reduced for a given situation.

[0060] For example, an example deep learning neural network can be trained on an expert classification dataset. This dataset builds the first parameters for the neural network, and this will be the stage of supervised learning. During the supervised learning stage, it can be tested whether the desired behavior of the neural network has been achieved.

[0061] Once the desired neural network behavior has been achieved (e.g., the machine has been trained to operate according to specified thresholds, etc.), the machine can be deployed for use (e.g., testing the machine with new / updated data, etc.). During operation, the neural network classification can be confirmed or rejected (e.g., by an expert user, an expert system, a reference database, etc.) to continue to improve the neural network behavior. The example neural network is then in a state of transfer learning because the parameters used to determine the classification of the neural network behavior are updated based on the ongoing interaction. In some examples, the neural network can provide direct feedback to another process. In some examples, the neural network outputs data that is buffered (e.g., via the cloud, etc.) and verified before it is provided to another process.

[0062] When interacting with doctors, deep learning machines can leverage transfer learning to offset the small datasets available in supervised training. These deep learning machines can improve their computer-aided diagnosis over time through training and transfer learning. However, larger datasets lead to more accurate and robust deployed deep neural network models, which can be applied to transform different medical data into actionable results (e.g., system configuration / settings, computer-aided diagnosis results, image enhancement, etc.).

[0063] One or more such models / machines may be developed and / or deployed with respect to prepared and / or managed data. For example, features (e.g., liver biomarkers, blood / plasma concentrations, etc.) may be extracted from EHR / EMR data tables, etc. For example, managed data extracts structured data from unstructured sources (e.g., diagnosis dates from medical records, etc.). The extracted features form the basis for label creation. Time series measurements are extracted and aggregated to produce statistical descriptors of the time series data. A table of such data is then used to train a model for prediction. In some examples, the data may be resampled if necessary, desired, or determined. The model is validated using robust cross-validation (such as leave-one-out cross-validation).

[0064] For example, a selection, input, or goal may determine whether to maximize the Fl score or recall using a given model. When the Fl score is to be maximized by the model, then no resampling of the data is performed. When the recall is to be maximized by the model, then resampling of the data may be performed. For example, a decision may be driven by the deployment environment of the model. When the model is to be deployed in a system that facilitates drug trials, then the Fl score is to be maximized because pharmaceutical companies want patients with the greatest chance of responding well to a given drug. In a clinical treatment setting, a high recall is even more important, where the system hopes to eliminate as much toxicity as possible. For example, in a treatment setting for hepatitis, low model accuracy is acceptable due to the inexpensive nature of hepatitis treatment.

[0065] The associated systems and methods can be used to assess the probability or reliability of predictions generated by the model. For example, the predictions and associated confidence levels or other reliability can be output to another processing system. The possible response to immunotherapy treatment can be modeled based on available patient data collected in clinical practice.

[0066] Figure 1 An example model generation device 100 is illustrated, including an example input processor circuit system 110, an example model trainer circuit system 120, an example model comparator circuit system 130, an example model storage circuit system 140, and an example model deployer circuit system 150. Figure 1As shown in the example of , the example input processor circuit system 110 processes input data extracted from the record to form a candidate feature set. The example model trainer circuit system 120 uses the candidate feature set to train at least a first model and a second model. The example model comparator circuit system 130 tests at least the first model and the second model to compare the performance of the first model and the second model. Based on the comparison, the example model comparator circuit system 130 selects at least one of the first model or the second model based on the comparison. The example model storage circuit system 140 provides a memory circuit system to store the selected first and / or second model. The example model deployer circuit system 150 deploys the selected first model and / or second model to predict the possibility of toxicity due to immunotherapy according to the patient's treatment plan, such as pneumonia, hepatitis, colitis, etc.

[0067] In operation, the example input processor circuit system 110 processes input data related to one or more patients, such as laboratory results, diagnosis codes, billing codes, etc. The input can originate from one or more external systems 160, such as EHR, EMR, etc. For example, the example input processor circuit system 110 can extract and organize the input for each patient in a time series. In some examples, the input processor circuit system 110 aligns the input data with respect to anchor points to organize the input data in a time series. In some examples, the input processor circuit system 110 generates labels for the input data to form a candidate feature set.

[0068] In some examples, the input processor circuit system 110 performs feature engineering on the candidate feature set and / or the base data. For example, the input processor circuit system performs feature engineering on the candidate feature set by, for example, regularizing, transforming, and / or extracting from the candidate feature set. In some examples, the input processor circuit system 110 selects from the candidate feature set to form a patient feature set to train and / or test at least the first model and the second model based on feature engineering. In some examples, the input processor circuit system 110 generates a feature matrix to train and / or test / validate the first model and / or the second model based on feature engineering.

[0069] In some examples, the model deployer circuit system 150 deploys the selected first and / or second model as an executable tool with an interface to facilitate the collection of patient data and interaction with the selected first and / or second model. The deployed model(s) can be used with the tool to select patients for clinical trials, start immunotherapy treatment courses according to a treatment plan or protocol, adjust a current immunotherapy treatment plan, etc. In some examples, the deployed model(s) can be stored / utilized in one or more external systems 160 such as EHRs, EMRs, scheduling systems, clinical information systems, etc.

[0070] Thus, the example model generator device 100 may pre-process data from one or more external systems 160 via the example input processor circuit system 110. The example model generator device 100 then uses the model trainer circuit system 120 and the model comparator circuit system 130 to train and validate multiple models. The example model generator device 100 uses the model storage circuit system 140 and the model deployer circuit system 150 to post-process, store and deploy one or more of the trained and validated models. Input data from EHR, EMR and / or other external systems 160 is transformed into multiple models, which can be stored and deployed to predict treatment efficacy, toxicity risks and / or other side effects, etc. from immunotherapy treatment of a specific patient. For example, data from multiple patients can be utilized through processing and modeling to generate indicator predictions for a specific patient.

[0071] In some examples, the model trainer circuit system 120 trains and verifies a boosted decision tree model, other ensemble and / or regression-based learning models, and the like. For example, a boosted decision tree model utilizes multiple weak learning decision trees to create a strong learning AI model. For example, a tree in the model can correct errors in other trees of the model, and the entire set of trees in the model works together to generate a predicted output. The model can also be formed as a random forest (RF) model, a gradient boosting (GB) model, and the like. For example, the data is conformed to a general model for comparison by the model trainer circuit system 120, which is robust to outliers and can be scaled.

[0072] In some examples, the input data processor 110 regularizes the input data value to the upper limit of normal, so that the values ​​(e.g., blood work values, liver function test values, other laboratory work values, etc.) are all on the same scale. Each laboratory value may have a certain range of values. For example, the input data processor 110 regularizes the value of a specific laboratory type within a range so that the data for that laboratory can be compared across patients. The input data processor 110 can also organize the data into a time series. The time series data can be regularized and / or otherwise adjusted by the input data processor 110 based on anchor points (e.g., common or reference events such as hospitalization, symptoms, birth, diagnosis date, first ICI administration (administration) date, etc.).

[0073] For example, the patient data received may be asynchronous. Selecting anchor points / inflection points / influence points allows data to be aligned for the same, similar, and / or different patients. For example, in a patient population with diabetes, ten years of patient history are reviewed while excluding data occurring after diagnosis. The diagnosis point is selected as an anchor point for retrospectively reviewing patient history data (e.g., when blood sugar levels were not obtained or insulin was not taken, etc.). As another example, data from the first immunotherapy treatment can be used to align patient data.

[0074] Thus, for example, data is organized for use in identifying relationships and comparisons. For example, patterns can be identified in the data.

[0075] In some examples, the time series data is formed into multiple features by the input data processor 110. These features can be a set of candidate features from which features are selected to train and validate the model. For example, the feature engineering performed by the input data processor 110 can form multiple features based on codes (e.g., ICD-10 codes, etc.). For example, the ICD-10 codes of patients in a given year can be processed to identify codes related to lung and / or respiratory function (C34, C78, ​​etc.), and the time series of those codes can form a function for calculating the relative likelihood of lung disease in the patient. Similarly, multiple features can be formed to predict the development of pneumonia, hepatitis, colitis, and / or other toxicities from immunotherapy treatment.

[0076] Features based on lung function, liver biomarkers, blood work (e.g., concentrations in blood, plasma, etc.), etc. can form the basis for creating labels and / or other statistical descriptors for model training. For example, the collected data can be compared to a baseline set of true values ​​for the identified labels to generate labels for the data. Resampling and validation (e.g., leave-one-out cross-validation, other cross-validation, other validation, etc.) can be performed to generate a robust model.

[0077] The example model trainer circuit system 120 uses features and / or other data to train one or more AI models, such as toxicity prediction models (e.g., hepatitis prediction models, colitis prediction models, pneumonia prediction models, etc.), efficacy prediction models (e.g., immunotherapy efficacy models, etc.), and the like. The model may be a high recall with low precision, a low recall with high precision, a harmonic mean (Fl) maximization, a majority of claims undersampling, and the like. Determinations may be made by and / or for the example model trainer circuit system 120 (e.g., based on settings, modes, types, etc.). For example, the model trainer circuit system 120 trains one or more models to maximize the Fl score or maximize the recall. When focusing on Fl, resampling may not be necessary. When focusing on recall, resampling (e.g., using a training data set and a test data set, multiple training data sets, and / or multiple test data sets, etc.) may be desirable to improve the model.

[0078] For example, when configuring participation in a drug trial, the focus is on maximizing the Fl score to identify patients with the highest chance of responding well to a given immunotherapy drug. However, when determining clinical treatment for a patient, the goal is to eliminate as many toxicities as possible from the patient. Therefore, models developed with high recall are more important, while low precision is acceptable as cheaper treatment options for hepatitis, colitis, pneumonia, etc.

[0079] Model comparator circuitry 130 may compare and / or otherwise assess multiple training models to validate the models and assess the likelihood of predictions generated by each model. Such assessment / assessment may be used to select one or more models to be deployed. For example, models may be compared and / or otherwise assessed based on average output values, standard deviations of model outputs, etc. One or more models may be selected for storage in model storage circuitry 140 and / or other memory circuitry, deployment in a tool via model deployer circuitry 150, output to one or more external systems 160, etc.

[0080] In some examples, external system 160 may utilize one or more deployed models from model deployer circuitry 150 to drive a tool to evaluate efficacy and / or toxicity associated with an immunotherapy for a patient and / or patient population, etc. Using one or more deployed models, a patient may be assessed at the beginning of an immunotherapy treatment plan, during administration of an immunotherapy treatment plan, as a candidate for an immunotherapy clinical trial, etc. In some examples, a patient's prior efficacy and / or toxicity model predictions may be used with the patient's updated efficacy and / or toxicity model predictions (e.g., combined between prior and current results, with prior results as input to generating current results, etc.).

[0081] Figure 2-14is a representation of an operation that can be stored in memory circuitry and executed by processor circuitry to implement and actuate Figure 1 An example model is a flowchart of an example process of computer readable instructions of the device 100. Figure 2 The example process 200 of begins at block 210, where the input processor circuit system 110 processes data to form (one or more) inputs for the model trainer circuit system 120. For example, the input processor circuit system 110 may align the data relative to an anchor or reference point (e.g., a date, event, milestone or other marker, etc.). The input processor circuit system 110 may form a time series from the data. Alternatively or additionally, for example, the input processor circuit system 110 may label and / or perform feature engineering on the data to form an input feature set to train the (one or more) models.

[0082] At block 220, the model is trained using the input. For example, a plurality of AI models are trained by the model trainer circuitry 120 using the input feature set and / or other input provided from the input processor circuitry 110. For example, by identifying patterns and correlations in the input, the model trainer circuitry 120 forms and weights nodes, layers, connections, etc. in a model (e.g., an RF model, a GB model, a boosted decision tree model, etc.).

[0083] At block 230, the trained model is validated by the model trainer circuitry 120. For example, additional inputs (e.g., features, resampled input data, etc.) are used to validate (e.g., test, confirm, etc.) that the trained model works as expected with a threshold level of precision, accuracy, etc.

[0084] At box 240, the model comparator circuit system 130 evaluates the validated models to select one or more of the models for deployment. For example, the model comparator circuit system 130 can compare and / or otherwise assess multiple trained, validated (e.g., tested, etc.) models to assess the likelihood of predictions generated by each model, the suitability of each model for a specific purpose (e.g., prediction of immunotherapy treatment efficacy, prediction of immunotherapy toxicity, immunotherapy clinical trial selection, immunotherapy treatment plan adjustment, etc.). One or more models can be selected by the model comparator circuit system 130 using such an assessment / evaluation. For example, the model can be compared and / or otherwise assessed based on the average output value, the standard deviation of the model output, etc. In some examples, the model comparator circuit system 130 selects a model for use / deployment. In other examples, the model comparator circuit system 130 selects multiple models (e.g., toxicity prediction models and efficacy prediction models, etc.) for use / deployment.

[0085] At block 250, the one or more selected models may be stored by the model storage circuitry 140. Thus, the selected model(s) may be saved for later use, deployment, etc. In some examples, the unselected model(s) may also be stored by the model storage circuitry 140, as the unselected model(s) may be suitable for other use / deployment in response to a subsequent request.

[0086] At block 260, one or more selected models are deployed by the model deployer circuitry 150 to an external system 160. The model(s) may be deployed as part of a tool for immunotherapy treatment and / or clinical trial planning to provide toxicity predictions, efficacy predictions, and the like.

[0087] Figure 3 An example implementation of input processing for model development is illustrated (e.g., Figure 2 At box 310, the input data for each of the plurality of patients is arranged in one or more time series (e.g., placed in chronological order to show evolution, progression, correlation, and / or other patterns). At box 320, the time series data for each of the plurality of patients is aligned relative to an anchor point or other reference event. For example, the various sets of time series data may be aligned according to birth, onset of illness, events such as hospitalization, and the like. For example, by aligning the time series, the data in the time series may be more meaningfully evaluated, compared, and / or otherwise analyzed.

[0088] At block 330, the aligned time series data is labeled. For example, a target label and associated rating may be generated for each set of data values ​​(eg, blood test values, liver function values, lung function values, etc.).

[0089] At block 340, feature engineering is performed on the data to prepare the data into a set of features with labels for training, testing, and / or otherwise validating a model. For example, feature engineering can regularize, transform, and extract the data into a set of patient features for application to a model. In some examples, the features form a set of candidate features that are evaluated to select a subset of patient features for model training, validation, etc.

[0090] Figure 4 illustrates an example implementation of model selection for deployment (e.g., Figure 2At box 410, the goal or purpose is checked. For example, the request for the AI ​​model may be for clinical trial evaluation, prediction of immunotherapy efficacy, prediction of toxicity and / or other side effects associated with immunotherapy, etc. Such a request may drive the selection of models with certain characteristics, such as high Fl score, high recall, etc. Based on the requested goal or purpose, a subset of available models may be selected. In other words, based on the requested goal or purpose, a subset of available models may be excluded, leaving a subset of models that may be used for further evaluation and selection.

[0091] At box 420, a test is performed using each remaining model. For example, a permutation test, a binomial test, and / or other comparison operations are performed using each model (e.g., evaluating a lung function model with respect to smoking, non-smoking, other binomial and / or permutation tests, etc.). At box 430, the output of each model for which the test was performed is evaluated based on one or more criteria. For example, each model can be evaluated based on an average, a standard, and / or other comparison criteria to determine which model best meets one or more criteria.

[0092] At box 440, one or more models are selected based on a comparison of the output and / or other model performance. For example, a single model can be selected for deployment based on an assessment of how the model performs and / or outputs relative to one or more criteria. In some examples, multiple models can be selected based on one or more criteria and / or other factors. For example, multiple models can meet (one or more) criteria. As another example, a request can include a request for therapeutic efficacy and the possibility of associated toxicity. In response to such a request, both an efficacy model and a toxicity model can be selected. In some examples, a single model can be trained for both toxicity and efficacy to output a combined prediction.

[0093] Figure 5 An example sequential procedure 500 for model building and evaluation (eg, by developing a feature set, etc.) is illustrated. Figure 5As shown in the example of , at 1, the initial data set 510 is divided into at least two partitions. Separate data analysis is performed on the first part 520 of the data and the second part 525 of the data. The first data partition 520 is divided into at least two loops, such as an inner loop 522 and an outer loop 524 for data analysis. At 2, a first test such as a binomial test evaluates the data values ​​in the inner loop 522 and the outer loop 524 (e.g., if the data value Ho includes "smoking", then reject, etc.) to train the model. At 3, a permutation test is used instead of using a separate loop to process a separate "non-contact" test data set 525. For example, the data set 525 can be processed within multiple permutations with criteria such as smoking and non-smoking to identify that the possibility of smoking is greater than the possibility of non-smoking. At 4, the model is evaluated to report the average, standard and / or other evaluation / performance statistics about the first and second data partitions 520, 525 and the associated tests.

[0094] In some examples, one or more of loops 522, 524 are used to evaluate the first data partition 520 via exploratory data analysis, which has a K-fold cross validation (CV) comparing the first feature set M1 and the second feature set M2. Through sequential iterations, if the K-fold CV performance of M1 is less than M2, feature F is added to the first feature set M1. Once the features have been evaluated accordingly, the second data partition 525 is used to test the resulting model performance. If both loops 522, 524 are utilized, M1 and M2 can be compared based on the results of the inner loop 522 (e.g., a binomial test). When M2 is selected, a permutation test is performed to assess whether M2 is sufficiently better than M1 on the validation data set 525. Then, for example, model performance can be evaluated on the outer loop 524. Therefore, the initial feature set M1 or the extended feature set M2 can be selected for model training.

[0095] Figure 6An example process 600 for preparing input data for model training is illustrated. Using the example process 600, data from different observation databases can be transformed into a common format (e.g., a data model) and a common representation (e.g., common terms, vocabulary, coding schemes, etc.) for system analysis according to a library of analysis routines. The example process 600 accepts input of (one or more) concept tables 605 including concept identifiers, concept names, concept descriptions, etc. For example, the concepts in this example relate to liver function tests. The clinical table 615 is also input with database extracts of patient demographics, hospital visits, laboratory measurements, etc. At box 610, patient liver function test values ​​(e.g., represented in patient blood test history) are extracted from electronic medical records, etc., and are cleaned, quality checked, and organized in a time series format for one or more patients. Such extraction and processing can also be applied to other patient information (e.g., lung function, drug efficacy, etc.).

[0096] At box 620, a label definition algorithm is executed to assign adverse event (AE) levels to blood test / liver function value sets (e.g., ALT, AST, TBILIRUBIN, ALKPHOS, etc.) and create binary target labels for the AI ​​model. Such label definitions can also be applied to other patient test values ​​(e.g., lung function, etc.).

[0097] At box 630, feature engineering is performed. For example, blood test values ​​and / or other values ​​can be regularized to an upper limit of "normal" to aid in feature generation, comparison, and other analysis. Such a "normal" range can be determined for a particular patient based on specific blood work, other lab tests, etc. Feature engineering transforms the regularized values ​​into discrete symbolic representations, such as modified symbol set approximations, etc. In some examples, motifs can be extracted from a series of symbolic representations as n-grams, and counts from patient history can be used as features.

[0098] At box 640, the AI ​​model can be trained and evaluated based on features, etc., so as to dynamically predict the risk of immune-related adverse events based on patient data. For example, the risk of immune-related adverse events can be predicted by a model from a fixed-length blood test history, etc. The output may include a model performance metric 625, a data set of AE risk prediction 635, etc. For example, the risk prediction data set 635 may include a risk prediction of immune-related hepatitis adverse events for each blood test value history. That is, for each patient, based on their blood test history, etc., the risk of developing AE hepatitis is predicted until the patient's next ICI treatment appointment. Based on the associated patient data training model, the data set 635 may alternatively include risk predictions for pneumonia, colitis, hospitalization, etc.

[0099] Figure 7An example process 700 for building an example classification model is illustrated. At box 710, data from an input data set 705 is divided into multiple partitions. For example, structured EHR-derived tabular data, unstructured data tables, measurement features aggregated over time (e.g., 30 days, 60 days, 1 year, etc.), managed labels (e.g., pneumonia labels, hepatitis labels, colitis labels, etc.), other features aggregated over time (e.g., 30 days, 60 days, 1 year, etc.) (e.g., condition, smoking status, etc.), etc. can be collected and partitioned. For example, 90% of the available data can be organized in a first partition, while the remaining 10% of the data can be organized in a second partition. Based on the typical, expected, or desired number of features in the model, a model size input 715 can help adjust the partitions, etc.

[0100] At block 720, a sequential forward selection procedure is performed to evaluate multiple models. The example procedure iterates to find patterns in the data to form potential predictive features. For example, the first partition may be evaluated to identify candidate features based on associations between labels and features. A model is selected at each iteration, and then evaluation continues.

[0101] At block 730 , model performance is evaluated. For example, the model selected at the last iteration of the selection procedure 720 is cross-validated, such as on an outer loop of a nested cross-validation using the input data set 705 .

[0102] Figure 8 Provided Figure 7 700 of the example process 700. At block 810, a candidate feature F is identified in a first data partition. For example, the candidate feature F can be identified based on an association between a target label and an existing / identified feature. At block 820, a candidate feature F is identified for a model having two models M1 and M2. i , M2 i ={M1 i , F} performs nested cross validation (CV) on the first partition. That is, model M2 i Including Model M1 i features plus new candidate features F.

[0103] At block 830, the inner loop result is evaluated to test whether M2 i Is it better than M1? i For example, a binomial test is performed to assess M2 based on the results of the inner loop. i Is it better than M1? i If not, control returns to block 810 to identify another candidate feature in the first partition. i Better than M1 i, then at block 840, a permutation test is performed to assess whether M2 is on the second partition. i Is it better than M1? i For example, M2 can be evaluated based on the p-value i and M1 i , the p-value represents the part of the permutation of the candidate feature F, where M1 i The performance is not less than M2 i performance. When M2 i Performed “significantly better” than the M1 i , (e.g., with a standard significance level defined as 0.05 or 0.1), then, at block 850, select M2 i Otherwise, control returns to block 810 to identify another candidate feature in the first partition.

[0104] At block 860, the model size is evaluated (e.g., by comparing the current feature set to the model size input 715, etc.). If the model size has been reached, the process may proceed to block 730. However, if the model size has not yet been met, control returns to block 810 to select new candidate features in the next iteration of the example process.

[0105] Fig. 9 Provided Figure 7 The example implementation of the robust performance evaluation 730 of the example process 700 is shown in FIG. At block 730 , the finally selected model M1 is evaluated using nested cross validation (CV). i or M2 i performance. Fig. 9 As shown in the example of , the input data set 705 is used as an external test set to verify the performance of the selected model and its features. Thus, for example, the inner loop of the partitioned data can be used to evaluate various candidate models, while the outer loop of the test data can be used to verify the performance of the selected model.

[0106] Fig.10 An example process 1000 of preparing data from multiple sources for use in machine learning and / or other AI model training is illustrated. For example, database extraction and / or expert supervision can be used to generate data from / in an input data set 1005. Data can be collected from multiple structured and / or unstructured data sources to form the data set 1005.

[0107] At block 1010, one or more input data sources 1005 are preprocessed to prepare the input data (e.g., by cleaning and extracting features from the data, labeling features, etc.). The input data and resulting features may include smoking history, medication administration, (one or more) medical conditions, radiation treatment history, (one or more) laboratory measurements, anthropometric data, etc. For example, at block 1012, structured data (e.g., medication administration, medical conditions, laboratory measurements, clinical procedures, anthropometric data, etc.) from the input data set 1005 is filtered and cleaned in a common data format (e.g., a common data model such as the OMOP data model format, etc.). At block 1014, curated data (e.g., related to downstream tasks, smoking history, specific condition history, specific medical history, etc.) from the input data set 1005 is filtered and cleaned into a common data format.

[0108] The preprocessed data set forms the input to block 1020, where patient vectors are generated from the filtered, cleaned, aggregated data set. For example, feature filtering can be used to provide a model-agnostic set of aggregated patient vectors. Model-agnostic non-aggregate tables with temporal data structures can be used for feature filtering and vector generation. Patient vectors and / or non-aggregated tables can form part of an output data set 1025.

[0109] At block 1030, the interface filters and aggregates the data. For example, the data may be filtered and arranged around anchor points and / or other references for aggregation and forming a patient vector.

[0110] Fig.11 An example implementation of generating a patient vector is provided (e.g., box 1020 of example process 1000). At box 1110, the processed (e.g., filtered, cleaned, etc.) data is evaluated to determine whether a feature in the data is to be used? When the feature is not to be used, at box 1120, the feature is omitted. When the feature is to be used, at box 1130, the available data (source) is evaluated to determine whether multiple sources (e.g., structured and / or managed data sources, etc.) are to be used. When multiple data sources are to be used, at box 1140, features from multiple data sources are combined and coordinated. At box 1150, an unrestricted patient vector is formed from an unaggregated table that is agnostic to the downstream model that retains the temporal structure. For example, the intermediate data set output 1105 may include one or more data sets separated by the input domain and retaining the temporal structure.

[0111] Fig.12An example implementation of a filtering and aggregation interface is provided (e.g., block 1030 of the example process 1000). At 1210, the patient is evaluated to determine whether the patient is a patient for further evaluation and processing. For example, such a determination can be based at least in part on external patient eligibility criteria 1205 related to downstream tasks. For example, whether the patient meets certain health criteria, age criteria, condition criteria, likelihood of success, etc. When the patient is not eligible, at block 1220, the patient is omitted from further processing.

[0112] When the patient qualifies, at block 1230, an anchor is set for further evaluation. For example, the determination of the anchor can be based on downstream, task-related, external patient qualification criteria 1215. Anchors are patient timeline events that align multiple patients for comparison. For example, events in a patient's life and / or medical history (such as diagnosis, symptoms, birth, hospitalization, initial treatment, etc.) can be driven by the framework to align time series data for relational processing and comparison. For example, an anchor can be a configurable parameter associated with a downstream task 1205.

[0113] At block 1240, an aggregation period is set. For example, the framework may configure and / or otherwise set an aggregation period before a prediction point (e.g., an anchor point and / or other time point and / or patient's data). For example, the aggregation period may include an aggregation of patient data 200 days before the prediction time, and so on.

[0114] At block 1250, an aggregation period is set for the measurement. For example, the interface framework allows different aggregation periods to be constructed for laboratory measurements, other measurements, etc., since such measurements can change more dynamically.

[0115] At block 1260, one or more restricted aggregated patient vectors are formed as outputs of the interface framework. For example, the patient vectors may be customized for multiple downstream tasks. In some examples, for a given set of inputs, the patient vectors may be dynamically generated as downstream tasks change.

[0116] Fig.13 An example process 1300 for constructing a predictive model of efficacy associated with ICI is illustrated. Such a model can assess multiple efficacy surrogate endpoints to predict response to ICI therapy. Models can be created for individual cancer indications and / or combinations thereof.

[0117] At block 1310, input data from multiple sources is prepared for processing. For example, as described in more detail above, data can be cleaned and features extracted from sources such as structured data sets 1305, managed data sets 1315, etc. For example, raw, automatically exported EHR data can be converted into a common format with managed data to form a feature set.

[0118] At box 1320, a baseline true value prediction label is generated. The prediction label may include ICI treatment time (TOT), next treatment time (TNET) (e.g., after ICI cessation), overall survival (OS), etc. For example, the listed baseline true value endpoints can be derived from the input data 1325 and can be generated on a continuous scale, such as expressed in days past the anchor point. For example, the patient timeline can be aligned based on similarities in the ICI treatment process. For example, the default anchor point can be the first date of the start of ICI treatment. The generated baseline true value labels can be used as is and / or with a modified granularity (e.g., past weeks, months, years, etc.) for training regression, survival analysis-based models, etc. Discretization of the baseline true value can be performed for binary and / or multi-class classification (e.g., responders vs. non-responders, 5-year survival rate, etc.).

[0119] At box 1330, model building is performed on the generated feature matrix, where the benchmark truth values ​​are used as independent modules of the framework. In some examples, the efficacy endpoints can be modeled separately (e.g., individuals, etc.). For example, modeling can be performed without assumptions and / or using different machine learning algorithms. Alternatively or additionally, modeling can be completed on a continuous scale using survival analysis methods. In addition, for example, modeling can follow a multi-class classification method for all endpoints. Example model building and selection are further described above.

[0120] Fig.14 An example method 1400 for model selection and generation is illustrated. Fig.14 In some examples, model generation is driven at least in part by the intent, purpose, or objective of the model. For example, the objective can be to generate a model with a high (e.g., maximized) Fl ​​harmonic score (e.g., the harmonic mean of recall and precision) such as a GB model, a model with a high (e.g., maximized) recall (e.g., the ratio of true positives found) such as an RF model, and so on. In some examples, majority class undersampling can be combined with time series data aggregation to obtain a well-balanced and static data set to feed the model. In some examples, the model creates probability estimates for labels, not just the discrete labels themselves.

[0121] At box 1410, input is prepared, such as by extracting data from one or more EHR data tables 1405. Measurements such as liver biomarker concentrations in plasma, other blood concentration values, etc. can be extracted. For example, time series measurements and / or other data can be put into a single complex data structure (e.g., aggregation). The aggregated data can be processed using feature engineering to describe the time series data in terms of mean, standard deviation, minimum, maximum, etc. For example, lags and features can be created as well as labels. In some examples, dates of ICI treatment are generated 1415 to help align and anchor the time series data.

[0122] At block 1420, the goal or objective of the model(s) is evaluated. As described above, the objective of maximizing Fl results in a different training model (e.g., a gradient boosting model, such as a gradient boosting decision tree, etc.) than the objective of maximizing recall (e.g., a random forest model, etc.). Based on this determination, at block 1430, data set resampling occurs to balance the unbalanced data set from block 1410 for the purpose of maximizing recall. For example, when the objective is to maximize recall, random majority class undersampling is performed on the data set. When the objective is to maximize the Fl score, resampling may be omitted.

[0123] At blocks 1440 and / or 1450, the models are trained. For example, at block 1440, the RF model is trained for recall maximization, and at block 1450, the GB model is trained for Fl score maximization. For example, the respective models are validated, such as using leave-one-out cross validation, where an individual predicts each sample, where the remaining samples are used as a training data set.

[0124] Although example implementations are described in the present application and associated appendices, one or more of the elements, processes and / or devices described may be combined, divided, rearranged, omitted, eliminated and / or implemented in any other manner. In addition, the example elements may be implemented only by hardware, or by hardware in combination with software and / or firmware. Therefore, for example, any of the example elements may be implemented by processor circuitry, (one or more) analog circuits, (one or more) digital circuits, (one or more) logic circuits, (one or more) programmable processors, (one or more) programmable microcontrollers, (one or more) graphics processing units ((one or more) GPUs), (one or more) digital signal processors ((one or more) DSPs), (one or more) application specific integrated circuits ((one or more) ASICs), (one or more) programmable logic devices ((one or more) PLDs) and / or (one or more) field programmable logic devices ((one or more) FPLDs) such as field programmable gate arrays (FPGAs). Further, in addition to or in place of those described, the example elements may include one or more elements, processes and / or devices, and / or may include more than one of any or all of the elements, processes and devices described.

[0125] In some examples, hardware logic circuit systems, machine-readable instructions, hardware-implemented state machines, and / or any combination thereof may implement (one or more) systems and / or perform the methods disclosed herein. Machine-readable instructions may be one or more executable programs or (one or more) parts of executable programs for execution by a processor circuit system. The program may be embodied in software stored on one or more non-transitory computer-readable storage media, such as a compact disk (CD), a floppy disk, a hard disk drive (HDD), a solid-state drive (SSD), a digital versatile disk (DVD), a Blu-ray disc, a volatile memory (e.g., any type of random access memory (RAM), etc.), or a non-volatile memory (e.g., an electrically erasable programmable read-only memory (EEPROM), a flash memory, a HDD, a SSD, etc.) associated with a processor circuit system located in one or more hardware devices, but the entire program and / or its parts may alternatively be executed by one or more hardware devices other than the processor circuit system and / or embodied in firmware or dedicated hardware. Machine-readable instructions may be distributed across multiple hardware devices and / or executed by two or more hardware devices (e.g., a server and a client hardware device). For example, a client hardware device may be implemented by an endpoint client hardware device (e.g., a hardware device associated with a user) or an intermediate client hardware device (e.g., a radio access network (RAN) gateway that may facilitate communications between a server and an endpoint client hardware device). Similarly, a non-transitory computer-readable storage medium may include one or more media located in one or more hardware devices. In addition, the order of execution may be changed, and / or some of the boxes described may be changed, deleted, or combined. Additionally or alternatively, any or all code boxes may be implemented by one or more hardware circuits (e.g., processor circuit systems, discrete and / or integrated analog and / or digital circuit systems, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.) that are configured to perform corresponding operations without executing software or firmware. The processor circuit system can be distributed in different network locations and / or local to the following devices: one or more hardware devices (e.g., a single-core processor (e.g., a single-core central processing unit (CPU)), a multi-core processor in a single machine (e.g., a multi-core CPU, XPU, etc.), multiple processors distributed across multiple servers in a server rack, multiple processors distributed across one or more server racks, a CPU and / or FPGA located in the same package (e.g., the same integrated circuit (IC) package or two or more separate housings, etc.), and the like.

[0126] The machine-readable instructions described herein may be stored in one or more of a compressed format, an encrypted format, a segmented format, a compiled format, an executable format, a packaged format, etc. The machine-readable instructions as described herein may be stored as data or data structures (e.g., as parts of instructions, codes, code representations, etc.) that can be used to create, manufacture, and / or generate machine-executable instructions. For example, the machine-readable instructions may be segmented and stored on one or more storage devices and / or computing devices (e.g., servers) located at the same or different locations of a network or a collection of networks (e.g., in the cloud, in an edge device, etc.). The machine-readable instructions may need to be installed, modified, adapted, updated, combined, supplemented, configured, decrypted, decompressed, unpacked, distributed, reassigned, compiled, etc., so that they can be directly read, interpreted, and / or executed by a computing device and / or other machine. For example, the machine-readable instructions may be stored in multiple parts that are individually compressed, encrypted, and / or stored on separate computing devices, wherein the parts, when decrypted, decompressed, and / or combined, form a set of machine-executable instructions that implement one or more operations, which together may form a program such as the program described herein.

[0127] In another example, the machine-readable instructions may be stored in a state where they are readable by the processor circuitry, but require the addition of a library (e.g., a dynamic link library (DLL)), a software development kit (SDK), an application programming interface (API), etc., in order to execute the machine-readable instructions on a particular computing device or other device. In another example, the machine-readable instructions may need to be configured (e.g., stored settings, data inputs, recorded network addresses, etc.) before the machine-readable instructions and / or corresponding program(s) can be executed in whole or in part. Thus, as used herein, a machine-readable medium may include machine-readable instructions and / or program(s) regardless of the particular format or state of the machine-readable instructions and / or program(s) when stored or otherwise at rest or in transport.

[0128] The machine-readable instructions described herein may be represented by any past, present, or future instruction language, scripting language, programming language, etc. For example, the machine-readable instructions may be represented using any of the following languages: C, C++, Java, C#, Perl, Python, JavaScript, Hypertext Markup Language (HTML), Structured Query Language (SQL), Swift, etc.

[0129] As mentioned above, the example operations disclosed herein may be implemented using executable instructions (e.g., computer and / or machine readable instructions) stored on one or more non-transitory computer and / or machine readable media (such as optical storage devices, magnetic storage devices, HDDs, flash memory, read-only memory (ROM), CDs, DVDs, caches, any type of RAM, registers, and / or any other storage devices or storage disks where information is stored for any duration (e.g., for an extended period of time, permanently, for a brief moment, for temporary buffering, and / or for caching of information)). As used herein, the terms non-transitory computer-readable medium, non-transitory computer-readable storage medium, non-transitory machine-readable medium, and non-transitory machine-readable storage medium are expressly defined to include any type of computer-readable storage device and / or storage disk, and exclude propagating signals and exclude transmission media. As used herein, the terms "computer-readable storage device" and "machine-readable storage device" are defined to include any physical (mechanical and / or electrical) structure for storing information, but exclude propagating signals and exclude transmission media. Examples of computer-readable storage devices and machine-readable storage devices include any type of random access memory, any type of read-only memory, solid-state memory, flash memory, optical disks, magnetic disks, disk drives, and / or redundant arrays of independent disks (RAID) systems. As used herein, the term "device" refers to a physical structure, such as a mechanical and / or electrical device, hardware, and / or circuit system, which may or may not be configured by computer-readable instructions, machine-readable instructions, etc., and / or is manufactured to execute computer-readable instructions, machine-readable instructions, etc.

[0130] "Include" and "comprising" (and all forms and tenses thereof) are used herein as open-ended terms. Thus, whenever a claim adopts any form of "include" or "comprising" (e.g., comprises, comprising, including, having, etc.) as an introduction or within any kind of claim recitation, it is to be understood that additional elements, terms, etc. may be present without falling outside the scope of the corresponding claim or recitation. As used herein, when the phrase "at least" is used as a transitional term, such as in the introduction of a claim, it is open-ended in the same manner as the terms "include" and "comprising" are open-ended. The term "and / or" when used, for example, in a form such as A, B, and / or C, refers to any combination or subset of A, B, C, such as (1) only A, (2) only B, (3) only C, (4) A and B, (5) A and C, (6) B and C, or (7) A and B and C. As used herein in the context of describing structures, components, items, objects, and / or things, the phrase “at least one of A and B” is intended to refer to implementations that include any of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in the context of describing structures, components, items, objects, and / or things, the phrase “at least one of A or B” is intended to refer to implementations that include any of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. As used herein in the context of describing the execution or performance of processes, instructions, acts, activities, and / or steps, the phrase “at least one of A and B” is intended to refer to implementations that include any of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in the context of describing the execution or performance of a process, instruction, action, activity, and / or step, the phrase "at least one of A or B" is intended to refer to an implementation that includes any of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.

[0131] As used herein, singular references (e.g., "a," "an," "first," "second," etc.) do not exclude the plural. As used herein, the term "a" or "an" object refers to one or more of that object. The terms "a" (or "an"), "one or more," and "at least one" are used interchangeably herein. In addition, although individually listed, multiple components, elements, or method actions may be implemented by, for example, the same entity or object. In addition, although individual features may be included in different examples or claims, these may be combined, and the inclusion in different examples or claims does not imply that the combination of features is not feasible and / or advantageous.

[0132] Fig.15 1 is a block diagram of an example processor platform 1500 that is configured to execute and / or instantiate the machine-readable instructions and / or operations disclosed and described herein. The processor platform 1500 may be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cellular phone, a smart phone, a tablet such as an iPad™), a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a game console, a personal video recorder, a set-top box, a headset (e.g., an augmented reality (AR) headset, a virtual reality (VR) headset, etc.) or other wearable device, or any other type of computing device.

[0133] The processor platform 1500 of the illustrated example includes a processor circuit system 1512. The processor circuit system 1512 of the illustrated example is hardware. For example, the processor circuit system 1512 can be implemented by one or more integrated circuits, logic circuits, FPGAs, microprocessors, CPUs, GPUs, DSPs, and / or microcontrollers from any desired series or manufacturer. The processor circuit system 1512 can be implemented by one or more semiconductor-based (e.g., silicon-based) devices.

[0134] The processor circuit system 1512 of the illustrated example includes a local memory 1513 (e.g., cache, registers, etc.). The processor circuit system 1512 of the illustrated example communicates with a main memory (including a volatile memory 1514 and a non-volatile memory 1516) via a bus 1518. The volatile memory 1514 may be a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), Dynamic Random Access Memory The non-volatile memory 1516 may be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 1514, 1516 of the illustrated example is controlled by a memory controller 1517.

[0135] The processor platform 1500 of the illustrated example also includes an interface circuit system 1520. The interface circuit system 1520 may be implemented by hardware according to any type of interface standard, such as an Ethernet interface, a universal serial bus (USB) interface, interface, a near field communication (NFC) interface, a peripheral component interconnect (PCI) interface, and / or a peripheral component interconnect express (PCIe) interface.

[0136] In the illustrated example, one or more input devices 1522 are connected to the interface circuitry 1520. The input device(s) 1522 permit a user to enter data and / or commands into the processor circuitry 1512. The input device(s) 1522 can be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touch screen, a trackpad, a trackball, an isochronous device, and / or a speech recognition system.

[0137] One or more output devices 1524 are also connected to the interface circuit system 1520 of the illustrated example. The output device(s) 1524 can be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube (CRT) display, an in-place switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. Thus, the interface circuit system 1520 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics processor circuit (such as a GPU).

[0138] The interface circuitry 1520 of the illustrated example also includes communication devices (such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces) to facilitate the exchange of data with external machines (e.g., any kind of computing device) over a network 1526. Communications can occur over, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a direct-line wireless system, a cellular telephone system, an optical connection, etc.

[0139] The processor platform 1500 of the illustrated example also includes one or more mass storage devices 1528 that store software and / or data. Examples of such mass storage devices 1528 include magnetic storage devices, optical storage devices, floppy disk drives, HDDs, CDs, Blu-ray disk drives, redundant array of independent disks (RAID) systems, solid-state storage devices (e.g., flash memory devices and / or SSDs), and DVD drives.

[0140] The example computer-readable instructions 1532 may be stored in the mass storage device 1528, in the volatile memory 1514, in the non-volatile memory 1516, and / or on a removable non-transitory computer-readable storage medium such as a CD or DVD.

[0141] Fig.16 yes Fig.15 1512. In this example, Fig.15The processor circuit system 1512 is implemented by a microprocessor 1600. For example, the microprocessor 1600 may be a general-purpose microprocessor (e.g., a general-purpose microprocessor circuit). The microprocessor 1600 executes some or all of the machine-readable instructions to effectively instantiate the circuit system described herein as a logic circuit so as to perform operations corresponding to those machine-readable instructions. In some such examples, the circuit system is instantiated by the hardware circuit of the microprocessor 1600 in combination with the instructions. For example, the microprocessor 1600 may be implemented by a multi-core hardware circuit system (such as a CPU, DSP, GPU, XPU, etc.). Although it may include any number of example cores 1602 (e.g., 1 core), the microprocessor 1600 of this example is a multi-core semiconductor device including N cores. The cores 1602 of the microprocessor 1600 may operate independently or may cooperate to execute machine-readable instructions. For example, a machine code corresponding to a firmware program, an embedded software program, or a software program may be executed by one of the cores 1602, or may be executed by multiple cores in the core 1602 at the same or different times. In some examples, machine code corresponding to a firmware program, embedded software program, or software program is split into threads and executed in parallel by two or more of the cores 1602. The software program may correspond to a portion or all of the machine-readable instructions and / or operations disclosed herein.

[0142] The core 1602 can communicate via a first example bus 1604. In some examples, the first bus 1604 can be implemented by a communication bus to implement communications associated with (one or more) cores in the core 1602. For example, the first bus 1604 can be implemented by at least one of an inter-integrated circuit (I2C) bus, a serial peripheral interface (SPI) bus, a PCI bus, or a PCIe bus. Additionally or alternatively, the first bus 1604 can be implemented by any other type of computing or electrical bus. The core 1602 can obtain data, instructions, and / or signals from one or more external devices via the example interface circuit system 1606. The core 1602 can output data, instructions, and / or signals to one or more external devices via the example interface circuit system 1606. While the cores 1602 of this example include example local memory 1620 (e.g., a level 1 (L1) cache that may be split into a level 1 (L1) data cache and a level 1 (L2) instruction cache), the microprocessor 1600 also includes example shared memory 1610 (e.g., a level 2 (L2) cache) that may be shared by the cores for high-speed access to data and / or instructions. Data and / or instructions may be transferred (e.g., shared) by writing to and / or reading from the shared memory 1610. The local memory 1620 and the shared memory 1610 of each of the cores 1602 may be part of a hierarchy of storage devices that include cache memory and main memory (e.g., Fig.15 1514, 1516). Typically, higher levels of memory in the hierarchy exhibit lower access times and have smaller storage capacities than lower levels of memory. Changes to the various levels of the cache hierarchy are managed (e.g., coordinated) by a cache coherence policy.

[0143] Each core 1602 may be referred to as a CPU, DSP, GPU, etc., or any other type of hardware circuit system. Each core 1602 includes a control unit circuit system 1614, an arithmetic and logic (AL) circuit system (sometimes referred to as an ALU) 1616, a plurality of registers 1618, a local memory 1620, and a second example bus 1622. Other structures may exist. For example, each core 1602 may include a vector unit circuit system, a single instruction multiple data (SIMD) unit circuit system, a load / store unit (LSU) circuit system, a branch / jump unit circuit system, a floating point unit (FPU) circuit system, etc. The control unit circuit system 1614 includes a semiconductor-based circuit configured to control (e.g., coordinate) data movement within the corresponding core 1602. The AL circuit system 1616 includes a semiconductor-based circuit configured to perform one or more mathematical and / or logical operations on the data within the corresponding core 1602. The AL circuit system 1616 of some examples performs integer-based operations. In other examples, the AL circuit system 1616 also performs floating-point operations. In still other examples, AL circuit system 1616 may include a first AL circuit system that performs integer-based operations and a second AL circuit system that performs floating-point operations. In some examples, AL circuit system 1616 may be referred to as an arithmetic logic unit (ALU). Register 1618 is a semiconductor-based structure for storing data and / or instructions (such as the results of one or more of the operations performed by AL circuit system 1616 of corresponding core 1602). For example, register 1618 may include (one or more) vector registers, (one or more) SIMD registers, (one or more) general registers, (one or more) flag registers, (one or more) segment registers, (one or more) machine-specific registers, (one or more) instruction pointer registers, (one or more) control registers, (one or more) debug registers, (one or more) memory management registers, (one or more) machine check registers, etc. Register 1618 may be arranged in a manner such as Fig.16 1602. Alternatively, registers 1618 may be organized in any other arrangement, format, or structure, including being distributed throughout core 1602 to reduce access time. Second bus 1622 may be implemented by at least one of an I2C bus, an SPI bus, a PCI bus, or a PCIe bus.

[0144] Each core 1602 and / or more generally the microprocessor 1600 may include additional and / or alternative structures to those shown and described above. For example, one or more clock circuits, one or more power supply devices, one or more power gates, one or more cache home agents (CHA), one or more aggregation / common grid stops (CMS), one or more shifters (e.g., (one or more) barrel shifters) and / or other circuit systems may exist. The microprocessor 1600 is a semiconductor device made to include many transistors, which are interconnected to implement the structure described above in one or more integrated circuits (ICs) contained in one or more packages. The processor circuit system may include one or more accelerators and / or cooperate with one or more accelerators. In some examples, the accelerator is implemented by a logic circuit system to perform certain tasks faster and / or more efficiently than what can be done by a general-purpose processor. Examples of accelerators include ASICs and FPGAs, such as those discussed herein. GPUs or other programmable devices may also be accelerators. The accelerator may be on a board of the processor circuit system, in the same chip package as the processor circuit system, and / or in one or more packages separated from the processor circuit system.

[0145] Fig.17 yes Fig.15 15. In this example, processor circuit system 1512 is implemented by FPGA circuit system 1700. For example, FPGA circuit system 1700 can be implemented by an FPGA. FPGA circuit system 1700 can be used, for example, to perform operations that could otherwise be performed by a processor executing corresponding machine-readable instructions. Fig.16 However, once configured, FPGA circuitry 1700 instantiates machine-readable instructions in hardware and can therefore often perform operations faster than if the operations could be performed by a general-purpose microprocessor executing corresponding software.

[0146] More specifically, Fig.16 Compared to the microprocessor 1600 of FIG. 1 (which is a general-purpose device that is programmed to execute some or all of the machine-readable instructions disclosed herein but whose interconnections and logic circuitry are fixed once fabricated), Fig.17The example FPGA circuit system 1700 includes interconnections and logic circuit systems that can be configured and / or interconnected in different ways after fabrication to instantiate, for example, some or all of the machine-readable instructions disclosed herein. In particular, the FPGA circuit system 1700 can be considered as an array of logic gates, interconnections, and switches. The switches can be programmed to change how the logic gates are interconnected by interconnections, thereby effectively forming one or more dedicated logic circuits (unless and until the FPGA circuit system 1700 is reprogrammed). The configured logic circuits enable the logic gates to cooperate in different ways to perform different operations on the data received by the input circuit system. Those operations may correspond to some or all of the software disclosed herein. Therefore, the FPGA circuit system 1700 can be configured to effectively instantiate some or all of the machine-readable instructions as dedicated logic circuits so as to perform operations corresponding to those software instructions in a dedicated manner similar to an ASIC. Therefore, the FPGA circuit system 1700 can perform operations corresponding to some or all of the machine-readable instructions faster than when a general-purpose microprocessor can perform operations corresponding to some or all of the machine-readable instructions.

[0147] exist Fig.17 In the example of FPGA circuit system 1700, FPGA circuit system 1700 is configured to be programmed (and / or reprogrammed one or more times) by an end user via a hardware description language (HDL) such as Verilog. Fig.17 FPGA circuit system 1700 includes an example input / output (I / O) circuit system 1702, which is used to obtain data from and / or output data to example configuration circuit system 1704 and / or external hardware 1706. For example, configuration circuit system 1704 can be implemented by an interface circuit system, which can obtain machine-readable instructions to configure FPGA circuit system 1700 or (one or more) parts thereof. In some such examples, configuration circuit system 1704 can obtain machine-readable instructions from a user, a machine (e.g., a hardware circuit system (e.g., a programming or dedicated circuit system) that can implement an artificial intelligence / machine learning (AI / ML) model to generate instructions), etc. In some examples, external hardware 1706 can be implemented by an external hardware circuit system. For example, external hardware 1706 can be implemented by Fig.16 FPGA circuit system 1700 also includes an array of example logic gate circuit system 1708, multiple example configurable interconnects 1710, and example storage circuit system 1712. Logic gate circuit system 1708 and interconnect 1710 are configurable to instantiate one or more operations that may correspond to at least some of the machine-readable instructions and / or other desired operations. Fig.17The logic gate circuit system 1708 shown in is made in groups or blocks. Each block includes a semiconductor-based electrical structure that can be configured as a logic circuit. In some examples, the electrical structure includes a logic gate (e.g., an AND gate, an OR gate, a NOR gate, etc.) that provides a basic building block for the logic circuit. An electrically controllable switch (e.g., a transistor) is present in each of the logic gate circuit systems 1708 to enable the configuration of the logic gates and / or electrical structures to form a circuit that performs the desired operation. The logic gate circuit system 1708 may include other electrical structures, such as a lookup table (LUT), a register (e.g., a flip-flop or a latch), a multiplexer, etc.

[0148] The illustrated example configurable interconnect 1710 is a conductive path, trace, via, or the like, which may include an electrically controllable switch (e.g., a transistor) whose state can be changed by programming (e.g., using an HDL instruction language) to activate or deactivate one or more connections between one or more of the logic gate circuit systems 1708 in order to program the desired logic circuit.

[0149] The storage circuit system 1712 of the illustrated example is configured to store (one or more) results of one or more of the operations performed by the corresponding logic gates. The storage circuit system 1712 can be implemented by registers, etc. In the illustrated example, the storage circuit system 1712 is distributed among the logic gate circuit system 1708 to facilitate access and increase execution speed.

[0150] Fig.17 The example FPGA circuit system 1700 also includes an example special operation circuit system 1714. In this example, the special operation circuit system 1714 includes a special circuit system 1716, which can be called to implement common functions to avoid the need to program those functions on site. Examples of such special circuit systems 1716 include memory (e.g., DRAM) controller circuit systems, PCIe controller circuit systems, clock circuit systems, transceiver circuit systems, memory, and multiplier-accumulator circuit systems. Other types of special circuit systems may exist. In some examples, the FPGA circuit system 1700 may also include an example general programmable circuit system 1718, such as an example CPU 1720 and / or an example DSP 1722. Additionally or alternatively, other general programmable circuit systems 1718 such as GPUs, XPUs, etc. may exist, which can be programmed to perform other operations.

[0151] Although Fig.16 and Fig.17 Pictured Fig.15 These are two example implementations of the processor circuitry 1512, but many other implementations are contemplated. For example, as mentioned above, a modern FPGA circuitry may include an onboard CPU, such as Fig.17 One or more of the example CPUs 1720. Thus, Fig.15 The processor circuit system 1512 can additionally be combined by Fig.16 An example microprocessor 1600 and Fig.17 In some such hybrid examples, the first portion of the machine-readable instructions may be implemented by Fig.16 The second portion of the machine-readable instructions may be executed by one or more of the cores 1602 of Fig.17 The FPGA circuit system 1700 of the present invention may be executed by the FPGA circuit system 1700, and / or the third part of the machine-readable instructions may be executed by the ASIC. It should be understood that, therefore, some or all of the circuit system may be instantiated at the same or different times. For example, some or all of the circuit system may be instantiated in one or more threads executed concurrently and / or serially. In addition, in some examples, some or all of the circuit system may be implemented in one or more virtual machines and / or containers executed on a microprocessor.

[0152] In some examples, Fig.15 The processor circuit system 1512 may be in one or more packages. For example, Fig.16 The processor circuit system 1600 and / or Fig.17 The FPGA circuit system 1700 may be in one or more packages. In some examples, the XPU may be composed of a Fig.15 The XPU may be implemented by processor circuitry 1512. For example, the XPU may include a CPU in one package, a DSP in another package, a GPU in yet another package, and an FPGA in yet another package.

[0153] The diagram is used to Fig.15 A block diagram of an example software distribution platform 1805 for distributing example machine readable instructions 1532 such as software to a hardware device owned and / or operated by a third party is provided in Fig.18 The example software distribution platform 1805 may be implemented by any computer server, data facility, cloud service, etc. that is capable of storing software and delivering software to other computing devices. A third party may be a customer of the entity that owns and / or operates the software distribution platform 1805. For example, the entity that owns and / or operates the software distribution platform 1805 may be a Fig.15The third party may be a consumer, user, retailer, OEM, etc. who purchases the software and / or licenses the software for use and / or resale and / or sublicense. In the illustrated example, the software distribution platform 1805 includes one or more servers and one or more storage devices. As described above, the storage device stores the machine-readable instructions 1532, which may correspond to Fig.15 The example machine-readable instructions 1532 of the example software distribution platform 1805 are provided. One or more servers of the example software distribution platform 1805 communicate with the example network 1810, which may correspond to any one or more of the Internet and / or any of the example networks described above. In some examples, as part of a business transaction, one or more servers respond to a request to transfer software to a requesting party. The delivery, sale, and / or payment of a license for the software may be processed by one or more servers of the software distribution platform and / or by a third-party payment entity. The server enables a purchaser and / or licensee to download the machine-readable instructions 1532 from the software distribution platform 1805. For example, the server may correspond to Fig.15 The software of the example machine readable instructions 1532 can be downloaded to the example processor platform 1500, which is to execute the machine readable instructions 1532 to implement the systems and methods described herein. In some examples, one or more servers of the software distribution platform 1805 periodically provide, transmit and / or force updates to the software (e.g., the example machine readable instructions 1532) to ensure that improvements, patches, updates, etc. are distributed and applied to the software at the end-user devices.

[0154] In some examples, deployed models and / or patient data may be uploaded via a cloud-based platform 1805 for remote execution, rather than downloading machine-readable instructions 1832 to a local processor platform 1800. In some examples, the example platform 1805 may host one or more models accessible by a network 1810, and the processor platform 1500 may provide input to the models and receive results, predictions, and / or other outputs.

[0155] From the foregoing, it will be appreciated that example systems, methods, devices, and articles of manufacture that implement model generation and deployment to drive processes for therapy prediction and treatment execution have been disclosed. The disclosed systems, methods, devices, and articles of manufacture improve the efficiency of using computing devices by implementing model generation and deployment to drive processes for therapy prediction and treatment execution. Thus, the disclosed systems, methods, devices, and articles of manufacture are directed to one or more improvements in the operation of machines such as computers or other electronic and / or mechanical devices.

[0156] Certain examples provide systems and methods for training and evaluating AI models to predict adverse risk events based on fixed-length patient history data. The example system and associated methods include submodules for extracting patient blood test history from electronic medical records, cleaning history, and performing data quality checks. The example system and associated methods include a label definition submodule, which instantiates an algorithm for assigning hepatitis adverse event levels to blood test value sets (e.g., ALT, AST, TBILIRUBIN, ALKPHOS) and creating binary target labels for the AI ​​model. The example system and method include a feature engineering submodule, which is used to regularize blood test values ​​to the upper limit of normal values ​​(e.g., specific to patients, laboratories, and / or blood tests, etc.). The example feature engineering submodule is to transform the normalized value into a discrete symbolic representation, such as a modified version of a symbol set approximation, etc. The example feature engineering submodule is to extract motifs in the form of n-grams from a symbol sequence, and use counts in recent patient history as features. The example system and method include submodules for training and evaluating an AI model to dynamically predict immune-related hepatitis adverse event risks from a fixed-length blood test history.

[0157] Certain examples provide systems and methods for building classification models (e.g., pneumonia classification models, etc.) using sequential procedures. Example systems and methods include preprocessing structured EHRs and unstructured data tables. For example, at the time of the first ICI administration, patient timelines are aligned. Statistics are used to aggregate laboratory measurements within a time window (e.g., a 60-day time window, etc.) before the first ICI. For example, other features (e.g., conditions, smoking status, etc.) can use different time windows (e.g., a 1-year time window, etc.).

[0158] Example systems and methods include finding patterns in data to identify potential predictive features associated with the development of ICI-related toxicities such as pneumonia. In the case of a small and noisy data set, many data points are utilized. The data is segmented with a first partition (e.g., 90% partition, 80% partition, 95% partition, etc.) to identify candidate features based on the association between pneumonia labels and features.

[0159] Example systems and methods include deciding between two model versions at each iteration of the procedure, one with the original feature set (M1) and one expanded with candidate features (M2). Nested cross validation is performed on the first (e.g., 90%, etc.) partitions, and the inner loop results are used to compare M1 and M2. A binomial test is performed to assess whether M2 is significantly better than M1.

[0160] Example systems and methods include assessing whether M2 has better performance on a second partition (e.g., 10% partition, 5% partition, 20% partition, etc.) that is retained when M2 is significantly better in step 3). A permutation test is performed to estimate the probability of observing better performance by random chance alone. This step serves as a safety measure to avoid overfitting to the first (e.g., 90%, etc.) data partition. If, based on step 4), M2 has significantly better performance, then M2 is selected.

[0161] The example system and method includes continuing to test new candidate features until the desired model size is reached. The performance of the final model is assessed on an outer loop. In this way, a performance estimator with a smaller variance is obtained, and the variability of the test predictions and the instability of the model can be assessed. If the final model has promising performance, the model is evaluated on a sufficiently large external test set.

[0162] Certain examples provide systems and methods that form an automated framework to prepare multi-source electronic health record data for use in machine learning model training. Example methods and associated systems include preparing input data from multiple sources: This portion of the framework focuses on cleaning and extracting features from multiple data sources. This step receives raw, automatically exported EHR data in a data model format (e.g., OMOP data model format, etc.) and multiple expert-curated data sources for additional features and labels. This step is open to extension and includes, but is not limited to, modules for preparing smoking history, medication administration, medical conditions, radiation therapy history, laboratory measurements, and anthropometric data.

[0163] Example methods and associated systems include generating a combined time-correlated unrestricted intermediate output. This step compresses the input data in a uniform format, preserving the timestamps of the individual data items for each patient. This intermediate step provides plug-in possibilities for modules that prepare time-correlated input data for sequence modeling algorithms (not implemented) and provides flexible inputs for the aggregation step of the framework.

[0164] Example methods and associated systems include Aggregate Features for Time-Independent Models. This step is a target-agnostic, highly configurable plug-in module for creating time-independent, aggregated inputs for machine learning models. This step is open to extension, and configurable parameters include, but are not limited to, prediction time points, aggregation length, data sources involved, and feature types involved.

[0165] Certain examples provide systems and methods for building predictive models for efficacy surrogate endpoints associated with immune checkpoint inhibitor therapy. Example methods and associated systems include preparing input data from multiple sources. This part of the framework involves cleaning and extracting features from multiple data sources. For example, this step receives raw, automatically exported EHR data in the OMOP data model format, as well as multiple expert-managed data sources for additional features. Example methods and associated systems include generating benchmark true value prediction labels, such as ICI treatment time (TOT), next treatment time (after ICI interruption) (TNET), overall survival (OS), etc. The listed benchmark true value endpoints are generated on a continuous scale, expressed in days past the anchor point (patient timelines are aligned based on similarities during ICI treatment). The default anchor point is the first date of ICI treatment initiation. The generated benchmark true values ​​can be used as is, or used to train regression or survival analysis-based models at a modified granularity (past weeks, months, years, etc.). Discretization of the ground truth may be performed for binary or multi-class classifications (eg, responders vs. non-responders, 5-year survival rate, etc.).

[0166] Example methods and associated systems include model building based on generated feature matrices and ground truth values ​​as independent modules of the framework. Endpoints can be modeled individually. For example, modeling can be performed without assumptions and with different machine learning algorithms. Model building and selection workflows can be used to generate predictive models for immune checkpoint inhibitor-associated pneumonia and sequential procedures for model building.

[0167] Certain examples provide systems and methods for hepatitis prediction. The example system and associated methods include preparing inputs by extracting relevant portions of blood features from received electronic health record (EHR) data tables. These are measurements of liver biomarker concentrations in plasma (such as ALT, AST, alkaline phosphatase, and bilirubin) and other concentration values ​​in blood. This step is followed by placing the time series data into a single complex data structure, which is an efficient option to then continue to aggregate this information into a final data table, now ready for preprocessing and transformation steps. The aggregation step of feature engineering includes describing the time series data of blood particles with their mean, standard deviation, minimum, and maximum values. Lag features can also be created by taking the last available liver biomarker measurements before treatment. Use definitions obtained from medical experts to create labels, for example, when the level of at least one liver biomarker exceeds 3 times the upper limit of normal within a predetermined window, someone is classified as positive, otherwise negative. The date of ICI treatment used in this scheme can be output from different workflows.

[0168] Example systems and methods include dataset resampling. For example, the dataset obtained from step 1 is unbalanced, so if the goal is to maximize the recall value, a random majority class us is performed on the dataset. If the Fl score is the maximized theme, the resampling step is ignored.

[0169] The example system and method include training and prediction. For example, on the final data obtained from step 2, the model is trained, the model is RF for maximizing the recall rate, and GB for maximizing the Fl score. Validation is performed using leave-one-out cross-validation, where each sample is individually predicted, and the remaining samples are used as training data sets.

[0170] Further aspects of the disclosure are provided by the subject matter of the following clauses:

[0171] Example 1 is a device comprising: a memory circuit system; instructions; and a processor circuit system for executing the instructions to: process input data extracted from a record to form a candidate feature set; use the candidate feature set to train at least a first model and a second model; at least test the first model and the second model to compare the performance of the first model and the second model; select at least one of the first model or the second model based on the comparison; store at least one of the selections in the first model or the second model; and deploy at least one of the selections in the first model or the second model to predict the likelihood of at least one of the following: a) toxicity due to immunotherapy according to a treatment plan, or b) the effectiveness of the treatment plan on a patient.

[0172] Example 2 includes the apparatus of any of the preceding clauses, further comprising deploying at least one of the selections in the first model or the second model in a tool having an interface to facilitate collection of patient data and interaction with at least one of the selections in the first model or the second model.

[0173] Example 3 includes the apparatus of any preceding clause, wherein the input data comprises at least one of a laboratory test result, a diagnosis code, or a billing code.

[0174] Example 4 includes the apparatus of any preceding clause, wherein the toxicity comprises at least one of pneumonia, colitis, or hepatitis.

[0175] Example 5 includes the apparatus of any preceding clause, wherein the effectiveness of a treatment plan for a patient is measured by at least one of patient survival or treatment time.

[0176] Example 6 includes the apparatus of any preceding clause, wherein the processor circuitry is to extract and organize the input data in a time series.

[0177] Example 7 includes the apparatus of any preceding clause, wherein the processor circuitry is to align the input data relative to the anchor point to organize the input data in a time series.

[0178] Example 8 includes the apparatus of any preceding clause, wherein the processor circuitry is to generate labels for input data to form a candidate feature set.

[0179] Example 9 includes the apparatus of any preceding clause, wherein the processor circuitry is to feature engineer the candidate feature set by at least one of regularizing, transforming, or extracting from the candidate feature set.

[0180] Example 10 includes an apparatus as described in any of the preceding clauses, wherein the processor circuit system is to select from a set of candidate features to form a set of patient features to train or test at least one of the first model and the second model based on feature engineering.

[0181] Example 11 includes an apparatus as described in any of the preceding clauses, wherein the processor circuit system is to generate a feature matrix based on feature engineering to at least train or test at least one of the first model and the second model.

[0182] Example 12 includes the apparatus of any preceding clause, wherein the processor circuitry is to deploy the selected at least one of the first model or the second model as an executable tool having an interface.

[0183] Example 13 includes at least one computer-readable storage medium comprising instructions that, when executed by a processor circuit system, cause the processor circuit system to at least: process input data extracted from a record to form a candidate feature set; use the candidate feature set to train at least a first model and a second model; test at least the first model and the second model to compare the performance of the first model and the second model; select at least one of the first model or the second model based on the comparison; store at least one of the selections in the first model or the second model; and deploy at least one of the selections in the first model or the second model to predict the likelihood of at least one of: a) toxicity resulting from immunotherapy according to a treatment plan, or b) the efficacy of the treatment plan on a patient.

[0184] Example 14 includes at least one computer-readable storage medium as described in any of the preceding clauses, wherein the instructions, when executed, cause the processor circuit system to deploy at least one of the selections of the first model or the second model in a tool having an interface to facilitate collection of patient data and interaction with at least one of the selections of the first model or the second model.

[0185] Example 15 includes at least one computer-readable storage medium of any preceding clause, wherein the instructions, when executed, cause the processor circuitry to extract and organize the input data in a time series relative to an anchor point.

[0186] Example 16 includes at least one computer-readable storage medium of any preceding clause, wherein the instructions, when executed, cause the processor circuitry to generate labels for input data to form a candidate feature set.

[0187] Example 17 includes at least one computer-readable storage medium as described in any of the preceding clauses, wherein the instructions, when executed, cause the processor circuit system to perform feature engineering on the candidate feature set by at least one of regularizing, transforming, or extracting from the candidate feature set.

[0188] Example 18 is a computer-implemented method comprising: processing input data extracted from a record to form a set of candidate features; using the set of candidate features to train at least a first model and a second model; testing at least the first model and the second model to compare the performance of the first model and the second model; selecting at least one of the first model or the second model based on the comparison; storing at least one of the selections in the first model or the second model; and deploying at least one of the selections in the first model or the second model to predict the likelihood of at least one of: a) toxicity resulting from immunotherapy according to a treatment plan, or b) the efficacy of the treatment plan on a patient.

[0189] Example 19 includes the method of any preceding clause, wherein deploying comprises deploying at least one of the selections in the first model or the second model in a tool having an interface to facilitate collection of patient data and interaction with at least one of the selections in the first model or the second model.

[0190] Example 20 includes the method of any preceding clause, further comprising extracting and organizing the input data in a time series relative to an anchor point.

[0191] Example 21 includes the method of any preceding clause, further comprising generating labels for the input data to form a candidate feature set.

[0192] Example 22 includes the method of any preceding clause, further comprising feature engineering the candidate feature set by at least one of regularizing, transforming, or extracting from the candidate feature set.

[0193] Example 23 is a device comprising: a memory circuit system; instructions; multiple models for predicting at least one of: a) toxicity in response to immunotherapy or b) the efficacy of immunotherapy, the multiple models being trained and validated using data from previous patients; and a processor circuit system for executing instructions to: accept input of data associated with a first patient via an interface; use at least one of the multiple models to generate a prediction of at least one of: a) toxicity produced during immunotherapy according to a treatment plan for the first patient, or b) the efficacy of the treatment plan on the first patient; and output a recommendation regarding the treatment plan for the first patient.

[0194] Example 24 includes the apparatus of any preceding clause, wherein the toxicity comprises at least one of pneumonia, colitis, or hepatitis.

[0195] Example 25 includes the apparatus of any preceding clause, wherein therapeutic efficacy is defined relative to patient survival.

[0196] Example 26 includes the apparatus of any preceding clause, wherein efficacy is measured by progression-free survival or by time on treatment.

[0197] Example 27 includes the apparatus of any preceding clause, wherein both a prediction of toxicity and a prediction of efficacy are generated for the first patient to enable assessment of a risk-to-benefit ratio for the first patient relative to a treatment plan.

[0198] Example 28 includes the apparatus of any preceding clause, wherein the processor circuitry is to extract and organize patient-associated data in a time series for provision to the model.

[0199] Example 29 includes the apparatus of any preceding clause, wherein the processor circuitry is to align data associated with the patient relative to the anchor point to organize the patient data in a time series.

[0200] Example 30 includes the apparatus of any preceding clause, wherein the treatment plan comprises a clinical trial involving the first patient.

[0201] Example 31 includes the apparatus of any preceding clause, wherein the treatment plan is part of clinical care for the first patient.

[0202] Example 32 includes an apparatus as described in any of the preceding clauses, wherein the input is a first data input at a first time and the prediction is a first prediction, and wherein the processor circuit system is to process a second data input from a first patient at a second time to generate a second prediction using the model, and the processor circuit system is to compare the second prediction with the first prediction to adjust the recommended output for the patient.

[0203] Example 33 includes the apparatus of any preceding clause, wherein the first prediction is used by at least one of the plurality of models to generate the second prediction.

[0204] Example 34 includes the apparatus of any preceding clause, wherein the processor circuitry is to compare the second prediction, the first prediction, and the image data to adjust a recommendation output for the patient.

[0205] Example 35 includes the apparatus of any of the preceding clauses, further comprising an interface circuit system for connecting to an electronic medical record to perform at least one of retrieving data associated with the first patient or storing the prediction.

[0206] Example 36 includes the apparatus of any preceding clause, wherein the processor circuitry is to obtain feedback regarding the recommendation for adjusting the model.

[0207] Example 37 includes at least one computer-readable storage medium comprising instructions that, when executed by a processor circuit system, cause the processor circuit system to at least: accept input of data associated with a first patient via an interface; generate a prediction of at least one of: a) toxicity produced during immunotherapy according to a treatment plan for the first patient, or b) the efficacy of the treatment plan on the first patient, using at least one of a plurality of models, the plurality of models being used to predict at least one of: a) toxicity in response to immunotherapy or b) the efficacy of immunotherapy, the plurality of models being trained and validated using data from previous patients; and output a recommendation for the first patient relative to the treatment plan.

[0208] Example 38 includes at least one computer-readable storage medium of any preceding clause, wherein the instructions, when executed, cause the processor circuitry to extract and organize patient data in a time series to provide to the model.

[0209] Example 39 includes at least one computer-readable storage medium of any preceding clause, wherein the instructions, when executed, cause the processor circuitry to align the patient data relative to the anchor point to organize the patient data in a time series.

[0210] Example 40 includes at least one computer-readable storage medium as described in any of the preceding clauses, wherein the input is a first data input at a first time, and the prediction is a first prediction, and wherein the processor circuit system is to process a second data input from a first patient at a second time to generate a second prediction using the model, and the processor circuit system is to compare the second prediction with the first prediction to adjust the recommended output for the patient.

[0211] Example 41 is a method comprising: accepting input of data associated with a first patient via an interface; generating a prediction of at least one of: a) toxicity produced during immunotherapy according to the treatment plan for the first patient, or b) the efficacy of the treatment plan on the first patient by executing instructions using a processor and at least one of a plurality of models, wherein the plurality of models are used to predict at least one of: a) toxicity in response to immunotherapy or b) the efficacy of immunotherapy, wherein the plurality of models are trained and validated using data from previous patients; and outputting a recommendation for the first patient relative to the treatment plan by executing instructions using a processor.

[0212] Example 42 includes the method of any preceding clause, further comprising extracting and organizing patient data in time series to provide to the model.

[0213] Example 43 includes the method of any preceding clause, further comprising aligning the patient data relative to the anchor point to organize the patient data in a time series.

[0214] Example 44 includes the method of any of the preceding clauses, wherein the input is a first data input at a first time, and the prediction is a first prediction, and further includes processing a second data input from a first patient at a second time to generate a second prediction using the model; and comparing the second prediction to the first prediction to adjust the recommended output for the patient.

[0215] Example 45 is a device comprising components for processing input data extracted from a record to form a set of candidate features; components for training at least a first model and a second model using the set of candidate features; components for testing at least the first model and the second model to compare the performance of the first model and the second model; components for selecting at least one of the first model or the second model based on the comparison; components for storing at least one of the selections in the first model or the second model; and components for deploying at least one of the selections in the first model or the second model to predict the likelihood of at least one of the following: a) toxicity resulting from immunotherapy according to a treatment plan, or b) the effectiveness of the treatment plan on a patient.

[0216] As disclosed and described herein, the processor circuit system provides components for processing (e.g., including components for processing input data, components for training models, components for selection, components for deployment, etc.) and the memory circuit system provides components for storage. As described above, various circuit systems can be implemented by the processor circuit system and by the memory circuit system.

[0217] The accompanying claims are hereby incorporated into this detailed description by this reference. Although certain example systems, methods, devices, and articles have been disclosed herein, the scope of coverage of this patent is not limited thereto. On the contrary, this patent covers all systems, methods, devices, and articles that fully fall within the scope of the claims of this patent.

Claims

1. A device comprising: Memory circuit system; instruction; as well as processor circuitry for executing said instructions to: processing input data extracted from the records to form a set of candidate features; Using the candidate feature set to train at least a first model and a second model; testing at least the first model and the second model to compare performance of the first model and the second model; selecting at least one of the first model or the second model based on the comparing; storing a selected at least one of the first model or the second model; as well as The selected at least one of the first model or the second model is deployed to predict at least one of: a) a likelihood of toxicity due to an immunotherapy according to a treatment plan, or b) an efficacy of the treatment plan on a patient.

2. The apparatus of claim 1 , further comprising deploying at least one of the selected ones of the first model or the second model in a tool having an interface to facilitate collection of patient data and interaction with at least one of the selected ones of the first model or the second model.

3. The device according to claim 1, wherein: The input data includes at least one of a laboratory test result, a diagnosis code, or a billing code.

4. The device according to claim 1, wherein: The toxicity includes at least one of pneumonitis, colitis, or hepatitis.

5. The device according to claim 1, wherein: The efficacy of the treatment plan on the patient is measured by at least one of patient survival rate or treatment time.

6. The device according to claim 1, wherein: The processor circuit system is to extract and organize the input data in a time series.

7. The device according to claim 6, wherein: The processor circuitry is to align the input data with respect to an anchor point to organize the input data in the time sequence.

8. The device according to claim 1, wherein: The processor circuit system is to generate labels for the input data to form the candidate feature set.

9. The device according to claim 1, wherein: The processor circuitry is to feature engineer the candidate feature set by at least one of regularizing, transforming, or extracting from the candidate feature set.

10. The device according to claim 9, wherein: The processor circuit system is to select from the candidate feature set to form a patient feature set to train or test at least one of the first model and the second model based on the feature engineering.

11. The device according to claim 9, wherein: The processor circuit system is to generate a feature matrix based on the feature engineering to at least train or test at least one of the first model and the second model.

12. The device according to claim 1, wherein: The processor circuitry is to deploy the selected at least one of the first model or the second model as an executable tool having an interface.

13. At least one computer-readable storage medium comprising instructions that, when executed by a processor circuit system, cause the processor circuit system to at least: processing input data extracted from the records to form a set of candidate features; Using the candidate feature set to train at least a first model and a second model; testing at least the first model and the second model to compare performance of the first model and the second model; selecting at least one of the first model or the second model based on the comparing; storing a selected at least one of the first model or the second model; as well as The selected at least one of the first model or the second model is deployed to predict at least one of: a) a likelihood of toxicity due to an immunotherapy according to a treatment plan, or b) an efficacy of the treatment plan on a patient.

14. The at least one computer-readable storage medium of claim 13, wherein: The instructions, when executed, cause the processor circuit system to deploy at least one of the selected ones of the first model or the second model in a tool having an interface to facilitate collection of patient data and interaction with at least one of the selected ones of the first model or the second model.

15. The at least one computer-readable storage medium of claim 13, wherein: The instructions, when executed, cause the processor circuitry to extract and organize the input data in a time series relative to an anchor point.

16. The at least one computer-readable storage medium of claim 13, wherein: The instructions, when executed, cause the processor circuitry to generate labels for the input data to form the candidate feature set.

17. The at least one computer-readable storage medium of claim 13, wherein: The instructions, when executed, cause the processor circuitry to feature engineer the candidate feature set by at least one of regularizing, transforming, or extracting from the candidate feature set.

18. A computer-implemented method comprising: processing input data extracted from the records to form a set of candidate features; Using the candidate feature set to train at least a first model and a second model; testing at least the first model and the second model to compare performance of the first model and the second model; selecting at least one of the first model or the second model based on the comparing; storing a selected at least one of the first model or the second model; as well as The selected at least one of the first model or the second model is deployed to predict at least one of: a) a likelihood of toxicity due to an immunotherapy according to a treatment plan, or b) an efficacy of the treatment plan on a patient.

19. The method according to claim 18, wherein: The deploying includes deploying at least one of the selected ones of the first model or the second model in a tool having an interface to facilitate collection of patient data and interaction with at least one of the selected ones of the first model or the second model.

20. The method of claim 18, further comprising extracting and organizing the input data in a time series relative to an anchor point.

21. The method of claim 18, further comprising generating labels for the input data to form the candidate feature set.

22. The method of claim 18, further comprising feature engineering the candidate feature set by at least one of regularizing, transforming, or extracting from the candidate feature set.

Images