Unsupervised learning and prediction of line of treatment from high-dimensional time-series drug data

Abstract

In one aspect, the present disclosure provides a method for labeling one or more medications co-administered to a patient as a line of therapy, the method including: identifying a patient's medical record from a plurality of digital records; creating a plurality of treatment intervals from a subset of the medical records, the plurality of treatment intervals including at least one medication administered to the patient and a time interval; associating medications of the one or more therapies with each treatment interval when administration of the medication falls within the time interval; refining the time interval for each treatment interval when one of the one or more therapies falls outside the time interval but within an extension period; identifying one or more potential lines of therapy from the plurality of treatment intervals; and labeling the potential line of therapy with the highest maximum likelihood estimate as the line of therapy.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 890,178, filed August 22, 2019, entitled "Unsupervised Learning And Prediction Of Lines Of Therapy From High-Dimensional Longitudinal Medications Data," the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to computer-implemented methods and systems for labeling lines of therapy present in patient records, and more particularly to an unsupervised approach to labeling lines of therapy from medication data. [Background technology]

[0003] Line of therapy (LoT) is the standard nomenclature for discussing treatment with antineoplastic agents. Both the National Comprehensive Cancer Network and the Association for Clinical Oncology (ASCO), the organizations that issue standard of care (SoC) treatment guidelines, present the results of studies within the framework of LoT. Oncologists scrutinize these guidelines when formulating treatment for patients. In addition, the composition of LoT is considered by regulators, payers (both private and institutional), and provider groups when planning, approving, and paying for new anticancer drugs. As such, pharmaceutical companies also approach planning and trial design by considering LoT and the potential impact / benefit to patients realized by the new drug. Physicians frequently summarize a patient's history to another physician by highlighting the LoT prescribed to the patient, any adverse effects, progressions, or intervening events, and any subsequent changes to the LoT to compensate or adapt the treatment to improve the patient's outcome. Unfortunately, this type of informal summary is never entered into the patient's electronic medical / health record (EMR / EHR). When a physician agrees to provide a patient's EMR, it is desirable to analyze the provided records to extract the LoT as well as significant intervening (superseding) events (including progression, regression, metastasis, length of time) and provide them to the physician for his or her convenience and to improve the physician's understanding of the LoT history for each patient.

[0004] This is a difficult task to accomplish because the information recoverable from the EHR and / or progress notes alone is never complete. There are many inaccuracies, inconsistencies, missing records, and other incomplete items that may (or may not) appear in the records and need to be considered. For example, an oncologist may consider two LoTs, one is the drug / treatment / therapy combination of cyclophosphamide, doxorubicin, and paclitaxel as a line of treatment or as monotherapy in the course of treatment. The patient's insurance company may refuse to include paclitaxel due to cost, so the patient receives cyclophosphamide and doxorubicin. After the course of treatment, the patient may determine that this combination is detrimental to his / her overall health, and the patient may move to maintenance treatment with paclitaxel alone. In the EMR, all of these drugs may appear to be recorded for several months, even though the patient never even received paclitaxel in the first few entries in the EMR and only received cyclophosphamide for part of that time. The oncologist may then order a CT scan to monitor the tumor growth. When the patient returns 6 months later with worsening symptoms, the doctor may document the worsening as a progression event in the progress note and list the drugs carboplatin and gemcitabine, or the doctor may list only the drug gemcitabine and leave the record vague about the previous drug. From an abstraction perspective, the EMR simply records that the prior drug was prescribed and then gemcitabine was prescribed 6 months later. The EMR may also record the CT scan and even the worsening in the 6 month time frame. The difficulty of creating a LoT from these records is manifold. 1) If a patient does not take their medication or stops taking it, the LoT that shows they took the medication is not reliable from a data science perspective. 2) From an industry perspective, a change in LoT where the medication is merely adjusted to avoid an adverse effect on the medication is not a new LoT, but the same LoT. Therefore, a medication change does not necessarily indicate a new LoT. Identifying whether a medication change coincides with an exacerbation event, a worsening of symptoms, or any other significant intervening event can be tricky; for example, if a patient does not take a medication due to insurance issues or side effect issues, this may be difficult to correct for a worsening of symptoms as a change in LoT, or may only be to avoid a side effect versus the original LoT. 3) From a data science perspective, even after a new medication is prescribed, it can be difficult to impute whether the medication was continued in part or in full throughout the year. 4) From a clinician's perspective, some drugs may have name changes but be considered essentially the same drug. 5) Patients receive many drugs that are called "supportive" drugs as part of their treatment, but these are not relevant for LoT assignment. Moreover, distinguishing between these is not always straightforward, because what is considered "supportive" vs. "primary treatment" varies by cancer type. 6) Heterogeneity of data sources. Curation from EHRs and follow-up records varies from source to source and requires harmonization to a common standard before LoT determination. 7) Overcoming the burden and complexity of spotty data. Few patients have complete cancer care records covered by both the EHR and curated progress notes. Often only one or the other is available, and when both exist, they describe incongruent parts of the patient's timeline. This complicates the issue, especially when records generally document the beginning of a set of medications but rarely record when they were stopped.

[0005] Currently, there are no algorithms to predict, digest, or impute LoT from EMR. This generally requires experienced practitioners to manually review files and make these decisions on a case-by-case basis for every patient, which is costly and time-consuming. Machine learning can be applied to consider all medications across all patients based on the frequency and common occurrence of medication changes for a particular diagnosis due to intervening events that typically reflect that LoT can be predicted from incomplete data. To address this, machine learning approaches that synthesize heuristics such as rule-based structures and clinical insights, as well as expectation-maximization (EM) algorithms that use machine learning algorithms (MLAs) to make effective predictions, can be considered. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] U.S. Patent Application Serial No. 16 / 732,168 Summary of the Invention [Means for solving the problem]

[0007] In one aspect, the disclosure provides a method for labeling one or more medications co-administered to a patient as a line of therapy, the method including: identifying a medical record of the patient from a plurality of digital records, a subset of the medical records reflecting the patient's medical history for one or more treatments following diagnosis of the patient's medical condition; creating a plurality of treatment intervals from the subset of the medical records, the plurality of treatment intervals including at least one medication administered to the patient and a time interval; associating a medication of the one or more treatments with each treatment interval when administration of the medication falls within the time interval; refining the time interval of each treatment interval when one of the one or more treatments falls outside the time interval but within an extended period, where no interval priority event occurs in the refined time interval; identifying one or more potential treatment lines from the plurality of treatment intervals via an unsupervised artificial intelligence engine, each potential treatment line including one or more of the plurality of treatment intervals and a maximum likelihood estimate; and labeling, via the unsupervised artificial intelligence engine, the potential treatment line with the highest maximum likelihood estimate as the treatment line.

[0008] The method may further include labeling, via an unsupervised artificial intelligence engine, additional potential lines of treatment in successive time intervals as incrementally numbered lines of treatment from the potential lines of treatment that occur chronologically after each preceding line of treatment and have the highest maximum likelihood estimate for each of the potential lines of treatment in each successive time interval. The method may further include labeling, via an unsupervised artificial intelligence engine, each incrementally numbered line of treatment for a plurality of patients, identifying a plurality of cohorts of patients, and reporting the incrementally numbered lines of treatment for each of the plurality of cohorts of patients. The method may further include reporting facility-wide and physician-specific compliance with standardized treatment guidelines. The method may further include reporting changes in the lines of treatment over time based at least in part on each time interval for each line of treatment. The method may further include reporting an analysis of differences in treatment effects between a first cohort of patients and a second cohort of patients, the cohorts identifying patients as having one or more similar treatment dates, treatment lines, diagnoses, outcomes, geographic locations, treating physicians, treatment facilities, genomic markers, or clinical characteristics. The method may further include reporting an analysis of progression-free survival rates between the first cohort and the second cohort. The method may further include generating a progression-free survival curve for the first cohort of patients having at least one labeled treatment line in common, generating a progression-free survival curve for the second cohort of patients having at least one other labeled treatment line in common, and displaying the progression-free survival curves for the first cohort and the second cohort on the same survival graph. In this method, the labeled treatment line may include one or more of a drug name, a therapeutic rollup class, a dosage, or a method of administration.

[0009] The method may further include generating a report, the report including each line of therapy labeled with the patient and respective time interval.

[0010] In this method, medical condition diagnoses may include cancer, cardiac disease, depression, mental health, diabetes, infectious diseases, epilepsy, dermatological, and autoimmune diseases.

[0011] In this method, the medical condition diagnosis can be cancer.

[0012] In this method, the plurality of digital records may include one or more EHR medical records and one or more curated medical records. In this method, the curated medical records may be images of records not included in the one or more EHR medical records. The method may further include binarizing, optical character recognizing, and encoding the curated medical records into a structured format.

[0013] The method may further include identifying, via the treatment identifier, the treatment administered to the patient and the date the treatment was administered.

[0014] In this method, the treatment may include one or more of the administration of a drug, agent, molecule, or chemical; implantation of a medical device; use of a medical device; biological therapy; viral therapy; phage therapy; plant therapy; gene therapy; epigenetic therapy; protein therapy; enzyme replacement therapy; hormone therapy; cell therapy; immunotherapy; antibody therapy; nutritional therapy; use of electromagnetic radiation wave therapy or radiation therapy; surgical procedures; or radiosurgery.

[0015] In another aspect, the present disclosure provides a system for labeling one or more drugs co-administered to a patient as a line of therapy. The system includes at least one computer having at least one processor, the processor being programmed to: identify a patient's medical records from a plurality of digital records, a subset of the medical records reflecting the patient's medical history for one or more treatments following diagnosis of the patient's disease condition; create a plurality of treatment intervals from the subset of the medical records including at least one medication administered to the patient and a time interval; associate a medication of the one or more treatments with each treatment interval when administration of the medication falls within the time interval; refine the time interval of each treatment interval when one of the one or more treatments is outside the time interval but within an extended period of time, wherein no interval precedence event occurs in the refined time interval; and the at least one computer includes an unsupervised artificial intelligence engine, the unsupervised artificial intelligence engine being programmed to: identify one or more potential lines of treatment from the plurality of treatment intervals, each potential line of treatment including one or more of the plurality of treatment intervals and a maximum likelihood estimate; and label the potential line of treatment with the highest maximum likelihood estimate as that line of treatment.

[0016] In the system, the unsupervised artificial intelligence engine may be programmed to label additional potential lines of treatment in successive time intervals as incrementally numbered lines of treatment from the potential line of treatment that occurs chronologically after each preceding line of treatment and has the highest maximum likelihood estimate for each of the potential lines of treatment in each successive time interval. In the system, the unsupervised artificial intelligence engine may be programmed to label each incrementally numbered line of treatment for a plurality of patients, identify a plurality of cohorts of patients, and report the incrementally numbered line of treatment for each of the plurality of cohorts of patients. In the system, the at least one computer may be further programmed to report facility-wide and physician-specific compliance with standardized treatment guidelines. In the system, the at least one computer may be programmed to report changes in the line of treatment over time based, at least in part, on the respective time intervals of each line of treatment. In the system, the at least one computer may be further programmed to report an analysis of differences in treatment effects between a first cohort of patients and a second cohort of patients, the cohorts identifying patients as having one or more similar treatment dates, treatment lines, diagnoses, outcomes, geographic locations, treating physicians, treatment facilities, genomic markers, or clinical characteristics. In the system, the at least one computer may be further programmed to report an analysis of progression-free survival rates between the first cohort and the second cohort. In the system, the at least one computer may be further programmed to generate a progression-free survival curve for the first cohort of patients having at least one labeled treatment line in common, generate a progression-free survival curve for the second cohort of patients having at least one other labeled treatment line in common, and display the progression-free survival curves for the first cohort and the second cohort on the same survival graph. In the system, the labeled treatment line may include one or more of a drug name, a therapeutic roll-up class, a dosage, or a method of administration.

[0017] In the system, medical condition diagnoses may include cancer, cardiac disease, depression, mental health, diabetes, infectious diseases, epilepsy, dermatology, and autoimmune diseases.

[0018] In the system, the plurality of digital records may include one or more EHR medical records and one or more curated medical records. In the system, the at least one computer may be further programmed to binarize, optically recognize, and encode the curated medical records into a structured format.

[0019] In the system, the treatment may include one or more of administration of a drug, agent, molecule, or chemical; implantation of a medical device; use of a medical device; biological therapy; viral therapy; phage therapy; plant therapy; gene therapy; epigenetic therapy; protein therapy; enzyme replacement therapy; hormone therapy; cell therapy; immunotherapy; antibody therapy; nutritional therapy; use of electromagnetic radiation wave therapy or radiation therapy; surgical procedure; or radiosurgery.

[0020] The file of this patent contains at least one drawing / photograph executed in color. Copies of this patent with color drawing / photographs will be provided by the Office upon request and payment of the necessary fee. [Brief description of the drawings]

[0021] [Figure 1] 1 is a flow chart illustrating one embodiment of preparing, training, and generating Line of Treatment (LoT) predictions. [Diagram 2] 1 is a flow chart illustrating one embodiment of preparing, training, and generating Line of Treatment (LoT) predictions using an MLA approach with rule-based pre-processing. [Diagram 3] FIG. 1 is an illustration of LoT prediction for a first patient diagnosed with metastatic non-small cell lung cancer. [Figure 4] FIG. 13 is an illustration of LoT prediction for a second patient diagnosed with ovarian cancer. [Diagram 5]FIG. 13 is an illustration of LoT prediction for a third patient diagnosed with breast cancer. [Figure 6] FIG. 6 is a flow chart illustrating one embodiment of a system 600 for preparing, training, and assigning a Line of Treatment (LoT) using an artificial intelligence engine approach with rules-based pre-processing that implements a method to impute complex curated fields from curated patient history and apply them to non-curated patients or patients with small amounts of data. [Figure 7] 1 is a flow chart illustrating one embodiment of preparing, training, and generating a Line of Treatment (LoT) model across three stages. [Figure 8] FIG. 1 is an illustration of LoT prediction for a first patient diagnosed with metastatic non-small cell lung cancer. [Figure 9] FIG. 13 is an illustration of LoT prediction for a second patient diagnosed with ovarian cancer. [Figure 10] FIG. 13 is an illustration of LoT prediction for a third patient diagnosed with breast cancer. [Figure 11A] FIG. 1 illustrates the first step of the flow. [Figure 11B] FIG. 11B illustrates the number of dosing intervals included in the second step of the flow of FIG. 11A. [Figure 11C] FIG. 11B illustrates the application of heuristics to refine one or more dosing intervals in the third step of the flow of FIG. 11A. [Figure 11D] FIG. 11B is a diagram illustrating outputting line allocation in the fourth step of the flow in FIG. 11A. [Figure 12A] FIG. 1 illustrates the first step of the flow. [Figure 12B] FIG. 12B illustrates the number of dosing intervals included in the second step of the flow of FIG. 12A. [Figure 12C] FIG. 12B illustrates the application of heuristics to refine one or more dosing intervals in the third step of the flow of FIG. 12A. [Figure 12D]FIG. 12B illustrates the application of an expectation-maximization model to refine one or more dosing intervals in the fourth step of the flow of FIG. 12A. [Figure 12E] FIG. 12B illustrates the application of an expectation-maximization model to refine one or more dosing intervals in the fourth step of the flow of FIG. 12A. [Figure 12F] FIG. 12B is a diagram illustrating outputting line allocation in the fifth step of the flow of FIG. 12A. [Figure 13A] FIG. 1 illustrates the first step of the flow. [Figure 13B] FIG. 13B illustrates the number of dosing intervals included in the second step of the flow of FIG. 13A. [Figure 13C] FIG. 13B illustrates the application of heuristics to refine one or more dosing intervals in the third step of the flow of FIG. 13A. [Figure 13D] FIG. 13B illustrates the application of an expectation-maximization model to refine one or more dosing intervals in the fourth step of the flow of FIG. 13A. [Figure 13E] FIG. 13B illustrates the application of an expectation-maximization model to refine one or more dosing intervals in the fourth step of the flow of FIG. 13A. [Figure 13F] FIG. 13B is a diagram illustrating outputting line allocation in the fifth step of the flow of FIG. 13A. [Figure 14] 1 is a Sankey diagram illustrating lines of treatment as applied to patients with at least an immuno-oncology (IO) regimen. [Figure 15] 1 is a progression-free survival (PFS) graph depicting survival curves for patients receiving different first lines of treatment. [Figure 16] FIG. 13 is an illustration of a treatment line editing tool for a user to edit a treatment line for a patient according to one embodiment. [Figure 17] FIG. 13 is an illustration of a structured data representation of a patient timeline after LoT labeling has been performed according to one embodiment. [Figure 18] FIG. 1 is a box plot of the mean duration of LoT identified from cohorts of patients in the lung and breast cancer cohorts. [Figure 19] FIG. 1 is an illustration of LoT frequencies based on the first LoT for breast cancer therapeutics administered to patients. [Figure 20] FIG. 1 is an illustration of LoT frequencies based on a second LoT for breast cancer therapeutics administered to patients. [Figure 21] FIG. 1 is an illustration of LoT frequency based on the first LoT for non-small cell lung cancer (NSCLC) therapeutics administered to patients. [Figure 22] FIG. 1 is an illustration of LoT frequency based on a second LoT for non-small cell lung cancer therapeutics administered to patients. [Figure 23] FIG. 1 is a bar graph depiction of the proportion of patients for each primary cancer who were assigned to a cohort that contained at least one of the listed LoTs. [Figure 24] FIG. 1 is a block diagram illustration of one implementation of a computer system in which some implementations of the present disclosure may operate. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] The phrase "disease condition" refers to the state of a disease, such as cancer, cardiac disease, depression, mental health, diabetes, infectious disease, epilepsy, dermatological, autoimmune disease, or other disease. A disease condition may reflect the presence or absence of a disease in a subject, and may further reflect the severity of the disease when present.

[0023] FIG. 1 illustrates one embodiment of a computer-implemented system 100 for labeling a line of care for a patient based on medications administered and clinical records. The line of care may be generated from patient information represented by a clinical health record 110 implemented by the system architecture 100. The system 100 may be a content server (also referred to as a LoT engine), which is hardware or a combination of both hardware and software. A user, such as a healthcare provider or a patient, is given remote access through GUIs 170a-170n to view, update, and analyze information about the patient's medical condition using the user's own local device (e.g., a personal computer or a wireless handheld device). The user may interactively operate the system to generate electronic records, update electronic records, and issue instructions to perform other actions. The content server is configured to receive a variety of information in different formats, which converts the information into a standardized format suitable for processing by operation of a line of care module 120 on or in conjunction with the content server. Thus, information obtained from patient electronic medical records (EMRs), unstructured text, gene sequencing, image processing, and a variety of other information can be converted into features that are used to train one or more artificial intelligence engines or machine learning models.

[0024] The information obtained, processed, and generated by the content server 100 is stored in one or more of the network-based storage devices. A user can interact with the content server to access the information stored in the network-based storage devices, and the content server can receive user-provided information, apply one or more models stored in the network-based storage to the information, and provide the results of the model application in electronic form to the user on a graphical user interface of the user device. The electronic information is transmitted in a standardized format over the computer network to the user accessing the information. In this manner, the user can easily adapt medical diagnosis and treatment strategies according to the system's predictions, which can be generated automatically. Additionally, the system generates recommendations to the user regarding patient diagnosis and treatment.

[0025] In some embodiments, the described system and method are implemented as part of a digital and laboratory healthcare platform. The platform may automatically generate molecular reports as part of targeted medical care precision medicine therapy. In some embodiments, the system according to embodiments of the present disclosure runs on one or more microservices, which may be microservices of an order management system. In some embodiments, the system is implemented in conjunction with one or more microservices of a cell type profiling service.

[0026] The clinical health record 110 may store a collection of features or condition characteristics generated for some or all patients for which information exists in the system 100. These features may be used to identify medications and priority events using the system 100. While the feature range across all patients is informationally dense, a patient's Feature Set may be sparsely populated throughout the aggregate feature range of all features across all patients. For example, the feature range across all patients may extend to tens of thousands of features, but a patient's unique Feature Set may include a subset of hundreds or thousands of features from the aggregate feature range based on the records available for that patient.

[0027] The features present in the clinical health record 110 may include a diverse set of fields available in the patient's health record. The clinical information may be based on fields entered into an electronic medical record (EMR) or electronic health record (EHR), which may be done automatically or manually, for example, by a doctor, nurse, or other medical professional or representative. Other clinical information may be digitally curated information 114 obtained from other sources, such as, for example, gene sequencing reports (e.g., from molecular fields), progress reports, lab results, insurance billing codes and / or records, and other medical documentation for the patient. Sequencing may include next generation sequencing (NGS), which may be long read, short read, or other forms of sequencing the patient's somatic cells and / or normal genome. The comprehensive collection of features in the additional clinical health record may combine various features together across various medical disciplines, which may include diagnosis, response to treatment regimens, genetic profiles, clinical and phenotypic features, and / or other medical, geographic, demographic, clinical, molecular, or genetic features. For example, the subset of features may include molecular data features, such as features derived from RNA or DNA sequencing, pathological features derived from cell review by a pathologist, or other images.

[0028] Image features may include features identified through review of specimens by a pathologist, such as review of stained H&E or IHC slides. As another example, a subset of features may include derived features obtained from analysis of the individual and combined results of such feature sets. Features derived from DNA and RNA sequencing may include genetic variants that can be identified in the sequenced samples. Further analysis of genetic variants may include steps such as identifying single or multiple nucleotide polymorphisms, identifying whether the mutation is an insertion or deletion event, identifying loss or gain of function, identifying fusions, calculating copy number changes, calculating microsatellite instability, calculating tumor mutation burden, or other structural changes in DNA and RNA. Analysis of slides for H&E or IHC staining may reveal features such as tumor invasion, programmed cell death ligand 1 (PD-L1) status, human leukocyte antigen (HLA) status, or other immunological related features.

[0029] Features derived from the structured, curated, and / or electronic medical or health records 112 and 114 may include clinical features such as diagnosis, symptoms, treatment, outcome; patient demographics such as patient name, date of birth, sex, ethnicity, date of death, address, smoking status, date of diagnosis of cancer, disease, illness, diabetes, depression, other physical or mental illness; medical history, family medical history, clinical diagnosis such as date of first consultation, date of metastasis diagnosis; cancer stage, tumor characterization, tissue of origin, line of treatment, treatment group, clinical trial, medication prescribed or taken, surgery, radiation therapy, imaging, side effects, associated outcomes; genetic testing and laboratory information such as performance score, laboratory tests, pathology results, prognostic indicators; genetic testing date, testing provider used, testing method used such as gene sequencing method or gene panel, genetic results such as genes included, variants, expression levels / status, or corresponding dates associated with any of the above.

[0030] The machine learning model and / or artificial intelligence engine may perform classification labeling and / or analysis by selecting from any feature. One method for enhancing a patient's feature set may include dimensionality reduction, such as collapsing a feature set from tens of thousands of features into a handful of features. Performing dimensionality reduction without losing information may be approached in an unsupervised or supervised manner. Unsupervised methods may include RNA variational autoencoder, SVD, PCA, KernelPCA, SparsePCA, DictionaryLearning, Isomap, NMF, UMAP, feature agglomeration, patient correlation clustering, KMeans, Gaussian Mixture, or spherical KMeans. Performing dimensionality reduction in a supervised manner may include linear discriminant analysis, neighborhood component analysis, MLP transfer learning, or tree supervised embedding.

[0031] The machine learning model or artificial intelligence engine may be implemented via a gradient boosting model, a random forest model, a neural network (NN), a regression model, a naive Bayes model, or a machine learning algorithm (MLA). The MLA or NN may be trained from a training dataset, such as multiple matrices with feature vectors for each patient or images and features. In an exemplary predictive profile, the training dataset may include patient images, pathology, clinical, and / or molecular reports and details, such as those curated from EHRs or gene sequencing reports. MLA includes supervised algorithms (e.g., algorithms where features / classifications in the dataset are annotated) using linear regression, logistic regression, decision trees, classification and regression trees, naive Bayes, nearest neighbor clustering, unsupervised algorithms (e.g., algorithms where features / classifications in the dataset are not annotated) using a priori, means clustering, principal component analysis, random forests, adaptive boosting, semi-supervised algorithms (e.g., algorithms where features / classifications in the dataset are not annotated) using generative approaches (e.g., mixtures of Gaussians, mixtures of multinomial distributions, hidden Markov models), sparse separation, graph-based approaches (e.g., mincut, harmonic functions, manifold regularization), heuristic approaches, or support vector machines. NN includes conditional random fields, convolutional neural networks, attention-based neural networks, deep learning, long short-term memory networks, or other neural models, and the training dataset includes pathology reports covering multiple tumor samples, RNA expression data for each sample, and imaging data for each sample. Although MLA and neural network identify different approaches to machine learning, these terms may be used interchangeably herein. Thus, unless otherwise specified, a reference to an MLA may include a corresponding NN, or a reference to a NN may include a corresponding MLA, and an artificial intelligence engine may embody one or more of each in a hierarchical architecture.

[0032] In some embodiments, the artificial intelligence engine may be a trained machine learning model. In some embodiments, the trained machine learning model may be generated by training a machine learning model on medical records having one or more LoTs. In some embodiments, the trained machine learning model may be a regression model. In some embodiments, the trained machine learning model may be a clustering model. In some embodiments, the trained machine learning model may be a dimensionality reduction model. In some embodiments, the trained machine learning model may be a classification model.

[0033] Training may include providing the optimized dataset as a matrix of feature vectors for each patient, labeling these features as supervisory signals when they occur in the patient record, and training the MLA to predict the output classification, prediction, or label. Artificial NNs are powerful computational models and have shown strength in solving difficult problems in artificial intelligence. They have also been shown to be universal approximators (they can represent a wide range of functions when given appropriate parameters). Some MLAs identify important features and identify coefficients, or weights, for them. The coefficients are multiplied with the frequency of occurrence of the feature to generate a score, and some classifications may be predicted by the MLA when the score of one or more features exceeds a threshold. The coefficient schema may be combined with a rule-based schema to generate more complex predictions, such as predictions based on multiple features. For example, 10 important features may be identified in different classifications. A list of coefficients may exist for the important features and a rule set may exist for the classification. The rule set may be based on the number of occurrences of features, scaled weights of features, or other qualitative and quantitative evaluations of features coded with logic known to those skilled in the art. In other MLAs, the features may be organized in a binary tree structure. For example, a key feature that distinguishes most classifications may be present as the root of the binary tree and each subsequent branch in the tree until a classification can be given based on reaching a terminal node of the tree. For example, the binary tree may have a root node that tests a first feature. The occurrence or non-occurrence of this feature must be present (a binary decision) and the logic may traverse the branch that is true for the item being classified. Additional rules may be based on thresholds, ranges, or other qualitative and quantitative tests. Supervised methods are useful when the training dataset has a large number of known values ​​or annotations, but due to the nature of EMR / EHR documents, a large number of annotations may not be given. When exploring large amounts of unlabeled data, unsupervised methods are beneficial for binning / bucketing instances in the dataset.A single instance of the above model, or two or more such instances combined, may constitute a model for purposes of the models or engines described herein.

[0034] The line of care module 120 may include multiple modules: a record reconciliation module 122, an interval generation module 124, and a line of care assignment module 126. The record reconciliation module 122 may reconcile records contained in the EMR 112 and the digitally curated records 114 to remove duplicates, combine similar records, and take additional reconciliation steps, as disclosed in more detail below with respect to FIG. 3. The interval generation module 124 may receive the reconciled records and generate and refine a plurality of treatment intervals based on the reconciled records, as disclosed in more detail below with respect to FIG. 4. The line of care assignment module 126 may receive a plurality of treatment intervals, identify potential lines of care, and apply a trained artificial intelligence engine to the potential lines of care to identify and label one or more lines of care to be administered to the patient.

[0035] A Line of Treatment (LoT) store 150 may receive the LoTs generated from the LoT module 120 and store them for use by the system 100. The LoTs may be stored in a structured format for retrieval by a user interface, such as, for example, a web form-based interactive user interface, which in some embodiments may include web forms 160a-160n. The web forms may support a GUI that may be displayed by a computer to a user of the computer system for performing a number of analytical functions, including initiating or viewing an instant LoT from the module 120, or initiating or adjusting a cohort of patients from which the module 120 may perform an analysis. Electronic reports 170a-170n may be generated and provided to a user via a Graphical User Interface (GUI) 165. It should be understood that the GUI 165 may be presented on a user device connected to the content server / prediction engine 100 via a network.

[0036] The reports 170a-170n may be provided to a user as part of a network-based patient management system that collects patient information from various physicians and healthcare providers (including laboratories), converts it into a standardized format, consolidates it, stores it in a network-based storage device, and generates a message that includes an electronic report after the report is generated according to an embodiment of the present disclosure. In this manner, a user (e.g., a physician, oncologist, or any other healthcare provider, or a patient) receives computer-generated predictions, analyses, and other reports related to the generated LoT.

[0037] In some embodiments, the electronic report may include recommendations to the physician to treat the patient using the next LoT or treatment that correlates with a predicted response based on the response of similar patients who have received similar treatments.

[0038] 2 illustrates a flow chart of a method for generating one or more lines of care from a plurality of digital records. In step 210, the system 200 may identify medical records for a patient from a plurality of digital records. These records may be recorded in an EMR / EHR or digitally curated from unstructured data. The medical records may be limited to those from a particular time period, including a time period beginning with a diagnosis of a medical condition, such as cancer. Additional time periods may include specific yearly intervals when performing an analysis of LoT development over time.

[0039] In step 220, the system 200 may create a number of treatment intervals with at least one medication in the time interval. These medications have been identified from the patient record and further identified as being related to the diagnosis. The time intervals are associated with a start date and / or an end date, which may be assigned directly from the patient record or may be imputed from additional supporting information such as learned intervals from similar medications, standard of care, or an artificial intelligence engine trained on similar patient records.

[0040] In step 230, the system 200 may associate a drug with a respective treatment interval when the administration of the drug falls within the time interval. For example, if a treatment interval exists from June 1 to June 23, each drug administered on June 15 may be associated with a treatment interval, but a drug administered on September 1 may not be associated with a treatment interval because its administration does not fall within the time interval.

[0041] In step 240, the system 200 may refine the time intervals of each respective treatment interval. The time intervals may be refined by combining overlapping intervals, splitting intervals, or adjusting start and / or end dates according to one or more heuristics, rules, or trained models.

[0042] In step 250, system 200 may identify one or more potential lines of therapy from a plurality of treatment intervals. Identification may be performed using a trained model, exhaustive enumeration with likelihood-based ranking, or other methods disclosed herein.

[0043] In step 260, system 200 may label the potential line of treatment with the highest maximum likelihood estimate as the line of treatment. For example, given a ranked list of potential lines of treatment, the line of treatment with the highest ranking (most confident) may be selected as the line of treatment.

[0044] FIG. 3 illustrates an example record reconciliation module 300 that implements a sub-process of record reconciliation 122 of FIG.

[0045] Record reconciliation may include one or more modules, such as a record whitelist and / or record blacklist module 322 , a record association module 324 , a record combination module 326 , and a heuristic processing module 328 .

[0046] The record whitelist and / or record blacklist module 322 may include or exclude records based on their content, quality, reliability, or other evaluation metrics. For example, a record may be included when its source can be identified as imaging results from a respective facility, excluded if the image resolution is below an acceptable threshold for readability, or excluded if the record contains patient inconsistencies such as inconsistencies in gender, race, or other immutable characteristics / demographics.

[0047] The record association module 324 may check records for redundancy and remove records that appear multiple times or contain the same information as other records. Associations with medications, start dates, or end dates may be assigned or imputed.

[0048] The record combination module 326 may combine records that have sufficiently overlapping associations. Sufficiently overlapping associations may include the same or similar medications (such as generic vs. brand name, competing brand names, known equivalents, or other similar). Multiple records with the same start date for similar medications may be "rolled up" into a single record.

[0049] The heuristic processing module 328 may incorporate a rule-based or artificial intelligence driven record reconciliation structure. In one example, an artificial intelligence engine may be trained on drug names to identify similar drugs, or on dosing intervals to predict the duration of the most commonly prescribed drug for patients for each drug type. The model may additionally include rule sets to apply set thresholds to the assignment of start and end dates, or enforce minimum or maximum time periods for each record.

[0050] Additional record reconciliation steps are disclosed below with respect to data capture and reconciliation.

[0051] FIG. 4 illustrates an example interval generation module 400 that implements a sub-process of interval generation 124 of FIG.

[0052] Interval generation may include one or more modules such as diagnosis relevance filter 422 , interval date assignment 424 , interval date imputation 426 , interval date extension 428 , or interval date segmentation 430 .

[0053] The diagnosis relevance filter 422 may incorporate a knowledge database that includes drugs known to treat diagnoses. These drugs may be curated from FDA approvals, clinical trials, publications, or artificial intelligence analysis of patient records. Drugs that are not related to the treatment of a diagnosis may be excluded from processing against the LoT, as they are unlikely to be administered for the purpose of treating the diagnosis.

[0054] Interval date assignment 424 may include assigning an interval to each medication record that has not been deleted from one of the preceding modules in record curation, reconciliation, or relevance filtering. Medical records that include a medication and at least a start date may be assigned to an interval with no medication or with the same or similar medication.

[0055] Interval date imputation 426 may include imputing a start or end date for an interval to reflect an accurate estimate of the administration of the medication for that interval. For example, if the medication is expected to continue for a set number of days (such as a course of antibiotics, steroids, or other medications with known periodicity), an end date may be imputed for the interval that each medication is assigned based on the expected duration, such as after 7 days, 14 days, 3 months, etc.

[0056] Interval date extension 428 may include identifying when consecutive intervals overlap and extending the duration of the first overlapping interval to encompass other overlapping intervals when the medications are the same or similar. Additionally, when medications are assigned or imputed to intervals, the start or end date of each interval may be extended to include additional dosing dates when the extension falls within a threshold time period. In one example, the threshold time period may include a set duration, such as 22 days, or a time period per medication based on the dosing of the interval, similar to record combination 326 above.

[0057] Interval date segmentation 430 may include identifying when an interval priority event occurred in one or more intervals and dividing each respective interval by the date of the priority event. For example, in cancer diagnosis, the occurrence of a metastasis may be referred to as an interval priority event, and any interval that extends beyond the date of the metastasis may be divided into a first interval with the date of the metastasis as the original interval start and end dates, and a second interval with the date of the metastasis as the interval start date and the original interval end date. Priority events include events that cause a line of treatment change and may be identified through artificial intelligence analysis of patient records, or through rule sets, or through those events as may be curated from FDA approvals, clinical trials, publications.

[0058] FIG. 5 illustrates an example line of care allocation module 500 for implementing a sub-process of line of care allocation 126 of FIG.

[0059] The line of treatment assignment may include one or more modules such as LoT assignment estimation 522, LoT assignment probability 524, LoT ranking 526, or LoT labeling 528.

[0060] The LoT allocation estimation 522 may include performing an evaluation of integer combinations of available intervals such that intervals may be considered in part and in whole relative to one another to identify potential lines of therapy. In another embodiment, the artificial intelligence engine may receive multiple treatment intervals and output multiple potential lines of therapy.

[0061] LoT assignment probability 524 may include calculating the expected maximum probability of each potential line of treatment to identify the likelihood that each potential line of treatment will be included as a consecutively numbered line of treatment (first line of treatment, second line of treatment, third line of treatment, etc.).

[0062] LoT ranking 526 may include ranking multiple potential lines of treatment based on their likelihood such that the potential lines of treatment with the highest probability are at the top of the ranking and the potential lines of treatment with the lowest probability are at the bottom of the ranking.

[0063] LoT labeling 528 may identify the most likely potential line of treatment as the line of treatment, or evaluate successive potential lines of treatment to identify successive incrementally numbered lines of treatment that occur chronologically after each preceding line of treatment and have the highest maximum likelihood estimate of each of the potential lines of treatment in each successive time interval. For example, if a first potential line of treatment begins in January and a second potential line of treatment begins in May, the first potential line of treatment may be labeled as the first line of treatment and the second potential line of treatment may be labeled as the second line of treatment when both lines of treatment are the highest ranked potential lines of treatment in the intervals before and after May, respectively. Additionally, an artificial intelligence engine may be trained to identify and label lines of treatment and / or successive lines of treatment iteratively.

[0064] The modules of FIGS. 3-5 may be configured to perform one or more of the following steps or processes:

[0065] In one example, a set of rules may guide the process of labeling one or more LoTs from a patient record. The rules may include one or more "hard" or "soft" rules.

[0066] Hard Rules After an initial diagnosis of prostate cancer, it may be common for patients to be prescribed leuprolide and administered for life. Thus, even if the LoT is changed after leuprolide appears in the EMR, it may always remain part of the LoT. Another exemplary hard rule may include some intervening event that requires a change in LoT, such as treatment discontinuation, metastasis, or progressive disease outcome. The opposite may also be coded as a hard rule. For example, patients with a recorded change in medication to abemaciclib generally only appear due to metastasis of breast cancer. Even if the EMR is incomplete and does not state that the patient's cancer has spread by metastasis, this may be imputed and added to the EMR and segment the respective treatment interval, potential LoT, or post-detection LoT. Similarly, progression from one class of drug to another class of drug may be implemented as a hard rule and may force segmentation of the respective treatment interval, potential LoT, or LoT.

[0067] Soft Rules When rules are formed but can be applied conditionally, they can be coded as soft rules. In one example, note that oncologists each define LoT differently, i.e., some may consider carboplatin and cisplatin together as LoT, but then maintenance pembrolizumab as not a core part of LoT, while others may consider cisplatin and carboplatin together with maintenance pembrolizumab as one LoT. Some may consider deviation as a new LoT, even if it is for side effect avoidance, while others may not. These types of preferences can be generated as soft rules that give additional weight to LoT, but do not require a new LoT to be generated when viewed in the EMR. If two drugs are similar, such as having the same active ingredient or achieving the same effect with different active ingredients, the change from one to the other may be weighted to determine whether or not there is a LoT change. Additionally, the preference for applying one soft rule over applying another soft rule may be recorded and applied on a per physician or per institution basis, such that a particular soft rule may be applied conditionally based on patients under the care of a particular physician or institution that chooses to use that soft rule.

[0068] Algorithm methodology and example output In one approach, the EMR and curated progress reports may be analyzed for all diagnoses, medications, significant events, and other features that may be relevant to identifying any LoT based on combinations of medications, significant events, and other combinations of medications. MLA works in two broad steps. The first step consists of synthesizing and harmonizing disparate data sources, including medications, outcomes, and diagnoses across the EHR and curated progress notes, to create unique intervals of patient care. The second step considers all possible combinations of these intervals and assigns LoT accordingly. In a lexical example where medication history is represented as the string THECATSAT, each character may represent a unique medication combination after digestion by the first step. After training across patient data, the goal of the second step is to recognize common medication patterns and separate this into "THE" + "CAT" + "SAT" given a heuristic. An example based on a patient EMR is described below with respect to Figures 8-13F.

[0069] Training the model Training the model may consist of defining a unique treatment interval (the "letter" from the example above, such as "T" or "H") for each patient in a large training cohort, iteratively considering all combinations of aggregates (T+HEC+AT vs. THE+CAT), assigning these aggregates, calculating the frequency of these resulting aggregates (words), and reassigning the aggregates until a termination condition is reached. These combinations of aggregates are enumerated using an integer combination approach. In a simplified example with a LoT representation of five such drugs, we split into 16 different possible LoT groupings using an integer combination approach.

[0070] The composition of 16 of 5 is as follows: ● 5 ● 4 + 1 ● 3 + 2 ● 3 + 1 + 1 ● 2 + 3 ● 2 + 2 + 1 ● 2 + 1 + 2 ● 2 + 1 + 1 + 1 ● 1 + 4 ● 1 + 3 + 1 ● 1 + 2 + 2 ● 1 + 2 + 1 + 1 ● 1 + 1 + 3 ● 1 + 1 + 2 + 1 ● 1 + 1 + 1 + 2 ● 1 + 1 + 1 + 1 + 1.

[0071] This frequency analysis can be combined with hard / soft rules along with diagnostic information, clinical information, and significant events to provide a global probability estimate for each combination. This most likely combination (THE+CAT+SAT) is then output as the assigned LoT.

[0072] One goal is to identify the cut-off points for this string after studying a large corpus of text, so that it can be read as THE|CAT|SAT. Such a problem would be easier if each letter of the alphabet were mapped to some sub-hierarchy, such as consonants (C) and vowels (V), so that the above could be read as T→C, H→V, E→C, C→C, A→V, T→C, S→C, A→V, T→C. In this case, it would be much easier to learn the separators CVC|CVC|CVC. For this, the Anatomical Therapeutic Chemical (ATC) Classification System is a drug classification system that classifies the active ingredients of drugs according to the organ or system in which they act and their therapeutic, pharmacological, and chemical properties. It is maintained by the World Health Organization's Drug Statistics and Methodology Collaborating Center (WHOCC) and was first published in 1976. This drug coding system separates drugs into different groups according to the organ or system in which they act, their therapeutic intent or nature, and the chemical properties of the drug. Different brands share the same code if they have the same active substance and indication. Each lower level ATC code represents a substance or combination of substances used as a medicine in a single indication (or use). This means that a drug can have multiple codes, e.g. acetylsalicylic acid (aspirin) has A01AD05 (WHO) as a drug for oral topical treatment, B01AC06 (WHO) as a platelet inhibitor, and N02BA01 (WHO) as an analgesic and antipyretic, and even a code can represent multiple active ingredients, e.g. C09BB04 (WHO) is a combination of perindopril and amlodipine, the two active ingredients having their own codes (C09AA04 (WHO) and C08CA01 (WHO) respectively) when prescribed alone. The ATC classification system is strictly hierarchical, i.e. each code has exactly one parent code, except for the top 14 codes, which have no parent. These codes are semantic identifiers, i.e. they represent a complete lineage of parents in themselves. The ATC hierarchy provides a third and fourth level hierarchy for a given drug.If each drug could be remapped into this hierarchy, improvements in defining the LoT in these simpler representations could be realized, especially in contexts where the dataset is smaller or there is very high drug cardinality. A notable specific case would be hormone maintenance therapy and other therapies that are very common in one cancer, but difficult to learn using traditional MLA techniques.

[0073] For example, in the breast cancer LoT, a cohort may have approximately 55 unique antineoplastic drugs associated with breast cancer. Grouping these drugs into their underlying ATC hierarchy, either through levels 3 or 4, results in a significant reduction of this drug space to 20 under ATC level 4 and 9 under ATC level 3. The number of LoT states identified (classification of LoT from the entire breast cancer patient set) is 1117 using drug names alone, 656 under ATC level 4 and 289 under ATC level 3. Roughly speaking, employing increasingly non-specific drug grouping reduces the number of unique "drugs" by a factor of two and correspondingly reduces the number of drug states in the output model by roughly a factor of two.

[0074] The equivalence provided by the ATC hierarchy was implemented to solve the "maintenance" problem observed primarily in breast cancer. We often see patients with first-line chemotherapy (doxycycline and cyclophosphamide) followed by tamoxifen or anastrozole (hormonal therapy), all of which are considered one LoT. However, a pure frequency-based approach to identifying LoT from basic patient data would almost always say that chemotherapy and hormone therapy are separate LoTs, because hormone therapy is ubiquitous, half of all patient medication states are hormone therapy (i.e., half the time, if the patient is taking at least one drug, it is hormone therapy), and secondly, patients who get chemotherapy as first-line will transition to hormone therapy within 180 days, half the time. Returning to the verbal example (THECATSAT), this is like saying that a person sees E almost all the time, sees TH at a regular pace, and sees THE almost half the time. In pure frequency, TH|E is the outcome. This is in part because of the ubiquity of E (prob(E) is high), so prob(TH)*(E)>prob(THE). In practice, this is because we observe hundreds of variations with an E after it, or simply an E, where one would think that E was only a single letter.

[0075] Therefore, the maintenance problem is a result of the core problems of the pure frequency model, and secondly, when the occurrence of E is not important, and secondly, when E is generally seen alone, and TH, in fact, there are some cases where it can become one "word" and must always be separated. Both of these can be encoded in the probability expression. THE is a word when prob_1(TH)*prob_2(E)<prob_1(THE), where prob_1 represents the probability of observing a given character as the first (1), second (2), etc. The latter can be encoded as the "word-ness" of THE being prob(E|TH)>prob(X|TH)*prob(E|X), that is, when a given TH is in front and the presence of E is significantly higher compared to being accidental (or any other given agent, X), and in some cases modulated by the frequency of E.

[0076] Selecting which model to implement is an example of evaluating the performance of the underlying system. Treatment lines are often the result of NCCN guidelines, and oncologists select the "plan" at a given time with only minor changes, change this after the outcome is observed, and / or the patient's response to treatment is not good. Therefore, the latter is a more appropriate way to represent the probability space. The former representation means prob_1(TH)*prob_2(E)!=prob_2(TH)*prob_3(E), meaning that chronological ordering strictly changes the probability. This is likely to be the case as well, but the more desirable feature to capture is not that seeing only E or TH is important for predicting whether THE is a word, but rather comparing TH(X) and (Z)E and predicting whether there is a significant enrichment when X = E and Z = TH.

[0077] As explained above, in its raw form, the probabilistic model has an imprecise mapping to the way we normally impute progression events. In the worst case, it may consistently impute progression events when it rarely actually sees them. To further improve the agreement between the model and the outcome, the training of the augmented model and the corresponding model is performed using the underlying probabilistic model as follows: prob(A→B)=prob(A)*prob(B)*prog_w(A,B ) vs prob(AB)*[1-prog_w(A,B )], Here, prog_w(A,B) represents an estimate of the observed progression incidence between medication state A and medication state B. The initiation of a LoT break for said state is modified by the incidence of progression typically observed in the transition from A to B, and the more common this state is, the more likely it is compared to the situation where both are one LoT. The model may implement prog_w(A,B) as the number of progressions observed in this transition relative to the total number of times this transition has been observed. However, the vast majority of transitions are largely unobserved, which would significantly bias the model. A "Bayesian" estimate (i.e., leaving the decision to the underlying frequency space of the model, rather than the progression incidence) is made that tends towards 0.5. The larger the sample size, the more one would want to use the observed estimate, and the smaller, the more one would want to use 0.5. The classical way to implement this is additive smoothing, where prog_w can be: prog_w(A→B)=(prog_f)±abs(prog_f-0.5)*(upper_95(prog_f)-lower_5(prog_f)). where prog_f is the number of progressions observed in the A → B transition divided by the overall incidence, and upper_95(prog_f), lower_5(prog_f) are the 95th and 5th binomial proportion CI estimates based on the Clopper Pearson method for prog_f. As the sample size increases, less "centering" is applied to prog_f, as the confidence interval approaches 0. As the sample size increases, the confidence interval diverges (up to 1.0), which means that the largest shift is applied, and prog_w has a tendency towards 0.5.

[0078] The ATC and enhanced models show the highest agreement with common transitions. Most importantly, the model next shows the least significant discrepancy with the data (trastuzumab → trozole), with an incidence of 1 / 319 in the data compared to 30 of 319 in the output, a very small difference, or the injection of ~29 additional events such as taxane → trozole. This is quite common as a transition from a strong and rapid initiation therapy such as chemotherapy to a maintenance therapy. The advantage of the instant LoT labeling artificial intelligence engine is that it allows nuance in the soft rules covering three possible LoTs based on the request entity. The LoT can include initiation therapy, maintenance therapy, or a combination of initiation and maintenance therapy.

[0079] Thus, the enhanced model would eliminate spurious LoT changes for maintenance therapy or reduce the occurrence of discrepancies between the model and observed data while preserving the possibility of higher order interactions.

[0080] In another embodiment, an artificial intelligence model, such as an unsupervised model, can be trained on the patient's health information or other treatment information. The unsupervised model can be trained on the patient's medication or other therapy and progression information, or information including treatment interval priority events. In some examples, the progression events relevant to model training are those that occur after a first therapy is stopped and before a second therapy is started.

[0081] In some embodiments, the line of treatment identified by the model trained using the patient's health data follows a predetermined standard of care guideline, such as guidelines published by the National Comprehensive Cancer Network (NCCN), FDA guidelines, clinical trials, or other publications.

[0082] In some embodiments, a semi-supervised approach may be implemented where multiple lines of therapy are curated from NCCN guidelines, FDA recommendations, clinical trials, drug fact sheets, or other publications, and then used as a starting point to identify labels in patients where labels were generated that do not match the labels curated in the unsupervised portion of the semi-supervised approach. Those skilled in the art will recognize that the semi-supervised approach requires additional time during operation to set up and run, but provides a more reliable LoT given the starting criteria, including precise LoT designation.

[0083] In some embodiments, the treatments may be generated by generating a concatenated treatment from treatments having the same treatment identifier when the administration dates of two or more treatments are within a time period of each other. In some embodiments, the time period may be based at least in part on a medical condition. In some embodiments, the time period may be based at least in part on a medical condition and a treatment identifier. In some embodiments, the time period may be based at least in part on a medical condition and a treatment identifier.

[0084] For any particular patient record, the line of treatment identified from the record may be aligned to the line of treatment published in a standard of care guideline. For example, in ovarian cancer, a standard of care guideline may recommend niraparib as maintenance therapy for patients with platinum-sensitive disease who have received two or more platinum-based lines of treatment and had a complete or partial response to the most recent recurrent line of treatment. Analyses such as those performed via web pages 160A-160N or GUI 170 may provide physicians or facilities with compliance statistics to guidelines from NCCN, FDA, or other guideline setting entities.

[0085] Some of the lines of treatment are consistent with standard of care guidelines, such as those identified in the NCCN guidelines. Others of the lines of treatment may not be consistent with standard of care guidelines. In these cases, the LoT determination may be used for quality improvement purposes to identify data in the medical record that may be incorrect. In other cases, the LoT determination may identify patients who participated in clinical trials, such as trials testing FDA-approved combinations of therapies. In other instances, the LoT determination may identify new trends in clinical practice that are not yet reflected in current standard of care guidelines. For example, a line of treatment in ovarian cancer for a patient receiving a LoT chemotherapy regime followed by a LoT maintenance targeted therapy (such as a targeted inhibitor like a PARP inhibitor or an EGFR inhibitor).

[0086] Another semi-supervised approach may enforce bucketing on the timeline of the LoT. For example, in 1980, the first LoT with two drugs may be standard, but in 1993, those two drugs are discontinued and replaced as LoT. A semi-supervised approach may capture guideline changes from public records or identify years when new treatments were released based on inclusion in clinical trials, and bound the unsupervised LoT curation to fall between the years of LoT change so that the evolution of the LoT is tracked over the years. In one embodiment, those boundaries may be set as sliding windows and automatically determined based on an optimized EM similar to the EM shown in more detail with respect to FIG. 7. Similar bucketing may be performed based on geographic criteria such as by physician, facility, state, region, country, or other common characteristics for a cohort of patients. Unsupervised learning excels in these analyses due to the inherently variable nature of trying to identify new windows based on time or location to derive a meaningful LoT from existing patient records. The analysis may be performed using windowing to identify regions, facilities, or years in which a particular LoT is most effective.

[0087] In some embodiments, the unsupervised approach allows the system to be fine-tuned according to different cancers and their medications. For example, treatment of prostate cancer involves oral medications taken at longer intervals than intravenous medications administered to treat lung cancer, which traditionally have shorter intervals. Restrictive structured models may have difficulty learning labels, as nuances of cancer subtypes and labels may be limited to only those categories of medications enumerated by the programmer during training or available from the system of curated data.

[0088] FIG. 6 is a flow chart illustrating one embodiment of a system 600 for preparing, training, and assigning lines of treatment (LoT) using an artificial intelligence engine approach with rules-based pre-processing that implements methods to impute complex curated fields from curated patient history and apply them to non-curated patients or patients with small amounts of data.

[0089] A patient's medical history is captured as a clinical medical record 110 through a combination of curated medical records (MR) 114 derived from a combination of machine learning and medical expert annotations of digital documents such as progress notes and electronic health records (EHR) MR 112. Features of the patient's medical record may be structured and sent to multiple modules within the system 600. The system 600 is focused on taking this redundant, repetitive, and sometimes inconsistent raw input and transforming it into a set of condensed records that capture the most prominent medications used to treat the patient's cancer. The following steps are an exemplary method for performing this transformation.

[0090] In another embodiment, the record reconciliation 122 may include a medication filtering module 605, a medication combination module 610, and a medication reconciliation module 615. The medication filtering 605 may include filtering the medications in the clinical health record 110 from the curated records 114 and the EHR 112 to drug cancer therapies, such as antineoplastic drugs. Filters may also be applied to other medication characteristics, such as to identify medications related to a line of therapy. Filtering medications that are not typically part of a line of therapy reduces the computational burden required to process the vast number of patient records available. For example, an electronic health record (EHR) consists of all medications ordered and administered to a patient, such as painkillers, antibiotics, placebos, multivitamins, and more. The filtering stage 605 removes any medications that are not identified as antineoplastic (anti-cancer) drugs and stored in an internal value set. This value set may be defined by the clinical team or curated from machine learning models as well as internal and external publications, clinical trials, FDA approvals, NCCN guidelines, or drug reports.

[0091] The drug combination 610 may combine drug records (MR) within a threshold number of days (d), such as 22 days. The threshold may be set by drug, drug class, active ingredient, or learned from the patient record via an artificial intelligence engine. EHR MRs are highly repetitive periodic records that occur every time a drug is ordered and administered. Many antineoplastic therapies are administered every 22d (called a cycle), so by combining records within 22d, significant record compression can be achieved that captures the entire duration that the patient has been administered the drug. Procedurally, one process for date imputation may be performed for each record, which includes: a) If the MR has an annual start or end date, or no start date, proceed to Reconciliation 615. b) If the MR has a start or end date of a month only, impute it on the 15th of that month. c) If the MR has a missing end date, set the end date to the start date. For the native and curated recordings separately (i.e., perform the following steps for all drugs in the native recordings and also for all drugs in the curated recordings), perform the following for each drug: i) Sort each MR of the drug by start date. ii) For each MR of a drug: (1) Compare the end of the record with the beginning of the subsequent record. If the MRs are within 22d of each other, combine the records. This can be accomplished by setting the end of the record to the end of the subsequent record and deleting the intervening MRs for that drug. (2) Go to the next record.

[0092] Harmonization 615 may identify and remedy redundant and sometimes contradictory information. "Harmonization" refers to a set of machine learning algorithms, heuristics learned from internal and external publications, and medical experts, to remove these redundancies or contradictions and establish a consistent set of MRs that combines the high temporal resolution of the EHR MRs 112 with the expert knowledge of the curated MRs 114. Procedurally, one embodiment of harmonization may involve: a) Correct for poor temporal resolution in any curated EMR and impose the following imputation rules to meet the resolution of the MR to be accurate to within a day, week, or month as appropriate for harmonization. The following approach assumes a maximum duration for these records and allows EHR MRs with potentially higher resolution to disambiguate these dates. i) If starting with month only, set the day to the 1st of the month. If starting with year only, set the month to January and the day to the 1st. ii) For month only ending, set the day to the 15th. For year only ending, set the month to 12th and the day to the 31st. b) If all recordings are native or curated and no harmonization is required, proceed to interval generation 124. c) Sort the combined records by start date. d) For each medication type in the patient record: i) Create an empty "output list" (the last entry in the list is referred to as output[-1]). ii) For each record of this medication type: (1) If this is the first entry, append it to the output. (2) If the current record occurs after output[-1], then append the record to the output. (3) If the record has a start or end date with a higher resolution than output[-1] (such as month or year resolution, and the record has day resolution), replace the lower resolution date with the higher resolution date. (4) If a record appears within the time window of output[-1] (e.g., a 1-day record that appears within several months of curated records), exclude this extraneous record from the output list. e) Return the output list as a reconciled medication table 620 with medications for each MR categorized and structured.

[0093] In another embodiment, interval generation 124 may include modules for primary cancer (PC) relevance filter 625, date padding 630, combination of MRs using PC curation model 635, and interval generation 640. A line of treatment is best described by periods of continuum of medical care that describe the administration of one or more medications (such as, but not limited to, a "regimen"). Here, these periods of continuum of care are referred to as "intervals", or "treatment intervals". "Interval generation" is the process of converting harmonized MRs into these treatment / medication intervals (MIs). These simply represent the duration that a patient is taking one set of medications, and a new MI begins when a change, addition, or subtraction of an anti-neoplastic drug occurs that is relevant to the patient's primary cancer type.

[0094] A PC relevance filter 625 may further filter the harmonized medication table by medications associated with the patient's primary cancer (PC), diagnosis, or other identifier to bias the line of treatment. This list is defined by machine learning models, internal and external publications, and medical experts. This removes medications such as denosumab that are considered supportive care for some cancers (such as prostate cancer) but prominent for others (such as osteosarcoma). One embodiment of PC filtering may involve: a) Filter the patient MR using a PC relevance filter. b) Filter patients with significant uncertainty in the remaining MR. This includes: i) Medication records by year only. ii) Vague or generic drug names (e.g., “platinum compounds” or “antineoplastic agents”) present from the curated progression notes rather than actual drug names.

[0095] Date Padding 630 may identify and pad dates. Many MRs consist of only a start date, or an end date that is equivalent to a start date. To construct a dosing interval, a minimum duration is required, so these records are given an end date using an interval distinction threshold (DATE_PAD, typically 21d, where DATE_PAD is a tunable hyperparameter that can be set by the user based on the user's threshold selection). One embodiment may involve: A) If the record has no end date or end date == start date, set end date = start date + DATE_PAD.

[0096] The MR combination using the PC curation model 635 may perform drug-specific rollups. Many antineoplastic drugs are administered approximately every 21 d, while some hormonal drugs are administered monthly or less frequently. To account for this, drug-specific rollups (typically 22d-180d) may be learned for each drug from a combination of machine learning and medical expert knowledge. This process is described in more detail below with respect to the PC curation model section. One embodiment may involve: a) The learned drug specific rollups are applied to each medication record. For each drug and each record: i) If the end of the previous record and the beginning of the next record are within this rollup, combine the records and stitch the two records together.

[0097] Interval generation 640 may include a conversion of MR to a dosing interval. A patient's medical history typically consists of multiple antineoplastic drug MRs that overlap in time. This conversion process creates predefined intervals of homogenous treatment of one or more drugs, and starts a new dosing interval whenever an antineoplastic drug is added or subtracted from the patient's medication record. One embodiment may involve: a) Sort the resulting medication records by start date and for each record do the following: i) Create a new interval if one of the following is true: (1) Initial medication record; (2) Medication recording begins after the end of the last interval, or (3) Medication recording begins within the interval distinction threshold (e.g., 22 days) of the end of the last interval end, and either the start of recording or the end of the previous interval has monthly resolution. ii) If the medication record overlaps with the last interval, add this medication record to that interval. iii) If an interval of at least the interval discrimination threshold cannot be constructed due to multiple overlapping records, then the LoT cannot be determined for that patient and a failure status is returned. iv) Otherwise, the medication record appears across multiple other intervals, so add this medication to any overlapping records. If recording continues after the end of the last interval, create a new interval for the remainder of the record. b) Print Medication Intervals 645, describing all records including treatment intervals, start and end, and associated medications.

[0098] Those skilled in the art will appreciate that the above steps represent embodiments in which the present disclosure may work. Other systems and methods consistent with the disclosure herein may also be used.

[0099] The Line of Treatment (LoT) assignment 126 may comprise one or more modules for refining intervals with outcomes 650, modeled LoT frequencies 655, estimated probabilities 660, LoT labeling 665, or post-processing LoT 670. A patient's line of treatment captures the treatment strategy adopted by the oncologist to manage the person's cancer. Each line is one or more planned antineoplastic drugs, and a new set of drugs or a new "line of treatment" is proposed when the patient experiences an interval-preferred event (e.g., an imaging result indicating a worsening of the patient's prognosis or a metastatic event). These unplanned events are often missing in the patient MR due to incomplete response data (often present solely in progress reports). The main goal is to learn common treatment patterns and apply these when response data is scarce to generate computationally derived LoT assignments. Here, the intervals generated from the LoT assignment 126 are annotated with response data from the patient's clinical health record 110, when present, to generate a refined interval list. Using a combination of integers (COI) approach, the most likely combination of these interval sets is determined, taking into account the patient's incrementally quantified LoT. In some embodiments, one or more LoTs may be annotated. In some embodiments, one or more LoTs may not be annotated. In some embodiments, the treatment may be annotated. In some embodiments, the treatment may not be annotated.

[0100] Refining intervals with outcomes 650 may identify a number of "treatment interval priority events" or patient events, such as outcomes, that are absolute indicators of change in LoT. These are added to the dosing intervals to generate a refined interval list consisting of one or more partitioned sets of intervals. One such process for completion may include: a) Collect all patient outcomes and filter to outcomes and interventions relevant to LoT. These can be cancer type specific, such as castration-resistant prostate cancer (CRPC) diagnosis, or generic, such as metastatic diagnosis or progressive disease outcomes. Commonly removed outcomes include complete and partial responses (indicating LoT was successful and should be continued). b) Iterate over each outcome and patient interval, and if an outcome occurs within the interval separation threshold of the start of a new interval, separate this patient interval list into two separate sets of intervals. If an outcome occurs within the thresholds of two intervals, separate by the interval that is closer in time and select the latter. c) Iterate through the patient refinement interval list and separate into additional sets if a treatment line maximum duration threshold (e.g., 180 days) separation occurs.

[0101] The modeled LoT frequencies 655 may enumerate integer division combinations for each treatment interval or generate all sequential permutations of drug divisions into potential treatment lines. In one example, this may include an artificial intelligence engine trained on patient records.

[0102] The estimated probability 660 may receive a collection of potential LoTs and treatment intervals to generate a respective likelihood for each LoT. In some embodiments, 655 and 660 may be combined into a single stage such that the artificial intelligence engine receives refined intervals based on outcome data, separates potential LoTs from the intervals, and generates a probability that measures the likelihood that each potential LoT is a LoT. In one example, the process may include: a) After separating the refined interval list into sets given outcomes, iterate over each set and do the following (set LoT counter = 1): i) Calculate all possible combinations of intervals (potential LoTs) using an integer combination approach or an unsupervised artificial intelligence engine trained to identify LoTs from patient data. ii) Calculate the probability of each of these interval combinations using the frequencies learned during model training. The total probability of a given interval combination is the product of the individual combined intervals (for 2 intervals this is Prob(Interval 1)*Prob(Interval 2) and Prob(Interval 1 followed by Interval 2)). iii) The combination of intervals with the highest probability is considered as the LoT assignment. (If Prob(Interval 1)*Prob(Interval 2)>Prob(Interval 1 followed by Interval 2), then Interval 1 is the first LoT and Interval 2 is the second LoT.)

[0103] LoT labeling 665 identifies the most likely combination of the set of dosing intervals (the potential LoT with the highest likelihood for each successive time period), which may be considered the LoT for the patient. In one example, the process may include: i) Number each of these interval combinations with a LoT counter, incrementing each time (in the case of 2 intervals, if the preceding interval was assigned LoT1, then interval 1 will be assigned LoT2, interval 2 will be assigned LoT3, etc.).

[0104] Post-processing LoT 670 may include identifying LoTs that can be corrected or modified. For example, in rare cases, consecutive combined intervals may have the same set of drugs, but were separated by a time period that exceeds the combined threshold for the drug or interval, but do not have any therapeutic interval priority events, and therefore may be considered to be the same LoT. Any LoT that falls into a pattern that can be corrected or adjusted in post-processing may be assigned to the same LoT, and subsequent LoTs may be incrementally adjusted. In one example, this may include: a) Combine all the most probable interval combinations into a final LoT list. b) Iterate over each of the interval joins and do the following: i) If two consecutive LoTs have the same drug, assign them the same LoT (e.g. [doxorubicin, cyclophosphamide and doxorubicin]|[cyclophosphamide and doxorubicin, cyclophosphamide and doxorubicin and paclitaxel, anastrozole] will generate the final output list [doxorubicin, cyclophosphamide and doxorubicin, cyclophosphamide and doxorubicin, cyclophosphamide and doxorubicin and paclitaxel, anastrozole], which will be transformed to [doxorubicin, cyclophosphamide and doxorubicin, cyclophosphamide and doxorubicin and paclitaxel, anastrozole], where the "|" bar indicates the outcome, the "," comma indicates the new LoT, and drugs grouped with "and" are combinations).

[0105] In some embodiments, the output LoT table, such as the LoT table stored in LoT store 150, may be in a structured format, such as a json file, xml file, or other format, and may have a subset of the following structured data fields: ● patient_id: A patient_id that is unique across tables. ● interval_start: Start of the medication interval. ● interval_end: End of medication interval. ● medication: Relevant medication (maximum one per line). ● lot: The LoT assigned (1~n, integer). ● complete_lot_start: The start of a comprehensive LoT, spanning one or more intervals. ● complete_lot_end: The end of a comprehensive LoT, spanning one or more intervals. ● complete_lot_medications: Lists associated medications in a given LoT, linked with the "+" sign. ● emr_derived: 1 if the entire LoT is derived from medication records present only in the EHR, 0 otherwise. ● success: whether a LoT could be successfully defined for the patient. This is unsuccessful if the patient has only the required years of medication for the LoT or has a medication record that results in a treatment interval.

[0106] Once generated, the LoT table may be stored in the LoT store 150 or combined with structured data from the patient's EHR for subsequent use in analytical models.

[0107] Primary Cancer (PC) Curation Model Algorithm As explained above, different drugs have different cycles that may vary by primary cancer type. The next step may be used to estimate this cycle time for a particular drug of interest for a primary cancer type by leveraging the medical knowledge present in the curated MR. This is by capturing the typical duration of drug administration as described by the curated MR and using this to propose a cycle time for that drug that repeats this typical duration when applied to the EHR MR. This has the effect of transforming a highly repetitive EHR administration record into a curated MR duration "look" that an abstractor may curate. Without loss of generality, this process is described below for a single drug for a single primary cancer type, but may be extended to multiple cancer types and multiple drugs without detracting from the present disclosure. 1. Input: Curated MR and EHR MR for a given agent for a given primary cancer type. 2. Filter the curated MR with the following characteristics: a. End date different from start date b. Start and end dates with at least monthly resolution. 3. Calculate the duration of medication administration spanning these curated MRs, defined as MR end date - MR start date. 4. Calculate the median duration of drug administration across the curated MRs. 5. Starting with the minimum interval threshold (typically 22d), do the following: a. Combine the EHR records within this threshold for each patient to generate the duration of medication administration described by one or more records (as the case may be). b. Calculate the duration (end to start) of drug administration for each of these records. c. Calculate the median duration across all these recordings. d. If this median is equal to the curated MR median (calculated in 4), return this interval threshold as the "cycle time", otherwise increment the interval threshold. e. If the increment threshold is greater than the maximum interval threshold (typically set to 180d), then exit the loop.

[0108] The above approach can be generalized to compute a kernel density approximation of the curated duration distribution, rather than the median, and return a threshold that minimizes a distance metric (Euclidean distance, geometric distance, or Kullback-Leibler divergence) of the two distributions.

[0109] FIG. 7 is a flow chart illustrating one embodiment of preparing, training, and generating a Line of Treatment (LoT) model across three stages 710, 720, and 730.

[0110] In the model data preparation stage 710, sources of patient information related to diagnosis, prognosis, treatment, and outcome are aggregated across sources in a curated clinical health record 110 including EMR / EHR records. Analysis from these records produces cancer-specific clinical insights 705, patient medications 714, and curated progression and treatment deviations 712. For example, if a patient is diagnosed with prostate cancer, prostate cancer clinical insights (such as drug, treatment, or therapy type that directly links to treatment of prostate cancer, effects of specific drugs, treatments, or therapies on prostate cancer, list of drug, treatment, or therapy names that link to codes related to LoT in prostate cancer, progression events that may establish LoT breaks for each cancer type such as progression to a castration-resistant prostate cancer (CRPC) diagnosis, etc.) may be retrieved from internal and external sources such as one or more databases or publications in elements 705 and 712. Medications curated from documents such as progress notes, lab results, or other written documents may then be filtered to include only those associated with the diagnosed cancer for further processing such as those having effects on treatment. For example, irrelevant medications such as amphetamines for headaches, antihistamines for allergies, seratonin receptor inhibitors for depression, or medications taken by the patient for purposes other than treating the diagnosed cancer may be filtered from the LoT prediction. Filtering to LoT relevant medications by cancer type in module 716 may be performed to filter depression medications from cancer diagnoses such that modeled dates and intervals 718 are calculated only from treatments related to the diagnosis for each patient. Modeled dates and intervals 718 may include corresponding start and end dates, which may be generated for each of the medications remaining in the filtered LoT medication list, and an initial timeline interval may be generated. The collection of modeled dates and intervals 718 may be curated for all patients in the dataset with an associated diagnosis of, for example, all cancers, organ-specific cancers (breast, lung, bone, blood, liver, prostate, etc.), or even a specific diagnosis such as non-small cell lung cancer, ductal carcinoma in situ (DCIS), invasive ductal carcinoma (IDC), triple-negative breast cancer, or other diagnoses. do.

[0111] In the training phase 720, additional cancer-specific clinical insights are retrieved in element 705 and the dosing intervals calculated in elements 710 to 718 are received in drug measurement interval frequency (w) 722. The additional cancer-specific clinical insights may include the number of days that constitute an automatic LoT break (e.g., for fast-growing cancers this may be 90 days, for slower-growing cancers this may be 180 days, and even progression events that may establish a LoT break for each cancer type, such as a CRPC diagnosis for prostate cancer). In element 722, the treatment / dosing intervals may be compared across all patient data to identify occurrence frequencies. For example, in patients with prostate cancer, the treatment / dosing intervals may also be refined according to a rolling window with a width corresponding to the number of days (w). This process may be performed over one or more window sizes (w=30 days, 90 days, 120 days, etc.) to identify the most representative set of LoTs in the patient population. For each dosing interval identified in the patient population, expectation maximization (EM) may be calculated to identify the reliability of the interval. These intervals may be assigned in LoT partition assignment 724 after looping through the EM of frequencies. After the optimal window size is determined, the dosing frequency may be refined in dosing frequency refinement 726. Refinement may include another loop through the EM of frequencies until convergence at the specified window size (w) is achieved. The set of treatment intervals with the highest expectation may be selected as the best representative set of LoTs (potential LoTs). After all patients have been processed to identify the basis set of LoTs, stage 130 may output the estimated LoT frequencies from the modeled LoT frequencies 655.

[0112] In a model LoT assignment stage 730, the collection of estimated LoT frequencies 655 across all patients may be compared to each new patient's dosing interval. In a LoT generation stage 732, the treatment intervals designated as potential LoTs may then be assigned to a first, second, third, etc. LoT split corresponding to each patient's ranking of potential LoTs. Each potential LoT assignment 732, enumerated according to the integer combination method, may be ranked by the corresponding popularity of the most likely LoT labeled as the patient's LoT in LoT and LoT labeling 734.

[0113] Exemplary applications of LoT labeling are described below with respect to Figures 8-13. Exemplary embodiments may be generated and labeled according to any of the methods and systems disclosed herein, including the systems and methods of Figures 1-7. The following examples are provided to illustrate how different methods and approaches herein may be applied to different examples of patient records.

[0114] 8-10 are diagrams illustrating patient medical records, calculated medication intervals, and predicted lines of therapy (LoT) from a combined EMR and curated record system according to one embodiment. The type and source of the patient record may be considered when extracting medications, outcomes, treatment interval priority events, and other LoT features. These illustrations highlight the interaction between the EMR and the curated records to identify one or more treatment intervals and potential LoTs within the records.

[0115] FIG. 8 is an illustration of LoT prediction for a first patient diagnosed with metastatic non-small cell lung cancer. The first patient was diagnosed with NSCLC at primary diagnosis 802 in August 2016. Throughout the next year of treatment, the oncologist treating the first patient generated several progress notes (represented above as "Progress Notes") that describe the patient, the first patient's medications, and associated outcomes. This data is complemented by electronic medical health records (EHR) of medication administration 804.

[0116] A medical professional (the curator) reviewed each of these progress notes and recorded the displayed medications and associated outcomes from each note 806. Next, the contents of the progress notes are described. Progress Note 1 (PN1): The curator recorded the patient's primary diagnosis in August 2016 and the date of the first administration of triplet chemotherapy with pemetrexed, bevacizumab, and carboplatin. PN2: The curator noted completion of triplet therapy 808 in PN1 in October 2016, partial response to related therapy (not shown, October 2016), and initiation of pemetrexed / bevacizumab maintenance therapy 810 in November 2016. PN3: In January 2017, the curator recorded a progressive disease outcome 812 with pemetrexed / bevacizumab maintenance therapy 810 and its associated termination in December 2016. The oncologist also noted an intention to place the patient on nivolumab 814 in February 2017. PN4: The curator recorded a progressive disease outcome816 (meaning that the therapy was unsuccessful in treating the patient's cancer) for nivolumab814 in February 2017, and the initiation of gemcitabine therapy818 beginning in March 2017. PN5: The end of gemcitabine818 due to toxicity in late March 2017 was recorded, and the start of paclitaxel820 was also recorded at the same time. PN6: Completion of paclitaxel 820 was recorded in June 2017, with an associated "complete response" to therapy (indicating the patient was found to be cancer-free). PN7: At the follow-up visit in August 2017, another “complete response” 822 was recorded, indicating the patient continued to be found to be cancer-free. These curated medications were entered into the medication table and outcomes entered into the outcomes SQL table for downstream analysis.

[0117] Inclusion and EHR Harmonization Next, medication administration records present in the EHR from the hospital were added to the medication data table (All medication administrations 804 and events bar according to record reconciliation 122 in Figure 1). Dexamethasone administration 824 was additionally recorded. These medications are not curated as they are supportive care medications that patients commonly receive in conjunction with chemotherapy to relieve side effects. While displayed, dexamethasone was filtered and not considered in subsequent calculations since supportive care medications are not considered salient to the LoT interval generation module 124. EHR administration records of pemetrexed, bevacizumab, and carboplatin 808 from September and October 2016 were added to the curated records of these medications to generate "native and curated" records for these medications in record reconciliation 122. The EHR administration of paclitaxel 820 was used to augment the curated record in April-June 2017, generating another “native and curated” record. An additional administration of paclitaxel in August (in 804a) was added to the patient record in record harmonization 122.

[0118] Dosage interval indication The written medication records were then converted into unique medication intervals (represented by dashed bars 830a-830e). These intervals represent a collection of medications taken at the same time with defined start and end dates. Medication intervals are created whenever a medication change occurs (see interval generation module 124). Interval 830a: The first dosing interval begins with the triplet regimen pemetrexed / bevacizumab / carboplatin and ends with the discontinuation of carboplatin in October 2016 (Interval Creation Module 124). Interval 830b: consists of pemetrexed / bevacizumab, the interval starting with the discontinuation of carboplatin and continuing until the end of this doublet therapy (Interval Generation Module 124). Interval 830c: This captures a patient administration of Nivolumab in February 2017 (Interval Generation Module 124). Interval 830d: This interval begins with the administration of gemcitabine in March 2017 and ends with its cessation and the initiation of paclitaxel (interval generation module 124). Interval 830e: Starts with the curated, native paclitaxel record and continues to the EHR record of paclitaxel administration in July 2017. Although the patient received dexamethasone beginning in May 2017, this medication is not considered relevant to the LoT assignment, and therefore does not cause a sixth interval from May 2017 to July 2017 to be generated (Interval Generation Module 124).

[0119] Spacing LoT Allocation A combination of probabilistic selection and heuristics is then applied to determine the LoT for the generated dosing intervals (1-5) (LoT assignment module 126).

[0120] The outcomes are taken into account to separate the different intervals: the outcome for January 2017 is used to separate interval 830b from interval 830c (LoT allocation module 126). The outcomes for late February 2017 are used to separate interval 830c from interval 830d (LoT allocation module 126). Next, consider interval 830a-830b. In this case, a probabilistic selection is made considering the relative frequency of each of these intervals across the training population. The probability of seeing interval 830a alone (representing triplet therapy) is 10% and the probability of seeing interval 830b alone (pemetrexed / bevacizumab) is 5%, so the combined probability is 0.5% (10%*5%). However, seeing interval 830a followed by interval 830b is 2%. Since 2%>0.5%, this interval is combined and defined as 1 LoT (LoT allocation module 126). Since interval 830c is separated in outcome by intervals 830b and 830d, this interval alone becomes LoT 2 (LoT assignment module 126). Intervals 830d and 830e are then considered from a probabilistic selection perspective: the probability of seeing intervals 830d and 830e in isolation is 7.5% (30% and 25%, respectively; 30%*25%=7.5%), while the probability of seeing interval 830d~830e across the dataset is 1%. Since 7.5%>1%, the intervals are considered isolated LoTs, and interval 830d is assigned to LoT3 and interval 830e is assigned to LoT4 (LoT assignment module 126).

[0121] FIG. 9 is an illustration of the LoT prediction for a second patient diagnosed with ovarian cancer. Demographics and clinical data sources. PN1, March 2015: Primary diagnosis and surgery, initiation of carboplatin, bevacizumab, and paclitaxel. PN2, September 2015: Started bevacizumab, finished carboplatin, paclitaxel. PN3, March 2016: Outcomes of Advanced Disease PN4, June 2016: Progressive disease outcome, end of bevacizumab (month only), initiation of carboplatin, bevacizumab, gemcitabine. PN5, October 2016: End of carboplatin, bevacizumab, gemcitabine. EHR Source: Anastrozole, April 2017.

[0122] Interval Construction Interval 930a: Triplet therapy carboplatin / bevacizumab / paclitaxel. Interval 930b: Bevacizumab (separated from interval 930a by the interval generation module 124). Interval 930c: carboplatin / bevacizumab / gemcitabine. Interval 930d: Anastrozole (separated from interval 930a by the interval generation module 124).

[0123] LoT Allocation: The June 2016 outcome is used to inform the separation between intervals 930a-930b and 930c-930d (LoT assignment module 126). Intervals 930c and 930d are separated due to the separation between the end of the previous and the start of the next (LoT assignment module 126). The refined interval list then consists of [(interval 930a, interval 930b), (interval 930c), (interval 930d)].

[0124] For the first set, the probability of interval 930a and interval 930b alone and the probability of the combination are compared. Since the combined interval is more probable, the set becomes (interval 930a + interval 930b) and is assigned LoT1 (LoT assignment module 126). Interval 930c is assigned LoT2 and interval 930d is assigned LoT3 (LoT assignment module 126, only one possible combination).

[0125] FIG. 10 is an illustration of the LoT prediction for a third patient diagnosed with breast cancer. Demographics and clinical data sources. PN1, January 2017: Primary diagnosis, breast, and initiation of docetaxel, trastuzumab, pertuzumab, and carboplatin. PN2, July 2017: Completed docetaxel, trastuzumab, pertuzumab, and carboplatin. EMR, July 2017: Trastuzumab Administration EMR, December 2017-June 2018: Capecitabine, several doses of trastuzumab, and two doses of tamoxifen.

[0126] Interval Construction Interval 1030a: Quadruplet regimen of docetaxel, trastuzumab, pertuzumab, and carboplatin. Interval 1030b: Trastuzumab (separated by interval 1030a via the interval generation module 124). Interval 1030c: trastuzumab and capecitabine (separated from interval 1030b by the new drug and interval generation module 124). Interval 1030d: Trastuzumab (separated from interval 1030c by the infusion of capecitabine, the interval generation module 124 references the continuation of trastuzumab). Interval 1030e: trastuzumab and tamoxifen (separated from interval 1030d by the introduction of tamoxifen, interval generation module 124, continuing through interval generation module 124, tamoxifen employs drug specific rollup of 207d).

[0127] LoT assignment: Since this record does not have an outcome, all possible combinations of the 5 intervals are considered probabilistically (in this case 2(5-1) or 16 possible combinations, LoT assignment module 126) and the most probable one is selected as the sequence of LoTs. In this case, 2+1+2 was selected as the most likely interval combination.

[0128] 11-13 illustrate line of therapy labeling from patient records according to another embodiment that includes four steps. 1) Combining medical records to identify outcomes such as medications, cancer diagnoses, and interval priority events. 2) Define multiple dosing intervals at the initiation of each new therapy or medication in the patient record. 3) Applying an artificial intelligence engine trained in an unsupervised manner from multiple patient records and applying learned heuristics against the complete listing of the LoT. 4) Outputting the LoT assignment according to the identified LoT from the artificial intelligence engine.

[0129] 11-13 include a listing of the primary diagnosis (pDX) at time 0, multiple medications / therapies that occurred after the primary diagnosis at a time and have a duration of at least 1 day, and any interval-priority events such as surgery, or outcome listings from the patient record, each with a pDX start and end date to indicate the interval of the therapy, medication, or other event.

[0130] 11A-11D, an exemplary flow 1100 of treatment line generation is shown. FIG. 11A illustrates a first step of the flow 1100. The first step of the flow 1100 can include combining one or more medications and / or treatments, one or more diagnoses (e.g., cancer diagnosis), and one or more outcomes (e.g., progressive disease outcome). In the exemplary flow 1100, the primary diagnosis can define the start of a timeline to which events and / or treatments can be mapped to provide one or more time intervals to a user (e.g., oncologist). As shown, the one or more medications and / or treatments can include carboplatin, pemetrexed, nivolumab, and surgery. In some embodiments, the flow 1100 as shown in FIG. 11A can be generated using steps 210 and 220 of FIG. 2.

[0131] FIG. 11B illustrates the number of dosing intervals included in the second step of flow 1100. Flow 1100 of FIG. 11B may be generated based on one or more dosing and / or treatments of FIG. 11A. Some drugs may be associated with a common interval for one or more new treatments. As shown, carboplatin and pemetrexed may be associated with a first dosing interval that is about 70 days after the primary diagnosis. The first dosing interval may define the start of the first new treatment. Other drugs may be associated with a second dosing interval. As shown, nivolumab may be associated with a second dosing interval. The second dosing interval may define the start of the second new treatment. In some embodiments, flow 1100 as shown in FIG. 11B may be generated using step 230 of FIG. 2.

[0132] FIG. 11C illustrates applying heuristics to refine one or more dosing intervals in a third step of flow 1100. The heuristics can refine one or more dosing intervals by combining overlapping intervals, splitting intervals, or adjusting start and / or end dates according to one or more heuristics, rules, or trained models. For example, carboplatin and pemetrexed can be associated with separate dosing intervals, and flow 1100 can associate both carboplatin and pemetrexed with a first dosing interval. The heuristics can refine one or more dosing intervals based on a rule associated with a progressive disease outcome. For example, the heuristics can set a second dosing interval to start about 130 days after diagnosis based on a progressive disease outcome at about 130 days after diagnosis. For continuation of the same medication (e.g., nivolumab), the line of treatment may be the same for all dosing intervals, such as the third interval associated with the second treatment using nivolumab. In some embodiments, flow 1100 as shown in FIG. 11C may be generated using step 240 of FIG.

[0133] FIG. 11D illustrates outputting a line assignment in the fourth step of flow 1100. The line assignment may be associated with one or more potential lines of therapy from one or more dosing intervals. The identification may be performed using a trained model, exhaustive enumeration with likelihood-based ranking, or other methods disclosed herein. The lines of therapy may include a first line associated with a first dosing interval (e.g., carboplatin and pemetrexed therapy) and a second line associated with a second dosing interval (e.g., nivolumab therapy). In some embodiments, flow 1100 as shown in FIG. 11D may be generated using steps 250 and 260 of FIG. 2.

[0134] 12A-12F, an exemplary flow 1200 of treatment line generation is shown. FIG. 12A illustrates a first step of the flow 1200. The first step of the flow 1200 can include combining one or more medications and / or treatments and one or more diagnoses (e.g., cancer diagnosis). In the exemplary flow 1200, the primary diagnosis can define the start of a timeline to which events and / or treatments can be mapped to provide one or more time intervals to a user (e.g., oncologist). As shown, the one or more medications and / or treatments can include doxorubicin, cyclophosphamide, paclitaxel, tamoxifen, and surgery. In some embodiments, the flow 1200 as shown in FIG. 12A can be generated using steps 210 and 220 of FIG. 2.

[0135] FIG. 12B illustrates the number of dosing intervals included in the second step of flow 1200. Flow 1200 of FIG. 12B may be generated based on one or more dosing and / or treatments of FIG. 12A. Some drugs may be associated with a common interval for one or more new treatments. As illustrated, doxorubicin and cyclophosphamide may be associated with a first dosing interval that is about 50 days after the primary diagnosis. The first dosing interval may define the start of the first new treatment. Other drugs may be associated with a second dosing interval or a third dosing interval. As illustrated, paclitaxel may be associated with a second dosing interval that is about 120 days after the primary diagnosis. The second dosing interval may define the start of the second new treatment. As illustrated, tamoxifen may be associated with a third dosing interval that is about 150 days after the primary diagnosis. The third dosing interval may define the start of the third new treatment. In some embodiments, a flow 1200 as shown in FIG. 12B may be generated using step 230 of FIG.

[0136] FIG. 12C illustrates applying heuristics to refine one or more dosing intervals in a third step of flow 1200. The heuristics can refine one or more dosing intervals by combining overlapping intervals, splitting intervals, or adjusting start and / or end dates according to one or more heuristics, rules, or trained models. For example, doxorubicin and cyclophosphamide can be associated with separate dosing intervals, and flow 1200 can associate both doxorubicin and cyclophosphamide with the first dosing interval. The heuristics can refine one or more dosing intervals based on rules associated with hormone therapy. For example, the heuristics can set the second dosing interval to include a third therapy (e.g., tamoxifen) because the third therapy is a hormone therapy. In some embodiments, flow 1200 as shown in FIG. 12C can be generated using step 240 of FIG. 2.

[0137] 12D illustrates applying an expectation maximization model to refine one or more dosing intervals in a fourth step of flow 1200. For each dosing interval, an expectation maximization model may be used to calculate an EM to identify the reliability of the dosing interval. In some embodiments, flow 1200 as shown in FIG. 12D may be generated using step 240 of FIG. 2.

[0138] FIG. 12E illustrates applying an expectation maximization model to refine one or more dosing intervals in the fourth step of flow 1200. Specifically, flow 1200 may include applying an expectation maximization model to a first dosing interval transition "A" and a second dosing interval transition "B" to determine whether the first dosing interval should be combined with the second dosing interval or left separate to maximize the estimation performance. The probability of observing one or more given intervals may be denoted as P(X). The probability of observing a progression event between two intervals may be denoted as P_prog(X→Y). The probability of seeing a progression event when splitting the interval into two lines (P(A)P(B)P_prog(A→B)) may be compared to the probability of seeing a progression event when maintaining one line (P(AB)(1-P_prog(A→B))), and the more probable option may be selected as the line assignment. In the example shown in FIG. 12E, it was determined that the more likely option was to split the interval into two lines, so that interval "A" and interval "B" each have a corresponding medication line.

[0139] FIG. 12F illustrates outputting a line assignment in the fifth step of flow 1200. The line assignment may be associated with one or more potential lines of therapy from one or more dosing intervals. The identification may be performed using a trained model, exhaustive enumeration with likelihood-based ranking, or other methods disclosed herein. The lines of therapy may include a first line associated with a first, second, and third dosing interval. In some embodiments, flow 1200 as shown in FIG. 12F may be generated using steps 250 and 260 of FIG. 2.

[0140] 13A-13F, an exemplary flow 1300 of treatment line generation is shown. FIG. 13A illustrates a first step of flow 1300. The first step of flow 1300 can include combining one or more medications and / or treatments, one or more progressive disease outcomes, and one or more diagnoses (e.g., cancer diagnosis). In the exemplary flow 1300, the primary diagnosis can define the start of a timeline onto which events and / or treatments can be mapped to provide one or more time intervals to a user (e.g., an oncologist). As shown, the one or more medications and / or treatments can include dexamethasone, lenalidomide, bortezomib, carfilzomib, pomalidomide, daratumumab, and stem cell transplant. In some embodiments, flow 1300 as shown in FIG. 13A can be generated using steps 210 and 220 of FIG. 2.

[0141] FIG. 13B illustrates the number of dosing intervals included in the second step of flow 1300. Flow 1300 of FIG. 13B may be generated based on one or more dosing and / or treatments of FIG. 13A. Some drugs may be associated with a common interval for one or more new treatments. As illustrated, dexamethasone, lenalidomide, and bortezomib may be associated with a first dosing interval that is about 5 days after the primary diagnosis. The first dosing interval may define the start of the first new treatment. Other drugs may be associated with additional dosing intervals. As illustrated, carfilzomib and pomalidomide may be associated with a second dosing interval that is about 180 days after the primary diagnosis. The second dosing interval may define the start of the second new treatment. As illustrated, stem cell transplant may be associated with a third dosing interval that is about 360 days after the primary diagnosis. The third dosing interval may define the start of the third new treatment. As shown, lenalidomide may be associated with a fourth dosing interval that is about 540 days after the primary diagnosis. The fourth dosing interval may define the initiation of a fourth new therapy. As shown, daratumumab and dexamethasone may be associated with a fifth dosing interval that is about 700 days after the primary diagnosis. The fifth dosing interval may define the initiation of a fifth new therapy. As shown, pomalidomide may be associated with a sixth dosing interval that is about 750 days after the primary diagnosis. The sixth dosing interval may define the initiation of a sixth new therapy. As shown, daratumumab may be associated with a seventh dosing interval that is about 930 days after the primary diagnosis. The seventh dosing interval may define the initiation of a seventh new therapy. In some embodiments, flow 1300 as shown in FIG. 13B may be generated using step 230 of FIG. 2.

[0142] FIG. 13C illustrates applying heuristics to refine one or more dosing intervals in a third step of flow 1300. The heuristics can refine one or more dosing intervals by combining overlapping intervals, splitting intervals, or adjusting start and / or end dates according to one or more heuristics, rules, or trained models. The heuristics can refine one or more dosing intervals based on rules associated with progressive disease outcomes. For example, the heuristics can set the fifth dosing interval to start at about 700 days after diagnosis based on a progressive disease outcome at about 700 days after diagnosis. In some embodiments, flow 1300 as shown in FIG. 13C can be generated using step 240 of FIG. 2.

[0143] 13D illustrates applying an expectation maximization model to refine one or more dosing intervals in a fourth step of flow 1300. For each dosing interval, an expectation maximization model may be used to calculate an EM to identify the reliability of the dosing interval. In some embodiments, flow 1300 as shown in FIG. 13D may be generated using step 240 of FIG. 2.

[0144] FIG. 13E illustrates applying an expectation maximization model to refine one or more dosing intervals in a fourth step of flow 1300. Specifically, flow 1300 can include applying an expectation maximization model to a first dosing interval transition "C", a second dosing interval transition "D", and a third dosing interval transition "E" to determine whether dexamethasone, pomalidomide, and daratumumab should be combined into a single interval or kept separate to maximize estimation performance. The probability of the new line in transitions CD and DE can be written as P(C)P(D)P(E)P_prog(C→D)P_prog(D→E). The probability of the new line in transition CD can be written as P(CD)P(E)(1-P_prog(C→D))P_prog(D→E). The probability of a new line at transition DE can be written as P(C)P(DE)P_prog(C→D)(1-P_prog(D→E)). The probability of transitions C, D, and E being on one line can be written as P(CDE)(1-P_prog (C→D))(1-P_prog (D→E)). In the example shown in FIG. 13E, it was determined that the probability of C, D, and E being on one line is the highest.

[0145] FIG. 13F illustrates outputting a line assignment in the fifth step of flow 1300. The line assignment may be associated with one or more potential lines of therapy from one or more dosing intervals. The identification may be performed using a trained model, exhaustive enumeration with likelihood-based ranking, or other methods disclosed herein. The lines of therapy may include a first line associated with the first dosing interval, a second line associated with the second, third, and fourth dosing intervals, and a third line associated with the fifth, sixth, and seventh dosing intervals. In some embodiments, flow 1300 as shown in FIG. 13F may be generated using steps 250 and 260 of FIG. 2.

[0146] FIG. 14 is a Sankey diagram 1400 illustrating lines of treatment as applied to patients with at least an immuno-oncology (IO) regimen.

[0147] For the patient record in this example, the first line of treatment may include IO monotherapy such as atezolizumab, nivolumab, or pembrolizumab, combination IO and IO such as ipilimumab and nivolumab, or combination IO monotherapy and chemotherapy such as carboplatin, pembrolizumab, pemetrexed, carboplatin, paclitaxel, pembrolizumab, carboplatin, protein-bound paclitaxel, pembrolizumab, carboplatin, pembrolizumab, pemetrexed, or any other listed combination. The second line of treatment includes additional combinations of IO therapy, hormonal therapy, or maintenance therapy. The third line of treatment includes docetaxel, ramucirumab, vinorelbine, and gemcitabine. And the fourth line of treatment includes docetaxel and gemcitabine.

[0148] After the dataset has been labeled according to which line of therapy each patient received, additional analyses can be performed.

[0149] FIG. 15 is a progression free survival (PFS) graph 1500 depicting survival curves for patients receiving different first lines of treatment.

[0150] Cohorts of patients may be generated based on a common diagnosis and one or more lines of therapy. Cohorts may also be filtered based on any of the features included in the patient records. Exemplary lines of therapy 1510, 1520, and 1530. A physician may select a cohort using a web portal, such as web portals 160A-160N of FIG. 1, that includes a collection of features found from patient records that include LoTs labeled according to any of the systems and methods disclosed herein, and populate a PFS graph to compare patient survival over time for the cohort of patients with the identified characteristics. Here, it may be empirically observed that patients receiving the line of therapy associated with PFS curve 1510 respond better to treatment than patients receiving the line of therapy associated with PFS curve 1520, and both cohorts respond better to treatment than patients receiving the line of therapy regimen associated with PFS curve 1530. Based on this information and the collection of cohorts, a physician may be more likely to avoid prescribing the LoT associated with curve 1530 and more likely to recommend prescribing the LoT associated with curve 1510.

[0151] An example of cohort selection based on characteristics found in patient records via an online web portal is described in U.S. patent application Ser. No. 16 / 732,168, filed Dec. 31, 2019, entitled "A Method And Process For Predicting And Analyzing Patient Cohort Response, Progression And Survival," which is incorporated by reference herein for all purposes.

[0152] With reference to the exemplary online web portal of US Patent Application No. 16 / 732,168, the cohort selection of Figures 1-9, the analysis of Figures 10-28, and the customizable web portal (notebook) implementation of Figures 29-33 are referred to in their entirety as, for example, the cohort selection to select and receive features from the clinical health record 110 of Figure 1 in the web portals 160a-160n or GUIs 170a-170n, the analysis to display based on at least one LoT identified for the patient, and any additional cohort features. The customizable notebooks suitable for displaying customized analyses to the user may include custom target generation for PFS, outliers, efficacy of surgery, molecular features of the patient on LoT performance, adverse event analysis, effect from duration of LoT, or other cohort analysis, etc., which may be based on the labeled LoT. The exemplary notebooks allow a programming approach to design a custom interface with R, SQL, or other database software backend for the analysis. The custom interface may include identifying off-label uses of drugs and treatments that may be particularly effective. The notebook may identify and present informatics for initiating and establishing investigations into new clinical trials to validate the effectiveness of off-label uses of treatments. In one example, the notebook may identify that each LoT is associated with a generic drug name, a brand name, and a therapeutic roll-up class (type of treatment [targeted, initiation, maintenance, chemotherapy, radiation therapy, tyrosine kinase inhibitor (TKI), etc.], dosage of the drug in the LoT, and / or a method of administration of the treatment, such as oral, intravenous, radiation exposure, etc.).

[0153] FIG. 16 is an illustration of a line of treatment edit tool 1600 for a user to edit a line of treatment for a patient according to one embodiment.

[0154] The line of treatment editing tool may receive any patient from the set of patients having a line of treatment identified according to the methods and systems herein. After the patient is loaded, a timeline similar to the timeline of Figures 11-13 is loaded. In another embodiment, a timeline similar to the timeline of Figures 8-10 may be loaded as well. The user may use the cursor to perform several operations that remedy perceived deficiencies in the LoT labeling performed by the unsupervised artificial intelligence engine.

[0155] For example, the user may drag and drop 1610 any line of therapy designation to recalibrate the new LoT based on the new position of the listed LoT. The user may edit the drug name 1620 to correct a spelling error or to replace a brand name with a generic name or class of drug. The user may right-click 1630 at the primary diagnosis axis to insert or delete a LoT at the mouse cursor position. The user may drag and drop 1640 a dosing interval to edit the start or end date of the interval. The user may right-click 1650 a drug to insert a new drug or delete a drug. The user may right-click 1660 an outcome to insert a new outcome into the record, such as a surgery, procedure, or other priority event.

[0156] Although the controls 1610-1660 are illustrated with respect to mouse events such as click and drag and right click, those skilled in the art will recognize that the controls may be manipulated using different events such as keyboard controls, touch screen controls, or similar interactive manipulation of the input / output devices of a PC or a mobile device interface.

[0157] FIG. 17 is an illustration of a structured data representation of a patient timeline 1700 after LoT labeling has been performed according to one embodiment.

[0158] The patient timeline 1700 represents all significant interval anchors for a patient record into which LoT labels have been inserted. In this example, the patient received a primary cancer diagnosis on 1 / 1 / 2000, started a first line of treatment on 2 / 29 / 2000, and approximately 16 months later, on 8 / 11 / 2001, a progression event was observed, which was recognized as a treatment-internal priority event, and starting on 12 / 9 / 2001, a second line of treatment was administered, and the patient underwent genetic sequencing on 12 / 16 / 2002, likely due to the progression event, such as tumor metastasis to another organ site. As of the sequencing date, the timeline is made available to the treating physician, for example, through an exhaustive web portal 160A-160N. The physician may then utilize the portal to review future LoT outcomes based on similar cohorts of patients to personalize the patient's next LoT to improve the chances of a positive response. Other analyses include evaluation of other treatments, procedures other than those labeled as LoT, evaluation of genetic variants in treatment outcomes, identifying the level of risk that a patient has of developing metastasis or progression at any point during treatment, and other analyses to personalize patient treatment and improve positive outcomes.

[0159] FIG. 18 is a box plot diagram 1800 of the mean duration of LoT identified from a cohort of patients in a lung cancer cohort and a breast cancer cohort.

[0160] As mentioned above, cancer-specific intervals may be learned from training of an artificial intelligence engine on multiple patients. After such learning relates to the average duration of treatment so that the correct dates can be imputed, the intervals are appropriately sized to provide meaningful potential LoT selections for the model to label from. As an example, lung cancer patients experience the majority of treatment intervals at around 70 days, with very few exceeding 120 days. Analysis of LoTs 1-4 shows that there is negligible variation across each successive line of treatment in lung cancer diagnoses. Thus, the lung cancer interval threshold is around 70-120 days and does not change depending on the LoT interval being evaluated. However, for breast cancer diagnoses, the majority of treatment intervals are around 100 days, with the first LoT at 300 days, the second LoT at 290 days, the third LoT at 260 days, and the fourth LoT at around 210 days. Analysis of breast cancer has identified thresholds for drug duration and treatment intervals that are 2-3 times longer than those for lung cancer.

[0161] FIG. 19 is an illustration of LoT frequencies 1900 based on the first LoT for breast cancer therapeutics administered to a patient.

[0162] Based on the model generated via unsupervised learning, the majority of breast cancer patients will begin their first line of treatment with anastrozole, tamoxifen, or AC-T (treatment with the drugs doxorubicin hydrochloride (Adriamycin) and cyclophosphamide followed by paclitaxel (Taxol)). While there appears to be a large variability in the first LoT selection of breast cancer patients, Figure 20 presents the second LoT frequency distribution for breast cancer patients.

[0163] FIG. 20 is an illustration of LoT frequencies 2000 based on a second LoT for breast cancer therapeutics administered to patients.

[0164] Based on the model generated via unsupervised learning, the majority of breast cancer patients receive the first line of treatment in Figure 19 followed by a second line of treatment of anastrozole, tamoxifen, and letrozole. Similarly, note that it makes sense that the AC-T regimen is never prescribed as a second LoT for patients and therefore does not appear in the learned labeling space for second lines of treatment. There is little variability in second LoT selection for breast cancer patients, which is expected as maintenance therapy is a commonly prescribed treatment for the third line of treatment.

[0165] FIG. 21 is an illustration of LoT frequencies 2100 based on a first LoT for non-small cell lung cancer (NSCLC) therapeutics administered to a patient.

[0166] Based on the model generated via unsupervised learning, the majority of NSCLC patients start on carboplatin and paclitaxel, and some patients start on carboplatin and either pemetrexed or etoposide. Another point of accuracy includes that none of the NSCLC patients start on targeted therapy, such as those listed in Figure 22.

[0167] FIG. 22 is an illustration of LoT frequency 2200 based on a second LoT for a non-small cell lung cancer therapeutic administered to a patient.

[0168] Based on the model generated via unsupervised learning, the majority of NSCLC cancer patients will receive the first line of treatment in Figure 21 followed by a second line of treatment with nivolumab or pemetrexed, with a fairly uniform distribution attempting targeted therapy with gemcitabine, docetaxel, or topotecan as the second LoT.

[0169] 19-22 illustrate the frequency of occurrence of a particular LoT across all patients in each primary diagnosis cohort. Although the illustrative examples are limited to breast and lung cancer, the methods and systems disclosed herein may be applied to any cancer diagnosis without departing from the embodiments described herein. Furthermore, the ranking for intervals and potential treatment lines may be derived directly from the likelihood based on frequency. For example, if the first LoT occurs in 50% of patients as the first LoT, it may directly give a 50% likelihood when the respective treatment interval corresponds to it in the LoT labeling. In addition, the artificial intelligence engine may generate further likelihood estimates based on the frequency of LoTs that are consecutive to each other. For example, if a first LoT of carboplatin and paclitaxel is followed by a second LoT of nivolumab 80% of the time, with both carboplatin and paclitaxel and nivolumab in consecutive treatment intervals, the probability that the first LoT is carboplatin and paclitaxel and the second line of treatment is nivolumab will be scaled higher than if the record contained only a first line of carboplatin and paclitaxel or only a second line of nivolumab.

[0170] FIG. 23 is a bar graph representation 2300 of the proportion of patients for each primary cancer who were assigned to a cohort that included at least one of the listed LoTs.

[0171] In this example, breast cancer patients are distributed such that 70% are labeled with the first LoT, 15% are labeled with the second LoT, and only 5% are labeled with the third LoT. Conversely, for a primary diagnosis of colorectal cancer (CRC), only 60% are labeled with the first LoT, 15% are labeled with the second LoT, and 5% are labeled with the third LoT. Each unlabeled patient can be manually evaluated to identify what label, if any, applies to them.

[0172] 24 is an illustration of an exemplary machine of a computer system 2400 upon which a set of instructions may be executed to cause the machine to perform any one or more of the methods described herein. In alternative implementations, the machine may be connected (e.g., networked) to other machines in a LAN, an intranet, an extranet, and / or the Internet.

[0173] The machine may operate in the capacity of a server or client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment, or as a server or client machine in a cloud computing infrastructure or environment. The machine may be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), mobile phone, web appliance, server, network router, switch or bridge, or any machine capable of executing a set of instructions (sequentially or otherwise) that specify actions to be performed by the machine. Moreover, although a single machine is illustrated, the term "machine" shall be taken to include any collection of machines that individually or in concert execute a set of instructions (or multiple sets of instructions) to perform any one or more of the methodologies described herein.

[0174] The exemplary computer system 2400 includes a processing device 2402, a main memory 2404 (read-only memory (ROM), flash memory, dynamic random access memory (DRAM), such as synchronous dynamic random access memory (DRAM) (SDRAM) or DRAM, etc.), a static memory 2406 (flash memory, static random access memory (SRAM), etc.), and a data storage device 2418, which communicate with each other via a bus 2430.

[0175] The processing device 2402 represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit, or the like. More specifically, the processing device may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or a processor implementing a combination of instruction sets. The processing device 2402 may also be one or more special-purpose processing devices, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the like. The processing device 2402 is configured to execute instructions 2422 to perform the operations and steps described herein.

[0176] The computer system 2400 may further include a network interface device 2408 for connecting to a LAN, an intranet, the Internet, and / or an extranet. The computer system 2400 may also include a video display unit 2410 (such as a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 2412 (such as a keyboard), a cursor control device 2414 (such as a mouse), a signal generation device 2416 (such as a speaker), and a graphic processing unit 2424 (such as a graphics card).

[0177] The data storage device 2418 may be a machine-readable storage medium 2428 (also referred to as a computer-readable medium) on which is stored one or more sets of instructions or software 2422 that employs any one or more of the methods or functions described herein. The instructions 2422 may also reside, completely or at least partially, within the main memory 2404 and / or the processing device 2402 during execution by the computer system 2400, with the main memory 2404 and the processing device 2402 also constituting machine-readable media.

[0178] In one implementation, the instructions 2422 include a software library including instructions for a line of care module (such as the line of care module 120 of FIG. 1) and / or a method for functioning as a line of care module. The instructions 2422 may further include instructions for the record reconciliation module 122, the interval generation module 124, and the LoT assignment module 126 (such as the record reconciliation module 122, the interval generation module 124, and the LoT assignment module 126 of FIG. 1). Although the machine-readable storage medium 2428 is shown in the exemplary implementation as being a single medium, the term "machine-readable storage medium" should be interpreted as including a single medium or multiple media (such as a centralized or distributed database, and / or associated caches and servers) that store one or more sets of instructions. The term "machine-readable storage medium" should also be interpreted as including any medium that can store or encode a set of instructions that are executed by a machine and cause the machine to perform any one or more of the methods of the present disclosure. The term "machine-readable storage medium" shall be interpreted accordingly to include, but not be limited to, solid-state memory, optical media, and magnetic media. The term "machine-readable storage medium" shall accordingly exclude transitory storage media, such as signals, unless otherwise specified by identifying the machine-readable storage medium as a transitory storage medium or a transitory machine-readable storage medium.

[0179] In another implementation, the virtual machine 2440 may comprise modules that execute instructions for the record reconciliation module 122, the interval generation module 124, and the LoT allocation module 126. In computing, a virtual machine (VM) is an emulation of a computer system. Virtual machines are based on computer architectures and provide the functionality of a physical computer. Their implementation may involve dedicated hardware, software, or a combination of hardware and software.

[0180] Some portions of the preceding detailed description are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by data processing engineers to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, contemplated to be a self-consistent sequence of operations leading to a desired result. These operations are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0181] It should be remembered, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. As is apparent from the above description, unless otherwise specifically noted, throughout the description, descriptions utilizing terms such as "identifying" or "providing" or "calculating" or "determining" or similar terms will be understood to refer to the actions and processes of a computer system, or similar electronic computing device, that manipulates and converts data represented as physical (electronic) quantities in the computer system's registers and memory into other data that are similarly represented as physical quantities in the computer system's memory or registers or other such information storage devices.

[0182] The present disclosure also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the intended purposes, or may comprise one or more general-purpose computers selectively activated or reconfigured by a computer program stored in the computer. Such computer programs may be stored in a computer-readable storage medium, such as any type of disk, including, but not limited to, a floppy disk, an optical disk, a CD-ROM, and a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic or optical card, or any type of medium suitable for storing electronic instructions, each of which is coupled to a computer system bus.

[0183] The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform the methods. A structure for a variety of these systems may appear as set forth in the description that follows. In addition, the present disclosure has not been described with reference to any particular programming language. It will be understood that a variety of programming languages ​​may be used to implement the teachings of the present disclosure as described herein.

[0184] The present disclosure may be provided as a computer program product or software, which may include a machine-readable medium having instructions stored thereon that may be used to program a computer system (or other electronic device) to perform a process according to the present disclosure. A machine-readable storage medium comprises a mechanism for storing information in a form readable by a machine (e.g., a computer). For example, machine-readable (e.g., computer-readable) media includes machine (e.g., computer) readable storage media such as read-only memory ("ROM"), random access memory ("RAM"), magnetic disk storage media, optical storage media, flash memory devices, and the like.

[0185] In the above specification, implementations of the present disclosure have been described with reference to certain exemplary implementations. It will be apparent that various modifications may be made thereto without departing from the broader spirit and scope of the implementations of the present disclosure as defined in the following claims. Accordingly, the specification and drawings should be considered in an illustrative rather than a restrictive sense. [Explanation of symbols]

[0186] 100 Computer-implemented systems 110 Clinical Health Records 112 EMR 114 Records, Information 120 Treatment Line Module 122 Record Harmonization Module 124 Interval Generation Module 126 Treatment Line Allocation Module 150 Line of Treatment (LoT) Store 160a~160n Web Form 160A~160N Webpage 165 Graphical User Interface (GUI) 170a~170n GUI, electronic reports 200 Systems 300 Record Harmonization Module 322 Record Whitelist and / or Record Blacklist Module 324 Record Association Module 326 Recording Combination Module 328 Heuristic Processing Module 400 Interval Generation Module 422 Diagnosis Relevance Filter 424 Interval Date Allocation 426 Interval Date Imputation 428 Interval Date Extension 430 Interval Date Segmentation 500 Interval Generation Module, Treatment Line Allocation Module 522 LoT Allocation Estimate 524 LoT allocation probability 526 LoT ranking 528 LoT labeling 600 System 605 Drug Filtering Module 610 Drug Combination Module 615 Drug Reconciliation Module 620 Harmonized Medication Table 625 Primary Cancer (PC) Relevance Filter 630 Date Padding 635 PC Curation Model 640 Interval Generation 645 Dosage Interval 650 Outcome 655 Modeled LoT Frequency 660 Estimated Probability 665 LoT labeling 670 Post-processing LoT 705 Clinical Insight 710 Model data preparation stage 712 Curated Progression and Treatment Deviations 714 Patient Medication 716 Module 718 Modeled Dates and Intervals 720 Training Phase 722 Drug measurement interval frequency (w) 724 LoT split allocation 726 Refine medication frequency 730 Model LoT Allocation Phase 732 LoT generation stage 734 LoT labeling 802 Primary diagnosis 804 Drug Administration 806 Outcome 808 Triplet chemotherapy 810 Pemetrexed / bevacizumab maintenance therapy 812 Progressive Disease Outcome 814 Nivolumab 816 Progressive Disease Outcome 818 Gemcitabine therapy 820 Paclitaxel 822 Complete Response 824 Dexamethasone administration 830a Spacing 830b interval 830c spacing 830d interval 830e spacing 930a Spacing 930b spacing 930c spacing 930d interval 1030a spacing 1030b spacing 1030c spacing 1030d spacing 1030e spacing 1100 Flow 1200 Flow 1300 Flow 1400 Sankey Diagram 1500 Progression-free survival (PFS) graph 1510, 1520, 1530 Treatment Lines 1600 Treatment Line Editing Tool 1610 Control 1620 Drug name, control 1630 Control 1640 Control 1650 Control 1660 Control 1700 Patient Timeline 1800 Box Plots 1900 LoT Frequency 2000 LoT Frequency 2100 LoT Frequency 2200 LoT Frequency 2300 Bar Graph 2400 Computer System 2402 Processing Device 2404 Main memory 2406 Static Memory 2408 Network Interface Device 2410 Video Display Unit 2412 Alphanumeric Input Device 2414 Cursor Control Device 2416 Signal Generating Device 2418 Data Storage Device 2422 Instruction set or software, instructions 2424 Graphic Processing Unit 2428 Machine-readable storage medium 2430 Bus 2440 Virtual Machines

Claims

1. 1. A computer-implemented method for labeling one or more drugs co-administered to a patient as a line of therapy, comprising: deriving curated fields from digital records via an artificial intelligence engine trained on patient histories, the digital records including multiple different record types or multiple records from different sources; identifying medical records of the patient from a plurality of digital records, the plurality of digital records including records that include the derived plurality of curated fields, the subset of medical records reflecting the patient's medical history with respect to one or more treatments following diagnosis of the patient's medical condition; generating a plurality of treatment intervals from the subset of medical records, the treatment intervals including at least one medication administered to the patient and a time interval, wherein a given time interval can be associated with a plurality of treatment intervals; associating said one or more therapeutic agents with respective treatment intervals when said administration of said agents falls within said time intervals; refining the time interval for each treatment interval when one of the one or more treatments is outside the time interval but within an extension period, where no interval priority event occurs during the refined time interval; identifying, via an unsupervised artificial intelligence engine, one or more potential lines of treatment from the plurality of treatment intervals, each potential line of treatment comprising one or more of the plurality of treatment intervals and a maximum likelihood estimate, the one or more of the plurality of treatment intervals of the potential line of treatment being a line of treatment; and labeling, via the unsupervised artificial intelligence engine, the potential line of treatment having the highest maximum likelihood estimate as the line of treatment.

2. 2. The method of claim 1, further comprising labeling, via the unsupervised artificial intelligence engine, additional potential lines of treatment in successive time intervals as lines of treatment occurring chronologically after each preceding line of treatment and numbered incrementally from the potential line of treatment having the respective highest maximum likelihood estimate of the potential line of treatment in each successive time interval.

3. performing the labeling step for a plurality of patients; Identifying a plurality of cohorts of patients; reporting the incrementally numbered lines of therapy for each of the multiple cohorts of patients; The method of claim 2 further comprising:

4. 4. The method of claim 3, further comprising reporting facility-wide and physician-specific compliance with standardized treatment guidelines.

5. 4. The method of claim 3, further comprising reporting changes in lines of therapy over time based at least in part on the respective time intervals for each line of therapy.

6. 4. The method of claim 3, further comprising reporting an analysis regarding differences in treatment impact between a first cohort of patients and a second cohort of patients, the cohorts identifying patients as having one or more of similar treatment dates, lines of treatment, diagnoses, outcomes, geographic locations, treating physicians, treating facilities, genomic markers, or clinical characteristics.

7. 7. The method of claim 6, further comprising reporting an analysis regarding progression-free survival between the first cohort and the second cohort.

8. generating progression-free survival curves for a first cohort of patients having at least one labeled line of treatment in common; generating progression-free survival curves for a second cohort of patients having at least one other labeled line of treatment in common; 4. The method of claim 3, further comprising displaying the progression-free survival curve for the first cohort on a survival graph and the progression-free survival curve for the second cohort on the same survival graph.

9. 9. The method of claim 8, wherein the labeled line of therapy includes one or more of a drug name, a therapeutic roll-up class, a dosage, or a method of administration.

10. The method of claim 1 , further comprising the step of generating a report, the report including each line of therapy labeled for the patient and the respective time interval.

11. 10. The method of claim 1, wherein the medical condition diagnosis includes cancer, heart disease, depression, mental health, diabetes, infectious diseases, epilepsy, dermatology, and autoimmune diseases.

12. The method of claim 1 , wherein the plurality of digital records includes one or more EHR medical records and one or more curated medical records.

13. The method of claim 12 , wherein curated medical records are images of records not included in the one or more EHR medical records.

14. 13. The method of claim 12, further comprising binarizing, optical character recognizing, and encoding the curated medical records into a structured format.

15. 10. The method of claim 1, further comprising identifying, by a treatment identifier, the treatment administered to the patient and the date on which the treatment was administered.

16. Treatment is administration of a drug, agent, molecule, or chemical; Implantation of medical devices, Use of medical devices, Use of biological therapy, viral therapy, phage therapy, phytotherapy, gene therapy, epigenetic therapy, protein therapy, enzyme replacement therapy, hormone therapy, cellular therapy, immunotherapy, antibody therapy, nutritional therapy, electromagnetic radiation therapy, or radiotherapy; Surgical procedures, or radiosurgery 2. The method of claim 1, comprising one or more of:

17. 1. A system for labeling one or more drugs co-administered to a patient as a line of therapy, comprising: At least one computer having at least one processor, the processor comprising: deriving a plurality of curated fields from a plurality of digital records via an artificial intelligence engine trained on a plurality of patient histories, the digital records including a plurality of different record types or a plurality of records from different sources; identifying medical records of the patient from a plurality of digital records, the plurality of digital records including records that include the derived plurality of curated fields, the medical record subset reflecting a medical history of the patient with respect to one or more treatments following diagnosis of the patient's medical condition; generating a plurality of treatment intervals from the subset of medical records, the treatment intervals including at least one medication administered to the patient and a time interval, wherein a given time interval can be associated with a plurality of treatment intervals; associating the one or more therapeutic agents with respective treatment intervals when the administration of the agents falls within the time intervals; refining the time interval for each treatment interval when one of the one or more treatments is outside the time interval but within an extension period, wherein no interval priority event occurs during the refined time interval; At least one computer programmed to: The at least one computer includes an unsupervised artificial intelligence engine, the unsupervised artificial intelligence engine comprising: identifying one or more potential lines of treatment from the plurality of treatment intervals, each potential line of treatment including one or more of the plurality of treatment intervals and a maximum likelihood estimate, the one or more of the plurality of treatment intervals of the potential line of treatment being a line of treatment; labeling the potential line of treatment having the highest maximum likelihood estimate as the line of treatment; A system that is programmed to:

18. 20. The system of claim 17, wherein the unsupervised artificial intelligence engine is programmed to label additional potential lines of treatment in successive time intervals as lines of treatment occurring chronologically after each preceding line of treatment and incrementally numbered from the potential line of treatment having the highest maximum likelihood estimate of the respective potential line of treatment in each successive time interval.

19. The unsupervised artificial intelligence engine includes: labeling a plurality of patients; Identifying multiple cohorts of patients; 20. The system of claim 18, programmed to report the incrementally numbered lines of therapy for each of the multiple cohorts of patients.

20. The at least one computer 20. The system of claim 19, further programmed to report facility-wide and physician-specific compliance with standardized treatment guidelines.

21. The at least one computer 20. The system of claim 19, programmed to report changes in lines of therapy over time based at least in part on the respective time intervals for each line of therapy.

22. The at least one computer 20. The system of claim 19, further programmed to report an analysis regarding differences in treatment impact between the first cohort of patients and the second cohort of patients, the cohorts identifying patients as having one or more of similar treatment dates, lines of treatment, diagnoses, outcomes, geographic locations, treating physicians, treating facilities, genomic markers, or clinical characteristics.

23. The at least one computer 23. The system of claim 22, further programmed to report an analysis regarding progression-free survival between the first cohort and the second cohort.

24. The at least one computer generating progression-free survival curves for the first cohort of patients having at least one labeled line of treatment in common; generating progression-free survival curves for a second cohort of patients having at least one other labeled line of treatment in common; 20. The system of claim 19, further programmed to display the progression-free survival curve for the first cohort on a survival graph and the progression-free survival curve for the second cohort on the same survival graph.

25. The labeled lines of therapy are:

25. The system of claim 24, including one or more of a drug name, a therapeutic rollup class, a dosage, or a method of administration.

26. 20. The system of claim 17, wherein medical condition diagnoses include cancer, heart disease, depression, mental health, diabetes, infectious diseases, epilepsy, dermatology, and autoimmune diseases.

27. 20. The system of claim 17, wherein the plurality of digital records includes one or more EHR medical records and one or more curated medical records.

28. The at least one computer 28. The system of claim 27, further programmed to binarize, optically character recognize, and encode the curated medical records into a structured format.

29. Treatment is administration of drugs, substances, molecules, or chemicals; Implantation of medical devices, Use of medical devices, Use of biological therapy, viral therapy, phage therapy, phytotherapy, gene therapy, epigenetic therapy, protein therapy, enzyme replacement therapy, hormone therapy, cellular therapy, immunotherapy, antibody therapy, nutritional therapy, electromagnetic radiation therapy, or radiotherapy; Surgical procedures, or radiosurgery 20. The system of claim 17, comprising one or more of:

30. A non-transitory computer-readable storage medium that, when executed by one or more processors, causes at least one computer to: deriving a plurality of curated fields from a plurality of digital records via an artificial intelligence engine trained on a plurality of patient histories, the digital records including a plurality of different record types or a plurality of records from different sources; identifying medical records of the patient from a plurality of digital records, the plurality of digital records including records that include the derived plurality of curated fields, the medical record subset reflecting a medical history of the patient with respect to one or more treatments following diagnosis of the patient's medical condition; generating a plurality of treatment intervals from the subset of medical records, the treatment intervals including at least one medication administered to the patient and a time interval, wherein a given time interval can be associated with a plurality of treatment intervals; associating the one or more therapeutic agents with respective treatment intervals when the administration of the agents falls within the time intervals; refining the time interval for each treatment interval when one of the one or more treatments is outside the time interval but within an extension period, wherein no interval priority event occurs during the refined time interval; and storing computer executable instructions for causing the The at least one computer includes an unsupervised artificial intelligence engine, the unsupervised artificial intelligence engine comprising: identifying one or more potential lines of treatment from the plurality of treatment intervals, each potential line of treatment including one or more of the plurality of treatment intervals and a maximum likelihood estimate, the one or more of the plurality of treatment intervals of the potential line of treatment being a line of treatment; labeling the potential line of treatment having the highest maximum likelihood estimate as the line of treatment; A non-transitory computer-readable storage medium programmed to perform the steps of:

Images