Prediction of Response to Immunotherapy Using Deep Learning Analysis of Image Data and Clinical Data

Abstract

The method includes providing a pre-treatment image of the target subject to at least one deep learning model that is uniquely trained to predict immunotherapy treatment response. The method further includes a processing device generating a predicted treatment response score to the treatment based on the single pre-treatment image and the at least one deep learning model. The method further includes providing a recommended treatment plan based on the predicted treatment response score.

Description

Technical Field

[0001] 〔Cross - Reference to Related Applications〕 This application claims the benefit of U.S. Provisional Patent Application No. 63 / 056,393, filed Jul. 24, 2020, which is hereby incorporated by reference in its entirety.

[0002] This disclosure relates to predicting immunotherapy treatment responses using deep learning analysis, and more specifically, to systems and methods for predicting responses to PD - [L]1 and CTLA - 4 immune checkpoint inhibitors using deep learning analysis of image data and clinical data.

Summary of the Invention

[0003] This disclosure will be more fully understood from the following detailed description and the accompanying drawings of various implementations of the disclosure.

Brief Description of the Drawings

[0004]

Figure 1

[0005]

Figure 2

[0006]

Figure 3A

[0007]

Figure 3B

[0008]

Figure 4

[0009]

Figure 5

DETAILED DESCRIPTION OF THE INVENTION

[0010] Embodiments of the present invention relate to the field of predicting immunotherapy treatment using deep learning analysis, and more specifically, to systems and methods for predicting responses to PD-[L]1 and CTLA-4 immune checkpoint inhibitors using deep learning analysis of image data and clinical data.

[0011] Immunotherapy has revolutionized cancer treatment in stage IV metastatic tumors such as non-small cell lung cancer and melanoma using checkpoint inhibitors of the programmed cell death 1 (PD-1) and anti-cytotoxic T lymphocyte antigen-4 (CTLA-4) classes (e.g., pembrolizumab, nivolumab, atezolizumab, ipilimumab, tremelimumab). However, in addition to unpredictable and low patient response rates, high drug costs and severe toxicities can impose a significant burden on the healthcare system, third-party payers, and patients. As PD-[L]1 and CTLA-4 checkpoint inhibitors continue to be adopted, there is no doubt that a diagnostic tool is needed to stratify patients according to response likelihood. Recent efforts to explore the utility of quantitative imaging biomarkers for predicting responses to PD-[L]1 and CTLA-4 immunotherapy have been effective.

[0012] However, PD-1 / PD-L1 and CTLA-4 checkpoint blockade therapies have many drawbacks, such as a low response rate of generally 15% - 20% in most diseases when used as monotherapies, high treatment costs globally (over $150,000 per year in the United States), and serious immune-mediated adverse events. As described above, in addition to unpredictable and low patient response rates, high drug costs and severe toxicity can impose a significant burden on the healthcare system, third-party payers, and patients.

[0013] Numerous approaches have been investigated to predict the response to PD-1 / PD-L1 and CTLA-4 checkpoint therapies, but the success cases are limited. In one embodiment, immunohistochemistry (IHC) assays are used to measure the level of PD-L1 protein expressed in tumor samples. In multiple clinical trials involving PD-[L]1, the amount of tumor gene mutations, the presence of tumor-infiltrating lymphocytes, and inflammatory cytokines are often studied in combination with additional immuno-oncology (IO) therapies such as CTLA-4 checkpoint inhibitors.

[0014] In some embodiments, efforts to explore the utility of quantitative imaging biomarkers for predicting the response to PD-[L]1 immunotherapy are considered promising. With such an approach, non-invasive image scans can provide insights and information regarding the entire tumor burden of a patient, rather than a sample of a subset of lesions (as obtained by biopsy or serum-based analysis). If relevant image features can be identified by further analyzing diagnostic images showing all treatable lesions using computational techniques such as machine learning and artificial intelligence, it may be possible to achieve an accurate comprehensive assessment of patient response to PD-[L]1 and / or CTLA-4 therapies.

[0015] The embodiments shown in this specification are not limited to the following, but by describing a multi-omic classifier for predicting responses to PD-1 / PD-L1 and CTLA-4 checkpoint blockade in various clinical indications, including non-small cell lung cancer (NSCLC), melanoma, bladder cancer, and breast cancer, the above and other problems are advantageously overcome. In one embodiment, the classifier is developed based on diagnostic image scans at baseline and follow-up intervals, and training data including existing biomarkers, related clinical data, molecular data, demographic data, response data, and survival data. Examples of existing biomarkers used in clinical practice include PD-L1 expression immunohistochemistry, tumor mutational burden (TMB), mismatch repair (MMR), microsatellite instability (MSI), and neutrophil-to-lymphocyte ratio (NLR). Furthermore, there is also initial evidence suggesting that clinical tests such as lactate dehydrogenase (LDH), S100 protein, and related serum proteins can predict immunotherapy response and pseudoprogression. In the near future, features and biomarkers extracted from the microbiome are also expected to play an important role.

[0016] To summarize the general method sequence of one embodiment of the present disclosure, when sufficient patient data is anonymized and accumulated, the image data (both baseline scans and follow-up scans) is annotated (segmented) to specify lesions, lymph nodes, surrounding organs, etc., and annotated to clinical notes and other calculated metrics (changes in tumor volume) to evaluate the response or disease progression in each lesion at the patient level, generating a (response evaluation criteria for solid tumors) RECIST score. The preprocessing layer normalizes the image data based on the reconstruction kernel and hardware parameters (slice width of the CT scanner), and a convolutional neural network (CNN) processes the annotated image data and clinical cohort features.

[0017] In one embodiment, terms such as "target", "target lesion", "target subject" can mean a nodule, lesion, tumor, metastatic mass or anatomical structure near (within some defined vicinity of) the treatment site. In another embodiment, the target can be a bone structure or bone metastasis. In yet another embodiment, the target can mean the soft tissue of the patient. The target can be any defined structure or site that can be identified and tracked as described herein (including the patient as a whole).

[0018] Furthermore, for the sake of convenience and brevity, PD-1 and CTLA-4 are frequently referred to, but the embodiments disclosed herein are not limited to the following, and are equally applicable to any other form of immunotherapy, chemotherapy and radiotherapy, including other forms of treatment. Furthermore, PACS as used herein means Picture Archiving and Communication System, and DICOM means Digital Imaging and Communications in Medicine in the medical field.

[0019] FIG. 1 is a diagram showing a machine learning system 100 used in an embodiment of the present disclosure. Although specific components are disclosed within the machine learning system 100, it should be understood that such components are only examples. That is, the embodiments of the present invention are also fully compatible with various other components or modified examples of components described within the machine learning system 100. It should be understood that the components within the machine learning system 100 can also operate with components other than the presented components, and not all components of the machine learning system 100 are necessary to achieve the goals of the machine learning system 100.

[0020] In one embodiment, system 100 includes server 101, network 106, and client device 150. Server 100 can include various components that enable prediction of responses to PD-1 checkpoint blockade (and other immunotherapy treatments) for image data and clinical data on a server device or client device using deep learning analysis. Each component can perform different functions, operations, actions, processes, methods, etc. for a web application and / or provide different services, functionality, and / or resources for a web application. Server 100 can include a machine learning architecture 127 of a processing device 120 that executes operations related to predicting responses to PD-1 checkpoint blockade for image data and clinical data using deep learning analysis using a trained model. In one embodiment, processing device 120 is one or more graphics processing units of one or more servers (including, for example, server 101). Further details of machine learning architecture 127 are shown with respect to the remaining figures of the present disclosure. Server 101 can further include network 105 and data store 130.

[0021] The processing device 120 and the data store 130 are operably coupled to each other via the network 105 (e.g., can be operably coupled, can be communicatively coupled, can convey data / messages to each other). The network 105 can be a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), or a combination thereof. In one embodiment, the network 105 can be provided by one or more wireless communication systems such as a Wi-Fi hotspot connected to the network 105 and / or a wireless carrier system that can be implemented using various data processing facilities, communication towers (e.g., cell towers), etc., and can include a wired or wireless infrastructure. The network 105 can carry communications (e.g., data, messages, packets, frames, etc.) between various components of the server 101. The data store 130 can be a persistent storage that can store data. The persistent storage can be a local storage unit or a remote storage unit. The persistent storage can be a magnetic storage unit, an optical storage unit, a solid state storage unit, an electronic storage unit (main memory), or a similar storage unit. Also, the persistent storage can be a monolithic / single device or a set of distributed devices.

[0022] Each component can include hardware such as a processing device (e.g., a processor, a central processing unit (CPU), a graphics processing unit (GPU)), a memory (e.g., a random access memory (RAM), a storage device (e.g., a hard disk drive (HDD), a solid state drive (SSD), etc.)), and other hardware devices (e.g., a sound card, a video card, etc.). Server 100 can include any suitable type of computer device or machine having a programmable processor, including, for example, a server computer, a desktop computer, a laptop computer, a tablet computer, a smartphone, a set-top box, etc. In some examples, Server 101 can include a single machine or multiple interconnected machines (e.g., multiple servers configured in the form of a cluster). Server 101 can be implemented by a common entity / organization or by different entities / organizations. For example, Server 101 can be operated by a first company / corporation, and a second server (not shown) can be operated by a second company / corporation. As will be described in detail below, each server can execute or include an operating system (OS). The OS of the server can manage the execution of other components (e.g., software, applications, etc.) and / or access to the hardware of the computer device (e.g., a processor, a memory, a storage device, etc.).

[0023] As described herein, server 101 can provide machine learning capabilities to client devices (e.g., client device 150). In one embodiment, server 101 is operably connected to client device 150 via network 106. Network 106 can be a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or wide area network (WAN)), or a combination thereof. In one embodiment, network 106 can include a wired or wireless infrastructure provided by one or more wireless communication systems such as a Wi-Fi hotspot connected to network 106 and / or a wireless carrier system that can be implemented using various data processing facilities, communication towers (e.g., cell towers), etc. Network 106 can carry communications (e.g., data, messages, packets, frames, etc.) between various components of system 100. Further implementation details of the operations performed by system 101 will be described with respect to the remaining figures of the present disclosure.

[0024] In one embodiment, system 101 can operate based on any of the following data and any other suitable data that may be contemplated. TIFF0007702477000001.tif229169

[0025] FIG. 2 is a flowchart of a method for predicting immunotherapy treatment using deep learning analysis according to an embodiment of the present disclosure. Generally, each method described herein (including method 200) can be performed by processing logic that can include hardware (e.g., a processing device, circuitry, dedicated logic, programmable logic, microcode, hardware of a device, etc.), software (e.g., instructions that operate or execute on a processing device), or a combination thereof. In some embodiments, these methods can be performed by the processing logic of machine learning architecture 127 of FIG. 1.

[0026] Method 200 begins, at block 201, by providing a pre-treatment image of a target subject optionally including lesion annotations or seed points to at least one deep learning model that is uniquely trained to predict a treatment response (e.g., an immunotherapy treatment) based on a single lesion or multiple lesions. In embodiments, other types of machine learning models can be used instead of or in combination with the at least one deep learning model. In some embodiments, after generating a large number of sets of predetermined images and clinical features, a feature selection algorithm (e.g., minimum redundancy maximum relevance (MRMR) or least absolute shrinkage and selection operator (LASSO)) is applied and a machine learning method (e.g., gradient boosting decision tree, random decision forest, or support vector machine) is used to fit these to generate a prediction model. Any lesion annotations or seed points provided to block 201 can be generated manually by a clinical user or automatically by an automatic segmentation and / or target detection method. An example of an automatic segmentation or target detection method is a convolutional neural network model. To predict the treatment response of a single lesion, multi-parametric optimization techniques such as the Stochastic Gradient Descent (SGD) algorithm, the RMSprop algorithm, or the Adaptive Momentum (Adam) algorithm are used to train the model to maximize the agreement between the lesion response predicted by the model and the lesion response determined by a human expert (e.g., a radiologist).

[0027] Examples of lesion responses can include numerical evaluations (e.g., changes in lesion volume, changes in one or more major dimensions of the lesion, changes in image intensity within the lesion), tumor growth rate (TGR), or categorical evaluations (e.g., responding lesion, stable lesion, progressive lesion, new lesion). By aggregating one or more lesion-level model predictions, treatment response prediction at the patient level is performed. In one embodiment, the aggregation from lesion-level response prediction to patient-level response prediction is performed by a series of rules and logical operations.

[0028] In an embodiment, after calculating a per-lesion response score for multiple lesions of a single patient, mathematical operations such as a maximum score, a minimum score, and / or an average score can be performed to convert the response prediction for each of the multiple lesions into a single patient-level response prediction. In one embodiment, the aggregation from lesion-level response prediction to patient-level response prediction is performed by a second model that is specially trained to take predictions from one or more lesion-level models as inputs and make a patient-level response prediction. In some embodiments, in order to consider variable lesions (e.g., model inputs), the input to the model can be a lesion-level prediction statistic (e.g., mean value, median value, standard deviation, etc.). In another embodiment, the model can be a recurrent neural network (RNN) model in which multiple lesion predictions are represented as a variable-length input sequence.

[0029] Examples of patient-level models can include, but are not limited to, artificial neural networks, random forest models, support vector machines, and logistic regression models. In another embodiment, a single machine learning model that considers multiple lesions simultaneously can be used. Such an embodiment can effectively remove the hierarchy of per-lesion models and per-patient models. In one embodiment, the pre-treatment image can be a two-dimensional anatomical image, a three-dimensional anatomical image, or a four-dimensional anatomical image. In another embodiment, two or more different types of treatment images can be used.

[0030] The treatment images can be taken at the time of diagnosis (before the start of treatment) or at any other suitable time. The treatment images can be, but are not limited to, computerized tomography (CT) scans, positron emission tomography (PET) scans, or magnetic resonance imaging (MRI) scans. At least one deep learning model can include any suitable various machine learning models, including, but not limited to, convolutional neural networks. In one embodiment, the models are trained based on the same data using different hyperparameters and optimization techniques. In another embodiment, the models are trained based on different data, such as using different techniques with different purposes, and the results can be aggregated in various ways.

[0031] The deep learning model can utilize various suitable training methods. For example, in one embodiment, the deep learning model uses a training subject population and a plurality of images associated with each of the plurality of training subjects as training data. In another embodiment, the deep learning model uses a calculated subject-specific model as training data. In yet another embodiment, the deep learning model uses a combination of the two methods described above.

[0032] In an embodiment, the treatment is PD-[L]1 immune checkpoint inhibitor treatment. The PD-[L]1 immune checkpoint inhibitor treatment can be PD-1-based treatment or PD-L1-based treatment. In yet another embodiment, the treatment is CTLA-4-immune checkpoint inhibitor treatment, or any other suitable treatment type (e.g., chemotherapy, targeted therapy, drug-based treatment, radiation therapy, etc.).

[0033] In block 203, processing logic generates a predicted treatment response score for immunotherapy treatment (e.g., on a scale representing from the least likely to result in a positive or negative effect to the most likely to result in a positive or negative effect) based on a deep learning model (e.g., by a processing device). In some embodiments, the predicted treatment response score can be a numerical value. In one embodiment, the processing logic generates a predicted treatment response score based on a single pre-treatment image and at least one deep learning model. For example, in one embodiment, results from different models can be combined (e.g., averaged, or combined in any other way) to generate a single response score. In one embodiment, one or more non-image features (e.g., genomic tests, electronic medical record information, PD-L1 immunohistochemistry assays, etc.) can be used to generate the predicted response score. In another embodiment, one or more non-image features can be combined with one or more image features to generate the predicted response score.

[0034] In one embodiment, the predicted treatment response score includes a prediction of patient progression with respect to a given pharmaceutical. In another embodiment, the predicted treatment response score indicates a prediction of one or more immune-related adverse events associated with immunotherapy treatment. In one embodiment, the predicted treatment response score can include the predicted likelihood (e.g., confidence) of a particular type of response and / or adverse event occurrence. In another embodiment, the response score can also include an indication of pseudoprogression characterized by a short-term and transient increase in tumor volume due to natural swelling and / or inflammation (e.g., in response to treatment) rather than disease progression. In another embodiment, the response score can reflect the likelihood of hyper-progression, a severe condition with rapid clinical deterioration where disease progression accelerates during treatment administration. In another embodiment, the response score can be formulated to indicate progression-free or overall patient survival in months or years.

[0035] In block 205, the processing logic provides a recommended treatment plan based on a predicted treatment response. For example, the recommended treatment plan can include instructions on whether a particular pharmaceutical should be used, the dosage of such a product, the timing associated with the administration of such a product, etc., based on the predicted treatment response. In an embodiment, this instruction can identify whether a patient is likely to respond to a particular pharmaceutical. In one embodiment, a lesion-specific treatment plan for enhancing the treatment effect of high-risk lesions is created by using the prediction of the response to immunotherapy and / or chemotherapy for each lesion and combining ongoing systemic therapy and local therapy. The local therapy can be any one of stereotactic ablative radiation therapy (SBRT), intensity modulated radiation therapy (IMRT), conformal radiation therapy (CRT), radiosurgery, surgical resection, thermal ablation, cryoablation, or high intensity focused ultrasound (HIFU) therapy. In another embodiment, the recommended treatment plan for a patient for whom a high progression risk is predicted by the model can be to combine chemotherapy or CTLA-4 immunotherapy with PD-[L]1 immunotherapy and add it to maximize the likelihood of treatment response. In another embodiment, the recommended treatment plan can be to discontinue one or all treatment methods to maximize the quality of life of the patient. In an embodiment, the processing logic can generate other outputs based on the predicted treatment response score, instead of or in combination with the recommended treatment plan. For example, the processing logic can generate a report based on the predicted treatment response score.

[0036] In block 207, the processing logic can receive an intra-treatment follow-up image.

[0037] In block 209, the processing logic can provide the intra-treatment follow-up images to the machine learning model.

[0038] In block 211, the processing logic can generate the latest predicted treatment response score.

[0039] In block 213, the processing logic can provide the latest recommended treatment plan based on the latest predicted treatment response score.

[0040] In various embodiments, the processing logic can perform any number of suitable preprocessing operations and postprocessing operations that can enhance the accuracy, efficiency, and / or compatibility of the machine learning model in the current context. For example, with regard to preprocessing, conventional radiomics methods may be susceptible to variations in scanner hardware and imaging protocols. The data preprocessing and data augmentation system described herein is designed to optimize the model generalizability and minimize the model's sensitivity to variations in imaging hardware and protocols.

[0041] In the field of machine learning, especially deep learning, strategies for improving the model generalizability are known. For each category, the following and other methods are envisioned.

[0042] 1. Select a model size (number of parameters) that achieves the optimal balance between underfitting and overfitting of the available training data. A) MLops (e.g., machine learning and operations) frameworks and infrastructure enable monitoring of the model's key performance indicators (KPIs) and allow for continuous adjustment of the model complexity and architecture as more data is acquired.

[0043] 2. Maximize the diversity of the training dataset. A) The training data is supplied from various institutions (academic institutions, small local centers, large payer / provider networks) and can reflect various clinical practice trends and various imaging hardware and radiation protocols (for example, among local cancer centers, some use a CT protocol with 5 mm thick slices, while research institutions tend to use thin slice scans with a high resolution of 1 - 2 mm). B) A database system can be used to internally catalog the training data to ensure an appropriate distribution of imaging hardware and protocols during model training.

[0044] 3. Normalize the input data. A) During model training and model inference, scans can be resampled to a consistent resolution (for example, a voxel spacing of 1.0×1.0×1.0 mm). This significantly reduces the dependence of model performance on CT slice thickness. B) Image voxel intensities can be normalized by excluding intensity outliers (such as metal artifacts from fiducials, pacemakers, wires, etc.) and rescaling the intensities to a consistent range (for example, an intensity distribution with a mean of 0 and a variance of 1). C) If multiple reconstruction protocols are available for a given imaging session, the reconstruction protocol that most closely matches the "gold standard" protocol can be used.

[0045] 4. Augment the training data by generating synthetic training examples that simulate realizable scenarios not represented in the available training data. A) An online augmentation strategy can be used that means new variations of training data are continuously generated as long as the model is being trained. In practice, this strategy means that the number of unique training examples is infinite and limited only by the time spent in the model training loop. The online augmentation loop performs model shift, rotation, rescaling operations, deformations, and intensity perturbations to generate new unique training cases. B) Physics-based principles can be used to generate noise and intensity variations and simulate differences between scanner hardware and scan protocols. Examples of physics-based methods include ray tracing and Monte Carlo photon simulations in existing clinical CT scans that generate diverse CT projection data, which can then be used to reconstruct new CT scans with alternative imaging protocols and simulated artifacts. Examples of simulated artifacts include different primary beam energies, beam scatter and hardening characteristics, patient motion artifacts, and imaging dose variations.

[0046] 5. Model input using multiple resolutions and region of interest (ROI) sizes. A) The CNN model may prefer one or more small regions (ROIs) of CT scans as input. Redundant representations of the input CT images (or small regions) near the tumor location can be created using ROIs of various sizes and resolutions. By using multiple ROI sizes, the model can accommodate tumors of different sizes and shapes. For example, if only an ROI covering 5×5×5 cm around the tumor is used, the model may not function well for large tumors. Conversely, if a 50×50×50 cm ROI is used, the classifier may not function well for small tumors that require high spatial resolution and high fidelity. By combining an ROI region with small spatial dimensions and an ROI region with large spatial dimensions in one model, complementary learning of image features in the local context (e.g., tumor shape, texture, intensity profile) and the global context (e.g., location of the lesion relative to the body and other organs, lymph node metastasis, patient's mass composition and muscle reserve, overall health or vital organs, microcalcifications, etc.) becomes easier, and ultimately a treatment response and survival prediction model with high predictability and robustness can be obtained.

[0047] Regarding post-processing, various techniques can be used to post-process individual model predictions to obtain the prediction accuracy and interpretability required by clinical end-users. Examples of post-processing methods that can be used include, but are not limited to, the following.

[0048] 1. Model Ensemble: Ensemble (or bagging) is a way to improve the stability and overall performance of a model. Instead of training a single model for a given task, multiple variations of the model are trained (by perturbing training hyperparameters, weight initialization, model architecture, training set distribution, etc.). Then, these models are used simultaneously (ensemble prediction) by calculating the consensus among the multiple models. In one embodiment, the average or median prediction from multiple models is on average more accurate than a single prediction. Examples of ensembling operations that combine multiple model predictions can be simple averaging, median calculation, the STAPLE algorithm (Simultaneous Truth and Performance Level Estimation by Warfield et al.), or a dedicated ensembling model such as a linear classifier, random forest, support vector machine, or neural network.

[0049] 2. Bottom - up Model Aggregation: In some clinical applications, the concept of training a classification model to predict the single - lesion response to a therapeutic agent may be desirable. In some clinical scenarios, it is a clinical requirement to predict the treatment response at the patient level (considering that some lesions respond while others continue to progress, whether this patient is likely to benefit from a given treatment overall). In this scenario, the concept of model ensemble may also be applicable. However, in this application, each single - lesion model (or sub - ensemble of models) contributes to the overall patient - level prediction estimated by ensembling the individual lesion predictions. Further, by combining the predictions of each model within a large - scale ensemble and incorporating other clinical factors, biomarkers, and / or image features, the processing logic can predict the treatment response at the patient level rather than at the lesion level.

[0050] 3. Explainability: The response of the deep convolutional network model can be decomposed into the activation of dominant features to highlight which spatial, texture, and morphological features had the most impact on the prediction. For example, the explanation can predict a "high risk of lesion progression" for reasons such as 1. the lesion volume exceeds 50 cc, 2. the lesion location is at the apex of the lung, 3. the tissue heterogeneity of the central and peripheral parts of the lesion is low, 4. there are metastatic bone lesions, etc. In related embodiments, the processing device can explain and assist in model response prediction or prediction of immune-related adverse events by presenting reference data and past cases of patients with similar presentations and medical history profiles.

[0051] Incorporation of temporal information: In one embodiment, the treatment prediction model can be considered as a "single-shot" prediction at the baseline time for determining the future treatment process, or a continuous integration process that incorporates image information and electronic medical record (EMR) information along the treatment course to provide continuous decision-making support to the clinician. In one embodiment, the treatment response model is trained to predict the likelihood of disease progression, pseudoprogression, or overprogression of the patient using the baseline and the first in-treatment follow-up scan. In this clinical scenario, using the model prediction can significantly shorten the timeline for making treatment decisions or adjustments such as switching the patient to another therapeutic agent, adding a secondary therapeutic agent, or stopping the treatment. In the case of a prediction model that incorporates multiple imaging time points, the temporal data can be integrated in various ways (two imaging time points can be used for illustrative purposes).

[0052] 1. Approach #1: Calculate the difference in image features between Scan #1 and Scan #2, and then use this to create a prediction model. In one embodiment, an image feature set can be calculated separately for Scan #1 and Scan #1. The weights or values of the features calculated from Scan #1 can be subtracted from the features or values calculated from Scan #2. The differences or changes in individual features can constitute a new set of "delta features" corresponding to the temporal changes in typical image features (e.g., changes as a function of time in shape, intensity, texture, etc.).

[0053] 2. Approach #2: Train a 4D CNN prediction model where the input ROI shape is [N x ,N y ,N z ,2] (where Nx, Ny, Nz are the number of voxels along each axis, and 2 corresponds to two (or more) imaging time points, each represented by a single 3D volume within a 4D input volume). This approach is similar to a multimodal CNN model. The most obvious example is a natural image in the RGB format where each color channel is represented separately. In this case, each channel is used to represent one event temporally.

[0054] 3. Approach #3: Calculate the intensity difference between spatially aligned Scan #1 and Scan #2 and then train a 3D CNN prediction model (the model input ROI shape is [N x ,N y ,N z ,1], where Nx, Ny, Nz are the number of voxels along each axis, and 1 corresponds to a single intensity channel).

[0055] 4. Approach #4: Train a model that combines a 3D CNN and an RNN (recurrent neural network) used to model a series of image inputs.

[0056] FIG. 3A is a diagram showing an example of a pre-treatment image 300 of a target according to an embodiment of the present disclosure. The pre-treatment image 300 can correspond to the pre-treatment image as described above with reference to FIG. 2. The pre-treatment image 300 can correspond to a lung lesion 302 of a patient during a baseline scan. In an embodiment, a baseline scan can be performed on a patient before treatment. In an embodiment, the pre-treatment image 300 can correspond to a CT image. In some embodiments, the pre-treatment image 300 can correspond to a PET image. In one embodiment, the pre-treatment image 300 can correspond to an MRI image. In some embodiments, other types of pre-treatment images can be used.

[0057] FIG. 3B is a diagram showing an example of a follow-up image 350 of a target according to an embodiment of the present disclosure. As described above, embodiments of the present disclosure can utilize one or more follow-up images such as a follow-up image 350 of a target captured after treatment. The follow-up image 350 includes a lung lesion 352 that can correspond to the lung lesion 302 after treatment. In an embodiment, the follow-up image 350 can be provided to a machine learning architecture 127 and used to determine whether the current treatment plan should be continued as effective, whether there are more effective treatment options, and / or whether the treatment should be discontinued based on an analysis of the follow-up image 350 relative to the pre-treatment image 300. In an embodiment, the follow-up image 350 can correspond to a CT image. In some embodiments, the follow-up image 350 can correspond to a PET image. In one embodiment, the follow-up image 350 can correspond to an MRI image. In some embodiments, other types of follow-up images can be used.

[0058] FIG. 4 is a diagram showing an example of an output 400 generated based on a predicted treatment response score according to an embodiment of the present disclosure. In the embodiment, as described above, the output 400 can be generated based on the predicted treatment response score. The output 400 shows the temporal relationship between a treatment course 402 (e.g., immunotherapy, chemotherapy, targeted therapy) and an imaging examination 404 (e.g., CT and PET images). The output 400 can indicate when different treatment courses 402 and / or imaging examinations 404 should be performed with respect to the treatment timeline.

[0059] The output 400 can also include treatment information 406. The treatment information 406 can indicate which type of immunotherapy, chemotherapy, and / or targeted therapy is recommended for use in the treatment. The output 400 can further include a patient profile 408 that includes information related to the patient receiving the treatment. Examples of information included in the patient profile 408 include, but are not limited to, the patient's age, the patient's gender, known genomic driver mutations, or a PD-L1 immunohistochemistry tissue proportion score (TPS).

[0060] Note that the output 400 is shown for illustrative purposes only and is not intended to limit the present disclosure. Embodiments of the present disclosure can also generate other types of outputs that can differ in appearance and / or information (e.g., treatment course 402, imaging examination 404, treatment information 406, patient profile 408) compared to the output 400 shown in FIG. 4 based on the predicted treatment response score.

[0061] FIG. 5 shows a diagrammatic representation of a machine in the form of a computer system 500 that can execute an instruction set 522 to cause the machine to execute any one or more of the methods described herein. In another embodiment, the machine can be connected (e.g., network-connected) to other machines within a local area network (LAN), intranet, extranet, or the Internet. The machine can operate as a server or client device in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine can be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), cellular telephone, web appliance, server, network router, switch or bridge, hub, access point, network access control device, or any machine capable of executing an instruction set (sequential or otherwise) that specifies actions to be taken by the machine. Further, although only one machine is shown in the figure, the term "machine" can also be construed to include a group of machines that individually or jointly execute an instruction set (or sets) for executing any one or more of the methods described herein. In one embodiment, the computer system 500 can represent a server computer system such as system 100.

[0062] The exemplary computer system 500 includes a processing device 502, a main memory 504 (e.g., read-only memory (ROM)), flash memory, dynamic random access memory (DRAM), static memory 506 (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage device 518, which communicate with each other via a bus 530. Signals supplied via the various buses described herein are all time-division multiplexed with other signals and can be supplied via one or more common buses. Also, the interconnections between circuit components or circuit blocks can be shown as a bus or a single signal line. Each bus can also be one or more single signal lines, and each single signal line can also be a bus.

[0063] The processing device 502 represents one or more general-purpose processing devices such as a microprocessor or a central processing unit. Specifically, the processing device can be a complex instruction set computer (CISC) microprocessor, a reduced instruction set computer (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor that executes other instruction sets, or a processor that executes a combination of instruction sets. The processing device 502 can also be one or more application-specific processing devices such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), or a network processor. The processing device 502 is configured to execute processing logic 526, which can be an example of the system 100 shown in FIG. 1 for performing the operations and steps described herein.

[0064] The data storage device 518 can include a machine-readable storage medium 528 that stores one or more instruction sets 522 (e.g., software) that embody any one or more of the methodological functions described herein for causing the processing device 502 to execute the system 100. The instructions 522 can also be fully or at least partially present in the main memory 504 that constitutes the machine-readable storage medium or in the processing device 502 during execution by the computer system 500. The instructions 522 can be further transmitted or received via the network interface device 508 over the network 520.

[0065] The machine-readable storage medium 528 can also be used to store instructions for performing the methods and operations described herein. Although the machine-readable storage medium 528 is shown as a single medium in the exemplary embodiment, the term "machine-readable storage medium" is intended to be interpreted to include a single medium or a plurality of media (e.g., a centralized database or a distributed database, or associated caches and servers) that store one or more instruction sets. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer), such as software, a processing application. Machine-readable media include, but are not limited to, magnetic storage media (e.g., floppy disk), optical storage media (e.g., CD-ROM), magneto-optical storage media, read only memory (ROM), random access memory (RAM), erasable programmable memory (e.g., EPROM and EEPROM), flash memory, or other types of media suitable for storing electronic instructions.

[0066] In the above description, numerous specific details, such as specific system examples, component examples, and method examples, have been described to enable a good understanding of multiple embodiments of the present disclosure. However, it will be apparent to those skilled in the art that at least some embodiments of the present disclosure can be implemented without these specific details. In other instances, well-known components or methods have not been described in detail so as not to unnecessarily obscure the present disclosure, or are shown in a simple block diagram format. Therefore, the described specific details are merely illustrative. Specific embodiments may differ from these illustrative details, but are still envisioned to be within the scope of the present disclosure.

[0067] Also, some embodiments can be implemented in a distributed computing environment where a machine-readable medium is stored in or executed by multiple computer systems. Additionally, the information transferred between computer systems can be pull-distributed or push-distributed across the communication medium connecting the computer systems.

[0068] Embodiments of the claimed subject matter include, but are not limited to, various operations described herein. These operations can be executed by hardware components, software, firmware, or combinations thereof.

[0069] Although the operations of the methods herein are illustrated and described in a particular order, the order of operations of each method can be changed, some operations can be executed in reverse order, or some operations can be executed at least partially concurrently with other operations. In another embodiment, the instructions or sub-operations of different operations can also be intermittent or alternating.

[0070] The description of the exemplary implementation of the present invention as described above, including the content described in the abstract, is not intended to be complete or to limit the present invention to the exact form disclosed. In this specification, specific embodiments and examples of the present invention have been described for illustrative purposes, but as will be recognized by those skilled in the art, various equivalent modifications are possible within the scope of the present invention. As used herein, the words "example" or "exemplary" are meant to serve as an example, instance, or illustration. Any aspect or design described herein as "example" or "exemplary" should not necessarily be construed as preferred or advantageous over other aspects or designs. Rather, the use of the words "example" or "exemplary" is intended to specifically illustrate the concept. The term "or" as used in this application is intended to mean inclusive "or" rather than exclusive "or". That is, unless otherwise expressly stated or clear from the context, the expression "X includes A or B" is intended to mean any natural inclusive substitution. That is, "X includes A or B" is satisfied under any of these instances if X includes A, if X includes B, or if X includes both A and B. Also, as used in this specification and the appended claims, the articles "a" and "an" should generally be construed to mean "one or more than one" unless otherwise expressly stated to the contrary or clear from the context. Further, throughout, the terms "an embodiment" or "one embodiment", or "an implementation" or "one implementation" are not intended to mean the same embodiment or implementation unless so stated. Additionally, the terms "first", "second", "third", "fourth", etc. as used herein are intended as labels to distinguish different elements and do not necessarily imply an order that follows these numerical designations in all cases.

[0071] It will be understood that the features and functions disclosed above, as well as other features and functions, or variants of these alternatives, can be combined in other different systems or applications. Also, various alternative, modified, deformed or improved forms that are currently unforeseen or unpredictable may later be realized by those skilled in the art, and these are also intended to be included in the following claims. The claims can include embodiments in hardware, software, or combinations thereof. In the foregoing specification, the disclosure has been described with reference to specific exemplary implementations. However, it will be apparent that various modifications and changes can be made to these implementations without departing from the broad spirit and scope of the disclosure as set forth in the appended claims. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a limiting sense.

Claims

1. The processing device independently trains a plurality of deep learning models using a plurality of sets of training data to predict an immunotherapy treatment response indicating the survival rate of a subject based on the volume change of a lesion of the subject, wherein each of the plurality of sets of training data shows a baseline, a follow-up interval, and a unique diagnostic image scan in a temporary volume change, The processing device provides a pre-treatment image of one target lesion of a target subject to the plurality of deep learning models independently trained using the plurality of sets of training data to generate the immunotherapy treatment response, The processing device combines the immunotherapy treatment responses of the plurality of deep learning models independently trained using the plurality of sets of training data to generate a predicted treatment response score for the treatment based on the consensus of the immunotherapy treatment responses of the plurality of deep learning models, The processing device generates a recommended treatment plan for the target lesion of the target subject based on the predicted treatment response score, A method comprising.

2. The processing device receives an in-treatment follow-up image, The processing device provides the in-treatment follow-up image to the at least one deep learning model to generate a further immunotherapy treatment response, The processing device generates a latest predicted treatment response score based on the further immunotherapy treatment response, The processing device generates a latest recommended treatment plan based on the latest predicted treatment response score, The method according to claim 1, further comprising.

3. The one pre-treatment image includes a plurality of image features, The method according to claim 1.

4. At least one deep learning model among the plurality of deep learning models includes a convolutional neural network, The method according to claim 1.

5. The treatment is PD-[L]1 immune checkpoint inhibitor treatment, The method according to claim 1.

6. The treatment is PD-[L]1 or CTLA-4 immune checkpoint inhibitor treatment, The method according to claim 1.

7. The treatment is a combination of PD-[L]1-based treatment or CTLA-4-based treatment and chemotherapy treatment, The method according to claim 1.

8. The treatment is a combination of a PD-[L]1-based treatment or a CTLA-4-based treatment and radiotherapy. The method according to claim 1.

9. The one pre-treatment image is either a three-dimensional anatomical image or a four-dimensional anatomical image. The method according to claim 1.

10. The predicted treatment response score indicates a prediction of response to a given pharmaceutical. The method according to claim 1.

11. The predicted treatment response score indicates a prediction of progression-free survival at the patient level and the lesion level for a given pharmaceutical. The method according to claim 1.

12. The predicted treatment response score indicates a prediction of overall survival at the patient level and the lesion level for a given pharmaceutical. The method according to claim 1.

13. The predicted treatment response score indicates a prediction of over-progression at the patient level and the lesion level for a given pharmaceutical. The method according to claim 1.

14. The predicted treatment response score indicates a prediction of pseudo-progression at the patient level and the lesion level for a given pharmaceutical. The method according to claim 1.

15. The predicted treatment response score indicates a prediction of one or more immune-related adverse events associated with the treatment. The method according to claim 1.

16. The processing device further includes providing one or more non-image features related to the target subject to at least one of the plurality of deep learning models, and the predicted treatment response score for the treatment is generated based on the one pre-treatment image, the one or more non-image features, and the at least one deep learning model. The method according to claim 1.

17. A memory storing a pre-treatment image of a target subject, A processing device operably coupled to the memory, Comprising, the processing device Independently trains a plurality of deep learning models using a plurality of sets of training data to predict an immunotherapy treatment response indicating the survival rate of the subject based on the volumetric change of the lesion of the subject, and each of the plurality of sets of training data shows a unique diagnostic image scan in a baseline, a follow-up interval, and a temporary volumetric change. Provides one pre-treatment image of the target lesion of the target subject to the plurality of deep learning models independently trained using a plurality of sets of training data to generate the immunotherapy treatment response. Combining the immunotherapy treatment responses of the plurality of deep learning models that are uniquely trained using a plurality of sets of the training data to generate a predicted treatment response score for treatment based on the consensus of the immunotherapy treatment responses of the plurality of deep learning models, Generating a recommended treatment plan for the target lesion of the target subject based on the predicted treatment response score, A treatment analysis system.

18. At least one deep learning model of the plurality of deep learning models includes a convolutional neural network, The treatment analysis system according to claim 17.

19. The treatment is PD-[L]1 immune checkpoint inhibitor treatment, The treatment analysis system according to claim 17.

20. The treatment is PD-[L]1 or CTLA-4 immune checkpoint inhibitor treatment, The treatment analysis system according to claim 17.

21. The treatment is a combination of PD-[L]1-based treatment or CTLA-4-based treatment and chemotherapy treatment, The treatment analysis system according to claim 17.

22. The predicted treatment response score indicates a prediction of a response to a predetermined pharmaceutical, The treatment analysis system according to claim 17.

23. The predicted treatment response score indicates a prediction of progression at the patient level and lesion level for a predetermined pharmaceutical, The treatment analysis system according to claim 17.

24. A non-transitory computer-readable storage medium including instructions that, when executed by a processing device, cause the processing device to, Independently train a plurality of deep learning models using a plurality of sets of training data to predict an immunotherapy treatment response indicating the survival rate of a subject based on the volume change of the lesion of the subject, wherein each of the plurality of sets of training data indicates a unique diagnostic image scan at a baseline, a follow-up interval, and a temporary volume change, and Provide a pre-treatment image of one of the target lesions of the target subject to a plurality of deep learning models that are independently trained using a plurality of sets of training data to generate the immunotherapy treatment response, Combine the immunotherapy treatment responses of the plurality of deep learning models that are independently trained using a plurality of sets of the training data to generate a predicted treatment response score for treatment based on the consensus of the immunotherapy treatment responses of the plurality of deep learning models, generating a recommended treatment plan for the target lesion of the target subject based on the predicted treatment response score; A non-transitory computer-readable storage medium for causing the above to be performed. **Claim 25** At least one of the plurality of deep learning models includes a convolutional neural network. The non-transitory computer-readable storage medium according to claim 24. **Claim 26** The treatment is PD-[L]1 immune checkpoint inhibitor treatment. The non-transitory computer-readable storage medium according to claim 24. **Claim 27** The treatment is PD-[L]1 or CTLA-4 immune checkpoint inhibitor treatment. The non-transitory computer-readable storage medium according to claim 24. **Claim 28** The treatment is a combination of PD-[L]1-based treatment or CTLA-4-based treatment and chemotherapy treatment. The non-transitory computer-readable storage medium according to claim 24. **Claim 29** The predicted treatment response score indicates a prediction of overprogression at the patient level and the lesion level for a predetermined pharmaceutical. The non-transitory computer-readable storage medium according to claim 24.

Images