Predicting the response to immunotherapy using deep learning analysis of image data and clinical data.

Abstract

To provide systems and methods for predicting responses to PD-[L]1 and CTLA-4 immune checkpoint inhibitors using deep learning analysis of imaging and clinical data.SOLUTION: A method disclosed herein comprises generating a predicted treatment response score for a treatment based on the single pre-treatment image and the at least one deep learning model using a processing device, and providing a recommended treatment plan based on the predicted treatment response score.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] Cross-Reference to Related Applications This application claims the benefit of U.S. Provisional Patent Application No. 63 / 056,393, filed on July 24, 2020, the entirety of which is incorporated herein by reference.

[0002] The present disclosure relates to prediction of immunotherapy treatment response using deep learning analysis, and specifically to systems and methods for predicting response to PD-[L]1 and CTLA-4 immune checkpoint inhibitors using deep learning analysis of image data and clinical data. SUMMARY OF THE INVENTION

[0003] The present disclosure will be more fully understood from the following detailed description and the accompanying drawings of various implementations of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] [Figure 1] FIG. 1 is a diagram illustrating a machine learning system for use with embodiments of the present disclosure.

[0005] [Figure 2] FIG. 2 is a flowchart of an immunotherapy treatment prediction method using deep learning analysis according to an embodiment of the present disclosure.

[0006] [Figure 3A] FIG. 3 is a diagram illustrating an example of a pre-treatment image of a target according to an embodiment of the present disclosure.

[0007] [Figure 3B] FIG. 4 is a diagram illustrating an example of a follow-up image of a target according to an embodiment of the present disclosure.

[0008] [Figure 4] FIG. 5 is a diagram illustrating an example of an output generated based on a predicted treatment response score according to an embodiment of the present disclosure.

[0009] [Figure 5] This figure shows examples of different systems that can be used to predict immunotherapy treatments using deep learning analysis, according to embodiments described herein. [Modes for carrying out the invention]

[0010] Embodiments of the present invention relate to the field of predicting immunotherapy treatment using deep learning analysis, and more specifically, to a system and method for predicting responses to PD-[L]1 and CTLA-4 immune checkpoint inhibitors using deep learning analysis of image data and clinical data.

[0011] Immunotherapy has revolutionized cancer treatment with programmed cell death-1 (PD-1) and anti-cytotoxic T lymphocyte antigen-4 (CTLA-4) class checkpoint inhibitors (e.g., pembrolizumab, nivolumab, atezolizumab, ipilimumab, tremelimumab) that show sustained responses in non-small cell lung cancer and stage IV metastatic tumors such as melanoma. However, unpredictable and low patient response rates, coupled with high drug costs and severe toxicity, can place a significant burden on healthcare systems, third-party payers, and patients. As PD-[L]1 and CTLA-4 checkpoint inhibitors continue to be adopted, there is undoubtedly a need for diagnostic tools to stratify patients according to their response likelihood. Recent efforts to explore the practicality of quantitative imaging biomarkers to predict responses to PD-[L]1 and CTLA-4 immunotherapy have proven effective.

[0012] However, PD-1 / PD-L1 and CTLA-4 checkpoint blockade therapy has many drawbacks, including a low response rate of approximately 15% to 20% in most diseases when used as monotherapy, high treatment costs worldwide (over $150,000 per year in the US), and serious immune-mediated adverse events. As mentioned above, the unpredictable and low patient response rates, combined with high drug costs and severe toxicity, can place a significant burden on healthcare systems, third-party payers, and patients.

[0013] Numerous methods have been explored to predict responses to PD-1 / PD-L1 and CTLA-4 checkpoint therapies, but success has been limited. In one embodiment, immunohistochemistry (IHC) testing measures the level of PD-L1 protein expressed in tumor samples. In several clinical trials involving PD-[L]1, tumor mutational burden, the presence of tumor-infiltrating lymphocytes, and inflammatory cytokines have been studied, often in combination with further immuno-oncology (IO) therapies such as CTLA-4 checkpoint inhibitors.

[0014] In some embodiments, efforts to explore the practical applications of quantitative imaging biomarkers for predicting response to PD-[L]1 immunotherapy appear promising. Such approaches allow non-invasive image scans to provide insights and information about the patient's overall tumor volume, rather than samples of lesion subsets (as obtained by biopsy or serum-based analysis). Further analysis of diagnostic images showing all treatable lesions using computational techniques such as machine learning and artificial intelligence could potentially identify relevant image features, leading to an accurate and comprehensive assessment of the patient's response to PD-[L]1 and / or CTLA-4 therapy.

[0015] The embodiments described herein, without limitation, advantageously overcome the above and other problems by describing a multi-omic classifier that predicts responses to PD-1 / PD-L1 and CTLA-4 checkpoint blockade in a variety of clinical indications, including non-small cell lung cancer (NSCLC), melanoma, bladder cancer, and breast cancer. In one embodiment, the classifier is developed based on diagnostic imaging scans at baseline and follow-up intervals, as well as training data including existing biomarkers, relevant clinical data, molecular data, demographic data, response data, and survival data. Examples of existing biomarkers used in clinical practice include PD-L1 expression immunohistochemistry, tumor mutational load (TMB), mutation mismatch repair (MMR), microsatellite instability (MSI), and neutrophil-to-lymphocyte ratio (NLR). Furthermore, there is early evidence suggesting that clinical tests such as lactate dehydrogenase (LDH), S100 protein, and related serum proteins can predict immunotherapy response and pseudoprogression. In the near future, features and biomarkers extracted from the microbiome are also expected to play an important role.

[0016] To summarize a rough method sequence of one embodiment of this disclosure, once sufficient patient data has been accumulated in an anonymized form, the image data (both baseline and follow-up scans) is annotated (segmented) to specify lesions, lymph nodes, surrounding organs, etc., clinical notes and other computational metrics (changes in tumor volume) are annotated to assess response or disease progression at the patient level in each lesion to generate a RECIST score (response evaluation criterion for solid tumors), a preprocessing layer normalizes the image data based on a reconstruction kernel and hardware parameters (slice width of the CT scanner), and a multilayer convolutional neural network (CNN) processes the annotated image data and clinical cohort features.

[0017] In one embodiment, terms such as “target,” “target lesion,” and “target subject” may mean a nodule, lesion, tumor, metastatic mass, or anatomical structure near (within some defined vicinity of) the treatment site. In another embodiment, the target may be a bone structure or bone metastasis. In yet another embodiment, the target may mean the patient’s soft tissue. The target may be any defined structure or site that can be identified and tracked (including the patient as a whole) as described herein.

[0018] Furthermore, for convenience and brevity, PD-1 and CTLA-4 will be frequently mentioned, but the embodiments disclosed herein are equally suitable for any other treatment, including other forms of immunotherapy, chemotherapy, and radiotherapy, though not limited to those described below. Additionally, as used herein, PACS means Picture Archiving and Communication System, and DICOM means Digital Imaging and Communications in Medicine.

[0019] Figure 1 shows a machine learning system 100 used in conjunction with embodiments of the present disclosure. While certain components are disclosed within the machine learning system 100, it should be understood that these components are merely examples. That is, embodiments of the present invention are fully compatible with a variety of other components or variations of components described within the machine learning system 100. Furthermore, it should be understood that the components within the machine learning system 100 may work together with components other than those presented, and that not all components of the machine learning system 100 are necessary to achieve the goals of the machine learning system 100.

[0020] In one embodiment, the system 100 includes a server 101, a network 106, and client devices 150. The server 100 may include various components that enable the prediction of responses to PD-1 checkpoint blockade (and other immunotherapy treatments) of image data and clinical data on the server or client devices using deep learning analysis. Each component may perform different functions, operations, actions, processes, methods, etc. for a web application and / or provide different services, functionalities and / or resources for a web application. The server 100 may include a machine learning architecture 127 of a processing unit 120 that performs operations related to the prediction of responses to PD-1 checkpoint blockade of image data and clinical data using deep learning analysis, using a trained model. In one embodiment, the processing unit 120 is one or more graphics processing units of one or more servers (including, for example, server 101). Further details of the machine learning architecture 127 are shown with respect to the remaining figures of this disclosure. The server 101 may further include a network 105 and a data store 130.

[0021] The processing device 120 and the data store 130 are operatively coupled to each other via the network 105 (for example, they can be operatively coupled, communicatively coupled, and can transmit data / messages to each other). The network 105 can be a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), or a combination thereof. In one embodiment, the network 105 may include a wired or wireless infrastructure that can be provided by one or more wireless communication systems such as Wi-Fi hotspots connected to the network 105, and / or by a wireless carrier system that can be implemented using various data processing facilities, communication towers (e.g., cell towers), and the like. The network 105 can carry communications (e.g., data, messages, packets, frames, etc.) between various components of the server 101. The data store 130 can be a persistent storage capable of storing data. The persistent storage can be a local storage unit or a remote storage unit. The persistent storage can be a magnetic storage unit, an optical storage unit, a solid-state storage unit, an electronic storage unit (main memory), or a similar storage unit. The persistent storage can also be a monolithic / single device or a distributed set of devices.

[0022] Each component may include hardware such as a processing unit (e.g., processor, central processing unit (CPU), graphics processing unit (GPU)), memory (e.g., random access memory (RAM), storage device (e.g., hard disk drive (HDD), solid state drive (SSD), etc.)), and other hardware devices (e.g., sound card, video card, etc.). Server 100 may include any suitable type of computer device or machine having a programmable processor, such as a server computer, desktop computer, laptop computer, tablet computer, smartphone, set-top box, etc. In some examples, Server 101 may include a single machine or multiple interconnected machines (e.g., multiple servers configured in the form of a cluster). Server 101 may be implemented by a common entity / organization or by different entities / organizations. For example, Server 101 may be operated by a first company / corporation, and a second server (not shown) may be operated by a second company / corporation. Each server may run or include an operating system (OS), as will be described in detail below. The server's OS can manage the execution of other components (e.g., software, applications, etc.) and / or access to the computer equipment's hardware (e.g., processor, memory, storage, etc.).

[0023] As described herein, the server 101 can provide a machine learning function to a client device (e.g., client device 150). In one embodiment, the server 101 is operatively connected to the client device 150 via a network 106. The network 106 can be a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), or a combination thereof. In one embodiment, the network 106 can include a wired or wireless infrastructure that can be provided by one or more wireless communication systems such as a Wi-Fi hotspot connected to the network 106 and / or by a wireless carrier system that can be implemented using various data processing facilities, communication towers (e.g., cell towers), and the like. The network 106 can carry communications (e.g., data, messages, packets, frames, etc.) between various components of the system 100. Further implementation details of the operations performed by the system 101 are described with respect to the remaining figures of the present disclosure.

[0024] In one embodiment, the system 101 can operate based on any of the following data, and any other suitable data contemplated. JPEG0007933959000001.jpg156167

[0025] Figure 2 is a flow diagram of a method for predicting immunotherapy treatment using deep learning analysis according to an embodiment of the present disclosure. In general, each method (including method 200) described herein can be executed by processing logic that may comprise hardware (e.g., processing devices, circuits, dedicated logic, programmable logic, microcode, device hardware, etc.), software (e.g., instructions that run or are executed on a processing device), or a combination thereof. In some embodiments, these methods can be executed by the processing logic of the machine learning architecture 127 of Figure 1.

[0026] Method 200 begins in block 201 by providing pre-treatment images of a target subject, optionally including lesion annotations or seed points, to at least one deep learning model, whose processing logic is independently trained to predict a therapeutic response (e.g., immunotherapy) based on a single or multiple lesions. In embodiments, other types of machine learning models may be used instead of or in conjunction with the at least one deep learning model. In some embodiments, after generating a large number of predetermined image and clinical feature sets, a feature selection algorithm (e.g., minimum redundancy maximum relevance (MRMR) or least absolute shrinkage and selection operator (LASSO)) is applied, and these are fitted using a machine learning method (e.g., gradient boosting decision tree, random decision forest, or support vector machine) to generate a predictive model. The optional lesion annotations or seed points provided in block 201 can be generated manually by the clinical user or automatically by an automated segmentation and / or target detection method. An example of an automated segmentation or target detection method is a convolutional neural network model. To predict the therapeutic response to a single lesion, multiparametric optimization techniques such as the stochastic gradient descent (SGD) algorithm, the RMSprop algorithm, or the adaptive momentum (Adam) algorithm are used to train the model to maximize the agreement between the lesion response predicted by the model and the lesion response determined by a human expert (e.g., a radiologist).

[0027] Examples of lesion responses include numerical evaluations (e.g., changes in lesion volume, changes in one or more major dimensions of the lesion, changes in image intensity within the lesion), tumor growth rate (TGR), or categorical evaluations (e.g., responsive lesion, stable lesion, progressive lesion, new lesion). Patient-level treatment response predictions are performed by aggregating one or more lesion-level model predictions. In one embodiment, the aggregation from lesion-level response predictions to patient-level response predictions is performed by a series of rules and logical operations.

[0028] In some embodiments, after calculating response scores for each lesion for multiple lesions in a single patient, mathematical operations such as calculating the maximum score, minimum score, and / or mean score can be performed to convert the multiple lesion-level response predictions into a single patient-level response prediction. In some embodiments, the aggregation from lesion-level response predictions to patient-level response predictions is performed by a second model specially trained to make patient-level response predictions by taking predictions from one or more lesion-level models as input. In some embodiments, to account for a variable number of lesions (e.g., model inputs), the input to the model can be lesion-level predictive statistics (e.g., mean, median, standard deviation, etc.). In other embodiments, the model can be a recurrent neural network (RNN) model in which multiple lesion predictions are represented as a variable-length input sequence.

[0029] Examples of patient-level models include, but are not limited to, artificial neural networks, random forest models, support vector machines, and logistic regression models. In another embodiment, a single machine learning model that considers multiple lesions simultaneously can be used. Such embodiments can effectively eliminate the hierarchy of lesion-specific models and patient-specific models. In one embodiment, pre-treatment images can be 2D anatomical images, 3D anatomical images, or 4D anatomical images. In another embodiment, two or more different types of treatment images can be used.

[0030] Therapeutic images can be acquired at the time of diagnosis (before treatment begins) or at any other suitable time. Therapeutic images may be, but are not limited to, computed tomography (CT) scans, positron emission tomography (PET) scans, or magnetic resonance imaging (MRI) scans. At least one deep learning model may include, but are not limited to, any suitable variety of machine learning models, including convolutional neural networks. In one embodiment, the models are trained on the same data using different hyperparameters and optimization techniques. In another embodiment, the models are trained on different data, etc., using different techniques with different purposes, and these results can be aggregated in various ways.

[0031] Deep learning models can utilize a variety of suitable training methods. For example, in one embodiment, the deep learning model uses a training subject population and multiple images associated with each of the multiple training subjects as training data. In another embodiment, the deep learning model uses a computed subject-specific model as training data. In yet another embodiment, the deep learning model uses a combination of the two methods described above.

[0032] In one embodiment, the treatment is PD-[L]1 immune checkpoint inhibitor therapy. PD-[L]1 immune checkpoint inhibitor therapy can be PD-1 based therapy or PD-L1 based therapy. In yet another embodiment, the treatment is CTLA-4 immune checkpoint inhibitor therapy, or any other preferred type of therapy (e.g., chemotherapy, targeted therapy, drug-based therapy, radiotherapy, etc.).

[0033] In block 203, the processing logic generates a predicted treatment response score for immunotherapy treatment (e.g., by the processing device) based on a deep learning model (e.g., on a scale representing the least likely to produce a positive or negative effect to the most likely to produce a positive or negative effect). In some embodiments, the predicted treatment response score can be numerical. In one embodiment, the processing logic generates a predicted treatment response score based on a single pretreatment image and at least one deep learning model. For example, in one embodiment, results from different models can be combined (e.g., by averaging or by any other method) to generate a single response score. In one embodiment, one or more non-image features (e.g., genomic analysis, electronic medical record information, PD-L1 immunohistochemical assay, etc.) can be used to generate a predicted response score. In another embodiment, one or more non-image features can be combined with one or more image features to generate a predicted response score.

[0034] In one embodiment, the predictive treatment response score includes a prediction of patient progression with respect to a given drug. In another embodiment, the predictive treatment response score indicates a prediction of one or more immune-related adverse events associated with immunotherapy treatment. In one embodiment, the predictive treatment response score may include a predictive likelihood (e.g., confidence level) of the occurrence of a particular type of response and / or adverse event. In another embodiment, the response score may include an indication of suspected progression, characterized by a short-term and transient increase in tumor volume due to spontaneous swelling and / or inflammation (e.g., in response to treatment), rather than disease progression. In another embodiment, the response score may reflect the likelihood of hyper-progression, a severe condition with rapid clinical deterioration in which disease progression accelerates during administration of treatment. In another embodiment, the response score may be formulated to indicate progression-free or overall patient survival rates on a monthly or yearly basis.

[0035] In block 205, the processing logic provides a recommended treatment plan based on the predicted treatment response. For example, the recommended treatment plan may include instructions on whether a particular drug should be used, the dosage of such product, and the timing associated with the administration of such product, based on the predicted treatment response. In one embodiment, this instruction can identify whether the patient is likely to respond to a particular drug. In one embodiment, lesion-specific immunotherapy and / or chemotherapy response predictions are used to create a lesion-specific treatment plan to enhance the therapeutic effect of high-risk lesions by combining ongoing systemic therapy with local therapy. Local therapy may be any of the following: stereotactic ablative radiation therapy (SBRT), intensity-modulated radiation therapy (IMRT), conformal radiation therapy (CRT), radiosurgery, surgical resection, thermal ablation, cryoablation, or high-intensity focused ultrasound (HIFU) therapy. In another embodiment, the recommended treatment plan for a patient with a model-predicted high risk of progression may be to maximize the likelihood of treatment response by adding chemotherapy or a combination of CTLA-4 immunotherapy and PD-[L]1 immunotherapy. In yet another embodiment, the recommended treatment plan may be to discontinue one or all treatments to maximize the patient's quality of life. In an embodiment, the processing logic may generate other outputs in place of or in conjunction with the recommended treatment plan based on the predicted treatment response score. For example, the processing logic may generate a report based on the predicted treatment response score.

[0036] In block 207, the processing logic can receive intra-treatment follow-up images.

[0037] In block 209, the processing logic can provide in-treatment follow-up images to a machine learning model.

[0038] In block 211, the processing logic can generate the latest predicted treatment response score.

[0039] In block 213, the processing logic can provide the latest recommended treatment plan based on the latest predicted treatment response score.

[0040] In various embodiments, the processing logic can perform any number of preferred pre-processing and post-processing operations that can improve the accuracy, efficiency, and / or compatibility of the machine learning model in the given context. For example, with respect to pre-processing, conventional radiomics methods may be susceptible to variability in scanner hardware and imaging protocols. The data pre-processing and data augmentation systems described herein are designed to optimize model generalizability to minimize the model's sensitivity to variations in imaging hardware and protocols.

[0041] In the field of machine learning, particularly deep learning, strategies are known for improving the generalizability of models. For each category, the following and other methods are considered:

[0042] 1. Select a model size (number of parameters) that achieves the optimal balance between underfitting and overfitting with the available training data. A) The MLops (e.g., Machine Learning and Operations) framework and infrastructure enable monitoring of the model's key performance indicators (KPIs) and allow for continuous adjustment of the model's complexity and architecture as more data becomes available.

[0043] 2. Maximize the diversity of training datasets. A) Training data can be supplied from various institutions (academic institutions, small regional centers, large payer / provider networks) and can reflect various clinical practice trends and various imaging hardware and radiology protocols (for example, some regional cancer centers use thick 5mm slice CT protocols, while research institutions tend to use high-resolution 1-2mm thin slice scans). B) A database system can be used to internally catalog training data and ensure the appropriate distribution of imaging hardware and protocols during model training.

[0044] 3. Normalization of input data. A) During model training and model inference, scans can be resampled to a consistent resolution (e.g., voxel spacing of 1.0 × 1.0 × 1.0 mm). This significantly reduces the model performance's dependence on CT slice thickness. B) Image voxel intensity can be normalized by eliminating intensity outliers (metal artifacts from references, pacemakers, wires, etc.) and rescaling the intensity to a consistent range (e.g., an intensity distribution with a mean of 0 and a variance of 1). C) If multiple reconstruction protocols are available for a given imaging session, the reconstruction protocol that best matches the "gold standard" protocol can be used.

[0045] 4. Enhance training data by generating synthetic training examples that simulate feasible scenarios not represented in the available training data. A) An online enhancement strategy can be used, meaning that new variations of training data are continuously generated as long as the model is being trained. In practice, this strategy means that the number of unique training examples is infinite and is limited only by the time spent in the model training loop. The online enhancement loop generates new unique training examples by performing model shifts, rotations, rescaling operations, deformations, and intensity perturbations. B) Noise and intensity variations can be generated using physics-based principles to simulate differences between scanner hardware and scanning protocols. Examples of physics-based methods include ray tracing and Monte Carlo photon simulations in existing clinical CT scans, which generate varied CT projection data that can then be used to reconstruct new CT scans with alternative imaging protocols and simulated artifacts. Examples of simulated artifacts include different primary beam energies, beam scattering and hardening characteristics, patient motion artifacts, and imaging dose variations.

[0046] 5. Model input using multiple resolutions and region of interest (ROI) sizes. A) CNN models may prefer one or more CT scan regions (ROIs) as input. Using ROIs of various sizes and resolutions can create redundant representations of input CT images (or regions) near the tumor location. Using multiple ROI sizes allows the model to accommodate tumors of different sizes and shapes. For example, if only an ROI extending 5x5x5cm around the tumor is used, the model may not function well for large tumors. Conversely, if a 50x50x50cm ROI is used, the classifier may not function well for small tumors requiring high spatial resolution and high fidelity. By combining ROI regions with small and large spatial dimensions within a single model, complementary learning of image features in local contexts (e.g., tumor shape, texture, intensity profile) and global contexts (e.g., lesion location within the body and other organs, lymph node metastasis, patient mass composition and muscle reserve, overall health or vital organs, microcalcification, etc.) becomes easier, ultimately leading to highly predictable and robust treatment response and survival prediction models.

[0047] Regarding post-processing, various techniques can be used to post-process individual model predictions to obtain the predictive accuracy and explainability required by clinical end-users. Examples of post-processing methods used include, but are not limited to, the following:

[0048] 1. Model Ensembles: Ensembles (or bagging) are methods to improve the stability and overall performance of a model. Instead of training one model for a given task, multiple variations of the model are trained (by perturbing training hyperparameters, weight initialization, model architecture, training set distribution, etc.). These models are then used simultaneously by computing a consensus among the multiple models (ensemble prediction). In one embodiment, mean or median predictions from multiple models are, on average, more accurate than a single prediction. Examples of ensembling operations that combine predictions from multiple models can be simple averaging, median calculation, the STAPLE algorithm (Simultaneous Truth and Performance Level Estimation by Warfield et al.), or dedicated ensemble models such as linear classifiers, random forests, support vector machines, or neural networks.

[0049] 2. Bottom-Up Model Aggregation: In some clinical applications, the concept of training classification models to predict single-lesion responses to therapeutic agents may be desirable. In some clinical scenarios, the clinical requirement is to predict treatment response at the patient level (whether this patient is likely to benefit from a given overall treatment, considering that some lesions respond while others continue to progress). In this scenario, the concept of a model ensemble may also be applicable. However, in this application, each single-lesion model (or sub-ensemble of models) contributes to an overall patient-level prediction estimated by ensembling the individual lesion predictions. Furthermore, by combining the predictions of each model within a large ensemble and incorporating other clinical factors, biomarkers, and / or imaging features, the processing logic can predict treatment response at the patient level rather than the lesion level.

[0050] 3. Explainability: The response of a deep convolutional network model can be decomposed into the activation of dominant features to highlight which spatial, textural, and morphological features had the greatest influence on prediction. For example, an explanation could predict a "high risk of lesion progression" for reasons such as: 1. lesion volume exceeding 50 cc, 2. lesion location in the lung apex, 3. low tissue heterogeneity in the central and peripheral parts of the lesion, and 4. presence of metastatic bone lesions. In relevant embodiments, the processing device can explain and support model response prediction or prediction of immune-related adverse events by presenting reference data and past cases of patients with similar presentations and medical history profiles.

[0051] Incorporation of Temporal Information: In one embodiment, a treatment prediction model can be considered as a "single" baseline prediction determining the future course of treatment, or as a continuous integration process incorporating imaging and electronic medical record (EMR) information along the course of treatment to provide clinicians with ongoing decision support. In one embodiment, a treatment response model is trained to predict the likelihood of disease progression, pseudo-progression, or over-progression in a patient using baseline and the first intra-treatment follow-up scan. In this clinical scenario, the model prediction can be used to significantly shorten the timeline for making treatment decisions or adjustments, such as switching the patient to a different therapy, adding a second-line therapy, or discontinuing treatment. In the case of prediction models incorporating multiple imaging time points, temporal data can be integrated in various ways (two imaging time points can be used for explanatory purposes).

[0052] 1. Approach #1: Calculate the difference in image features between scan #1 and scan #2, and then use this to create a predictive model. In one embodiment, the image feature sets can be calculated separately for scan #1 and scan #2. The feature weights or values ​​calculated from scan #1 can be subtracted from the features or values ​​calculated from scan #2. The differences or changes in individual features can constitute a new set of "delta features" corresponding to the temporal changes of typical image features (e.g., changes as a function of time such as shape, intensity, and texture).

[0053] 2. Approach #2: Input ROI shape is [N x ,N y ,N z We train a 4D CNN prediction model [Nx, Ny, Nz] (where Nx, Ny, Nz are the number of voxels along each axis, and 2 corresponds to two (or more) imaging time points, each represented by a single 3D volume within a 4D input volume). This approach is analogous to a multimodal CNN model, most obviously a natural image in RGB format where each color channel is represented individually. In this example, each channel is used to represent one event in time.

[0054] 3. Approach #3: Train a 3D CNN prediction model after calculating the intensity difference between spatially aligned scan #1 and scan #2 (model input ROI shape is [N x ,N y ,N z [,1], where Nx, Ny, and Nz are the number of voxels along each axis, and 1 corresponds to a single intensity channel).

[0055] 4. Approach #4: Train a model that combines a 3D CNN with an RNN (Recurrent Neural Network) used to model a series of image inputs.

[0056] Figure 3A shows an example of a pre-treatment image 300 of a target according to an embodiment of the present disclosure. The pre-treatment image 300 can correspond to a pre-treatment image as described above in Figure 2. The pre-treatment image 300 can correspond to a lung lesion 302 of a patient during a baseline scan. In the embodiment, a baseline scan can be performed on the patient before treatment. In the embodiment, the pre-treatment image 300 can correspond to a CT image. In some embodiments, the pre-treatment image 300 can correspond to a PET image. In some embodiments, the pre-treatment image 300 can correspond to an MRI image. In some embodiments, other types of pre-treatment images can be used.

[0057] Figure 3B shows an example of a follow-up image 350 of a target according to an embodiment of the present disclosure. As described above, embodiments of the present disclosure may utilize one or more follow-up images, such as a follow-up image 350 of a target taken up after treatment. The follow-up image 350 includes a lung lesion 352 that can correspond to a lung lesion 302 after treatment. In embodiments, the follow-up image 350 can be provided to a machine learning architecture 127, which can be used to determine, based on an analysis of the follow-up image 350 against a pre-treatment image 300, whether the current treatment plan is effective and should be continued, whether there are more effective treatment options, and / or whether treatment should be discontinued. In embodiments, the follow-up image 350 may correspond to a CT image. In some embodiments, the follow-up image 350 may correspond to a PET image. In some embodiments, the follow-up image 350 may correspond to an MRI image. In some embodiments, other types of follow-up images may be used.

[0058] Figure 4 shows an example of an output 400 generated based on a predicted treatment response score according to an embodiment of the present disclosure. In the embodiment, as described above, the output 400 can be generated based on a predicted treatment response score. The output 400 shows the temporal relationship between treatment courses 402 (e.g., immunotherapy, chemotherapy, targeted therapy) and imaging studies 404 (e.g., CT and PET images). The output 400 can indicate when different treatment courses 402 and / or imaging studies 404 should be performed relative to the treatment timeline.

[0059] Output 400 may also include treatment information 406. Treatment information 406 may indicate which types of immunotherapy, chemotherapy, and / or targeted therapy are recommended for use in treatment. Output 400 may further include a patient profile 408 containing information related to the patient receiving treatment. Examples of information included in the patient profile 408 may include, but are not limited to, the patient's age, sex, known genomic driver mutations, or PD-L1 immunohistochemistry tissue percentage score (TPS).

[0060] Output 400 is shown for illustrative purposes only and is not intended to limit the scope of this disclosure. Embodiments of this disclosure may also generate other types of outputs that may differ in appearance and / or information (e.g., treatment progress 402, imaging 404, treatment information 406, patient profile 408) from Output 400 shown in Figure 4, based on the predicted treatment response score.

[0061] Figure 5 shows a schematic representation of a machine in an example configuration of a computer system 500 capable of executing instruction set 522 that causes the machine to perform one or more of the methods described herein. In another embodiment, this machine can be connected to other machines (e.g., network connection) in a local area network (LAN), intranet, extranet, or internet. This machine can operate as a server or client device in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. This machine can be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), mobile phone, web appliance, server, network router, switch or bridge, hub, access point, network access control device, or any machine capable of executing an instruction set (sequential or otherwise) that specifies the actions the machine should take. Furthermore, although only one machine is shown in the figure, the term “machine” may also be interpreted to include a group of machines that individually or collaboratively execute an instruction set (or set of instructions) for performing one or more of the methods described herein. In one embodiment, computer system 500 may represent a server computer system such as system 100.

[0062] An exemplary computer system 500 includes a processing unit 502, main memory 504 (e.g., read-only memory (ROM)), flash memory, dynamic random access memory (DRAM), static memory 506 (e.g., flash memory, static random access memory (SRAM), etc.), and data storage device 518, which communicate with each other via a bus 530. Signals supplied via the various buses described herein can all be time-division multiplexed together with other signals and supplied via one or more common buses. Furthermore, interconnections between circuit components or circuit blocks can be represented as buses or single signal lines. Each bus may be one or more single signal lines, and each single signal line may be a bus.

[0063] The processing unit 502 represents one or more general-purpose processing units, such as a microprocessor or a central processing unit. Specifically, the processing unit can be a composite instruction set computer (CISC) microprocessor, a reduced instruction set computer (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor that executes other instruction sets, or a processor that executes a combination of instruction sets. The processing unit 502 can also be one or more application-specific processing units, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), or a network processor. The processing unit 502 is configured to execute processing logic 526, which can be an example of the system 100 shown in Figure 1 for performing the operations and steps described herein.

[0064] The data storage device 518 may include a machine-readable storage medium 528 storing one or more instruction sets 522 (e.g., software) that embody one or more of the methodologies of the functions described herein, including instructions for the processing unit 502 to execute the system 100. The instructions 522 may also reside entirely or at least partially in the main memory 504 or the processing unit 502, which also constitute a machine-readable storage medium, during execution by the computer system 500. The instructions 522 may be further transmitted or received via the network 520 through the network interface device 508.

[0065] The machine-readable storage medium 528 can also be used to store instructions for performing the methods and operations described herein. Although the machine-readable storage medium 528 is shown as a single medium in exemplary embodiments, the term “machine-readable storage medium” should be interpreted to include a single or multiple mediums (e.g., a centralized or distributed database, or associated caches and servers) that store one or more instruction sets. A machine-readable medium includes any mechanism that stores information in a form that can be read by a machine (e.g., a computer) (e.g., software, processing applications). A machine-readable medium includes, but is not limited to, magnetic storage media (e.g., floppy diskettes), optical storage media (e.g., CD-ROMs), magneto-optical storage media, read-only memory (ROM), random-access memory (RAM), erasable programmable memory (e.g., EPROM and EEPROM), flash memory, or other types of media suitable for storing electronic instructions.

[0066] The above description includes numerous specific details, such as examples of systems, components, and methods, to facilitate a good understanding of the various embodiments of this disclosure. However, it will be apparent to those skilled in the art that at least some embodiments of this disclosure can be implemented without these specific details. In other cases, well-known components or methods are not described in detail or are shown in simple block diagrams to avoid unnecessarily obscuring the disclosure. Accordingly, the specific details described are illustrative only. Certain embodiments may differ from these exemplary details, but are still expected to fall within the scope of this disclosure.

[0067] Furthermore, some embodiments can be implemented in a distributed computing environment where machine-readable media are stored in or run by multiple computer systems. Information transferred between computer systems can also be delivered via pull or push distribution across communication media connecting the computer systems.

[0068] Embodiments of the claimed subject matter include, but are not limited to, various operations described herein. These operations can be performed by hardware components, software, firmware, or a combination thereof.

[0069] Although the operation of the methods described herein is illustrated and described in a specific order, the order of operation of each method can be changed to execute some operations in reverse order, or some operations can be executed at least partially simultaneously with other operations. In another embodiment, the instructions or subordinate operations of different operations may be intermittent or alternating.

[0070] The above-described exemplary implementations of the present invention, including those described in the abstract, are not intended to be complete or to limit the invention to any specific form disclosed. While specific embodiments and examples of the invention have been described herein for illustrative purposes, various equivalent modifications are possible within the scope of the invention, as will be apparent to those skilled in the art. The words “example” or “exemplary” as used herein mean that they serve as examples, cases, or illustrations. Any aspect or design described herein as “example” or “exemplary” should not necessarily be construed as preferable or more advantageous than other aspects or designs. Rather, the use of the words “example” or “exemplary” is intended to illustrate the concept concretely. The term “or” as used in this application is intended to mean inclusive “or” rather than exclusive “or.” That is, unless otherwise explicitly stated or evident from the context, the expression “X includes A or B” is intended to mean either of the natural inclusive substitutions. In other words, if X includes A, if X includes B, or if X includes both A and B, “X includes A or B” is satisfied under any of these cases. Furthermore, the articles “a” and “an” as used herein and in the appended claims should generally be interpreted as meaning “one or more” unless otherwise explicitly stated or made clear from the context that they refer to a singular form. Moreover, throughout this text, the terms “a certain embodiment” or “one embodiment,” or “a certain implementation” or “one implementation” are not intended to mean the same embodiment or implementation unless described as such. Furthermore, terms such as “first,” “second,” “third,” and “fourth” as used herein are intended as labels to distinguish different elements from each other and do not necessarily imply an order that follows these numerical designations.

[0071] The features and functions disclosed above, as well as other features and functions, or variations thereof, will be understood to be able to be combined with other different systems or applications. Furthermore, various alternative forms, modifications, variations, or improvements that are not currently foreseeable or predictable may be realized later by those skilled in the art, and these are also intended to be included in the following claims. The claims may include embodiments in hardware, software, or combinations thereof. The above-mentioned specification has described the disclosure with reference to specific exemplary implementations. However, it will be evident that various modifications and changes can be made to these implementations without departing from the broad intent and scope of the disclosure as set forth in the appended claims. Accordingly, this specification and drawings should be understood as illustrative rather than restrictive.

Claims

1. Using a processing device, train one or more deep learning models independently using one or more training datasets related to multiple patients, each representing at least one of the following: baseline, follow-up interval, or transient change in lesion volume, to predict immunotherapy response indicating patient survival based on changes in the volume of a patient's lesions. To generate the immunotherapy response by providing a single pre-treatment image of the target lesion of a target patient to one or more deep learning models independently trained using the aforementioned training dataset, The process involves combining the immunotherapy response of one or more deep learning models, each independently trained using the aforementioned training dataset, to generate a predicted treatment response score. Methods that include...

2. Receiving in-treatment follow-up images, The in-treatment follow-up images are provided to the at least one deep learning model, To generate the latest predicted treatment response score, To provide the latest recommended treatment plan based on the aforementioned latest predicted treatment response score, The method according to claim 1, further comprising:

3. The aforementioned pre-treatment image includes multiple image features, The method according to claim 1.

4. The aforementioned at least one deep learning model includes a convolutional neural network. The method according to claim 1.

5. The aforementioned treatment is PD-[L]1 immune checkpoint inhibitor therapy. The method according to claim 1.

6. The aforementioned treatment is PD-[L]1 or CTLA-4 immune checkpoint inhibitor therapy. The method according to claim 1.

7. The aforementioned treatment is a combination of PD-[L]1-based therapy or CTLA-4-based therapy and chemotherapy. The method according to claim 1.

8. The aforementioned treatment is a combination of PD-[L]1-based therapy or CTLA-4-based therapy and radiotherapy. The method according to claim 1.

9. The aforementioned pre-treatment image is either a three-dimensional anatomical image or a four-dimensional anatomical image. The method according to claim 1.

10. The at least one deep learning model uses a training subject population and multiple images associated with each of the multiple training subjects as training data. The method according to claim 1.

11. The aforementioned predicted treatment response score indicates the prediction of the response to a given drug. The method according to claim 1.

12. The aforementioned predicted treatment response score indicates the prediction of progression-free survival at the patient level and the lesion level for a given drug. The method according to claim 1.

13. The aforementioned predicted treatment response score indicates the prediction of overall survival at the patient level and the lesion level for a given drug. The method according to claim 1.

14. The aforementioned predicted treatment response score indicates the prediction of over-progression at the patient level and the lesion level in relation to a given drug. The method according to claim 1.

15. The aforementioned predicted treatment response score indicates the prediction of one or more immune-related adverse events associated with treatment. The method according to claim 1.

16. The method further includes providing one or more non-image features related to the target subject to the at least one deep learning model, wherein the predicted treatment response score for the treatment is generated based on the single pre-treatment image, the one or more non-image features, and the at least one deep learning model. The method according to claim 1.

17. A memory that stores pre-treatment images of the target subject, A processing unit operably coupled to the memory, The apparatus is equipped with, To predict the immunotherapy response, which indicates patient survival rates, based on changes in the volume of a patient's lesions, one or more deep learning models are independently trained using one or more training datasets related to multiple patients, each representing at least one of the following: baseline, follow-up interval, or transient change in lesion volume. The immunotherapy response is generated by providing a single pre-treatment image of the target lesion of a target patient to one or more deep learning models that have been independently trained using a training dataset, The immunotherapy response is generated by combining the immunotherapy response of one or more deep learning models, each independently trained using the aforementioned training dataset. Treatment analysis system.

18. The aforementioned at least one deep learning model includes a convolutional neural network. The treatment analysis system according to claim 17.

19. The aforementioned treatment is PD-[L]1 immune checkpoint inhibitor therapy. The treatment analysis system according to claim 17.

20. The aforementioned treatment is PD-[L]1 or CTLA-4 immune checkpoint inhibitor therapy. The treatment analysis system according to claim 17.

21. The aforementioned treatment is a combination of PD-[L]1-based therapy or CTLA-4-based therapy and chemotherapy. The treatment analysis system according to claim 17.

22. The aforementioned predicted treatment response score indicates the prediction of over-progression at the patient level and the lesion level in relation to a given drug. The treatment analysis system according to claim 17.

23. A non-temporary computer-readable storage medium containing instructions, wherein, when the instructions are executed by the processing unit, the processing unit receives the instructions. Using the aforementioned processing apparatus, one or more deep learning models are independently trained using one or more training datasets related to multiple patients, each representing at least one of the following: baseline, follow-up interval, or transient change in lesion volume, to predict the immunotherapy response indicating patient survival rate based on changes in the volume of the patient's lesions. To generate the immunotherapy response by providing one or more deep learning models, which have been independently trained using a training dataset, with a single pre-treatment image of the target lesion of the target patient, The processing device generates a predicted treatment response score by combining the immunotherapy treatment response of one or more deep learning models, which have been independently trained using the training dataset, A non-temporary computer-readable storage medium that enables the following.

24. The aforementioned at least one deep learning model includes a convolutional neural network. The non-temporary computer-readable storage medium according to claim 23.

25. The aforementioned treatment is PD-[L]1 immune checkpoint inhibitor therapy. The non-temporary computer-readable storage medium according to claim 23.

26. The aforementioned treatment is PD-[L]1 or CTLA-4 immune checkpoint inhibitor therapy. The non-temporary computer-readable storage medium according to claim 23.

27. The aforementioned treatment is a combination of PD-[L]1-based therapy or CTLA-4-based therapy and chemotherapy. The non-temporary computer-readable storage medium according to claim 23.

28. The aforementioned predicted treatment response score indicates the prediction of over-progression at the patient level and the lesion level in relation to a given drug. The non-temporary computer-readable storage medium according to claim 23.

Images