Predictive modeling of therapeutic agent response using deep learning analysis on sequential images before and during treatment.

Abstract

The present invention relates to a system and method for performing at least one of predictive modeling or multimodal predictive modeling of therapeutic response using sequential images acquired before and during treatment. The method includes the steps of: acquiring pre-treatment features of one or more target lesions associated with a pre-treatment scan of the target site before treating the target site according to a treatment plan; determining a set of features that show changes in the one or more target lesions using the pre-treatment features; inputting the set of features into one or more predictive models that have been trained to predict therapeutic response based on the features of the target lesions; and generating a predicted therapeutic response score for the treatment plan using a processing device based on the set of features and the input results from the one or more predictive models before treating the target site according to the treatment plan.

Description

Cross - reference to related applications

[0001] This application claims priority to U.S. Patent Application No. 18 / 657383, filed on May 7, 2024, and further claims priority to U.S. Provisional Patent Application No. 63 / 465460, filed on May 10, 2023. The disclosures of these applications are hereby incorporated by reference in their entirety into this specification.

Technical Field

[0002] The present disclosure relates to a technique for predicting therapeutic agent response using deep learning analysis, and more particularly to a system and method for performing at least one of predictive modeling of therapeutic agent response or multimodal predictive modeling by utilizing sequential images before and during treatment. As used herein, the term "therapeutic agent" includes agents (chemical substances, biological preparations) used for treating, curing, preventing, or diagnosing diseases or symptoms, and in some embodiments, also includes physical means (treatment modalities based on radiation).

Brief Description of the Drawings

[0003] The present invention will be more fully understood from the following detailed description and the accompanying drawings related to each embodiment of the present disclosure.

[0004] [Figure 1] A diagram showing a machine learning system used in an embodiment of the present invention.

[0005] [Figure 2] A diagram showing the timeline of image acquisition and treatment of a patient according to some embodiments.

[0006] [Figure 3] A diagram showing a flowchart of a method for predicting therapeutic agent response using deep learning analysis for sequential images before and during treatment according to some embodiments.

[0007] [Figure 4] This figure shows examples of various systems used in some embodiments to predict the therapeutic response of immunotherapy by deep learning analysis. Detailed description of the invention

[0008] Embodiments of this disclosure relate to the art of predicting therapeutic response using deep learning analysis, and more particularly to systems and methods for performing at least one of either predictive modeling or multimodal predictive modeling of therapeutic response using sequential images before the start of treatment (pre-treatment stage) and during the treatment process.

[0009] Predictive modeling of therapeutic response can be performed using multiple methods. One approach involves a computing system using at least one of the following—pre-treatment images, a set of electronic medical record (EMR) features, or laboratory / measurement values ​​(e.g., from blood samples, urine samples, tissue biopsies, etc.)—to predict the most likely outcome of treatment, thereby helping physicians select the most appropriate treatment option for a particular patient. Another approach involves building (e.g., learning) a predictive model based on a continuous (e.g., longitudinal) set of features acquired before and during treatment. Such a continuous model (longitudinal predictive model) can be used to select the optimal treatment, adjust the treatment plan during the course of treatment, or provide early insights and assessments of the treatment response. Examples of features of continuous modeling include information spanning many different data areas, such as levels of specific serum proteins measured at different time points, scans taken before and during treatment (e.g., computed tomography (CT) scans), and the patient's cognitive function status assessed at each consultation. In some forms, scans may include radiographic images (e.g., CT scans, magnetic resonance imaging (MRI)) but not slide images (pathology slides).

[0010] Another approach involves using sequential CT images to predict overall survival (OS) in patients with advanced melanoma undergoing immunotherapy. Such radiographic models (so-called radiomics-based predictive models) can be constructed by incorporating image features that capture changes in tumor appearance and volume between CT scans acquired at two time points (e.g., baseline and during treatment). Predictive models based on changes in tumor appearance at two different time points (baseline and during treatment) tend to show higher predictive performance than models that incorporate only baseline tumor appearance. This finding is a core concept in the field of delta radiomics, where delta represents the change in image features between two image acquisition points.

[0011] However, conventional approaches to predicting such therapeutic responses rely on at least some sequential images (follow-up images) taken during treatment, i.e., while treatment is being administered. This increases the likelihood of selecting and initiating treatment with an inappropriate drug for the patient. As a result, it may take several months for a physician to realize that the drug is not adequately improving the patient's condition, during which time the disease may progress. Furthermore, selecting an inappropriate therapeutic agent can lead to side effects in patients that could have been avoided if the optimal agent had been selected from the outset. Therefore, providing a mechanism to optimize the prediction of therapeutic response before initiating treatment has been a long-standing challenge.

[0012] Each aspect of this disclosure solves the above-mentioned problems and other problems by performing at least one of either predictive modeling of the therapeutic response or multimodal predictive modeling of the therapeutic response using sequential images before and / or during the treatment process. As will be described later, embodiments of the present invention utilize sequential images acquired at multiple points in time before the start of treatment, and changes in features extracted from these sequential image data. A wide variety of situations (scenarios) can be envisioned in which multiple imaging points may exist before the start of treatment.

[0013] For example, a typical lung cancer treatment flow begins with obtaining diagnostic CT or positron emission tomography (PET-CT) images, which are used to make an initial diagnosis. Approximately 40% of all cases are diagnosed with advanced stage lung cancer (e.g., stage III or IV), in which case systemic treatment (e.g., chemotherapy, immunotherapy, or molecular targeted therapy) is indicated. Contrast-enhanced CT is a clinically standard imaging test for staging and follow-up. Therefore, patients may undergo both diagnostic PET-CT and contrast-enhanced CT scans before starting a treatment plan. Therefore, the situation in which multiple image scans are acquired before the start of treatment makes it possible to incorporate changes in the appearance of the lesion (e.g., volume, diameter, etc.) into a predictive model using these pre-treatment scans, and can be effectively utilized in embodiments of the present invention.

[0014] In another example, a patient may be diagnosed with an early stage of the disease (e.g., stage I or II) and treated with local therapy (e.g., surgery, radiation therapy, or ablation). In this case, after a certain period, follow-up imaging (e.g., observation) before the initiation of systemic treatment may detect that the disease has progressed to stage III or IV. In such cases, there are multiple image acquisition points that include information about the rate of disease progression, and in the embodiments of the present invention, this information can be used to improve the accuracy of predicting the therapeutic response.

[0015] As yet another example, certain therapeutic agents may be indicated as second-line (2L) or third-line (3L) treatment options. In October 2017, pembrolizumab (brand name: Keytruda registered trademark) was first approved as a first-line (1L: primary treatment) immunotherapy agent for patients with metastatic non-small cell lung cancer (NSCLC) in which tumors express a protein called PD-L1. Prior to this approval, pembrolizumab was indicated as a 2L or 3L treatment, and patients typically received chemotherapy as first-line treatment, and then pembrolizumab (or other PD-1 immune checkpoint inhibitors such as nivolumab or atezolizumab) as the disease progressed. In such situations, image data from 1L treatment can function as a sequence of pre-treatment images in embodiments of this disclosure, and this image data can be used to predict the response to 2L therapeutic agents.

[0016] In one embodiment, terms such as “target,” “target lesion,” and “target site” may refer to a nodule, lesion, tumor, metastatic mass, or anatomical structure located in the vicinity (within a predetermined proximity) of the treatment area. In another embodiment, the target may be a bone structure or bone metastasis. In yet another embodiment, the target may refer to the patient’s soft tissue. The target may be any defined structure or region (including the entire patient) that is identifiable and traceable, as described herein.

[0017] Furthermore, while therapeutic agents (e.g., programmed cell death protein 1 (PD-1) agents, cytotoxic T lymphocyte antigen 4 (CTLA-4) agents, etc.) are frequently mentioned herein for convenience and brevity, the embodiments disclosed herein are equally applicable to any other therapeutic method, including, but not limited to, other forms of immunotherapy, chemotherapy, radiotherapy, and the like.

[0018] [1. Machine learning system for predictive modeling of therapeutic response] Figure 1 shows a machine learning system 100 used in an embodiment of the present invention. While certain components are disclosed in the machine learning system 100, these components are merely examples and do not limit the present invention. That is, embodiments of this disclosure can be applied to include various other components, or variations thereof, in addition to those described in the machine learning system 100. Furthermore, the components in the machine learning system 100 may work in conjunction with other components not shown, and not all components are necessarily required to achieve the objectives of the machine learning system 100.

[0019] In one embodiment, the system 100 includes a server 101, a network 106, and a client device 150. The server 101 may have multiple components, which allow the use of pre-treatment or treatment-stage sequential images (e.g., those available on the server 101, client device 150, and / or data store 130) in at least one of the predictive modeling of therapeutic response or multimodal predictive modeling. Each component may perform different functions, operations, processes, methods, etc., to the web application and / or provide different services, functions, or resources. Server 101 includes a machine learning architecture 127 of the processing unit 120 and can perform processing to predict the response to one or more therapeutic agents based on deep learning analysis of pre-treatment or treatment-stage sequential images using a trained model. In one embodiment, the processing unit 120 may include one or more graphics processing units (GPUs) of one or more servers (e.g., including Server 101). Details of the machine learning architecture 127 will be described with reference to other drawings of this disclosure. Server 101 may further include a network 105 and a data store (data storage unit) 130.

[0020] The processing unit 120 and the data store 130 are operablely connected to each other via the network 105 (e.g., operable, communicative, and able to send and receive data and messages to and from each other). The network 105 may be a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or wide area network (WAN)), or a combination thereof. In one embodiment, the network 105 may include wired or wireless infrastructure and may be provided by a wireless communication system (e.g., a Wi-Fi hotspot connected to the network 105, or a wireless communication carrier system that can be implemented using various data processing units and communication towers (e.g., a mobile phone base station)). The network 105 can transmit communications (e.g., data, messages, packets, frames, etc.) between the components of the server 101. As the data store 130, persistent storage capable of storing data can be employed. The persistent storage may be a local storage unit or a remote storage unit, and may also be a magnetic storage unit, optical storage unit, solid-state storage unit, electronic storage unit (main memory), or a similar storage unit. Furthermore, the persistent storage may be monolithic / single device or a distributed group of multiple devices.

[0021] Each component may include hardware such as processing units (e.g., processors, central processing units (CPUs), graphics processing units (GPUs)), memory (e.g., random access memory (RAM)), storage devices (e.g., hard disk drives (HDDs), solid-state drives (SSDs), etc.), and other hardware devices (e.g., sound cards, video cards, etc.). Server 101 can be composed of any suitable type of computing device or machine equipped with a programmable processor, such as, for example, a server computer, a desktop computer, a laptop computer, a tablet computer, a smartphone, a set-top box, etc. In some examples, Server 101 may be composed of a single machine or may include a plurality of interconnected machines (e.g., a plurality of servers configured in a cluster). Server 101 may be implemented by the same business entity / organization or may be implemented by different business entities / organizations, respectively. For example, Server 101 may be operated by a first company / corporation, and a second server (not shown) may be operated by a second company / corporation. Each server may execute or include an operating system (OS) as described below. The server's OS may manage at least one of the execution of other components (e.g., software, applications, etc.) or access to the hardware of the computing device (e.g., processor, memory, storage device, etc.).

[0022] As described herein, server 101 can provide a machine learning function to client devices (e.g., client device 150). In one embodiment, server 101 is operably connected to client device 150 via network 106. Network 106 can be a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), or a combination thereof. In one embodiment, network 106 may include a wired or wireless infrastructure and may be provided by a wireless communication system (e.g., a Wi-Fi hotspot connected to network 106 or a wireless communication carrier system that can be implemented using various data processing devices and communication towers (e.g., cell phone base stations)). Network 106 may transmit communications (e.g., data, messages, packets, frames, etc.) between the various components of server 101. Further details of additional embodiments of the processes performed by server 101 will be described based on other figures of the present disclosure.

[0023] [1.1 Features of Machine Learning] The serial images in predictive modeling are based on the finding that the serial images capture changes in the appearance of lesions due to the therapeutic effect (or lack thereof) of administered anti-cancer drugs between pre-treatment images and post-treatment follow-up images. Embodiments of the present disclosure center around the finding that serial images obtained before the start of treatment may contain important implications (features) regarding the malignancy of each lesion (e.g., growth rate, volume, diameter). This is particularly important in advanced diseases with multiple tumor sites. For example, one tumor may remain mostly unchanged and stagnant while another tumor may exhibit an aggressive growth rate. The tumor growth rate (proliferation rate) quantified from pre-treatment images becomes a powerful predictive feature (feature quantity) available for predictive models for anti-cancer drugs (e.g., immunotherapy drugs or molecular target drugs).

[0024] [1.2 Response Evaluation (Label)] When therapeutic agents are initiated, some lesions may shrink in size, while highly aggressive lesions may only experience a decrease in their growth rate (proliferation rate). The latter (e.g., change in growth rate) can be expressed as an indicator equivalent to the second derivative of tumor volume with respect to time, which may allow for a more accurate quantification of drug effects than conventional absolute changes in lesion diameter (e.g., the Reactive Indicators of Therapeutic Response (RECIST) protocol in solid tumors). This concept, also known as lesion dynamics (a way of quantitatively and dynamically capturing changes over time), focuses on the comparative measurement and evaluation of tumor growth rate and acceleration (the difference between the two). This concept can be applied to a single lesion or to measure the overall dynamics of all lesions in a single patient. Furthermore, this method makes it possible to model (i.e. predict) various endpoints (e.g., outcomes), including overall survival (OS), progression-free survival (PFS), objective response rate (ORR), or individual tumor dynamics (e.g., velocity and acceleration), which are commonly used in cancer drug trials. Models incorporating these new features and evaluation labels can be constructed as either classification or regression models, depending on the nature of the prediction. The architecture of these models can take a variety of forms, from simple rule-based models, decision trees, random forests, and support vector machines to deep neural networks.

[0025] In one embodiment, a predictive model is trained using changes in features (sometimes called novel features) extracted from pre-treatment images of one or more target lesions to predict response evaluation labels, including criteria for treatment response (RECIST) and changes in tumor volume from baseline.

[0026] In other embodiments, the same predictive model utilizes multimodal features extracted from pre-treatment images (e.g., changes in blood test values, changes in urine test values, changes in image features, etc.).

[0027] In further embodiments, image models and multimodal models are trained to predict therapeutic responses quantified by changes in growth rate (proliferation rate). This response evaluation method using changes in growth rate (proliferation rate) is a unique evaluation method newly discovered by the inventors.

[0028] [2. Patient image acquisition and treatment] Figure 2 shows a timeline of patient image acquisition and treatment according to one embodiment. The timeline in Figure 2 includes multiple points in time (e.g., 202, 204, 206, 208) acquired before treatment (pre-treatment stage), point 210 indicating the start of treatment, and multiple points in time (e.g., 212, 214, 216) acquired after treatment (treatment process). In Figure 2, each point in time is shown divided into specific time units (e.g., -3 weeks, -1 week, etc.), but these points in time may be divided into any time unit (e.g., ± minutes, ± days, ± months, etc.).

[0029] At time point 202, the patient presents with symptoms consistent with malignancy. At time point 204 (e.g., -3 weeks), server 101 acquires a pre-baseline scan (a diagnostic scan acquired before baseline) 204s. This pre-baseline scan 204s is a diagnostic scan in which a suspicious lesion is detected. At time point 206, server 101 acquires (e.g., retrieves, receives) a set of patient laboratory data for solid tissue, biopsy, and blood biomarkers to confirm or rule out a cancer diagnosis. At time point 208 (e.g., -1 week), server 101 acquires a baseline scan 208s. The baseline scan 208s may be a contrast-enhanced CT or PET-CT and may include additional areas (e.g., anatomical structures) containing metastatic disease.

[0030] At time point 210 (e.g., t=0), server 101 determines a treatment plan (e.g., a specific treatment or medication) based on the pre-baseline scan 204s, patient examination data, and / or baseline scan 208s. Subsequently, server 101 initiates treatment for the patient according to the treatment plan.

[0031] At time point 212 (e.g., 6 weeks after the start of treatment), server 101 acquires a first follow-up scan 212s. This scan is intended to provide an early assessment of the patient's response to the treatment. At time point 214, server 101 may decide whether to adjust (e.g., modify) the treatment plan based on the radiographic findings (imaging findings) from the first follow-up scan 212s, or not to adjust the treatment plan. In some embodiments, radiographic findings include at least one of the following: tumor growth rate, tumor volume change, tumor diameter change, tumor shape change, etc. At time point 216, server 101 acquires a second follow-up scan, which assesses the patient's response to the applied treatment.

[0032] [2.1 Model Training Procedure] Server 101 can train its predictive model according to the following method.

[0033] In operation 1, server 101 creates a set of training cases (cases representing patients or lesions) (e.g., one or more). This set of training cases includes past longitudinal patient records, each containing at least one of the following: continuous image data, treatment history with medication, or non-image clinical features.

[0034] In operation 2, server 101 extracts model features and outcome labels (sometimes called "ground truth" or "correct data") for each training case (e.g., decision, classification).

[0035] In operation 3, the server 101 can extract model features according to the following method.

[0036] In operation 3a, the server 101 identifies the target lesion on the baseline scan 208s acquired immediately before the start of treatment 210. In operation 3b, the server 101 identifies the corresponding target lesion on the pre-baseline scan 204s. In operation 3c, the server 101 calculates baseline features (feature quantities) from the baseline scan 208s, including image features and non-image features (at the same scan time) (e.g., determination, measurement). In operation 3d, server 101 calculates prebaseline features (feature quantities) from the prebaseline scan 204s, including image features and non-image features (at the same scan time). In operation 3e, the server 101 calculates the difference or change in image features and non-image features between the pre-baseline scan 204s and the baseline scan 208s. Furthermore, the server 101 may generate a normalized (time-corrected) change (difference) by dividing the change (difference) of these image features and non-image features by the number of days between the acquisition dates of the pre-baseline scan 204s and the baseline scan 208s.

[0037] In operation 4, server 101 extracts outcome features according to the following method.

[0038] In operation 4a, the server 101 identifies (specifies) the target lesion on the baseline scan 208s acquired immediately before the start of treatment 210.

[0039] In operation 4b, the server 101 identifies (specifies) the corresponding target lesion on the first follow-up scan 212s or the second follow-up scan 216s.

[0040] In operation 4c, the server 101 calculates a response label for each of the aforementioned target lesions (on a lesion-by-lesion basis). In some embodiments, each label can be one of the following: • Categorical variables (e.g., progression (PD), stable (SD), partial response (PR), complete response (CR)), • A scalar variable corresponding to the change in tumor diameter. • A scalar variable corresponding to the absolute change in tumor volume. • A scalar variable corresponding to the relative change in tumor volume (e.g., percentage change). • A scalar variable corresponding to the growth rate (e.g., the linear or exponential change in tumor volume per unit time).

[0041] In operation 4d, server 101 calculates the patient-specific (patient-unit) response label using one of the following methods: (a) A method for determining the simple mean (or median) or minimum (or maximum) of the response labels for all lesion units. (b) A method using categorical variables that represent the following states, in accordance with known response evaluation protocols (e.g., RECIST 1.1, iRECIST, irRECIST, etc.): • Uniform response: When all target lesions respond to treatment. • Uniform progression: When all target lesions are growing and not responding to treatment. • Mixed response: When some target lesions respond, while others progress. In some embodiments, other patient-level outcome labels may include at least one of the following: overall survival (e.g., 6 months, 1 year, 2 years), whether treatment was changed, whether treatment was discontinued, or immune-related adverse events.

[0042] In some embodiments, the method for calculating features and labels uses two time points to represent a first-order difference (e.g., velocity). This framework is expandable, and the server 101 may use three or more time points to calculate and utilize a second-order difference (e.g., acceleration) of features and labels.

[0043] In some embodiments, the server 101 performs a feature selection method using a known algorithm to identify a subset of fewer features that are most strongly associated with the selected outcome labels.

[0044] In some embodiments, the server 101 trains a predictive model using an optimization algorithm (e.g., stochastic gradient descent, ADAM, etc.). This optimization algorithm optimizes the model so that the agreement between the outcome label and the model prediction generated from the predictive model and its inputs (e.g., features) is maximized (loss function: error is minimized) across all training cases.

[0045] [2.2 Model Inference Procedure] Server 101 can perform model inference according to the following method.

[0046] In operation 1, server 101 identifies the target lesion on the baseline scan 208s acquired immediately before the start of treatment 210. In operation 2, server 101 identifies lesions corresponding to the target lesions identified in operation 1 on the pre-baseline scan 204s. In operation 3, server 101 calculates baseline features (feature quantities) from the baseline scan 208s, including image features and non-image features (at the same scan time). In operation 4, the server 101 calculates prebaseline features (feature quantities) from the prebaseline scan 204s, including image features and non-image features (at the same scan time) (e.g., determination, measurement).

[0047] In operation 5, the server 101 calculates the changes in image and non-image features between the pre-baseline scan 204s and the baseline scan 208s. In some embodiments, the change in features between the pre-baseline scan 204s and the baseline scan 208s is normalized by the number of days between the two scans. For example, the server 101 calculates the normalized change (difference) by dividing the change in image features and non-image features by the number of days between the pre-baseline scan 204s and the baseline scan 208s.

[0048] In operation 6, the server 101 combines baseline features, pre-baseline features, and the difference (change) between these features as input to disease-level and patient-level treatment response prediction models, and predicts the treatment response at specific points in time after the start of treatment (e.g., +6 weeks, +12 weeks, etc.). In some embodiments, the lesion-level predictive model predicts the treatment response (e.g., growth kinetics) for each target lesion. In other embodiments, the patient-level predictive model calculates the patient-level treatment response (e.g., response based on RECIST evaluation criteria) by combining the predicted growth kinetics for each target lesion.

[0049] In some embodiments, the server 101 may use the predicted results for lesion level and patient level to create a recommended treatment plan.

[0050] In some embodiments, the server 101 may also collect observed lesion level and patient level outcome labels for use in adapting (updating) the online learning model.

[0051] Figure 3 shows a flow diagram of a method for predicting the response to a therapeutic agent using deep learning analysis on sequential images before and during treatment, according to some embodiments. Each of the methods described herein (including Method 300) can be implemented by processing logic that includes hardware (e.g., processors, circuits, proprietary logic, programmable logic, microcode, device hardware, etc.), software (e.g., instructions executed on the processor), or a combination thereof. In some embodiments, these methods may be performed by the processing logic of the machine learning architecture 127 shown in Figure 1.

[0052] As shown in Figure 3, method 300 includes the step in block 302 of acquiring baseline features (features) of one or more target lesions associated with baseline scans obtained from the patient before treatment. In some embodiments, the processing unit may acquire baseline features by obtaining (e.g., searching for, receiving) baseline features from other computing devices. In other embodiments, the processing unit may acquire baseline features by identifying one or more target lesions on a patient's baseline scan. A scan (e.g., pre-baseline scan, baseline scan, follow-up scan) may include one or more therapeutic images (diagnostic images). The therapeutic images are not limited to computed tomography (CT) scans, positron emission tomography (PET) scans, or magnetic resonance imaging (MRI) scans. The therapeutic images obtained from a scan may include two-dimensional, three-dimensional, or four-dimensional anatomical images. Furthermore, in some embodiments, two or more therapeutic images of multiple types (e.g., CT scans, PET scans, MRI scans, etc.) may be used in combination.

[0053] Method 300 includes the step in block 304 of obtaining pre-baseline features of one or more target lesions associated with a patient's pre-baseline scan. In some embodiments, the processing unit may acquire prebaseline features by obtaining (e.g., searching for, receiving) prebaseline features from other computing devices. In other embodiments, the processing unit may acquire prebaseline features by identifying one or more corresponding target lesions on a patient's prebaseline scan.

[0054] Method 300 includes the step of determining a set of features that indicate changes in one or more target lesions using baseline and pre-baseline features in block 306. In some embodiments, the processing device can determine a feature set by using a patient's baseline scan to determine baseline features of one or more target lesions, using a patient's pre-baseline scan to determine pre-baseline features of one or more target lesions, and further determining the difference by comparing (e.g., subtracting) the baseline features with the pre-baseline features. In other words, in some embodiments, the processing device can determine a feature set that indicates changes in one or more target lesions by calculating the difference between the pre-baseline features and the baseline features. Furthermore, the processing unit can generate a normalized difference by dividing the difference between image features and non-image features by the difference in days between the baseline scan and the pre-baseline scan.

[0055] Method 300 includes the step in block 308 of inputting the aforementioned feature set into one or more deep learning models (predictive models). These deep learning models are uniquely trained using multiple training datasets to predict therapeutic agent (e.g., immunotherapy) responses based on the aforementioned feature set, such as changes between sequential image data at multiple time points. In some embodiments, the training dataset includes imaging and / or non-imaging features associated with target lesions in multiple patients. In other embodiments, the training dataset also includes information indicating changes in lesion volume and / or lesion diameter, as well as / or other patient-level endpoints (e.g., progression-free survival (PFS), overall survival (OS), clinical benefit, objective response based on the RECIST protocol, etc.). Examples of predictive models (including deep learning models) include, but are not limited to, artificial neural networks, convolutional neural networks, random forest models, support vector machines, and logistic regression models. In other embodiments, a single predictive model may be used. In some embodiments, one or more predictive models may be trained using training data that includes image and non-image features associated with multiple target lesions in multiple patients. In some embodiments, one or more predictive models may also be further trained to predict therapeutic response based on changes in lesion volume.

[0056] In some embodiments, the computing system can train a predictive model to predict therapeutic response using one or more training datasets, as described herein. In some embodiments, the computing system can train a predictive model to predict therapeutic response showing pseudo-exacerbation (pseudo-progression) based on at least one of changes in the patient's lesion volume or lesion diameter, using one or more training datasets. In some embodiments, the predictive accuracy of the predictive model can be improved by training the predictive model using multiple pre-treatment scans associated with multiple patients.

[0057] Deep learning models can utilize a variety of appropriate learning methods, as described here. For example, in one embodiment, the deep learning model uses a group of subjects to be trained and multiple images associated with each subject as training data. In another embodiment, the deep learning model uses subject-specific models calculated for each subject as training data. In yet another embodiment, the deep learning model may be used in combination with the two methods described above. Furthermore, in other embodiments, these models may be trained using different data, different methods, and different objectives, and their results may be aggregated (model fusion) in various ways.

[0058] In one embodiment, the treatment may be a PD-1 (programmed cell death protein 1)-based treatment. In another embodiment, the treatment may be a PD-L1 (programmed cell death ligand 1)-based treatment. In yet another embodiment, the treatment may be a CTLA-4 (cytotoxic T lymphocyte antigen 4)-based treatment, or other appropriate forms of treatment (e.g., chemotherapy, drug-based treatment, radiotherapy, etc.).

[0059] Method 300 includes a step in step 310 in which the processing unit generates a predicted treatment response score for a treatment (e.g., a score on a scale representing a range from a low probability of a positive or negative effect occurring to a high probability) based on a feature set and one or more deep learning models (input results of the predictive models). In one embodiment, the processing logic generates a predicted treatment response score based on a single pre-treatment image and two or more deep learning models (input results of the predictive models). In one embodiment, a single response score can be generated by combining the results obtained from different models (e.g., averaging or integrating in any other way).

[0060] In one embodiment, the aforementioned predictive treatment response score includes a prediction of disease progression in a patient in response to a particular drug. In another embodiment, the aforementioned predictive treatment response score includes a prediction of the occurrence of one or more immune-related adverse events associated with immunotherapy. Also in one embodiment, the predictive treatment response score includes a predicted probability (e.g., confidence level) of the occurrence of a particular type of response and / or adverse event. In other embodiments, the response scores described above may include indicators of pseudo-exacerbation. Pseudo-exacerbation is a condition characterized by a short-term and transient increase in tumor volume, which is attributed to spontaneous swelling and / or inflammation (e.g., a response to treatment) rather than disease progression. In other embodiments, the response score may include the possibility of exacerbation. Exacerbation refers to a serious condition in which the progression of the disease is accelerated by the implementation of treatment. In yet another embodiment, the response score described above may be configured to indicate the patient's predicted progression-free survival (PFS) or overall survival (OS) in terms of months or years, in relation to the target site. In yet another embodiment, the generation of the aforementioned predictive treatment response score may be further based on pretreatment information indicating at least one of changes in blood test values, changes in urine test values, or changes in imaging features.

[0061] In some embodiments, the processing device can acquire post-treatment features of one or more target lesions associated with a post-treatment scan of the patient after treating the patient according to a treatment plan. In some embodiments, the processing device can use post-treatment features and at least one of baseline or pre-treatment features to determine a second set of features indicating a second change in one or more target lesions. In some embodiments, the processing unit can input a second set of features into one or more predictive models. In some embodiments, the processing unit can, after treating the patient according to a treatment plan, generate a second predicted treatment response score for the patient's second treatment plan based on a second feature set and one or more predictive models (input results of the predictive models).

[0062] Method 300 may include a step (block 316: not shown in Figure 3) of providing a recommended treatment plan based on a predicted treatment response. For example, based on the predicted treatment response, the recommended treatment plan may include indicators such as whether or not a particular drug should be used, the dosage of the drug, and the timing of administration of the drug. In one embodiment, lesion-specific treatment plans combining systemic and local therapies may be generated using lesion-specific response predictions to immunotherapy and / or chemotherapy to enhance treatment efficacy in high-risk lesions. These local therapies may include any of the following: stereotactic ablation radiotherapy (SBRT), intensity-modulated radiotherapy (IMRT), conformal radiotherapy (CRT), radiosurgery, surgical resection, thermal ablation, cryoablation, or high-intensity focused ultrasound (HIFU). In yet another embodiment, the recommended treatment plan may include discontinuing one or all treatments in order to maximize the patient's quality of life (QOL).

[0063] In one embodiment, the processing logic can perform various follow-up processes to improve the accuracy of at least one of the prediction or recommendation. For example, in one embodiment, the processing logic receives treatment process follow-up images, inputs the treatment process follow-up images into a machine learning model, and generates an updated predicted treatment response score. The processing logic can then provide an updated recommended treatment plan based on the updated predicted treatment response score. In one embodiment, the pre-treatment image and the follow-up image during the treatment process may each include multiple image biomarkers.

[0064] In various embodiments, the processing logic can perform any number of appropriate pre- and post-processing operations to improve the accuracy, efficiency, and / or compatibility of the machine learning model in the present invention. For example, with respect to pre-processing, conventional radiomics methods are susceptible to variations due to differences in scanner hardware and imaging protocols. The data pre-processing and data augmentation systems described herein are designed to optimize the generalization performance of the model and minimize the model's sensitivity to differences in image hardware and imaging protocols.

[0065] Figure 4 is a schematic diagram of an information processing device (e.g., a computer system 400) for executing the instruction set 422 for performing one or more methodologies described herein. In other embodiments, this information processing device may be connected to other information processing devices (e.g., network connection) in a local area network (LAN), intranet, extranet, or internet. This information processing device may operate as a server or a client device in a client-server network environment, or as a peer device in a peer-to-peer (or distributed) network environment. Furthermore, this information processing device may be a personal computer (PC), tablet PC, set-top box (STB), PDA (Personal Digital Assistant), mobile phone, web appliance, server, network router, switch or bridge, hub, access point, network access control device, or any other information processing device capable of executing a set of instructions (sequentially or otherwise) that defines the operations that the device should perform. Furthermore, although only a single information processing device is illustrated, the term “information processing device” should be interpreted to include a collection of multiple information processing devices that individually or collectively execute one or more instruction sets (or instruction sets) that implement the methodologies described herein. In one embodiment, the computer system 400 may represent a server computer system such as system 100.

[0066] The computer system 400 shown in Figure 4 comprises a processing unit 402, main memory 404 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM)), static memory 406 (e.g., flash memory, static random access memory (SRAM), etc.), and data memory 418, which communicate with each other via a bus 430. Signals provided via the various buses described herein may be time-division multiplexed with other signals and provided on one or more common buses. In addition, connections between circuit components or blocks may be shown as buses or as single signal lines. Each bus may alternatively be one or more signal lines, and each single signal line may alternatively be configured as a bus.

[0067] The processing unit 402 comprises one or more general-purpose processing units, such as microprocessors and central processing units (CPUs). More specifically, the processing unit 402 may be a CISC (Complex Instruction Set Computing) microprocessor, a RISC (Reduced Instruction Set Computer) microprocessor, a VLIW (Very Long Instruction Word) microprocessor, a processor implementing other instruction sets, or a processor implementing a combination of multiple instruction sets. Furthermore, the processing unit 402 may include one or more dedicated processing units, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), and network processors. The processing unit 402 is configured to execute processing logic 426, which constitutes an example of the system 100 shown in Figure 1 and is for performing the various operations and procedures described herein.

[0068] The data storage device 418 may include a machine-readable storage medium 428 which stores one or more instruction sets 422 (e.g., software) that implement at least one of the methods or functions described herein. When the instruction set 422 is executed by the computer system 400, all or at least part of it may reside in the main memory 404 or the processing unit 402, in which case the main memory 404 and the processing unit 402 also constitute a machine-readable storage medium. Furthermore, the instruction set 422 may be transmitted or received over the network 420 via the network interface device 408.

[0069] The machine-readable storage medium 428 can also be used to store instructions for performing the methods and operations described herein. Although the machine-readable storage medium 428 is shown as a single medium in exemplary embodiments, the term “machine-readable storage medium” is to be interpreted as encompassing a single or multiple medium (e.g., a centralized or distributed database, associated caches and servers) that stores one or more sets of instructions. Machine-readable media include any mechanism for storing information in a format that can be read by a machine (e.g., a computer) (e.g., software, processing applications). Machine-readable media include, but are not limited to, magnetic storage media (e.g., floppy disks), optical storage media (e.g., CD-ROMs), magneto-optical storage media, read-only memory (ROM), random-access memory (RAM), erasable programmable memory (e.g., EPROM and EEPROM), flash memory, and other media suitable for storing electronic instructions.

[0070] The foregoing description provides numerous specific examples of systems, components, and methods to enable a thorough understanding of the various embodiments of this disclosure. However, as will be apparent to those skilled in the art, at least some embodiments of this disclosure can be implemented without these specific details. Furthermore, for the sake of simplicity, detailed descriptions of well-known components and methods may be omitted or they may be shown in the form of simple block diagrams. Therefore, the specific details shown above are merely illustrative. Individual embodiments may differ from these exemplary details, but they are still included within the scope of this disclosure.

[0071] Furthermore, some embodiments may be implemented in a distributed computing environment. In this case, the machine-readable medium may be stored on or executed by multiple computer systems. In addition, the information transferred between computer systems may be sent and received via a communication medium connecting these systems, using either a pull or push method.

[0072] Embodiments of the invention relating to this disclosure include, but are not limited to, various operations described herein. These operations may be performed by hardware components, software, firmware, or a combination thereof.

[0073] Although the operations of the methods described herein are illustrated and described in a particular order, this order is not limiting. That is, the order of operations of each method may be changed, certain operations may be performed in reverse order, and certain operations may be performed at least partially simultaneously with other operations. In other embodiments, instructions or suboperations included in individual operations may be performed intermittently or alternately.

[0074] The above description of embodiments relating to the present invention, including its abstract conceptual description, does not limit the invention to these embodiments. The embodiments and specific examples described herein are provided for the purpose of describing the present invention, and various equivalent modifications can be made to the extent that a person skilled in the art will recognize. The terms “example” or “exemplary” used herein are used to serve as examples or illustrations. Whatever form or design is described as “example” or “exemplary,” it should not be interpreted as superior to other forms or designs. The use of the terms “example” or “exemplary” is intended to illustrate a concept in a concrete form. As used herein, the term "or" is intended to be interpreted as an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or evident from the context, the expression "X contains A or B" means any of the natural inclusive reorderings. In other words, if X contains A, if X contains B, or if X contains both A and B, then in all of the foregoing cases, the condition "X contains A or B" is satisfied. Furthermore, the articles “a” and “an” used in this specification and the claims of the appended patent claims shall be interpreted as meaning “one or more” unless otherwise specified, unless the context makes it clear that they are singular. Furthermore, where terms such as "1st," "2nd," "3rd," and "4th" are used in the specification, these terms are used as labels to distinguish different elements and do not necessarily indicate an order according to the numerical designation.

[0075] The features and functions disclosed above, or variations thereof, can be incorporated into various other systems or applications as appropriate. Various alternatives, modifications, changes, or improvements not currently foreseen or anticipated may be made later by those skilled in the art, and these are also intended to be covered by the claims set forth below. The claims may encompass embodiments of hardware, software, or a combination thereof. While this specification has described the invention with reference to certain exemplary embodiments, it is evident that various changes and modifications can be made to such embodiments without departing from the spirit and scope of the invention as set forth in the appended claims. Accordingly, this specification and the drawings are to be interpreted illustratively and not restrictively.

Claims

1. It is a method, The process includes obtaining pre-treatment characteristics of one or more target lesions associated with a pre-treatment scan of the target site before treating the target site according to the treatment plan, A step of determining a set of features that show changes in one or more target lesions using the pre-treatment features, The process involves inputting the aforementioned set of features into one or more predictive models that have been trained to predict the therapeutic response based on the characteristics of the target lesion, A method comprising the step of generating a predicted treatment response score for the treatment plan by a processing device based on the feature set and the input results from one or more predictive models, before treating the target site in accordance with the treatment plan.

2. In the method according to claim 1, A method wherein one or more predictive models are trained using training data that includes image and non-image features associated with multiple target lesions in multiple target sites.

3. In the method according to claim 1, The generation of the aforementioned predicted treatment response score further involves, Changes in blood test values, Changes in urine test values, or Changes in image features A method based on pretreatment information indicating at least one of the following.

4. In the method according to claim 1, A method wherein one or more predictive models are further trained to predict therapeutic response based on changes in lesion volume.

5. The method according to claim 1, A method further comprising the step of improving the prediction accuracy of a predictive model by training the predictive model with multiple pre-treatment scans associated with multiple target sites.

6. In the method according to claim 5, The step of obtaining pre-treatment characteristics of one or more target lesions associated with the pre-treatment scan of the target site is: A step of acquiring baseline features of one or more target lesions associated with a baseline scan taken before applying a treatment plan to the target site, A method comprising the step of obtaining corresponding pre-baseline features of the target lesion associated with a pre-baseline scan taken before applying the treatment plan.

7. In the method according to claim 6, The step of determining a set of features that show changes in one or more target lesions using the pre-treatment features is: A method comprising the step of calculating the difference between the pre-baseline features and the baseline features.

8. The method according to claim 7, A method further comprising the step of normalizing the difference between the pre-baseline features and the baseline features, which is the difference between the pre-baseline features and the baseline features, by dividing it by the number of days between the baseline scan and the pre-baseline scan, thereby generating a normalized difference.

9. The method according to claim 1, The process involves treating the target site according to a treatment plan, and then obtaining post-treatment characteristics of one or more target lesions associated with a post-treatment scan of the target site. A step of determining a second set of features that show a second change using the post-treatment features and at least one of the baseline features or the pre-treatment features, The process of inputting the second set of features into one or more predictive models, A method further comprising the steps of: treating the target site in accordance with the treatment plan, and then generating a second predicted treatment response score for the second treatment plan based on the second set of features and the input results from one or more predictive models.

10. In the method according to claim 1, The aforementioned predicted treatment response score is, An indicator showing pseudo-exacerbation associated with one or more target lesions, An indicator showing exacerbation associated with one or more of the aforementioned target lesions, or An index indicating the predicted overall survival of a patient in relation to the aforementioned target site. A method that includes at least one of the following.

11. A treatment analysis system, A memory unit for storing pre-treatment scans of the target site, The storage unit comprises a processing unit operably connected to the storage unit, The aforementioned processing apparatus is Before treating the target site according to the treatment plan, obtain pretreatment features of one or more target lesions associated with a pretreatment scan of the target site. Using the aforementioned pre-treatment characteristics, a set of features indicating changes in one or more target lesions is determined. The aforementioned set of features is input into one or more predictive models that have been trained to predict the therapeutic response based on the characteristics of the target lesion. A treatment analysis system characterized by generating a predicted treatment response score for the treatment plan based on the feature set and the input results from one or more predictive models, before treating the target site according to the treatment plan.

12. In the treatment analysis system according to claim 11, A therapeutic analysis system in which one or more predictive models are trained using training data that includes multiple image and non-image features associated with multiple target lesions in multiple target sites.

13. In the treatment analysis system according to claim 11, The generation of the aforementioned predicted treatment response score further involves, Changes in blood test values, Changes in urine test values, or Changes in image features A treatment analysis system based on pretreatment information indicating at least one of the following.

14. In the treatment analysis system according to claim 11, A therapeutic analysis system in which one or more predictive models are further trained to predict therapeutic response based on changes in lesion volume.

15. In the treatment analysis system according to claim 11, The aforementioned processing apparatus is A treatment analysis system that improves the prediction accuracy of a prediction model by training one or more prediction models using multiple pre-treatment scans associated with multiple target sites.

16. In the treatment analysis system according to claim 15, The processing apparatus further, in order to acquire pre-treatment characteristics of one or more target lesions associated with the pre-treatment scan of the target site, Obtain baseline features of one or more target lesions associated with baseline scans taken before applying a treatment plan to the target site, A treatment analysis system that acquires pre-baseline features of the corresponding target lesion associated with a pre-baseline scan taken before applying the treatment plan.

17. In the treatment analysis system according to claim 16, The aforementioned processing apparatus is In order to determine a set of features indicating changes in one or more target lesions using the aforementioned pre-treatment characteristics, The difference between the pre-baseline features and the baseline features is calculated. A treatment analysis system that normalizes the difference between the pre-baseline features and the baseline features, which are the differences between the image features and non-image features, by dividing them by the number of days between the baseline scan and the pre-baseline scan, thereby generating a normalized difference.

18. A treatment analysis system according to claim 11, The aforementioned processing apparatus is After treating the target site according to the treatment plan, one or more post-treatment features of the target lesion associated with the post-treatment scan of the target site are obtained. Using the post-treatment characteristics and at least one of the baseline characteristics or the pre-treatment characteristics, a second set of characteristics indicating a second change is determined. The second set of features is input into one or more prediction models. A treatment analysis system that, after treating the target site according to the treatment plan, generates a second predicted treatment response score for a second treatment plan based on the second feature set and the input results from one or more predictive models.

19. In the treatment analysis system according to claim 11, The aforementioned predicted treatment response score is, An indicator showing pseudo-exacerbation associated with one or more target lesions, An indicator showing exacerbation associated with one or more of the aforementioned target lesions, or An index indicating the predicted overall survival of a patient in relation to the aforementioned target site. A treatment analysis system including at least one of the following.

20. A non-temporary computer-readable storage medium wherein, when an instruction stored in the storage medium is executed by a processing unit, the processing unit performs the following: Before treating the target site according to the treatment plan, one or more pretreatment features of the target lesion associated with the pretreatment scan of the target site are obtained; Using the aforementioned pre-treatment characteristics, a set of features indicating changes in one or more target lesions is determined; The aforementioned set of features is input into one or more predictive models that have been trained to predict the therapeutic response based on the characteristics of the target lesion; Before treating the target site according to the treatment plan, a predicted treatment response score for the treatment plan is generated based on the feature set and the input results from one or more predictive models.

Images