Machine learning supported longitudinal analysis of positron emission tomography scans and computed tomography scans for assessing disease progression and therapeutic response

Abstract

The present disclosure provides a method that may include determining a first tumor mask corresponding to a first positron emission tomography (PET) scan and a first computed tomography (CT) scan present in the first PET scan and the first CT scan based on the first PET scan and the first CT scan from a first point in time. A second tumor mask corresponding to a second lesion present in a second PET scan and a second CT scan may be determined based on the second PET scan and the second CT scan from a second point in time. A longitudinal segmentation model may be applied to update each of the first tumor mask and the second tumor mask based on the first PET scan, the first CT scan, the second PET scan, and the second CT scan. A response to treatment for the disease may be determined based on at least one of the first updated tumor mask and the second updated tumor mask. Related systems and computer program products are also provided.

Description

Cross Reference to Related Applications

[0001] This application claims priority to U.S. Provisional Application 63 / 497,660, entitled “MACHINE LEARNING ENABLED LONGITUDINAL ANALYSIS OF POSITRON EMISSION TOMOGRAPHY AND COMPUTED TOMOGRAPHY SCANS FOR ASSESSMENT OF DISEASE PROGRESSION AND TREATMENT RESPONSE,” filed April 21, 2023, the disclosure of which is incorporated by reference herein in its entirety. TECHNICAL FIELD

[0002] The subject matter described herein relates generally to machine learning, and more specifically to machine learning based techniques for assessing disease progression and treatment response based on positron emission tomography (PET) scans and computed tomography (CT) scans. BACKGROUND

[0003] Medical imaging refers to techniques and procedures used to obtain data characterizing internal anatomy and pathophysiology of a subject, including images generated, for example, by detecting radiation passing through the body (e.g., x-rays) or emitted by an administered radiopharmaceutical (e.g., gamma rays from an intravenously administered radioactive tracer). By revealing internal anatomy obscured by other tissues such as skin, subcutaneous fat, and bone, medical imaging is an indispensable part of numerous medical diagnoses and / or treatments. Examples of medical imaging modalities include two-dimensional imaging such as x-ray flat panel, bone scintigraphy, and thermography. Examples of three-dimensional imaging modalities include magnetic resonance imaging (MRI), computed tomography (CT), cardiac sestamibi scans, and positron emission tomography (PET). SUMMARY

[0004] Systems, methods, and articles of manufacture (including computer program products) are provided herein for machine learning supported longitudinal analysis of positron emission tomography (PET) scans and computed tomography (CT) scans for assessing disease progression and treatment response. Implementations of the current subject matter can include, but are not limited to, methods consistent with the descriptions provided herein and articles of manufacture including a tangible, machine-readable medium operable to cause one or more machines (e.g., computers, etc.) to result in operations that implement one or more of the described features. Similarly, computer systems are also described that can include one or more processors and one or more memories coupled to the one or more processors. The memories can include, encode, store, and / or the like one or more programs that cause the one or more processors to perform one or more of the operations described herein. Computer implemented methods consistent with one or more implementations of the current subject matter can be implemented by one or more data processors existing within a single computing system or multiple computing systems. Such multiple computing systems can be connected, and can exchange data and / or commands or other suitable instructions or other instructions via one or more connections, including, for example, a connection over a network (e.g., the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, or the like), a connection over a direct connection (e.g., a direct connection between one or more of the multiple computing systems), and / or the like.

[0005] In one aspect, a system for machine learning supported longitudinal analysis of positron emission tomography (PET) scans and computed tomography (CT) scans for assessing disease progression and treatment response is provided. The system can include at least one processor and at least one memory. The at least one memory can include program code that, when executed by the at least one processor, provides operations. The operations can include determining, based at least on a first positron emission tomography (PET) scan and a first computed tomography (CT) scan from a first time point, a first tumor mask corresponding to a first lesion present in the first PET scan and the first CT scan; determining, based at least on a second PET scan and a second CT scan from a second time point, a second tumor mask corresponding to a second lesion present in the second PET scan and the second CT scan; applying a longitudinal segmentation model to update each of the first tumor mask and the second tumor mask based at least on the first PET scan, the first CT scan, the second PET scan, and the second CT scan; and determining, based on at least one of the first updated tumor mask and the second updated tumor mask, a response to a treatment for a disease.

[0006] In another aspect, a method for machine learning supported longitudinal analysis of positron emission tomography (PET) scans and computed tomography (CT) scans for assessing disease progression and treatment response is provided. The method can include determining, based at least on a first positron emission tomography (PET) scan and a first computed tomography (CT) scan from a first time point, a first tumor mask corresponding to a first lesion present in the first PET scan and the first CT scan; determining, based at least on a second PET scan and a second CT scan from a second time point, a second tumor mask corresponding to a second lesion present in the second PET scan and the second CT scan; applying a longitudinal segmentation model to update each of the first tumor mask and the second tumor mask based at least on the first PET scan, the first CT scan, the second PET scan, and the second CT scan; and determining a response to a treatment for a disease based on at least one of the first updated tumor mask and the second updated tumor mask.

[0007] In another aspect, a computer program product for machine learning supported longitudinal analysis of positron emission tomography (PET) scans and computed tomography (CT) scans for assessing disease progression and treatment response is provided. The computer program product can include a non-transitory computer readable medium storing instructions that, when executed by at least one data processor, cause operations. The operations can include determining, based at least on a first positron emission tomography (PET) scan and a first computed tomography (CT) scan from a first time point, a first tumor mask corresponding to a first lesion present in the first PET scan and the first CT scan; determining, based at least on a second PET scan and a second CT scan from a second time point, a second tumor mask corresponding to a second lesion present in the second PET scan and the second CT scan; applying a longitudinal segmentation model to update each of the first tumor mask and the second tumor mask based at least on the first PET scan, the first CT scan, the second PET scan, and the second CT scan; and determining a response to a treatment for a disease based on at least one of the first updated tumor mask and the second updated tumor mask.

[0008] In some variations of the methods, systems, and non-transitory computer readable media, one or more of the following features can optionally be included in any workable combination.

[0009] In some variations, the method can: determine, based at least on a first positron emission tomography (PET) scan and a first computed tomography (CT) scan from a first time point, a first tumor mask corresponding to a first lesion present in the first PET scan and the first CT scan; determine, based at least on a second PET scan and a second CT scan from a second time point, a second tumor mask corresponding to a second lesion present in the second PET scan and the second CT scan; apply a longitudinal segmentation model to update each of the first tumor mask and the second tumor mask based at least on the first PET scan, the first CT scan, the second PET scan, and the second CT scan; and determine, based on at least one of the first updated tumor mask and the second updated tumor mask, a response to a treatment for the disease.

[0010] In some variations, the first tumor mask can identify a first plurality of pixels in each of the first PET scan and the first CT scan that depict the first lesion, and wherein the second tumor mask identifies a plurality of pixels from the second PET scan and the second CT scan that depict the second lesion.

[0011] In some variations, the method can register the first CT scan, the first PET scan, the second CT scan, and the second PET scan to align the first CT scan and the first PET scan with the second CT scan and the second PET scan.

[0012] In some variations, the method can identify the second lesion as a new lesion based at least on the first updated tumor mask and the second updated tumor mask; and responsive to the second lesion being identified as a new lesion, the response to the treatment can be determined as a progression of the disease (PMD).

[0013] In some variations, the method can determine a distance between the first lesion and the second lesion based at least on the first updated tumor mask and the second updated tumor mask; identify the second lesion as a new lesion based at least on the distance between the first lesion and the second lesion satisfying one or more thresholds; and identify the second lesion as being the same lesion as the first lesion based at least on the distance between the first lesion and the second lesion failing to satisfy the one or more thresholds.

[0014] In some variations, the method can determine, responsive to determining that the first lesion and the second lesion are the same lesion, the response to the treatment for the disease based at least on a change in metabolic activity exhibited by the lesion between the first time point and the second time point.

[0015] In some variations, the change in metabolic activity between the first time point and the second time point can be determined by at least: determining, based at least on the first updated tumor mask and the first PET scan, a first metabolic activity level exhibited by the lesion at the first time point; determining, based at least on the second updated tumor mask and the second PET scan, a second metabolic activity level exhibited by the lesion at the second time point; and determining, based at least on the first metabolic activity level and the second metabolic activity level, the change in metabolic activity between the first time point and the second time point.

[0016] In some variations, based at least on the change in metabolic activity between the first time point and the second time point satisfying a first threshold, the response to the treatment is determined to be a disease metabolic progression (PMD).

[0017] In some variations, based at least on the change in metabolic activity between the first time point and the second time point, the response to the treatment can be determined to be a no metabolic response (NMR).

[0018] In some variations, based at least on the change in metabolic activity between the first time point and the second time point failing to satisfy the first threshold and a second threshold, the response to the treatment can be determined to be a partial metabolic response (PMR).

[0019] In some variations, the first metabolic activity level can correspond to a first standardized uptake value (SUV), and the second metabolic activity level corresponds to a second standardized uptake (SUV) value.

[0020] In some variations, each of the first metabolic activity level and the second metabolic activity level can correspond to a maximum, a minimum, a median, a mean, or a mode of the metabolic activity levels exhibited by the lesion at the corresponding time point.

[0021] In some variations, the first CT scan and the first PET scan can be performed prior to the treatment for the disease, and the second CT scan and the second PET scan can be performed after the treatment for the disease.

[0022] In some variations, the method can determine a tumor volume change based at least on the first updated tumor mask and the second updated tumor mask; and can determine the response to the treatment for the disease based at least on the tumor volume change.

[0023] In some variations, the method can determine a difference in the first metabolic activity change and / or the second tumor volume change across different lesions between the first time point and the second time point based at least on the first updated tumor mask and the second updated tumor mask; and can determine the response to the treatment based at least on the difference in the metabolic activity change and / or the tumor volume change exhibited by the different lesions between the first time point and the second time point.

[0024] In some variations, the method can determine a progression of the disease based at least on the first updated tumor mask and the second updated tumor mask.

[0025] In some variations, the first tumor mask can be determined by applying the segmentation model to the first PET scan and the first CT scan, and the second tumor mask can be determined by applying the segmentation model to the second PET scan and the second CT scan.

[0026] In some variations, the longitudinal segmentation model can be an artificial neural network or a visual transformer.

[0027] In some variations, each of the first CT scan, the first PET scan, the second CT scan, and the second PET scan can be a three-dimensional volume comprising a plurality of two-dimensional image slices.

[0028] In some variations, each pixel in the first PET scan and the second PET scan can be associated with an intensity value corresponding to a level of metabolic activity.

[0029] In some variations, each pixel in the first CT scan and the second CT scan can be associated with an intensity value corresponding to a tissue density or X-ray attenuation.

[0030] In some variations, the method can train the longitudinal segmentation model to update two or more tumor masks, each of the two or more tumor masks being generated from a positron emission tomography (PET) scan and a computed tomography (CT) scan from a single time point.

[0031] In some variations, the response to the treatment for the disease can be a complete metabolic response (CMR) or a non-complete metabolic response (non-CMR).

[0032] In some variations, the response to the treatment for the disease can be a responder or a non-responder.

[0033] In some variations, the response to the treatment for the disease can be a complete metabolic response (CMR), a partial metabolic response (PMR), a no metabolic response (NMR), or a progressive metabolic disease (PMD).

[0034] In some variations, the method can extract, from the first PET scan and the first CT scan, a first image patch comprising a first lesion associated with the first tumor mask; extract, from the second PET scan and the first CT scan, a second image patch comprising a second lesion associated with the second tumor mask; and apply the longitudinal segmentation model to the first image patch and the second image patch to update each of the first tumor mask and the second tumor mask.

[0035] The details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features and advantages of the subject matter described herein will be apparent from the description and drawings, and from the claims. While certain features of the currently disclosed subject matter are described for illustrative purposes with respect to fluorodeoxyglucose high-affinity (FDG-high affinity) cancers related to certain types of non-Hodgkin lymphoma (NHL), it is readily understood that such features are not intended to be limiting. The claims set forth below are intended to define the scope of the protected subject matter. BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings are incorporated in and constitute a part of this specification and, together with the description, explain certain aspects of the subject matter disclosed herein and, together with the description, help explain some of the principles associated with the disclosed implementations. In the drawings, FIG. 1 depicts a system diagram showing an example of a machine learning-based medical imaging analysis system, in accordance with some example embodiments; FIG. 2 depicts a schematic diagram showing an example of a process for machine learning-supported longitudinal analysis of positron emission tomography (PET) scans and computed tomography (CT) scans, in accordance with some example embodiments; FIG. 3 depicts a flow diagram showing an example of a process for machine learning-supported longitudinal analysis of positron emission tomography (PET) scans and computed tomography (CT) scans, in accordance with some example embodiments; FIG. 4 depicts a flow diagram showing another example of a process for machine learning-supported longitudinal analysis of positron emission tomography (PET) scans and computed tomography (CT) scans, in accordance with some example embodiments; FIG. 5 depicts a flow diagram showing another example of a process for machine learning-supported longitudinal analysis of positron emission tomography (PET) scans and computed tomography (CT) scans, in accordance with some example embodiments; FIG. 6 depicts a comparison of accuracy of complete metabolic response (CMR) assessment and objective response (OR) assessment by machine learning supported longitudinal analysis and radiologist-to-radiologist agreement across different clinical datasets, according to some example embodiments; FIG. 7 depicts a comparison of performance of machine learning supported longitudinal analysis and expert radiologists, according to some example embodiments; FIG. 8A depicts a comparison of accuracy of complete metabolic response (CMR) assessment and objective response (OR) assessment by machine learning supported longitudinal analysis and radiologist-to-radiologist agreement, according to some example embodiments; FIG. 8B depicts a comparison of accuracy of response assessment by machine learning supported longitudinal analysis and radiologist-to-radiologist agreement, according to some example embodiments; FIG. 8C depicts a comparison of F1 score of machine learning supported longitudinal analysis and radiologist analysis, according to some example embodiments; FIG. 8D depicts an evaluation of progression free survival (PFS) at end of treatment by machine learning supported longitudinal analysis and by expert radiologist panel across different clinical datasets, according to some example embodiments; FIG. 8E depicts a comparison of accuracy of complete metabolic response (CMR) assessment, objective response (OR) assessment, and four-class assessment by machine learning supported longitudinal analysis and by expert radiologists, according to some example embodiments; FIG. 8F depicts a comparison of accuracy of machine learning supported longitudinal analysis and expert radiologist analysis, according to some example embodiments; FIG. 8G depicts a comparison of overall (OS) at end of treatment by machine learning supported longitudinal analysis and adjudicated response across different clinical datasets, according to some example embodiments; FIG. 9 depicts overall survival (OS) at end of treatment and early discontinuation determined by machine learning supported longitudinal analysis across various clinical datasets, according to some example embodiments; FIG. 10 depicts a comparison of Dwass score (DS) assessment by machine learning supported longitudinal analysis and adjudicated response across different clinical datasets, according to some example embodiments.

[0037] FIG. 11 depicts examples of true positives, true negatives, false positives, and false negatives.

[0038] FIG. 12 depicts a block diagram showing an example of a computing system, according to some example embodiments.

[0039] the disclosure of which is incorporated by reference herein in its entirety.

[0040] When actual application is made, like reference numerals indicate like structures, features or elements. DETAILED DESCRIPTION

[0041] Various forms of medical imaging can be applied to obtain data characterizing the internal anatomy of a subject as well as pathophysiology. An example of a three-dimensional imaging modality is computed tomography (CT), in which a series of x-rays are captured to generate cross-sectional images (e.g., image blocks, slices, etc.) of bone, blood vessels, and soft tissue within a body. A computed tomography scan can be a three-dimensional volume formed from a series of two-dimensional images, in which each pixel is associated with an intensity value indicative of tissue density or x-ray attenuation at a corresponding location in the subject’s body. Another example of a three-dimensional imaging modality is positron emission tomography (PET), which captures radioactive signals indicative of cellular metabolic activity within a subject’s body. A positron emission tomography scan can be a three-dimensional volume formed from a series of two-dimensional images, in which each pixel is associated with an intensity value indicative of a level of cellular metabolic activity (e.g., glucose uptake) at a corresponding location in the subject’s body. In some cases, a single gantry that incorporates both a positron emission tomography (PET) scanner and a computed tomography (CT) scanner is capable of acquiring both a positron emission tomography (PET) scan and a computed tomography (CT) scan during the same session. The resulting positron emission tomography (PET) scan and computed tomography (CT) scan can be combined into a single superimposed (e.g., co-registered) image (e.g., a PET-CT scan), in which the spatial distribution of metabolic activity depicted in the positron emission tomography (PET) scan is aligned with the anatomical structure depicted in the computed tomography (CT) scan.

[0042] In some example embodiments, the analysis controller can perform longitudinal analysis of positron emission tomography (PET) scans and computed tomography (CT) scans to assess disease progression and treatment response. In some cases, the analysis controller can apply a longitudinal segmentation model trained to update two or more individual tumor masks generated from positron emission tomography (PET) scans and computed tomography (CT) scans from a single time point. For example, the longitudinal segmentation model can ingest a first tumor mask determined based on a first positron emission tomography (PET) scan and a first computed (CT) scan from a first time point and corresponding to a first lesion present in the first positron emission tomography (PET) scan and the first computed (CT) scan. Further, the longitudinal segmentation model can ingest a second tumor mask determined based on a second positron emission tomography (PET) scan and a second computed tomography (CT) scan from a second time point and corresponding to a second lesion present in the second positron emission tomography (PET) scan and the second computed tomography (CT) scan. The longitudinal segmentation model can update each of the first tumor mask and the second tumor mask based on the first positron emission tomography (PET) scan, the first computed tomography (CT) scan, the second positron emission tomography (PET) scan, and the second computed tomography (CT) scan. In doing so, the longitudinal segmentation model can refine the first tumor mask and the second tumor mask to reduce false positives (where one or more pixels that do not belong to a part of a lesion are erroneously identified as part of the lesion).

[0043] In some example embodiments, the analysis controller can determine a response to treatment for a disease (e.g., complete metabolic response (CMR), objective response (OR), four-class assessment, etc.) based at least on the updated tumor masks generated by the longitudinal segmentation model. Alternatively and / or additionally, the analysis controller can determine a progression of the disease based at least on the updated tumor masks generated by the longitudinal segmentation model. For example, in some cases, the analysis controller can determine a response to treatment for a disease associated with the first lesion and the second lesion based at least on the first updated tumor mask and the second updated tumor mask. In some cases, the analysis controller can determine that the response to treatment for the disease is a progressive metabolic disease (PMD), no metabolic response (NMR), partial metabolic response (PMR), or complete metabolic response (CMR) based at least on the first updated tumor mask and the second updated tumor mask. Alternatively, in some cases, the analysis controller can determine that the response to treatment for the disease is a complete metabolic response (CMR) or a non-complete metabolic response (non-CMR) based at least on the first updated tumor mask and the second updated tumor mask.

[0044] In some example embodiments, the analysis controller can identify the second lesion present in the second positron emission tomography (PET) scan and the second computed tomography (CT) scan as a new lesion not present in the first positron emission tomography (PET) scan and the first computed tomography (CT) scan based at least on the first updated tumor mask and the second updated tumor mask. For example, in some cases, the analysis controller can determine that the second lesion is a new lesion if a distance (e.g., a minimum distance, an average distance, etc.) between the first lesion and the second lesion satisfies one or more thresholds (e.g., a minimum distance of more than 10 millimeters). Otherwise, if the distance between the first lesion and the second lesion fails to satisfy the one or more thresholds, the analysis controller can determine that the first lesion and the second lesion are the same lesion. Accordingly, in the case that the second lesion is identified as a new lesion, the analysis controller can determine the response to the treatment for the disease as a metabolic response progression (PMR). In the case that the second lesion is identified as the same lesion as the first lesion, the analysis controller can further determine the response to the treatment for the disease based on a change in the metabolic activity level exhibited by the lesion between the first time point and the second time point. For example, the analysis controller can determine the response to the treatment for the disease as: a disease metabolic progression (PMD) where the change in the metabolic activity between the first time point and the second time point satisfies a first threshold; no metabolic response (NMR) where the change in the metabolic activity satisfies a second threshold but fails to satisfy the first threshold; and a partial metabolic response (PMR) where the change in the metabolic activity fails to satisfy both the first threshold and the second threshold.

[0045] In some example embodiments, the analysis controller can determine a change in the metabolic activity level exhibited by the lesion between the first time point and the second time point based at least on the first updated tumor mask and the second updated tumor mask. The change in the metabolic activity level can correspond to a change in various metrics derived based on the updated tumor masks. Examples of such metrics include a standard uptake value (e.g., a maximum standard uptake value, a minimum standard uptake value, a median standard uptake value, an average standard uptake value, a mode standard uptake value, etc.) and a lesion size. For example, in some cases, the change in the metabolic activity level exhibited by the lesion between the first time point and the second time point can correspond to a difference between a first maximum metabolic activity level at the first time point and a second maximum metabolic activity level at the second time point. Accordingly, in some cases, the analysis controller can determine the first maximum metabolic activity level at the first time point (e.g., a first maximum standard uptake value (SUVmax)) based at least on the first updated tumor mask and the second maximum metabolic activity level at the second time point (e.g., a second maximum standard uptake value (SUVmax)) based at least on the second updated tumor mask. SUV max). Additionally, the analytics controller can determine a second maximum metabolic activity level (e.g., a second maximum standardized uptake value (SUVmax) at the second time point based at least on the second updated tumor mask SUV max As described above, in some cases where the analytics controller fails to identify a new lesion, the analytics controller can determine a response to treatment based on whether a change in metabolic activity level exhibited by the lesion between the first time point and the second time point satisfies one or more thresholds.

[0046] In some example embodiments, the analytics controller can also determine a change in tumor volume (e.g., total metabolic tumor volume (TMTV)) between the first time point and the second time point based at least on the first updated tumor mask and the second updated tumor mask. In some cases, a response to treatment for the disease can be determined based on the change in tumor volume between the first time point and the second time point. Alternatively and / or additionally, the analytics controller can determine a difference in change in metabolic activity and / or change in tumor volume between the first time point and the second time point across different lesions based at least on the first updated tumor mask and the second updated tumor mask. In some cases, the analytics controller can further determine a response to treatment for the disease based on the difference in change in metabolic activity and / or change in tumor volume between the first time point and the second time point across different lesions.

[0047] FIG. 1 depicts a system diagram illustrating an example of a machine learning-based medical imaging analysis system 100, in accordance with some example embodiments. Referring to FIG. 1, the machine learning-based medical imaging analysis system 100 can include an analytics controller 110, one or more imaging devices 120, and a client device 130. As shown in FIG. 1, the analytics controller 110, the one or more imaging devices 120, and the client device 130 can be communicatively coupled via a network 140. The one or more imaging devices 120 can include, for example, a computed tomography (CT) scanner 121 and a positron emission tomography (PET) scanner 123. The client device 130 can be a processor-based device including, for example, a smartphone, a tablet computer, a wearable, a virtual assistant, an Internet of Things (IoT) appliance, and the like. The network 140 can be a wired and / or wireless network including, for example, a wide area network (WAN), a local area network (LAN), a virtual local area network (VLAN), a public land mobile network (PLMN), the Internet, and the like.

[0048] In some example embodiments, the analysis controller 110 can perform longitudinal analysis of positron emission tomography (PET) scans and computed tomography (CT) scans generated by one or more imaging devices 120 (e.g., a computed tomography scanner 121, a positron emission tomography (PET) scanner 123, etc.). In some cases, the analysis controller 110 can apply a longitudinal segmentation model 113 that can be trained to update two or more single tumor masks generated from positron emission tomography (PET) scans and computed tomography (CT) scans from a single time point. To further illustrate, FIG. 2 depicts a schematic diagram showing an example of a process 200 for machine learning supported longitudinal analysis of positron emission tomography (PET) scans and computed tomography (CT) scans, in accordance with some example embodiments.

[0049] Referring to FIG. 2, the analysis controller 110 can receive, from the one or more imaging devices 120, a first positron emission tomography (PET) scan 210a and a first computed tomography (CT) scan 220a from a first time point. Further, the analysis controller 110 can also receive, from the one or more imaging devices 120, a second positron emission tomography (PET) scan 210b and a second computed tomography (CT) scan 220b from a second time point. In some cases, the analysis controller 110 can include a pre-processing controller 110. In the example shown in FIG. 2, the pre-processing controller 110 can pre-process the first positron emission tomography (PET) scan 210a and the first computed tomography (CT) scan 220a to generate a first tumor mask 230a. Further, the pre-processing controller 110 can also pre-process the second positron emission tomography (PET) scan 210b and the second computed tomography (CT) scan 220b to generate a second tumor mask 230b.

[0050] In some example embodiments, the pre-processing can include registration to align the first positron emission tomography (PET) scan 210a and the first computed tomography (CT) scan 220a with the second positron emission tomography (PET) scan 210b and the second computed tomography (CT) scan 220b. For example, in some cases, the pre-processing engine 111 can perform an affine and block matching registration based on the first computed tomography (CT) scan 220a from the first time point and the second computed tomography (CT) scan 220b from the second time point. As such, the first positron emission tomography (PET) scan 210a can be overlaid (or co-registered) with the first computed tomography (CT) scan 220a, while the second positron emission tomography (PET) scan 210b can be overlaid (or co-registered) with the second computed tomography (CT) scan 220b. Further, the first positron emission tomography (PET) scan 210a and the first computed tomography (CT) scan 220a from the first time point can be overlaid (or co-registered) with the second positron emission tomography (PET) scan 210b and the second computed tomography (CT) scan 220b. In doing so, the pre-processing engine 111 can map a first pixel in the first positron emission tomography (PET) scan 210a to a second pixel in the first computed tomography (CT) scan 220a, a third pixel in the second positron emission tomography (PET) scan 210b, and a fourth pixel in the second computed tomography (CT) scan 220b.

[0051] In some example embodiments, the pre-processing can further include segmenting the first positron emission tomography (PET) scan 210a, the first computed tomography (CT) scan 220a, the second positron emission tomography (PET) scan 210b, and the second computed tomography (CT) scan 220b into two or more regions. For example, in some cases, the pre-processing engine 111 can segment the first positron emission tomography (PET) scan 210a, the first computed tomography (CT) scan 220a, the second positron emission tomography (PET) scan 210b, and the second computed tomography (CT) scan 220b into two or more anatomical regions based on one or more anatomical landmarks. Examples of anatomical regions include a head and neck region, a thoracic region, and an abdominal and pelvic region.

[0052] In some example embodiments, the pre-processing can also include segmenting the first positron emission tomography (PET) scan 210a and the first computed tomography (CT) scan 220a to generate a first tumor mask 230a, and segmenting the second positron emission tomography (PET) scan 210b and the second computed tomography (CT) scan 220b to generate a second tumor mask 230b. For example, in some cases, the pre-processing engine 111 can apply the segmentation model 112 to segment the first positron emission tomography (PET) scan 210a and the first computed tomography (CT) scan 220a by identifying at least a first plurality of pixels corresponding to one or more lesions present in the first positron emission tomography (PET) scan 210a and the first computed tomography (CT) scan 220a. Similarly, the pre-processing engine 111 can apply the segmentation model 112 to segment the second positron emission tomography (PET) scan 210b and the second computed tomography (CT) scan 220b by identifying at least a second plurality of pixels corresponding to one or more lesions present in the second positron emission tomography (PET) scan 210b and the second computed tomography (CT) scan 220b. As described above, each pixel in the positron emission tomography (PET) scan can be associated with an intensity value corresponding to a level of metabolic activity (e.g., a standard uptake value (SUV)), while each corresponding pixel in the co-registered computed tomography (CT) scan can be associated with an intensity value corresponding to a tissue density or x-ray attenuation. Thus, in some cases, the segmentation model described above can determine whether a pixel is part of a lesion based on the level of metabolic activity and the tissue density (or x-ray attenuation) exhibited by the pixel and one or more neighboring pixels.

[0053] In some example embodiments, the segmentation model 112 can include one or more machine learning models trained to segment two-dimensional images and / or three-dimensional volumes. For example, in some cases, the segmentation model 112 can include one or more artificial neural networks, such as convolutional neural networks, vision transformers, and / or the like. In cases where the segmentation model 112 includes one or more vision transformers, the segmentation model 112 can be applied to identify individual image patches containing a lesion, after which a threshold is applied to the image patches to select pixels exhibiting a level of metabolic activity (e.g., a standardized uptake value (SUV)), tissue density, and / or X-ray attenuation that satisfies one or more thresholds. For example, a pixel can be identified as delineating a lesion if its standardized uptake value (SUV) exceeds a particular minimum value (e.g., 2.5 or 4), exceeds a particular minimum value relative to the standardized uptake value (SUV) of the liver (e.g., 1.5 times the standardized uptake value (SUV) of the liver plus 2 standard deviations from the standardized uptake value (SUV) of the liver), a percentage of the maximum standardized uptake value (SUV) of the identified tumor region, and / or the like. SUV max ) of the identified tumor region, and / or the like.

[0054] In some cases, the segmentation model 112 can segment by object classification. In those cases, the segmentation model 112 can include one or more machine learning models trained to perform object classification (e.g., a logistic regression model, a tree-based classifier, a fully connected neural network, etc.). For example, to segment the first positron emission tomography (PET) scan 210a and the first computed tomography (CT) scan 220a, the pre-processing engine 111 can first apply a threshold to identify one or more objects present in the first positron emission tomography (PET) scan 210a and the first computed tomography (CT) scan 220a. For example, in some cases, the pre-processing engine 111 can identify pixels depicting objects in the first positron emission tomography (PET) scan 210a and the first computed tomography (CT) scan 220a based at least on an intensity value of each pixel. As described above, the intensity value of each pixel in a computed tomography (CT) scan corresponds to tissue density or X-ray attenuation, while the intensity value of a pixel in a positron emission tomography (PET) scan corresponds to a level of metabolic activity. Thus, in some cases, the pre-processing controller 110 can identify objects present in the first positron emission tomography (PET) scan 210a and the first computed tomography (CT) scan 220a by applying a first threshold to the intensity value of each pixel in the first positron emission tomography (PET) scan 210a and / or a second threshold to the intensity value of each pixel in the first computed tomography (CT) scan 210b. In doing so, the pre-processing engine 111 can identify objects exhibiting a threshold level of metabolic activity, a threshold level of tissue density, and / or a threshold level of X-ray attenuation. In identifying objects present in the first positron emission tomography (PET) scan 210a and the first computed tomography (CT) scan 220a, the pre-processing engine 111 can apply the segmentation model 112 to classify each of the objects as a lesion or a non-lesion.

[0055] Referring again to FIG. 2, in some example embodiments, the analysis controller 110 can apply the longitudinal segmentation model 113 to update the first tumor mask 230a and the second tumor mask 230b. For example, in some cases, a first image patch can be extracted from the first positron emission tomography (PET) scan 210a and the first computed tomography (CT) scan 220a to include a first lesion associated with the first tumor mask 230a, while a second image patch can be extracted from the second positron emission tomography (PET) scan 210b and the second computed tomography (CT) scan 220b to include a second lesion associated with the second tumor mask 230b. In some cases, the longitudinal segmentation model 113 can ingest the first image patch and the second image patch in their entirety, rather than the first positron emission tomography (PET) scan 210a, the first computed tomography (CT) scan 220a, the second positron emission tomography (PET) scan 210b, and the second computed tomography (CT) scan 220b, in order to increase a signal-to-noise ratio (SNR) associated with pixels that depict the first lesion and the second lesion.

[0056] In some example embodiments, the longitudinal segmentation model 113 can update the first tumor mask 230a and the second tumor mask 230b based at least on the first image patch and the second image patch to generate a first updated tumor mask 240a and a second updated tumor mask 240b. For example, in some cases, the longitudinal segmentation model 113 can determine whether a pixel is part of a lesion based on a metabolic activity level and a tissue density (or x-ray attenuation) exhibited by the pixel and one or more neighboring pixels across the first time point and the second time point. In doing so, the longitudinal segmentation model 113 can update the first tumor mask 230a and / or the second tumor mask 230b, including by updating a label assigned to one or more pixels in the first tumor mask 230a and / or the second tumor mask 230b. For example, in some cases, a pixel previously classified as part of a lesion (e.g., by a segmentation model) is reclassified by the longitudinal segmentation model 113 as not part of a lesion, and a pixel previously classified as not part of a lesion (e.g., by a segmentation model) is reclassified by the longitudinal segmentation model 113 as part of a lesion.

[0057] Referring again to FIG. 2, the analysis controller 110 can include an evaluation engine 115 that determines a treatment response 180 based on at least the first updated tumor mask 240a and the second updated tumor mask 240b. For example, where a lesion present in the first updated tumor mask 240a is not present in the second updated tumor mask 240b and no new lesion is present in the second updated tumor mask 240b, the treatment response 180 can be a complete metabolic response (CMR). Where some, but not all, lesions present in the first updated tumor mask 240a are present in the second updated tumor mask 240b and no new lesion is present in the second updated tumor mask 240b, the treatment response 180 can be a partial metabolic response (PMR). Where the second updated tumor mask 240b includes one or more new lesions, the treatment response 180 can be a progressive metabolic disease (PMD). Even where no new lesion is present in the second updated tumor mask 240b, the treatment response 180 can still be a progressive metabolic disease (PMD) where a change in metabolic activity level exhibited by a lesion in the first updated tumor mask 240a and the same lesion in the second updated tumor mask 240b satisfies one or more threshold values.

[0058] Accordingly, in some instances, the assessment engine 115 can determine the treatment response 180 to be a responder based on at least the first updated tumor mask 240a and the second updated tumor mask 240b, where, for example, the assessment engine 115 detects a complete metabolic response (CMR) or a partial metabolic response. Alternatively, the assessment engine 115 can determine the treatment response 180 to be a non-responder based on at least the first updated tumor mask 240a and the second updated tumor mask 240b, where, for example, the assessment engine 115 detects a non-metabolic response (NMR) or a progressive metabolic disease (PMD). In some instances, the assessment engine 115 can determine the treatment response 180 to be a complete metabolic response (CMR), a partial metabolic response (PMR), a non-metabolic response (NMR), or a progressive metabolic disease (PMD) based on at least the first updated tumor mask 240a and the second updated tumor mask 240b. Alternatively, the assessment engine 115 can determine the treatment response 180 to be a complete metabolic response (CMR) or a non-complete metabolic response (non-CMR) based on at least the first updated tumor mask 240a and the second updated tumor mask 240b. Further, in some instances, in addition to or instead of the treatment response 180, the assessment engine 115 can determine a disease progression, an overall survival (OS), and / or a progression-free survival (PFS) based on at least the first updated tumor mask 240a and the second updated tumor mask 240b.

[0059] In some example embodiments, the assessment engine 115 can determine the treatment response 180 based at least on determining whether a second lesion associated with the second updated tumor mask 240b is a new lesion or the same lesion as a first lesion associated with the first updated tumor mask 240a. For example, the assessment engine 115 can determine a distance between the first lesion and the second lesion based at least on the first updated tumor mask 240a and the second updated tumor mask 240b. In some cases, the distance between the first lesion and the second lesion can be quantified by one or more of a maximum, a minimum, a mean, a median, and / or a mode of distances between a first plurality of pixels in the first updated tumor mask 240a and a second plurality of pixels in the second updated tumor mask 240b. In instances where the distance between the first lesion and the second lesion satisfies one or more threshold values (e.g., exceeds a minimum distance of 10 millimeters), the assessment engine 115 can determine that the second lesion is a new lesion. Alternatively, where the distance between the first lesion and the second lesion fails to satisfy one or more threshold values (e.g., fails to exceed a minimum distance of 10 millimeters), the assessment engine 115 can determine that the second lesion is the same lesion as the first lesion.

[0060] In some example embodiments, where the second lesion is identified as a new lesion, the assessment engine 115 can determine the treatment response 180 to be a progression of metabolic disease (PMD). Alternatively, where the second lesion is identified as the same lesion as the first lesion, the assessment engine 115 can determine the treatment response 180 further based on a change in metabolic activity exhibited by the lesion between the first time point and the second time point. For example, in some cases, the assessment engine 115 can determine a first metabolic activity level exhibited by the lesion at the first time point based at least on the first updated tumor mask 240a and the first positron emission tomography (PET) scan 210a. Further, the assessment engine 115 can determine a second metabolic activity level exhibited by the lesion at the second time point based at least on the second updated tumor mask 240b and the second positron emission tomography (PET) scan 210b. In some cases, the metabolic activity level exhibited by the lesion at each time point can correspond to a maximum, a minimum, a mean, a median, and / or a mode of metabolic activity levels. For example, the metabolic activity level exhibited by the lesion at each time point can correspond to a maximum, a minimum, a median, a mean, and / or a mode of standard uptake values (SUVs) associated with those pixels in the positron emission tomography (PET) scan from the time point that are identified as being part of the lesion by the corresponding updated tumor mask.

[0061] In some cases, the change in metabolic activity exhibited by the lesion between the first time point and the second time point can correspond to a difference between a first level of metabolic activity exhibited by the lesion at the first time point and a second level of metabolic activity exhibited by the second time point. For example, the change in metabolic activity exhibited by the lesion between the first time point and the second time point can correspond to a difference in maximum standardized uptake value (SUVmax) across the two time points. In some cases, the change in metabolic activity exhibited by the lesion between the first time point and the second time point can correspond to a difference in metabolic tumor volume (MTV) across the two time points. SUV max Alternatively and / or additionally, the level of metabolic activity exhibited by the lesion at each time point can correspond to a size of the lesion observed at each time point. Thus, in some cases, the change in metabolic activity exhibited by the lesion can also be determined based at least on a difference between a first size of the lesion at the first time point and a second size of the lesion at the second time point.

[0062] As described above, in some example embodiments in which the second lesion is identified as being the same lesion as the first lesion, the assessment engine 115 can further determine the treatment response 180 based on a change in metabolic activity exhibited by the lesion between the first time point and the second time point. For example, in some cases in which the change in metabolic activity exhibited by the lesion between the first time point and the second time point satisfies a first threshold, the assessment engine 115 can determine that the treatment response 180 is progressive metabolic disease (PMD). Alternatively, in cases in which the change in metabolic activity exhibited by the lesion between the first time point and the second time point satisfies a second threshold but does not satisfy the first threshold, the assessment engine 115 can determine that the treatment response 180 is no metabolic response (NMR). Further, in cases in which the change in metabolic activity exhibited by the lesion between the first time point and the second time point does not satisfy either the first threshold or the second threshold, the assessment engine 115 can determine that the treatment response 180 is partial metabolic response (PMR).

[0063] FIG. 3 depicts a flowchart illustrating an example of a process 300 for machine learning supported longitudinal analysis of positron emission tomography (PET) scans and computed tomography (CT) scans, according to some example embodiments. With reference to FIGS. 1-3, the process 300 can be performed by the analysis controller 110. The analysis controller 110 can perform the process 300 to determine, for example, the treatment response 180. As indicated by the performance metrics shown by FIGS. 6-7, 8A-8G, 9, and 10, the analysis controller 110 performing the process 300 is able to achieve highly accurate results that are consistent with those determined by expert radiologists. The process 300 can be performed to achieve highly accurate results independently (e.g., without expert radiologist intervention), which means that the process 300 provides a more efficient diagnostic solution that requires fewer resources than conventional techniques for analyzing positron emission tomography (PET) scans and computed tomography (CT) scans.

[0064] At 302, the analysis controller 110 can train the longitudinal segmentation model 113 to update a tumor mask generated based on a positron emission tomography (PET) scan and a computed tomography (CT) scan from a single time point. In some example embodiments, the analysis controller 110 can train the longitudinal segmentation model 113 to update two or more tumor masks based at least on a training set, each of the two or more tumor masks being generated from a positron emission tomography (PET) scan and a computed tomography (CT) scan from a single time point. In some cases, the training set can include one or more annotated training samples, each of the one or more annotated training samples including positron emission tomography (PET) scans and computed tomography (CT) scans from two or more different time points and a corresponding ground truth tumor mask. Further, in some cases, each annotated training sample can include regions (e.g., head and neck regions, chest regions, abdominal and pelvic regions, etc.) extracted from positron emission tomography (PET) scans and computed tomography (CT) scans from two or more different time points and a corresponding ground truth tumor mask. In such cases, each pixel in the ground truth tumor mask can be associated with a ground truth label having a first value (e.g., “1”) indicating that the pixel is part of a lesion or a second value (e.g., “0”) indicating that the pixel is not part of a lesion.

[0065] At 304, the analytics controller 110 can apply the trained longitudinal segmentation model 113 to update the first tumor mask from the first time point and the second tumor mask from the second time point. In some example embodiments, as shown in FIG. 2, the analytics controller 110 can apply the trained longitudinal segmentation model 113 to update the first tumor mask 230a from the first time point and the second tumor mask 230b from the second time point based on at least the first positron emission tomography (PET) scan 210a and the first computed tomography (CT) scan 220a from the first time point and the second positron emission tomography (PET) scan 210b and the second computed tomography (CT) scan 220b from the second time point. As will be described in greater detail below, the first tumor mask 230a can be generated based on the first positron emission tomography (PET) scan 210a and the first computed tomography (CT) scan 220a, while the second tumor mask 230b can be generated based on the second positron emission tomography (PET) scan 210b and the second computed tomography (CT) scan 220b. The longitudinal segmentation model 113 can utilize information from across multiple time points (e.g., metabolic activity levels, tissue density (or x-ray attenuation), etc.) in order to refine each of the first tumor mask 230a and the second tumor mask 230b in order to reduce false positives (where one or more pixels that are not part of a lesion are erroneously identified as being part of a lesion in the first tumor mask 230a and / or the second tumor mask 230b).

[0066] At 306, the analysis controller 110 can determine a response to the treatment for the disease based at least on the first updated tumor mask and the second updated tumor mask. In some example embodiments, FIG. 2 shows that the evaluation engine 115 can determine the treatment response 180 based at least on the first updated tumor mask 240a and the second updated tumor mask 240b. As will be described in greater detail below, the evaluation engine 115 can determine the treatment response 180 based at least on whether the second lesion associated with the second updated tumor mask 240b is a new lesion or the same lesion as the first lesion associated with the first updated tumor mask 240a. If the evaluation engine 115 determines that the second lesion associated with the second updated tumor mask 240b is not a new lesion, but the same lesion as the first lesion associated with the first updated tumor mask 240a, the evaluation engine 115 can determine the treatment response 180 based on changes in metabolic activity as determined based on the first updated tumor mask 240a, the first positron emission tomography (PET) scan 210a, the second updated tumor mask 240b, and the second positron emission tomography (PET) scan 210b.

[0067] In some cases, in addition to or instead of the treatment response 180, the evaluation engine 115 can determine disease progression, such as non-Hodgkin lymphoma (NHL) or another fluorodeoxyglucose avid (FDG avid) cancer observable in positron emission tomography (PET) scans, based at least on the first updated tumor mask 240a and the second updated tumor mask 240b. In some cases, the evaluation engine 115 can also determine changes in tumor volume (e.g., total metabolic tumor volume (TMTV), etc.), which in turn is indicative of the treatment response 180 and / or disease progression, based at least on the first updated tumor mask 240a and the second updated tumor mask 240b. Alternatively and / or additionally, in some cases, the evaluation engine 115 can determine differences in first changes in metabolic activity and / or second changes in tumor volume between the first time point and the second time point across different lesions based at least on the first updated tumor mask 240a and the second updated tumor mask 240b.

[0068] FIG. 4 depicts a flowchart showing another example of a process 400 for machine learning supported longitudinal analysis of positron emission tomography (PET) scans and computed tomography (CT) scans, according to some example embodiments. With reference to FIGS. 1-4, the process 400 can be performed by the analysis controller 110 and can implement, for example, the operation 304 of the process 300.

[0069] At 402, the analysis controller 110 can determine, based at least on the first positron emission tomography (PET) scan and the first computed tomography (CT) scan from the first time point, a first tumor mask corresponding to a first lesion present in the first positron emission tomography (PET) scan and the first computed tomography (CT) scan. For example, in the example illustrated in FIG. 2, the pre-processing engine 111 can determine, based at least on the first positron emission tomography (PET) scan 210a and the first computed tomography (CT) scan 220a, a first tumor mask 230a corresponding to the first lesion depicted in each of the first positron emission tomography (PET) scan 210a and the first computed tomography (CT) scan 220a. In some cases, the pre-processing engine 111 can determine the first tumor mask 230a by applying at least a segmentation model to the first positron emission tomography (PET) scan 210a and the first computed tomography (CT) scan 220a. Further, in some cases, the pre-processing controller 110 can align the first positron emission tomography (PET) scan 210a and the first computed tomography (CT) scan 220a to generate a first superimposed (or co-registered) image (e.g., a first PET-CT scan). Alternatively and / or additionally, the pre-processing controller 110 can extract, from the first positron emission tomography (PET) scan 210a aligned with the first computed tomography (CT) scan 220a, a first image patch comprising the first lesion for ingestion by the longitudinal segmentation model 113.

[0070] At 404, the analysis controller 110 can determine a second tumor mask corresponding to a second lesion present in the second positron emission tomography (PET) scan and the second computed tomography (CT) scan based at least on the second positron emission tomography (PET) scan and the second computed tomography (CT) scan from the second time point. In the example shown in FIG. 2, the preprocessing engine 111 can determine a second tumor mask 230b corresponding to a second lesion depicted in each of the second positron emission tomography (PET) scan 210b and the second computed tomography (CT) scan 220b based at least on the second positron emission tomography (PET) scan 210b and the second computed tomography (CT) scan 220b. As with the first tumor mask 230a, in some cases, the preprocessing engine 111 can determine the second tumor mask 230b by applying at least a segmentation model to the second positron emission tomography (PET) scan 210b and the second computed tomography (CT) scan 220b. Further, in some cases, the preprocessing controller 110 can align the second positron emission tomography (PET) scan 210b with the second computed tomography (CT) scan 220b to generate a second superimposed (or co-registered) image (e.g., a second PET-CT scan). Alternatively and / or additionally, the preprocessing controller 110 can extract a second image patch including the second lesion for ingestion by the longitudinal segmentation model 113 from the second positron emission tomography (PET) scan 210b aligned with the second computed tomography (CT) scan 220b.

[0071] At 406, the analysis controller 110 can apply the longitudinal segmentation model 113 to update each of the first and second tumor masks based on at least the first positron emission tomography (PET) scan, the first computed tomography (CT) scan, the second positron emission tomography (PET) scan, and the second computed tomography (CT) scan. For example, in the example illustrated in FIG. 2, the analysis controller 110 can apply the longitudinal segmentation model 113 to update each of the first and second tumor masks 230a, 230b based on at least the first image patches extracted from the first positron emission tomography (PET) scan 210a and the first computed tomography (CT) scan 220a and the second image patches extracted from the second positron emission tomography (PET) scan 210b and the second computed tomography (CT) scan 220b. As described above, in some cases, the longitudinal segmentation model 113 can determine whether a pixel is part of a lesion based on a level of metabolic activity and tissue density (or x-ray attenuation) exhibited by the pixel and one or more neighboring pixels across the first and second time points. Accordingly, the longitudinal segmentation model 113 can update the first and / or second tumor masks 230a, 230b, including by updating a label assigned to one or more pixels in the first and / or second tumor masks 230a, 230b. For example, in some cases, the update can include the longitudinal segmentation model 113 reclassifying a pixel that was previously classified (e.g., by a segmentation model) as being part of a lesion as not being part of a lesion. Alternatively and / or additionally, the update can include the longitudinal segmentation model 113 reclassifying a pixel that was previously classified (e.g., by a segmentation model) as not being part of a lesion as being part of a lesion.

[0072] FIG. 5 depicts a flow diagram illustrating another example of a process 500 for machine learning supported longitudinal analysis of positron emission tomography (PET) scans and computed tomography (CT) scans, according to some example embodiments. With reference to FIGS. 1-3 and 5, the process 500 can be performed by the analysis controller 110 and can implement, for example, the operation 306 of the process 300.

[0073] In some cases, process 500 can implement a Lugano classification process. Thus, in some cases, the treatment response 180 generated by the analytics controller 110 can conform to the Lugano classification (or tumor staging) paradigm, which includes identifying a patient’s positron emission tomography (PET) scan and computed tomography (CT) scan as depicting progressive metabolic disease (PMD) (or non-complete metabolic response (non-CMR)), no metabolic response (NMR), or partial metabolic response (PMR). The performance metrics shown in FIGS. 6-7, 8A-8G, 9, and 10 indicate that the analytics controller 110 performing process 500 is capable of achieving highly accurate results. That is, the Lugano classification produced by the analytics controller 110 performing process 500 is in agreement with determinations made by expert radiologists. Moreover, process 500 can be performed to achieve highly accurate results independently (e.g., without expert radiologist intervention) and thus is more convenient, more efficient, and requires fewer resources than conventional techniques for analyzing positron emission tomography (PET) scans and computed tomography (CT) scans. The speed and efficiency of process 500 can speed up many downstream clinical tasks, including treatment decisions. In some cases, the treatment response 180 generated by process 500 can be applied to generate immediate treatment decisions, thereby eliminating a key bottleneck in conventional clinical workflows.

[0074] As described in greater detail below, process 500 can be performed that leverages insights drawn from longitudinal analysis of positron emission tomography (PET) scans and computed tomography (CT) scans based on machine learning pairs to generate a more nuanced Lugano classification than conventional techniques, particularly those that only review data from a single time point. For example, process 500 can be performed to provide accurate and precise differentiation between progressive metabolic disease (PMD) (or non-complete metabolic response (non-CMR)), no metabolic response (NMR), and partial metabolic response (PMR), which can be more insightful than binary classification (e.g., responders and non-responders). The accuracy and precision of the treatment response 180 generated by process 500 means that process 500 also improves the accuracy and precision of downstream clinical tasks, such as identification of relapsed and refractory patients and treatment decisions that rely on the output of process 500.

[0075] At 502, the analysis controller 110 can determine whether the second lesion associated with the second updated tumor mask is a new lesion or the same lesion as the first lesion associated with the first updated tumor mask based at least on the first updated tumor mask from the first time point and the second updated tumor mask from the second time point. In some example embodiments, the assessment engine 115 can determine a distance between the first lesion associated with the first updated tumor mask 240a and the second lesion associated with the second updated tumor mask 240b based at least on the first updated tumor mask 240a from the first time point and the second updated tumor mask 240b from the second time point. In some cases, the distance between the first lesion and the second lesion can be quantified by one or more of a maximum, a minimum, a mean, a median, and / or a mode of distances between a first plurality of pixels in the first updated tumor mask 240a and a second plurality of pixels in the second updated tumor mask 240b. Whether the second lesion associated with the second updated tumor mask 240b is a new lesion or the same lesion as the first lesion associated with the first updated tumor mask 240a can be determined based on whether the distance between the first lesion and the second lesion satisfies one or more thresholds. For example, in some cases, if the distance (e.g., maximum distance, etc.) between the first lesion and the second lesion exceeds a threshold (e.g., 10 millimeters, etc.), the assessment engine 115 can determine that the second lesion is a new lesion. Alternatively, if the distance (e.g., maximum distance, etc.) between the first lesion and the second lesion does not exceed the threshold (e.g., 10 millimeters, etc.), the assessment engine 115 can determine that the second lesion is not a new lesion, but the same lesion as the first lesion associated with the first updated tumor mask 240a.

[0076] At 503-Y, the analysis controller 110 can identify the second lesion associated with the second updated tumor mask as a new lesion. As described above, in some cases, if the distance between the first lesion and the second lesion satisfies one or more thresholds (e.g., the maximum distance between the first lesion and the second lesion exceeds 10 millimeters, etc.), the assessment engine 115 can determine that the second lesion associated with the second updated tumor mask 240b is a new lesion, and not the same lesion as the first lesion associated with the first updated tumor mask 240a. Accordingly, at 504, the analysis controller 110 can determine that the response to the treatment for the disease is a disease metabolic progression (PMD). For example, in cases where the second lesion associated with the second updated tumor mask 240b is identified as a new lesion, the assessment engine 115 can determine that the treatment response 180 is a disease metabolic progression (PMD), or in some cases, a non-complete metabolic response (non-CMR).

[0077] Alternatively, at 503-N, the analysis controller 110 can identify the second lesion associated with the second updated tumor mask as not a new lesion, but the same lesion as the first lesion associated with the first updated tumor mask. For example, in some instances, if a distance between the first lesion and the second lesion fails to satisfy one or more thresholds (e.g., a maximum distance between the first lesion and the second lesion is no more than 10 millimeters, etc.), the evaluation engine 115 can determine that the second lesion associated with the second updated tumor mask 240b is not a new lesion, but the same lesion as the first lesion associated with the first updated tumor mask 240a. Accordingly, at 506, the analysis controller 110 can determine a change in metabolic activity level exhibited by the lesion between the first time point and the second time point based at least on the first updated tumor mask, the first positron emission tomography (PET) scan from the first time point, the second updated tumor mask, and the second positron emission tomography (PET) scan from the second time point. For example, in some instances, the evaluation engine 115 can determine the change in metabolic activity level exhibited by the lesion between the first time point and the second time point by at least determining a first metabolic activity level exhibited by the lesion at the first time point based at least on the first updated tumor mask 240a and the first positron emission tomography (PET) scan 210a. Further, in some instances, the evaluation engine 115 can determine the change in metabolic activity level exhibited by the lesion between the first time point and the second time point by at least determining a second metabolic activity level exhibited by the lesion at the second time point based at least on the second updated tumor mask 240b and the second positron emission tomography (PET) scan 210b. The change in metabolic activity level exhibited by the lesion between the first time point and the second time point can correspond to a difference between the first metabolic activity level and the second metabolic activity level. It will be appreciated that, in some instances, the metabolic activity level exhibited by the lesion at any one time point can be quantified by a maximum, minimum, mean, median, and / or mode (e.g., standardized uptake value (SUV)) of the metabolic activity level exhibited by each pixel in the positron emission tomography (PET) scan that is identified as being part of the lesion by the corresponding updated tumor mask. In some instances, the change in metabolic activity level exhibited by the lesion between the first time point and the second time point SUV may be determined based on the following equation (1) (1) wherein SUV max t 1 represents at the first time point t1 a first metabolic activity level (e.g., screening, pre-treatment, etc.), and SUV max t 1 indicates a metabolic activity level at a second time point t 2 a second metabolic activity level (e.g., follow-up, post-treatment, etc.).

[0078] At 508, the analysis controller 110 can determine whether the change in metabolic activity level exhibited by the lesion between the first time point and the second time point satisfies a first threshold. For example, in some instances, the assessment engine 115 can determine whether a change in metabolic activity level exhibited by the lesion between the first time point and the second time point, such as a difference between a first maximum standardized uptake value (SUVmax) SUV max t 1) from the first time point and a second maximum standardized uptake value (SUVmax) SUV max t2 2) from the second time point satisfies a first threshold (e.g., Δ SUV > 0.5).

[0079] At 509-Y, the analysis controller 110 can determine that the change in metabolic activity exhibited by the lesion between the first time point and the second time point satisfies the first threshold. For example, in some instances, the assessment engine 115 can determine that a change in metabolic activity level exhibited by the lesion between the first time point and the second time point, such as a difference between a first maximum standardized uptake value (SUVmax) SUV max t 1) from the first time point and a second maximum standardized uptake value (SUVmax) SUV max t 2) from the second time point satisfies a first threshold (e.g., Δ SUV > 0.5). Accordingly, the process 500 can continue at operation 504, where the analysis controller 110 determines that the response to treatment for the disease is a disease metabolic progression (PMD). For example, where the change in metabolic activity level between the first time point and the second time point is determined to satisfy the first threshold (e.g., Δ SUV > 0.5), the assessment engine 115 can determine that the treatment response 180 is a disease metabolic progression (PMD), or in some instances a non-complete metabolic response (non-CMR).

[0080] Alternatively, at 509-N, the analysis controller 110 can determine that the change in metabolic activity exhibited by the lesion between the first time point and the second time point fails to satisfy the first threshold. For example, in some cases, the assessment engine 115 can determine that the change in metabolic activity level between the first time point and the second time point fails to satisfy the first threshold (e.g., 0.5 > Δ SUV > 0.5). Thus, at 510, the analysis controller 110 can determine whether the change in metabolic activity exhibited by the lesion between the first time point and the second time point satisfies a second threshold. For example, in cases where the change in metabolic activity level between the first time point and the second time point fails to satisfy the first threshold (e.g., 0.5 > Δ SUV > 0.5), the assessment engine 115 can further determine whether the change in metabolic activity level between the first time point and the second time point satisfies the second threshold (e.g., 0.5 > Δ SUV > -0.25).

[0081] At 511-Y, the analysis controller 110 can determine that the change in metabolic activity exhibited by the lesion between the first time point and the second time point satisfies the second threshold. For example, in some cases, the assessment engine 115 can determine that the change in metabolic activity level exhibited by the lesion between the first time point and the second time point, such as the difference between the first maximum standardized uptake value (SUV1) from the first time point SUV max t 1) and the second maximum standardized uptake value (SUV2) from the second time point, satisfies the second threshold but not the first threshold (e.g., 0.5 > Δ SUV max t 2) satisfies the second threshold but not the first threshold (e.g., 0.5 > Δ SUV > -0.25). Thus, at 512, the analysis controller 110 can determine that the response to treatment for the disease is no metabolic response (NMR). For example, in cases where the change in metabolic activity level exhibited by the lesion between the first time point and the second time point satisfies the second threshold but not the first threshold (e.g., 0.5 > Δ SUV > -0.25), the assessment engine 115 can determine that the treatment response 180 is no metabolic response (NMR).

[0082] Alternatively, at 511-N, the analysis controller 110 can determine that the change in metabolic activity exhibited by the lesion between the first time point and the second time point fails to satisfy the second threshold. For example, in some cases, the assessment engine 115 can determine that the change in metabolic activity level exhibited by the lesion between the first time point and the second time point, such as the difference between the first maximum standardized uptake value (SUV1) from the first time point SUV maxt 1) the second maximum standardized uptake value (SUVmax) from the second time point (SUVmax2) minus the first maximum standardized uptake value (SUVmax) from the first time point (SUVmaxi) (SUVmax2- SUVmaxi). SUV max t 2) fails to satisfy a second threshold value (e.g., ΔSUVmax < -0.25) other than the first threshold value. Thus, at 514, the analysis controller 110 can determine that the response to treatment for the disease is a partial metabolic response (PMR). For example, where the assessment engine 115 determines that the change in metabolic activity level exhibited by the lesion between the first time point and the second time point does not satisfy either the first threshold value or the second threshold value (e.g., ΔSUVmax < -0.25), the assessment engine 115 can determine that the treatment response 180 is a partial metabolic response (PMR). SUV <-0.25). Thus, at 514, the analysis controller 110 can determine that the response to treatment for the disease is a partial metabolic response (PMR). For example, where the assessment engine 115 determines that the change in metabolic activity level exhibited by the lesion between the first time point and the second time point does not satisfy either the first threshold value or the second threshold value (e.g., ΔSUVmax < -0.25), the assessment engine 115 can determine that the treatment response 180 is a partial metabolic response (PMR). SUV <-0.25). Thus, at 514, the analysis controller 110 can determine that the response to treatment for the disease is a partial metabolic response (PMR). For example, where the assessment engine 115 determines that the change in metabolic activity level exhibited by the lesion between the first time point and the second time point does not satisfy either the first threshold value or the second threshold value (e.g., ΔSUVmax < -0.25), the assessment engine 115 can determine that the treatment response 180 is a partial metabolic response (PMR).

[0083] As described above, the analysis controller 110 can determine the treatment response 180 with better efficiency and a high level of accuracy (as measured by its agreement with expert radiologist-generated results). The performance of the analysis controller 110 can be assessed based on datasets from different populations and treatment regimens. The accuracy of the analysis controller 110 was assessed based on 2,266 evaluable follow-up visits from 678 unique patients. The treatment responses 180 determined by the analysis controller 110 showed strong agreement with expert radiologist assessments of the same dataset. In six of nine experiments, no statistically significant difference was observed between the performance of the analysis controller 110 and the final response performance from the expert radiologist panel versus inter-radiologist agreement (Table 1, FIG. 7(B)). In two other experiments, the difference was less than 5%. The same comparison to F1 scores showed that 7 of 9 comparisons had no significant difference, and the difference in the other two comparisons was less than 5% (p = 0.06). FIG. 8C The following Table 1 shows the accuracy of the analysis controller 110 on the test set and the comparison of the final response from the radiology experts to the inter-radiology expert agreement.

[0084] Table 1 includes three data sets: (1) GOYA (NCT01287741), (2) GO29365 (NCT02257567), and (3) GO29781 (NCT02500407). For GOYA, the first PET scan (baseline scan) was performed 1 to 35 days prior to treatment, and the second PET scan (end-of-treatment scan) was performed 6 to 8 weeks after the last dose. For early discontinuation, the second PET scan was performed 4 to 8 weeks after the last dose. For GO29365, the first PET scan (baseline scan) was performed prior to treatment, and the second PET scan (interim scan) was performed at 6 weeks during treatment, 3 months during treatment, and every 3 months after treatment continued. In addition, the second PET scan (follow-up scan) was performed every 3 months for 18 months, and then every 12 months. For GO29781, the first scan (baseline scan) was performed prior to treatment. The second PET scan was performed at 6 weeks during treatment (interim scan), at the completion of treatment (end-of-treatment scan), and every 6 months for 2 years (follow-up scan).

[0085] Table 1 Figure 6 depicts the accuracy of complete metabolic response (CMR) assessments made by the analysis controller 110 in (A) clinical trial data set GO29781 / NCT02500407 and (C) clinical trial data set GO29365 / NCT02257567, and the accuracy of objective response (OR) assessments made by the analysis controller 110 in (E) clinical trial data set GO29781 / NCT02500407 and (G) clinical trial data set GO29365 / NCT02257567, compared to the inter-reader agreement for complete metabolic response (CMR) assessments in (B) clinical trial data set GO29781 / NCT02500407 and (D) clinical trial data set GO29365 / NCT02257567, and the inter-reader agreement for objective response (OR) assessments in (F) clinical trial data set GO29781 / NCT02500407 and (H) clinical trial data set GO29365 / NCT02257567, respectively.

[0086] FIG. 7(A) depicts a comparison of the accuracy of the analysis controller 110 and expert radiologists, while FIG. 7(B) depicts error bars for the difference between the accuracy of the analysis controller 110 and inter-reader agreement compared to the final response. FIG. 7(C) shows the overall survival (OS) and progression-free survival (PFS) hazard ratios for patients within each dataset identified by the analysis controller 110 and expert radiologists as exhibiting a complete metabolic response (CMR) (e.g., non-complete metabolic response) at the end of treatment.

[0087] The agreement between the treatment response 180 determined by the analytic controller 110 and the treatment response determined by expert radiologists for complete metabolic response (CMR) predicted on clinical trial dataset GO29781 / NCT02500407, clinical trial dataset GO29365 / NCT02257567, and GOYA holdout set (NCT01287741) (Table 1, FIG. 6, FIG. 7(A) to FIG. 7(B), and FIG. 8A) was 0.87 (95% confidence interval (CI): 0.84, 0.89), 0.83 (95% CI: 0.79, 0.86), and 0.87 (95% CI: 0.82, 0.92), respectively. These comparisons showed no statistically significant difference compared to the inter-reader agreement, estimated at -0.02 (95% CI: -0.06, 0.00); 0.01 (95% CI: -0.05, 0.05); and -0.03 (95% CI: -0.09, 0.02), respectively. Objective response (OR) and four response category accuracy and radiologist inter-agreement are seen in Table 1. The performance of the analytic controller 110 predicted for objective response (OR) in the GOYA holdout set (NCT01287741) showed no significant difference from the inter-reader agreement (FIG. 6 to FIG. 7) -0.02 (95% CI: -0.06, 0.02), with less than 5% difference observed in the other test sets (-0.04, 95% CI: -0.09, -0.01 in GO29365 / NCT02257567; and -0.05, 95% CI: -0.07, -0.02 in GO29781 / NCT02500407). No statistically significant difference was found for the four response category accuracy of the analytic controller 110 on the GOYA holdout set (NCT01287741) (-0.04, 95% CI: -0.11, 0.01) and GO29365 / NCT02257567 (-0.03, 95% CI: -0.08, 0.03), while -0.07 (95% CI: -0.11, -0.03) was observed in GO29781 / NCT02500407 (FIG. 6 to FIG. 7 and FIG. 8B).

[0088] An expert radiologist reviewed the interim and final outputs of the analysis controller 110 (including tumor masks, metabolic activity levels (e.g., maximum uptake value of the hottest lesion (SUVmax)), indication of new lesions, and predicted metabolic response at screening and follow-up) to make an assessment of the 114 interim or treatment end scan. In 81% of visits (95% CI: 74-88), the predicted response did not need to be modified. The average time for the radiologist review was 2.02 minutes per visit (range 1 to 15 minutes). The agreement of the model with the expert radiologist was similar to the agreement of the radiologist with the expert radiologist committee final response. The agreement of the radiologist with the complete metabolic response (CMR) prediction of the analysis controller 110 (Figures 8D-8E) was 88% (95% CI: 82-93). The agreement with the objective response prediction was 91% (95% CI: 86-96), and the agreement for the 4 response categories prediction was 81% (95% CI: 74-88). For the objective response assessment, no statistically significant difference was observed between the agreement between the radiologist and the proposed method and the agreement between the radiologist and the expert committee final response (Figure 8E) (-0.01, 95% CI: -0.07, 0.05) and for the 4 response categories assessment (-0.05, 95% CI: -0.14, 0.4), while for the complete metabolic response (CMR) assessment accuracy a difference of -0.09 (95% CI: -0.16, -0.03) was observed.

[0089] Survival analysis was performed for the three test datasets. The predicted complete metabolic response (CMR) had strong prognostic significance for overall survival (OS) and the investigator assessed progression-free survival (PFS) (Table 2, Figure 7(C) and FIG. 8D). Overall survival (OS) hazard ratios (HR) for patients identified by the analytics controller 110 as exhibiting complete metabolic response (CMR) at the end of treatment were: 0.123 (95% CI: 0.055, 0.276) in the GOYA Retention Set (NCT01287741), 0.205 (95% CI: 0.098, 0.426) in the R / R DLBCL bendamustine + rituximab (BR) and pola + BR cohorts of GO29365 / NCT02257567, and 0.054 (95% CI: 0.01, 0.442) in the Phase 2 R / R FL Expansion Cohort of GO29781 / NCT02500407. Progression-free survival (PFR) hazard ratios (HR) and risk ratios for expert radiology panel responses are shown in Table 2 below.

[0090] Table 2 Patients predicted by the analytics controller 110 to exhibit complete metabolic response were at higher or equal risk of death at landmark survival times compared to patients identified by the expert radiology panel as not exhibiting complete metabolic response (CMR) (Figure 9). Two-year overall survival for patients predicted to not exhibit complete metabolic response (non-CMR) in the GOYA Retention Set (NCT01287741) (Figure 9(A)) was 57% (95% CI: 40-81) compared to 69% (95% CI: 55-86) for patients identified by the expert radiology panel as not exhibiting complete metabolic response (CMR) (Figure 9(D)). Eighteen-month overall survival (OS) for patients identified by the analytics controller 110 as exhibiting complete metabolic response in the GO29781 / NCT02500407 dataset (Figure 9(B)) and the GO29365 / NCT02257567 dataset (Figure 9(C)) were 60% (95% CI: 41-88) and 34% (95% CI: 19-60), respectively, compared to 78% (95% CI: 63-97, Figure 9(E)) and 34% (95% CI: 19-61, Figure 9(F)), respectively, for those identified by the expert radiology panel. Patients identified by the analytics controller 110 as exhibiting complete metabolic response (CMR) had higher or equal survival at the same landmark survival times identified by the expert radiology panel in the three test sets (Figure 7). FIG. 9Kaplan-Meier curves and progression-free survival (PFS) estimates for progression-free survival (PFS) are presented in FIG. 10. Expert radiologists and the analytics controller 110 assessed the Deauville score (DS) based on FDG uptake for all three datasets.

[0091] Overall survival log-rank test based on the CMR of the analytics controller 110 versus non-CMR and final IRC response for each test set. These analyses showed that there was no significant difference between the results of the analytics controller 110 and the final IRC response for all groups except one (GO29781, final IRC response) because the p-value was less than 0.05 (Table 3).

[0092] Table 3 Segmentation and lesion detection performance of the exemplary longitudinal segmentation model 113 was performed on follow-up FDG-PET / CT scans in the GOYA (NCT01287741) set. The longitudinal segmentation model 113 was developed to refine tumor segmentation in registered follow-up FDG-Pet / CT scans. First, region-specific VNet models were trained to refine tumor segmentation for different regions (e.g., abdominal / pelvic region, thoracic region, and head / neck region). The input to the VNet consisted of 4 types of image patches centered on the tumor centroid predicted at follow-up from the single timepoint tumor segmentation (screening PET, screening CT, registered follow-up PET, and CT). The thoracic and head / neck VNet input was 96 96 96 4, while the abdominal / pelvic VNet input size was 128 128 128 4. The predicted image patch tumor masks were then inserted into the whole body tumor mask. Longitudinal lesion segmentation was also tested for the pre-trained models Unet and Swin UNETR on the registered FDG-PET / CT follow-up scans. An ablation study was also performed to assess the lesion segmentation performance obtained on the follow-up scans by adding the longitudinal segmentation model, as well as the lesion segmentation performance obtained on the follow-up scans by adding the screening FDG-PET / CT scan information as input to the longitudinal segmentation model.

[0093] Longitudinal tumor segmentation model training was performed on 1919 samples from 740 follow-up scans registered to their baseline scans using the GOYA training set (including 384 non-tumor scans); 1196 image patches (397 negative examples) in the abdomen / pelvis region; 379 image patches (54 negative examples) in the chest region, and 344 image patches (85 negative examples) in the head / neck region. The GOYA test set included 179 registered follow-up scans. In the test set, the Dice score (DSC) of the abdomen / pelvis VNet was achieved as 0.802, while the Dice scores of the chest and head / neck VNet were achieved as 0.813 and 0.778, respectively. The overall scan-level total DICE score was 0.796. The final tumor mask had an average number of 0.11 false positive (FP) lesions, 0.39 false negative (FN) lesions, and 1.47 true positive (TP) lesions per scan (Table 4). Adding the longitudinal segmentation VNet step can reduce false positives (1.21 FP lesions per scan on average when using the tumor mask from the single-timepoint tumor segmentation model). Using both screening and follow-up FDG-PET / CT information in the longitudinal segmentation model can improve the sensitivity to lesions in follow-up scans compared to models that use only follow-up scans as input (0.65 FN lesions per scan on average when using only follow-up scans as input). VNet showed superior performance for longitudinal segmentation compared to UNet and Swin UNETR. Detailed assessment of different model performance is shown in Table 4.

[0094] Table 4 FIG. 11 depicts examples of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). FIG. 11A and FIG. 11B show true negative examples of correctly classified CMRs, where A) there is no tumor in the true standard TMTV annotation and B) there is no tumor in the tumor segmentation of the method. FIG. 11C and FIG. 11D show false positive examples of CMRs, where C) there is a metabolically inactive tumor according to the IRC annotation and D) a false positive lesion from the method. FIG. 11E and FIG. 11F show true positive examples of correctly classified non-CMRs with the same lesion detected by E) the IRC and F) the model. FIG. 11G and FIG. 11H show false negative examples by the PMD of the IRC, where G) a small lesion is mis-evaluated as a CMR by the model H).

[0095] In view of the above implementations of the subject matter, the present application discloses the following list of examples, wherein a feature of an individual example or a combination of more than one feature of the examples, and optionally in combination with one or more features of one or more other examples, is also an other example falling within the disclosure of the present application: Item 1 : A computer-implemented method, the method comprising: determining, based at least on a first positron emission tomography (PET) scan and a first computed tomography (CT) scan from a first time point, a first tumor mask corresponding to a first lesion present in the first PET scan and the first CT scan; determining, based at least on a second PET scan and a second CT scan from a second time point, a second tumor mask corresponding to a second lesion present in the second PET scan and the second CT scan; applying a longitudinal segmentation model to update each of the first tumor mask and the second tumor mask based at least on the first PET scan, the first CT scan, the second PET scan, and the second CT scan; and determining a response to a treatment for a disease based on at least one of the first updated tumor mask and the second updated tumor mask.

[0096] Item 2: The method of item 1, wherein the first tumor mask identifies a first plurality of pixels in each of the first PET scan and the first CT scan that depict the first lesion, and wherein the second tumor mask identifies a plurality of pixels from the second PET scan and the second CT scan that depict the second lesion.

[0097] Item 3: The method of any one of items 1-2, further comprising: registering the first CT scan, the first PET scan, the second CT scan, and the second PET scan to align the first CT scan and the first PET scan with the second CT scan and the second PET scan.

[0098] Item 4: The method of any one of items 1-3, further comprising: identifying the second lesion as a new lesion based at least on the first updated tumor mask and the second updated tumor mask; and in response to the second lesion being identified as the new lesion, determining the response to the treatment as a progression of disease (PMD).

[0099] Item 5: The method of item 4, further comprising: determining a distance between the first lesion and the second lesion based at least on the first updated tumor mask and the second updated tumor mask; identifying the second lesion as the new lesion based at least on the distance between the first lesion and the second lesion satisfying one or more thresholds; and identifying the second lesion as being the same lesion as the first lesion based at least on the distance between the first lesion and the second lesion failing to satisfy the one or more thresholds.

[0100] Item 6: The method of any one of items 4 to 5, further comprising: in response to determining that the first lesion and the second lesion are the same lesion, determining the response to the treatment for the disease based at least on a change in metabolic activity exhibited by the lesion between the first time point and the second time point.

[0101] Item 7: The method of item 6, wherein the change in metabolic activity between the first time point and the second time point is determined by: determining a first metabolic activity level exhibited by the lesion at the first time point based at least on the first updated tumor mask and the first PET scan, determining a second metabolic activity level exhibited by the lesion at the second time point based at least on the second updated tumor mask and the second PET scan, and determining the change in metabolic activity between the first time point and the second time point based at least on the first metabolic activity level and the second metabolic activity level.

[0102] Item 8: The method of item 7, wherein the response to the treatment is determined to be a disease metabolic progression (PMD) based at least on the change in metabolic activity between the first time point and the second time point satisfying a first threshold.

[0103] Item 9: The method of item 8, wherein the response to the treatment is determined to be a no metabolic response (NMR) based at least on the change in metabolic activity between the first time point and the second time point satisfying a second threshold but failing to satisfy the first threshold.

[0104] Item 10: The method of item 9, wherein the response to the treatment is determined to be a partial metabolic response (PMR) based at least on the change in metabolic activity between the first time point and the second time point failing to satisfy the first threshold and the second threshold.

[0105] Item 11: The method of any one of items 7-10, wherein the first metabolic activity level corresponds to a first standardized uptake value (SUV), and the second metabolic activity level corresponds to a second standardized uptake (SUV) value.

[0106] Item 12: The method of any one of items 7-10, wherein each of the first metabolic activity level and the second metabolic activity level corresponds to a maximum, minimum, median, mean, or mode of metabolic activity levels exhibited by the lesion at a corresponding time point.

[0107] Item 13: The method of any one of items 1-12, wherein the first CT scan and the first PET scan are performed prior to the treatment for the disease, and wherein the second CT scan and the second PET scan are performed after the treatment for the disease.

[0108] Item 14: The method of any one of items 1-13, further comprising determining a tumor volume change based at least on the first updated tumor mask and the second updated tumor mask, and determining the response to the treatment for the disease based at least on the tumor volume change.

[0109] Item 15: The method of any one of items 1-13, further comprising determining a difference in first metabolic activity change and / or second tumor volume change between different lesions between the first time point and the second time point based at least on the first updated tumor mask and the second updated tumor mask, and determining the response to the treatment based at least on the difference in metabolic activity change and / or tumor volume change exhibited by different lesions between the first time point and the second time point.

[0110] Item 16: The method of any one of items 1-15, further comprising determining progression of the disease based at least on the first updated tumor mask and the second updated tumor mask.

[0111] Item 17: The method of any one of items 1-16, wherein the first tumor mask is determined by applying a segmentation model to the first PET scan and the first CT scan, and wherein the second tumor mask is determined by applying the segmentation model to the second PET scan and the second CT scan.

[0112] Item 18: The method of any one of items 1-17, wherein the longitudinal segmentation model is an artificial neural network or a visual transformer.

[0113] Item 19: The method of any one of items 1-18, wherein each of the first CT scan, the first PET scan, the second CT scan, and the second PET scan is a three-dimensional volume comprising a plurality of two-dimensional image patches.

[0114] Item 20: The method of any one of items 1-19, wherein each pixel in the first PET scan and the second PET scan is associated with an intensity value corresponding to a level of metabolic activity.

[0115] Item 21: The method of any one of items 1-20, wherein each pixel in the first CT scan and the second CT scan is associated with an intensity value corresponding to a tissue density or X-ray attenuation.

[0116] Item 22: The method of any one of items 1-21, further comprising training the longitudinal segmentation model to update two or more tumor masks, each of the two or more tumor masks being generated from a positron emission tomography (PET) scan and a computed tomography (CT) scan from a single time point.

[0117] Item 23: The method of any one of items 1-22, wherein the response to the treatment for the disease is a complete metabolic response (CMR) or a non-complete metabolic response (non-CMR).

[0118] Item 24: The method of any one of items 1-22, wherein the response to the treatment for the disease is a responder or a non-responder.

[0119] Item 25: The method of any one of items 1-22, wherein the response to the treatment for the disease is a complete metabolic response (CMR), a partial metabolic response (PMR), a no metabolic response (NMR), or a progressive metabolic disease (PMD).

[0120] Item 26: The method of any one of items 1-25, further comprising extracting, from the first PET scan and the first CT scan, a first image patch comprising the first lesion associated with the first tumor mask; extracting, from the second PET scan and the first CT scan, a second image patch comprising the second lesion associated with the second tumor mask; and applying the longitudinal segmentation model to the first image patch and the second image patch to update each of the first tumor mask and the second tumor mask.

[0121] Item 27: A system comprising at least one data processor, and at least one memory storing instructions that, when executed by the at least one data processor, cause operations comprising the method of any of items 1 to 26.

[0122] Item 28: A non-transitory computer-readable medium storing instructions that, when executed by at least one data processor, cause operations comprising the method of any of items 1 to 26.

[0123] FIG. 12 depicts a block diagram illustrating an example of a computing system 1200 consistent with implementations of the current subject matter. With reference to FIGS. 1-10, the computing system 1200 can be used to implement the analytics controller 110, the one or more imaging devices 120, the client device 130, and / or any components therein.

[0124] As shown in FIG. 12, the computing system 1200 can include a processor 1210, a memory 1220, a storage device 1230, and an input / output device 1240. The processor 1210, the memory 1220, the storage device 1230, and the input / output device 1240 can be interconnected via a system bus 1250. The processor 1210 is capable of processing instructions for execution within the computing system 1200. Such execution instructions can implement, for example, one or more components of the analytics controller 110, the one or more imaging devices 120, and the client device 130. In some example embodiments, the processor 1210 can be a single-threaded processor. Alternatively, the processor 1210 can be a multi-threaded processor. The processor 1210 is capable of processing instructions stored on the memory 1220 and / or the storage device 1230 to display graphical information for a user interface provided via the input / output device 1240.

[0125] Memory 1220 is a computer readable medium or media used for storage of information within computing system 1200, such as volatile or non-volatile computer readable media. For example, memory 1220 can store data structures representing configuration object databases. Storage 1230 is capable of providing persistent storage for computing system 1200. Storage 1230 can be a solid state drive, floppy drive, hard drive, optical drive, or tape drive, or other suitable persistent storage devices. Input / output devices 1240 provide input / output operations for computing system 1200. In some example embodiments, input / output devices 1240 include a keyboard and / or pointing devices. In various implementations, input / output devices 1240 include a display unit for displaying a graphical user interface.

[0126] According to some example embodiments, input / output devices 1240 can provide input / output operations for a network device. For example, input / output devices 1240 can include an Ethernet port or other networking port to communicate with one or more wired and / or wireless networks (e.g., local area network (LAN), wide area network (WAN), the Internet).

[0127] In some example embodiments, computing system 1200 can be used to execute various interactive computer software applications that can be used to organize, analyze, and / or store data in various formats. Alternatively, computing system 1200 can be used to execute any type of software application. These applications can be used to perform various functions, such as scheduling functions (e.g., generating, managing, editing spreadsheet documents, word processing documents, and / or any other objects, etc.), calculating functions, communication functions, etc. The applications can include various additional functions or can be standalone computing products and / or functions. Upon activation within an application, a function can be used to generate a user interface provided via input / output devices 1240. The user interface can be generated by computing system 1200 and presented to a user (e.g., on a computer screen monitor, etc.).

[0128] One or more aspects or features of the subject matter described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs, field programmable gate arrays (FPGAs) computer hardware, firmware, software, and / or combinations thereof. These various aspects or features can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device. The programmable system or computing system can include clients and servers. Generally, programs include routines, applets, components, or modules, which perform particular tasks and / or implement particular abstract data types. The subject matter described herein can also be practiced via communications over a transmission medium to, and / or from, one or more remotely disposed computers and / or storage systems. As used herein, the term "transmission medium" encompasses any intangible medium that is capable of storing, encoding, or carrying instructions for execution by a programmable processor and / or instructions that encode the instructions for execution by a programmable processor and includes a wired medium and / or a wireless medium.

[0129] These computer programs (also known as programs, software, software applications apps, components, or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms "machine-readable medium" "computer-readable medium" refers to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal that a programmable processor can use to provide machine instructions and / or data. Machine-readable media can store such machine instructions non-transitorily (e.g., as would a non-transient solid-state memory or a magnetic hard drive or any other non-transient medium) or can otherwise store such machine instructions such as for example as transitory as in the case of a signal being transmitted over a network.

[0130] To provide for interaction with a user, one or more aspects or features of the subject matter described herein can be implemented on a computer having a display device, e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD), or a light emitting diode (LED) monitor for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. Other possible input devices include touch screens or other touch-sensitive devices such as single or multi-point resistive or capacitive trackpads, voice recognition hardware and software, optical scanners, optical pointers, digital image capture devices and associated interpretation software, and the like.

[0131] In the descriptions above and in the claims, there can be phrases such as "at least one of," or "one or more of," followed by a list of elements or features. The term "and / or" can also appear in a list of two or more elements or features. Unless otherwise implicitly or explicitly contradicted by the context in which it used, such a phrase is intended to mean any of the listed elements or features individually or any of the recited elements or features in combination with any of the other recited elements or features. For example, the phrases "at least one of A and B;" "one or more of A and B;" and "A and / or B" are each intended to mean "A alone," "B alone," or "A and B together." A similar interpretation is also intended for lists including three or more items. For example, the phrase "at least one of A, B, and C;" "one or more of A, B, and C;" and "A, B, and / or C" are each intended to mean "A alone," "B alone," "C alone," "A and B together," "A and C together," "B and C together," or "A and B and C together." Use of the term "based on," above and in claims is intended to mean "based at least in part on," such that an unrecited feature or element is also a permissible part of the basis for the claim.

[0132] Depending on the desired configuration, the subject matter described herein can be implemented in systems, apparatus, methods, and / or articles depending on the desired configuration. The embodiments set forth in the foregoing description do not represent all of the embodiments consistent with the subject matter described herein. Instead, they are merely some examples consistent with aspects related to the described subject matter. Although a few variations have been described in detail above, other modifications or additions are possible. In particular, further features and / or variations can be provided in addition to those set forth herein. For example, the implementations described above can be directed to various combinations and sub-combinations of the disclosed features and / or combinations and sub-combinations of several further features disclosed above. In addition, the logic flows depicted in the accompanying figures and / or described herein do not necessarily require the particular order shown, or sequential order, to achieve desirable results. Other implementations can be within the scope of the following claims.

Claims

1. A computer-implemented method, comprising: Based at least on a first positron emission tomography (PET) scan and a first computed tomography (CT) scan from a first time point, a first tumor mask corresponding to a first lesion present in the first PET scan and the first CT scan is determined; Based at least on a second PET scan and a second CT scan from a second time point, a second tumor mask corresponding to a second lesion present in the second PET scan and the second CT scan is determined; A longitudinal segmentation model is applied to update each of the first tumor mask and the second tumor mask based at least on the first PET scan, the first CT scan, the second PET scan, and the second CT scan; as well as Based on at least one of a first updated tumor mask and a second updated tumor mask, the response to treatment for the disease is determined.

2. The method of claim 1, wherein the first tumor mask identifies a first plurality of pixels depicting the first lesion in each of the first PET scan and the first CT scan, and wherein the second tumor mask identifies a plurality of pixels depicting the second lesion from the second PET scan and the second CT scan.

3. The method according to any one of claims 1 to 2, further comprising: The first CT scan, the first PET scan, the second CT scan, and the second PET scan are registered to align the first CT scan and the first PET scan with the second CT scan and the second PET scan.

4. The method according to any one of claims 1 to 3, further comprising: Based at least on the first updated tumor mask and the second updated tumor mask, the second lesion is identified as a new lesion; as well as In response to the second lesion being identified as the new lesion, the response to the treatment is defined as disease progression (PMD).

5. The method of claim 4, further comprising: The distance between the first lesion and the second lesion is determined based at least on the first updated tumor mask and the second updated tumor mask; The second lesion is identified as the new lesion at least based on the distance between the first lesion and the second lesion satisfying one or more thresholds; as well as The second lesion is identified as the same lesion as the first lesion, at least based on the fact that the distance between the first lesion and the second lesion fails to meet one or more thresholds.

6. The method according to any one of claims 4 to 5, further comprising: In response to determining that the first lesion and the second lesion are the same lesion, the response to the treatment for the disease is determined at least based on changes in metabolic activity exhibited by the lesion between the first time point and the second time point.

7. The method of claim 6, wherein the change in metabolic activity between the first time point and the second time point is determined by at least one of the following: The level of first metabolic activity exhibited by the lesion at the first time point is determined based at least on the first updated tumor mask and the first PET scan. The level of second metabolic activity exhibited by the lesion at the second time point is determined based at least on the second updated tumor mask and the second PET scan; and The change in metabolic activity between the first time point and the second time point is determined based at least on the first metabolic activity level and the second metabolic activity level.

8. The method of claim 7, wherein the response to the treatment is determined as metabolic progression of disease (PMD) based at least on the change in metabolic activity between the first time point and the second time point satisfying a first threshold.

9. The method of claim 8, wherein the response to the treatment is determined to be non-metabolic response (NMR) based at least on the change in metabolic activity between the first time point and the second time point satisfying a second threshold but failing to satisfy the first threshold.

10. The method of claim 9, wherein the response to the treatment is determined to be a partial metabolic response (PMR) based at least on the fact that the change in metabolic activity between the first time point and the second time point fails to meet the first threshold and the second threshold.

11. The method according to any one of claims 7 to 10, wherein the first metabolic activity level corresponds to a first normalized intake value (SUV), and the second metabolic activity level corresponds to a second normalized intake value (SUV).

12. The method according to any one of claims 7 to 10, wherein each of the first metabolic activity level and the second metabolic activity level corresponds to the maximum, minimum, median, mean, or mode of the metabolic activity level exhibited by the lesion at the corresponding time point.

13. The method according to any one of claims 1 to 12, wherein the first CT scan and the first PET scan are performed prior to the treatment for the disease, and wherein the second CT scan and the second PET scan are performed after the treatment for the disease.

14. The method according to any one of claims 1 to 13, further comprising: Tumor volume changes are determined based at least on the first updated tumor mask and the second updated tumor mask; as well as The response to the treatment for the disease is determined at least based on the changes in tumor volume.

15. The method according to any one of claims 1 to 13, further comprising: The differences in first metabolic activity changes and / or second tumor volume changes across different lesions between the first and second time points are determined based at least on the first updated tumor mask and the second updated tumor mask. as well as The response to the treatment is determined at least based on the differences in metabolic activity and / or tumor volume changes exhibited by different lesions between the first and second time points.

16. The method according to any one of claims 1 to 15, further comprising: The progression of the disease is determined based at least on the first updated tumor mask and the second updated tumor mask.

17. The method of any one of claims 1 to 16, wherein the first tumor mask is determined by applying a segmentation model to the first PET scan and the first CT scan, and wherein the second tumor mask is determined by applying the segmentation model to the second PET scan and the second CT scan.

18. The method according to any one of claims 1 to 17, wherein the longitudinal segmentation model is an artificial neural network or a visual transformer.

19. The method according to any one of claims 1 to 18, wherein each of the first CT scan, the first PET scan, the second CT scan, and the second PET scan is a three-dimensional volume comprising a plurality of two-dimensional image blocks.

20. The method according to any one of claims 1 to 19, wherein each pixel in the first PET scan and the second PET scan is associated with an intensity value corresponding to a level of metabolic activity.

21. The method according to any one of claims 1 to 20, wherein each pixel in the first CT scan and the second CT scan is associated with an intensity value corresponding to tissue density or X-ray attenuation.

22. The method according to any one of claims 1 to 21, further comprising: The longitudinal segmentation model is trained to update two or more tumor masks, each of which is generated from positron emission tomography (PET) scans and computed tomography (CT) scans from a single time point.

23. The method according to any one of claims 1 to 22, wherein the response to the treatment for the disease is a complete metabolic response (CMR) or an incomplete metabolic response (non-CMR).

24. The method according to any one of claims 1 to 22, wherein the response to the treatment for the disease is a responder or a non-responder.

25. The method according to any one of claims 1 to 22, wherein the response to the treatment for the disease is a complete metabolic response (CMR), a partial metabolic response (PMR), a non-metabolic response (NMR), or metabolic progression of disease (PMD).

26. The method according to any one of claims 1 to 25, further comprising: Extract a first image patch including the first lesion associated with the first tumor mask from the first PET scan and the first CT scan; Extract a second image patch, including the second lesion associated with the second tumor mask, from the second PET scan and the first CT scan; as well as The longitudinal segmentation model is applied to the first image patch and the second image patch to update each of the first tumor mask and the second tumor mask.

27. A system comprising: At least one data processor; as well as At least one memory storing instructions that, when executed by the at least one data processor, cause operation including the method according to any one of claims 1 to 26.

28. A non-transitory computer-readable medium storing instructions that, when executed by at least one data processor, cause operation comprising the method according to any one of claims 1 to 26.

Images