Application of artificial intelligence features in cancer patients

Abstract

In some embodiments, the present disclosure relates to a method. The method includes accessing digitized pathology imaging data from a cancer patient. The digitized pathology imaging data identifies cancer nuclei within one or more tumor regions. A plurality of features are extracted from the digitized pathology imaging data. The plurality of features are extracted using nuclei angularity measurements of respective ones of the cancer nuclei. The plurality of features are provided to a machine learning model that has been trained to generate a medical prediction corresponding to the cancer patient.

Description

APPLICATION OF ARTIFICIAL INTELLIGENCE FEATURES IN CANCER PATIENTSBACKGROUND

[0001] Bladder cancer is the third most common cancer among men in the United States. About 25% of bladder cancers are muscle invasive bladder cancer, or MIBC. MIBC is a specific type of bladder cancer, where cancer cells grow within a muscle wall (e.g., the detruser muscle) of a bladder. MIBC presents a high risk of spreading and therefore an immediate threat to life.BRIEF DESCRIPTION OF THE DRAWINGS

[0002] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate various example operations, apparatus, methods, and other example embodiments of various aspects discussed herein. It will be appreciated that the illustrated element boundaries (e.g., boxes, groups of boxes, or other shapes) in the figures represent one example of the boundaries. One of ordinary skill in the art will appreciate that, in some examples, one element can be designed as multiple elements or that multiple elements can be designed as one element. In some examples, an element shown as an internal component of another element may be implemented as an external component and vice versa. Furthermore, elements may not be drawn to scale in some embodiments.

[0003] Fig. 1 illustrates some embodiments of a block diagram of a cancer assessment apparatus configured to utilize extracted features from a digitized pathology image to make a medical prediction corresponding to a cancer patient.

[0004] Fig. 2 illustrates some embodiments of a block diagram of a bladder cancer assessment apparatus configured to utilize extracted features from a digitized pathology image to make a medical prediction corresponding to a cancer patient’s responsiveness to chemo-immunotherapy.

[0005] Fig. 3 illustrates a flow diagram showing some embodiments of a method of making a medical prediction corresponding to a cancer patient’s responsiveness to chemo-immunotherapy using features extracted from a digitized pathology image.

[0006] Figs. 4-6 illustrate images corresponding to some embodiments of a disclosed method and / or apparatus that makes a medical prediction of a patient’s responsiveness to chemo-immunotherapy using features extracted from a digitized pathology image.

[0007] Figs. 7A-7B illustrate diagrams showing some embodiments of a measure of a nuclei angularity for cancer nuclei.

[0008] Fig. 8 illustrates an exemplary graph showing a survival probability determined by a disclosed cancer assessment apparatus as a function of time.

[0009] Figs. 9A-9B illustrate exemplary performance metrics corresponding to a disclosed bladder cancer assessment apparatus.

[0010] Fig. 10 illustrates a block diagram of some additional embodiments of a bladder cancer assessment apparatus configured to utilize extracted features from a digitized pathology image to make a medical prediction corresponding to a cancer patient’s responsiveness to chemo-immunotherapy.

[0011] Fig. 11 illustrates some embodiments of a block diagram of an apparatus configured to make a medical prediction corresponding to a cancer patient’s responsiveness to chemo-immunotherapy using features extracted from a digitized pathology image.DETAILED DESCRIPTION

[0012] The description herein is made with reference to the drawings, wherein like reference numerals are generally utilized to refer to like elements throughout, and wherein the various structures are not necessarily drawn to scale. In the following description, for purposes of explanation, numerous specific details are set forth in order to facilitate understanding. It may be evident, however, to one of ordinary skill in the art, that one or more aspects described herein may be practiced with a lesser degree of these specific details. In other instances, known structures and devices are shown in block diagram form to facilitate understanding.

[0013] Muscle invasive bladder cancer (MIBC) is an aggressive form of cancer that has a poor overall prognosis. A standard of care medical treatment for MIBC typically includes a radical cystectomy, or surgical removal of the bladder. However, even after radical cystectomy, the 5-year survival rate of patients with MIBC is between approximately 50% and 70%. Because MIBC has a high likelihood of spreading, it may often be followed by chemotherapy and / or radiation to try to kill cancer cells that have spread to other parts of the body.

[0014] In recent years, neoadjuvant chemo-immunotherapy has emerged as an alternative to the traditional use of chemotherapy as an adjuvant treatment of cancer. Neoadjuvant chemo-immunotherapy treats a patient using a combination ofimmunotherapy drugs and chemotherapy drugs to prior to surgery. For example, a bladder cancer patient may be treated with a combination of nivolumab plus gemcitabine-cisplatin before a radical cystectomy. Nivolumab is an immune checkpoint inhibitor that works by keeping cancer cells from suppressing a patient’s immune system and therefore allows the patient’s immune system to attack cancer cells. It has been associated with a significant improvement in the overall survival of cancer patients. However, not all cancer patients respond to immunotherapy drugs such as nivolumab.

[0015] In some embodiments, the present disclosure relates to a method and apparatus configured to make a medical prediction as to whether or not a cancer patient will respond to neoadjuvant chemo-immunotherapy (e.g., nivolumab plus gemcitabine and cisplatin). The method accesses digitized imaging data from a cancer patient (e.g., a patient having MIBC). The digitized imaging data comprises segmented imaging data that identifies individual cancer nuclei within one or more tumor regions of one or more digitized pathology images. A plurality of features are extracted from the individual cancer nuclei using nuclei angularity measurements of the cancer nuclei. The plurality of features are provided to a machine learning model that has been pre-trained to generate a medical prediction as to whether or not the cancer patient will respond to neoadjuvant chemo-immunotherapy. The medical prediction can be utilized by health care professions to make a more informed decision relating to the treatment of a patient, thereby allowing for the patient to have an improved quality of life (e.g., a lower risk of death, to avoid negative side effects of drugs that are not likely to produce positive outcomes, and / or the like).

[0016] Fig. 1 illustrates some embodiments of a block diagram of a cancer assessment system 100 configured to utilize extracted features from a digitized pathology image to make a medical prediction corresponding to a cancer patient.

[0017] The cancer assessment system 100 comprises an electronic memory 101 configured to store digitized pathology imaging data 102 from one or more cancer patients. In some embodiments, the one or more cancer patients may have muscle invasive cancer (e.g., cancer that is within a muscle layer of tissue). For example, the one or more cancer patients may have muscle invasive bladder cancer (MIBC). In other embodiments, the one or more cancer patients may have prostate cancer, breast cancer, and / or the like.

[0018] The digitized pathology imaging data 102 may include one or more digitized pathology images (e.g., one or more digitized biopsy slides) obtained from a pathological tissue sample taken from a cancer patient. In some embodiments, the tissue sample may be taken from the cancer patient prior to a radical cystectomy procedure. In some embodiments, the digitized pathology imaging data 102 may include segmented imaging data that has been segmented to identify one or more regions of interest (ROI) 104. For example, the digitized pathology imaging data 102 may include segmented imaging data that has been segmented to identify one or more ROI 104 including one or more tumor regions and / or individual cancer nuclei within one or more tumor regions.

[0019] In some embodiments, the cancer assessment system 100 may include a segmentation tool 106 configured to access the digitized pathology imaging data 102. The segmentation tool 106 is further configured to identify one or more tumor regions 109 and / or cancer nuclei 111 within one or more tumor regions 109. In some embodiments, the segmentation tool 106 comprises a first segmentation stage 108 configured to segment the digitized pathology imaging data 102 to identify the one or more tumor regions 109. The segmentation tool 106 further comprises a second segmentation stage 110 that is downstream of the first segmentation stage 108 and that is configured to segment the one or more tumor regions 109 to identify the cancer nuclei 111.

[0020] A feature extraction tool 112 is configured to extract a plurality of features 114 from the cancer nuclei 111. In some embodiments, the plurality of features 1 14 may comprise angular texture features 116 and / or spatial arrangement features 1 18. In some embodiments, the plurality of features 114 may be extracted (e.g., generated) using nuclei angularity measurements taken on respective ones of the cancer nuclei 111. The nuclei angularity measurements describe an orientation of the cancer nuclei 111. It has been appreciated that features generated using nuclei angularity measurements have a high prognostic ability in predicting an outcome for treatment a cancer patient using chemo-immunotherapy (e.g., whether or not the cancer patient will respond to neoadjuvant chemo-immunotherapy).

[0021] The plurality of features 114 are provided to a machine learning model 120 that has been pre-trained to generate a medical prediction 122 corresponding to the cancer patient. In various embodiments, the medical prediction 122 may be indicative as to whether or not the cancer patient will respond to chemo-immunotherapy, anoutcome of the patient, an overall survival of the cancer patient, a risk of death, and / or the like. In some embodiments, the medical prediction 122 may relate to a cancer patient's outcome to being treated with chemo-immunotherapy that uses a combination of chemotherapy drugs that include gemcitabine, cisplatin, carboplatin, MVAC, CMV, Fluorouracil, mitomycin, and / or the like and immunotherapy drugs that include nivolumab, pembrolizumab, atezolizumab, avelumab, durvalumab, ipilimumab, and / or the like.

[0022] The medical prediction 122 can be utilized by health care professions to make a more informed decision relating to the treatment of a cancer patient, thereby allowing for the cancer patient to have an improved quality of life (e.g., a lower risk of death, to avoid negative side effects of drugs that are not likely to produce positive outcomes, and / or the like). For example, the medical prediction may help a doctor to identify chemo and immune resistant patients and determine alternate courses of treatment for such patients.

[0023] Fig. 2 illustrates some embodiments of a block diagram of a bladder cancer assessment system 200 configured to utilize extracted features from a digitized pathology image to make a medical prediction corresponding to a cancer patient’s outcome in response to chemo-immunotherapy.

[0024] The bladder cancer assessment system 200 comprises an electronic memory 101 configured to store digitized pathology imaging data 102 for a cancer patient 202 having bladder cancer (e.g., muscle invasive bladder cancer (MIBC)). In some embodiments, the digitized pathology imaging data 102 comprises one or more digitized pathology images (e.g., one or more digitized biopsy slides) obtained from a pathological tissue sample taken from the cancer patient 202. For example, the tissue sample may be taken as part of a pre-treatment transurethral resection of bladder tumor tissues for a cancer patient that is expected to undergo a radical cystectomy. In some embodiments, the digitized pathology imaging data 102 may comprise one or more whole slide images (WSIs), patches of a WSI, or the like. In some embodiments, the electronic memory 101 may comprise solid state memory, SRAM (static random-access memory), DRAM (dynamic random-access memory), and / or the like.

[0025] In some embodiments, the digitized pathology imaging data 102 may comprise one or more digitized pathology images generated by an image generation stage 203 that is configured to digitize a stained slide generated from a tissue sample taken from the cancer patient 202. In some embodiments, the image generation stage203 may comprise a tissue resection tool 204 (e.g., a scalpel, a needle, scissors, and / or the like) that is used to surgically excise tissue from the cancer patient 202. The tissue is provided to a tissue sectioning and staining tool 206, which is configured to slice the tissue into thin slices that are placed on one or more transparent slides (e.g., one or more glass slides). The tissue on the one or more transparent slides is then stained to generate one or more tissue slides. The one or more tissue slides are subsequently converted to the one or more digitized pathology images by a slide digitization tool 208 (e.g., comprising an CMOS image sensor, a CCD camera, and / or the like). In some embodiments, the one or more digitized pathology images may comprise a whole slide image of a H&E (Hematoxylin and Eosin) stained slide.

[0026] In some embodiments, a pre-processing stage 210 may be configured to operate upon the one or more digitized pathology images. The pre-processing stage 210 may include a normalization tool 212 configured to normalize image characteristics (e.g., color, brightness, contrast, etc.) so as to mitigate batch effects (e.g., differences between images obtained from different sites). The pre-processing stage 210 may further include a patch generator 214 configured to break a WSI into a plurality of nonoverlapping patches (e.g., respectively having a size of 512 x 512 pixels, 256 x 256 pixels, or other similar values) that cover the WSI. In some embodiments, the plurality of patches may be stored as part of the digitized pathology imaging data.

[0027] In some embodiments, a segmentation tool 106 is configured to access the one or more digitized pathology images. The segmentation tool 106 is further configured to identify one or more tumor regions 109 and / or cancer nuclei 111 within one or more tumor regions 109. In some embodiments, the segmentation tool 106 comprises a first segmentation stage 108 configured to segment the one or more digitized pathology images to identify the one or more tumor regions 109 and a second segmentation stage 110 configured to segment the one or more tumor regions 109 to identify the cancer nuclei 1 11. In some embodiments, the segmentation tool 106 is configured to generate one or more binary masks that comprise one or more automatically segmented tumor regions and / or automatically segmented cancer nuclei. In some such embodiments, the one or more binary masks comprise images having a value of “1” in image units (e.g., pixels, voxels, etc.) identified as being within a tumor region and / or cancer nuclei and having a value of “0” in image units outside of the tumor region and / or cancer nuclei.

[0028] In some embodiments, the first segmentation stage 108 and the second segmentation stage 110 may comprise machine learning models. For example, the first segmentation stage 108 may comprise a first machine learning model that has been trained to perform tumor segmentation. In some embodiments, the first segmentation stage 108 may comprise a U-net model with an Adam optimizer and a loss that is binary cross-entropy. The second segmentation stage 110 may comprise a second machine learning model that has been trained to perform both nuclei segmentation and classification (e.g., as cancerous or non-cancerous). In some embodiments, the second segmentation stage 110 may comprise a HoVer-Net model. In some embodiments, the first segmentation stage 108 and the second segmentation stage 110 may be run on one or more processors e.g., a central processing unit including one or more transistor devices configured to operate computer code to achieve a result, a microcontroller, a graphics processing unit (GPU), and / or the like).

[0029] In some embodiments, a post-processing stage 215 may be configured to operate upon the output of the segmentation tool 106. The post-processing stage 215 may be configured to fill holes and / or remove small objects from within the segmented imaging data. Such post-processing operations can improve a quality of the segmented images to improve results of the bladder cancer assessment system 200.

[0030] A feature extraction tool 112 is configured to extract a plurality of features 114 from the cancer nuclei 111. In some embodiments, the plurality of features 1 14 may comprise angular texture features and / or spatial arrangement features (e.g., nuclear morphology and architecture features). In some embodiments, the feature extraction tool 112 may be implemented as computer code run by a processing unit (e.g., a central processing unit including one or more transistor devices configured to operate computer code to achieve a result, a microcontroller, or the like).

[0031] In some embodiments, at least some of the plurality of features 114 may be generated by measuring a nuclei angularity 216 of individual ones of the cancer nuclei 111. For example, a co-occurrence matrix 218 may be formed using the using the measured nuclei angularity 216. The co-occurrence matrix 218 may be formed to have values that are the measured nuclei angularity 216. This is in contrast to a gray level co-occurrence matrix, which uses image intensity values to form the gray level cooccurrence matrix. It has been appreciated that the formation of the co-occurrence matrix 218 using the measured nuclei angularity 216 allows for the plurality of features 114 to have a high prognostic ability in predicting an outcome of a cancer patient 202being treated with chemo-immunotherapy. The angular texture features may be subsequently extracted from the co-occurrence matrix 218 (e.g., using second order statistical analysis). The spatial arrangement features may be extracted from cell graphs / subgraphs and / or cell clusters formed using the cancer nuclei 111. In some embodiments, the plurality of features 114 may be extracted from a plurality of cooccurrence matrices that are formed using the nuclei angularity measured within respective ones of a plurality of cell clusters.

[0032] The plurality of features 114 are provided to a machine learning model 120 that has been trained to generate a medical prediction 122 as to whether or not the patient will respond to chemo-immunotherapy (e.g., neoadjuvant chemoimmunotherapy). In some embodiments, the machine learning model 120 may be pretrained by finding weighting values that solve the technical problem of providing better identification of patients who will respond to chemo-immunotherapy. In some embodiments, the machine learning model 120 may be trained to generate a medical prediction 122 as to whether or not a patient that has MIBC will respond to neoadjuvant nivolumab with gemcitabine-cisplatin prior to undergoing a radical cystectomy. In some embodiments, the machine learning model 120 may be run on one or more processors (e.g., a central processing unit including one or more transistor devices configured to operate computer code to achieve a result, a microcontroller, or the like, a GPU, and / or the like).

[0033] In some embodiments, the machine learning model 120 may be configured to perform survival analysis to generate a risk score 220 corresponding to a risk of death for the patient. The machine learning model 120 is further configured to compare the risk score 220 to a threshold 222 to identify the cancer patient 202 as a responder to neoadjuvant chemo-immunotherapy or as a non-responder to neoadjuvant chemoimmunotherapy. In some embodiments, the threshold 222 may be approximately 50%.

[0034] In some embodiments, the machine learning model 120 may accurately predict pathological downstaging with an area under a receiver operating characteristic curve of 0.83. This is significantly better than other existing methods of prediction, thereby allowing for the medical prediction 122 to be used by health care professionals to make a more informed decision relating to the treatment of the cancer patient 202. For example, the improved prediction provided by the disclosed bladder cancer assessment system 200 can produce the technical effect of improving the administration of chemo-immunotherapy by increasing the accuracy of and decreasingthe time required to determine if a patient is likely or unlikely to respond. Treatments and resources, including expensive immunotherapy or chemotherapy agents may be more accurately tailored to patients with a likelihood of benefiting from treatments and resources, so that more appropriate treatment protocols may be employed, and expensive resources are not wasted.

[0035] In some embodiments, the machine learning model 120 may be trained to generate the medical prediction 122 to have a correlation between a responder and tumor regions having cancer nuclei with nuclei angularity having a high degree of uniformity. In such embodiments, the medical prediction 122 indicates that a patient having a high degree of uniformity in nuclear angularity within the tumor tissue will have a high chance of a positive outcome e.g., a response) to neoadjuvant chemoimmunotherapy, while a patient having a lower degree of uniformity in nuclear angularity within the tumor tissue will have a lower chance of a positive outcome (e.g., a response) to neoadjuvant chemo-immunotherapy.

[0036] Fig. 3 illustrates a flow diagram showing some embodiments of a method 300 of making a medical prediction of a patient’s responsiveness to chemo-immunotherapy using features extracted from a digitized pathology image.

[0037] While the disclosed method 300 is illustrated and described herein as a series of acts or events, it will be appreciated that the illustrated ordering of such acts or events are not to be interpreted in a limiting sense. For example, some acts may occur in different orders and / or concurrently with other acts or events apart from those illustrated and / or described herein. In addition, not all illustrated acts may be required to implement one or more aspects or embodiments of the description herein. Further, one or more of the acts depicted herein may be carried out in one or more separate acts and / or phases.

[0038] At act 302, digitized pathology imaging data from a cancer patient is accessed. In some embodiments, the cancer patient may have muscle invasive bladder cancer (MIBC). In some embodiments, the digitized pathology imaging data may comprise segmented data that identifies one or more tumor regions and / or one or more cancer nuclei within the one or more tumor regions. In other embodiments, the digitized pathology imaging data may comprise unsegmented imaging data (e.g., a WSI or patches from a WSI) that may be subsequently segmented (e.g., according to act 306).

[0039] At act 304, one or more pre-processing operations may be performed on the digitized pathology imaging data. In various embodiments, the one or more preprocessing operations may include normalization, patch / tile generation, and / or the like.

[0040] At act 306, the digitized pathology imaging data is segmented to generate segmented imaging data that identifies cancer nuclei. In some embodiments, the one or more digitized pathology images may be segmented according to acts 308-312.

[0041] At act 308, a first segmentation process is performed to identify one or more tumor regions within the digitized pathology imaging data.

[0042] At act 310, a second segmentation process is performed to generate segmented imaging data that identifies and / or classifies cancer nuclei within the one or more tumor regions.

[0043] At act 312, the segmented imaging data may be stored in electronic memory as part of the digitized pathology imaging data.

[0044] At act 314, a plurality of features are extracted from the digitized pathology imaging data using nuclei angularities of the cancer nuclei. The plurality of features may comprise features relating to texture and / or spatial arrangements of the cancer nuclei. In some embodiments, the plurality of features may be extracted according to acts 316-324.

[0045] At act 316, cell subgraphs are generated using the cancer nuclei.

[0046] At act 318, cell clusters are generated using the cell subgraphs.

[0047] At act 320, a nuclei angularity is measured for the cancer nuclei.

[0048] At act 322, a co-occurrence matrix is formed using the measured nuclei angularity. The co-occurrence matrix may be formed to have values that are the measured nuclei angularity.

[0049] At act 324, the plurality of features are extracted from the co-occurrence matrix, the cell subgraphs, and / or the cell clusters. The features extracted from the cooccurrence matrix may include angular texture features. The features extracted from the cell subgraphs and / or the cell clusters may include features describing a spatial arrangement of the cancer nuclei.

[0050] At act 326, a machine learning model (e.g., a trained machine learning model) is operated on the plurality of features to generate a medical prediction relating to an outcome of the patient being treated with chemo-immunotherapy (e.g., nivolumab plus gemcitabine-cisplatin).

[0051] At act 328, a treatment is administered to the cancer patient based on the medical prediction. In some embodiments, a medical prediction indicating a likelihood of a pathological completely response to chemo-immunotherapy may cause a treatment including chemo-immunotherapy to be administered to the cancer patient, while a medical prediction indicating that a pathological complete response to chemoimmunotherapy is unlikely may cause a different treatment to be administered to the cancer patient.

[0052] Therefore, the disclosed method 300 utilizes features extracted by measuring angularities of cancer nuclei within cancerous tissue (e.g., cancerous bladder tissue) to make a medical prediction as to whether or not a patient will respond to neoadjuvant chemo-immunotherapy (e.g., nivolumab plus gemcitabine-cisplatin).

[0053] It will be appreciated that the disclosed methods and / or block diagrams may be implemented as computer-executable instructions, in some embodiments. Thus, in one example, a computer-readable storage device (e.g., a non-transitory computer-readable medium) may store computer executable instructions that if executed by a machine (e.g., computer, processor) cause the machine to perform the disclosed methods and / or block diagrams. While executable instructions associated with the disclosed methods and / or block diagrams are described as being stored on a computer-readable storage device, it is to be appreciated that executable instructions associated with other example disclosed methods and / or block diagrams described or claimed herein may also be stored on a computer-readable storage device.

[0054] Figs. 4-6 illustrate images corresponding to some embodiments of a disclosed method and / or apparatus that makes a medical prediction of a patient’s responsiveness to chemo-immunotherapy using features extracted from a digitized pathology image.

[0055] Fig. 4 illustrates a plurality of exemplary images 400-404 corresponding to pre-processing operations (e.g., corresponding to act 304 of Fig. 3).

[0056] Image 400 illustrates an exemplary whole slide image comprising bladder tissue.

[0057] Image 402 illustrates an exemplary whole slide image that has been normalized. In some embodiments, normalization of the whole slide image reduces batch effects. For example, whole slide images coming from different sites (e.g., different hospitals, labs, etc.) will be formed by different imaging tools (e.g., scanners) that have different technical specifications and / or parameters. Normalization of thewhole slide image may reduce variations between different whole slide images coming from different sites, so as to improve downstream analysis of the whole slide images.

[0058] Image 404 illustrates a plurality of patches extracted from the normalized whole slide image. The plurality of patches may comprise a minimum percentage of tissue (e.g., 50% tissue). In various embodiments, the plurality of patches may have a size of 256 x 256 pixels, 512 x 512 pixels, or other similar values.

[0059] Fig. 5 illustrates a plurality of images 500, 506, and 510 corresponding to an exemplary segmentation process (e.g., corresponding to act 306 of Fig. 3).

[0060] Image 500 illustrates an exemplary WSI comprising a plurality of different tumor regions that have been stained in different shades 502 and 504.

[0061] Image 506 illustrates a ground truth of a segmented WSI. The ground truth has been segmented to identify tumor regions 508 within the WSI. The ground truth has been generated by an expert human pathologist.

[0062] Image 510 illustrates an automated segmentation of a WSI by a first disclosed segmentation stage. The automated segmentation has been segmented to identify tumor regions 512 within the WSI. Comparison of the ground truth with the automated segmentation shows a good agreement, thereby showing that the disclosed segmentation is able to accurately identify tumor regions within digitize pathology images.

[0063] Fig. 6 illustrates a plurality of exemplary images corresponding to a generation of a co-occurrence matrix (e.g., corresponding to acts 314-320 of Fig. 3).

[0064] Image 600 illustrates a plurality of segmented cancer nuclei 602. Subgraphs 604 are constructed using the cancer nuclei 602. In some embodiments, centroids of the cancer nuclei 602 may be used as nodes that are connected by edges to form the subgraphs 604.

[0065] Image 606 illustrates a plurality of cell clusters 608. The cell clusters 608 may comprise polygonal clusters formed to comprise proximally situated nuclei identified in the subgraphs 604.

[0066] Image 610 illustrates a co-occurrence matrix have different values that correspond to different nuclei angularities. In some embodiments, the co-occurrence matrix may be formed by measuring nuclei angularities for cancer nuclei within a cell cluster and subsequently using the nuclei angularities to generate the co-occurrence matrix. A plurality of features can be extracted from the co-occurrence matrix. In some embodiments, the plurality of features may comprise first and / or second orderstatistical measures extracted from the co-occurrence matrix. In such embodiments, second order statistical analysis is applied to the co-occurrence matrix to generate the features.

[0067] In some embodiments, the plurality of features may include nuclei level features, cluster level features, and / or global-level features. For example, the plurality of features may include angular texture features from a single nuclei and / or that are statistical measures taken over a cell cluster and or over an entire digitized image. In some embodiments, the plurality of features may be generated from a plurality of cooccurrence matrices respectively generated using nuclei angularity measurements taken from one of a plurality of cell clusters within digitized pathology imaging data (e.g., within a digitized pathology image).

[0068] Fig. 7A illustrates a diagram 700 showing some embodiments of a measure of a nuclei angularity for a cancer nuclei.

[0069] As shown in diagram 700, a cancer nuclei 702 may have an oblong shape with a major axis D extending along a direction of the longest length of the cancer nuclei 602. A nuclei angularity is an angle 0 measured between the major axis D and an x- axis. The angle 0 of the nuclei angularity may have a value that is between 0° and 180°, inclusive.

[0070] Fig. 7B illustrates a series of exemplary images showing a measure of a nuclei angularity for a cancer nuclei within a digitized pathology image.

[0071] Image 704 is a digitized pathology image comprising tumor regions 706 and cancer nuclei 708. Image 710 shows segmented cancer nuclei. Image 714 illustrates a magnified section 712 of image 710. As can be seen in image 714, the cancer nuclei each have a major axis 716. The major axis 716 of each cancer nuclei has a nuclei angularity that can be measured with respect to an x-axis.

[0072] Fig. 8 illustrates an exemplary graph 800 showing a survival probability determined by a disclosed cancer assessment apparatus as a function of time.

[0073] Graph 800 shows survival probabilities along the y-axis and time along the y- axis. The survival probabilities are shown by a first line 802 corresponding to patients that were identified by a disclosed bladder cancer assessment apparatus as falling within a high risk group (e.g., a group that is not responsive to chemo-immunotherapy) and a second line 804 corresponding to patients that were identified by a disclosed bladder cancer assessment apparatus as falling within a low risk group (e.g., a group that is responsive to chemo-immunotherapy). As can be seen in graph 800, thepatients that were identified as being within the high risk group (first line 802) have a lower survival probability than the patients that were identified as being within the low risk group (second line 804). The survival probabilities were found to have a p-value of 6.9e-05 and an average hazard ratio of 1 .82 (with a range of between 1 .35 and 2.45). Therefore, the disclosed bladder cancer assessment apparatus is able to accurately identify patients as being responsive or non-responsive to chemo-immunotherapy.

[0074] Fig. 9A illustrates an exemplary confusion matrix 900 corresponding to a disclosed bladder cancer assessment apparatus.

[0075] The confusion matrix 900 compares predicted classes of response or nonresponse to chemo-immunotherapy to true classes of low risk (e.g., corresponding to response) and high-risk (e.g., corresponding to non-response). The confusion matrix 900 shows good agreement between the corresponding classes. For example, of 23 true class responders, the disclosed bladder cancer assessment apparatus correctly identified 21 as falling within the low risk category (e.g., corresponding to a responder).

[0076] Fig. 9B illustrates an exemplary receiver operating curve (ROC) 902 corresponding to a disclosed bladder cancer assessment apparatus.

[0077] The ROC 902 shows a true positive rate (along the y-axis) vs. a false positive rate (along the x-axis). An area under the curve (AUC) may be approximately 0.83, thereby showing a good prognostic ability for the disclosed bladder cancer assessment apparatus.

[0078] Fig. 10 illustrates a block diagram of some additional embodiments of a bladder cancer assessment apparatus 1000 configured to utilize extracted features from a digitized pathology image to make a medical prediction of a patient’s responsiveness to chemo-immunotherapy.

[0079] The bladder cancer assessment apparatus 1000 comprises an electronic memory 101 configured to store digitized pathology imaging data 102 including a plurality of digitized pathology images from patients that have bladder cancer (e.g. MIBC) and that are expected to undergo radical cystectomy. In various embodiments, the digitized pathology imaging data 102 may comprise a plurality of digitized pathology images obtained by an image generation stage 203 and / or from an on-line database 1002 and / or archive containing digitized pathology images from patients generated at different sites (e.g., different hospitals, research laboratories, and / or the like). Prior to including digitized pathology images within the digitized pathology imaging data 102, the digitized pathology images may be subjected to a pre-processing stage 210.

[0080] The digitized pathology imaging data 102 may include a plurality of digitized pathology images within a training set 102t and a validation set 102v. The training set 102t comprises digitized pathology images from a first plurality of patients. The validation set 102v comprises digitized pathology images from a second plurality of patients. In some embodiments, the electronic memory 101 may also be configured to store ground truth segmentation data (e.g., segmentation results provided by an expert human pathologist).

[0081] The training set 102t may be used to train a downstream machine learning models. For example, the training set 102t may be used to train a downstream segmentation tool 106 to perform segmentations that identify cancer nuclei 111. The training set 102t may also be used to train a downstream machine learning model 120 to generate a medical prediction 122 as to whether or not a cancer patient will respond to chemo-immunotherapy (e.g., neoadjuvant chemo-immunotherapy).

[0082] The validation set 102v may be used to validate the results of the segmentation tool 106 to perform segmentations that identify cancer nuclei 111. The validation set 102v may also be used to validate the results of the machine learning model 120 to generate the medical prediction 122.

[0083] In some embodiments, machine learning model 120 may include a feature selection element 1004 configured to select a set of most prognostic features to generate the medical prediction 122. For example, the features extraction tool 1 12 may extract a first number of features (e.g., 408 features) from the digitized pathology imaging data 102 and then select a smaller second number of the features (e.g., 17 features) that are most prognostic of an outcome (e.g., that have a most significant impact in determining a risk of death). In some embodiments, the second number of features may be used to train and validate the machine learning model 120. In some embodiments, the machine learning model 120 may comprise a Cox regression model (e.g., a Cox proportional hazards model). In some embodiments, the Cox regression model may comprise a LASSO (least absolute shrinkage and selection operator) algorithm (e.g., a LASSO Cox regression model) that is configured to operate as the feature selection element 1004. In other embodiments, a minimum redundancy maximum relevance (mRMR) feature selection method may be employed to select the second number of features and to operate as the feature selection element 1004.

[0084] Fig. 11 illustrates some embodiments of a block diagram of an apparatus 1100 configured to make a medical prediction of a patient’s responsiveness to chemoimmunotherapy using features extracted from a digitized pathology image.

[0085] The apparatus 1100 comprises a bladder cancer assessment apparatus 1102. The bladder cancer assessment apparatus 1102 is coupled to an image generation stage 203, which is configured to generate digitized pathology imaging data 102 from one or more tissue samples collected from a cancer patient 202 (e.g., that has bladder cancer).

[0086] The bladder cancer assessment apparatus 1 102 comprises a processor 1106 and a memory 1104. The processor 1106 can, in various embodiments, comprise circuitry such as, but not limited to, one or more single-core or multi-core processors. The processor 1106 can include any combination of general-purpose processors and dedicated processors (e.g., graphics processors, application processors, etc.). The processor(s) 1106 can be coupled with and / or can comprise memory (e.g., memory 1104) or storage and can be configured to execute instructions stored in the memory 1104 or storage to enable various apparatus, applications, or operating systems to perform operations and / or methods discussed herein.

[0087] The memory 1104 can be configured to store the digitized pathology imaging data 102 (e.g., comprising one or more digitized pathology images). The digitized pathology imaging data 102 may comprise a plurality of pixels, each pixel having an associated intensity. In some additional embodiments, the digitized pathology imaging data 102 may be stored in the memory 1104 as one or more training sets of digitized images for training a classifier and / or one or more test sets (e.g., validation sets) of digitized images.

[0088] The bladder cancer assessment apparatus 1 102 also comprises an input / output (I / O) interface 1108 (e.g., associated with one or more I / O devices), a display 11 10, one or more circuits 1114, and an interface 1112 that connects the processor 1106, the memory 1104, the I / O interface 1108, and the one or more circuits 11 14. The I / O interface 1112 can be configured to transfer data between the memory 1104, the processor 1 106, the one or more circuits 1114, and external devices, for example, the image generation stage 203.

[0089] In some embodiments, the one or more circuits 1114 may comprise one or more of a segmentation circuit 1116, a feature extraction circuit 1118, and a machinelearning circuit 1120. In some embodiments, the machine learning circuit 1120 may operate according to machine learning algorithms stored in the memory 1104.

[0090] In some embodiments, the segmentation circuit 1116 is configured to segment the digitized pathology imaging data 102 to identify cancer nuclei. The feature extraction circuit 1118 is configured to extract a plurality of features 114 from the cancer nuclei. In some embodiments, the feature extraction circuit 1118 is configured to measure a nuclei angularity 216 of the cancer nuclei and to generate a co-occurrence matrix 218 using the nuclei angularity 216. The plurality of features 114 may be extracted using the co-occurrence matrix 218. The machine learning circuit 1120 is configured to utilize the plurality of features 114 to generate a medical prediction 122 of a cancer patient 202. The display 1110 is configured to output or display the medical prediction of the bladder cancer assessment apparatus 1 102.Example use case:

[0091] Background: Background: BLASST-1 is a multi-center phase II trial evaluating neoadjuvant nivolumab (N) with gemcitabine-cisplatin (GC) for patients (pts) with MIBC undergoing radical cystectomy (RC) (NCT03294304). 41 patients with MIBC (cT2-T4a, N<1 , M0) were enrolled between Feb 2018 and June 2019; (cT2N0 90%, cT3N0 7%, cT4N1 3%). Patients received cisplatin (C) (70mg / m2) IV on D1 , gemcitabine (G) (1000mg / m2) on D1 , D8, and nivolumab (N) (360 mg) IV on D8 every 21 days for 4 cycles followed by RC within 8 wks. The primary endpoint was pathologic downstaging (PaR; <pT1 N0). Safety, Relapse-free survival (RFS), Progression-free survival (PFS) and biomarker analyses were secondary endpoints. PaR rate was 65.8%, the pCR (<pT is NO) rate was 49% and there were no safety concerns or delays to RC. Morphometric characteristics of the cell nucleus can be used to assess bladder cancer grading and gain insights into cellular functionalities. In this study, we sought to evaluate the ability of the Al model to identify non-responders to neoadjuvant chemoimmunotherapy. in the BLASST-1 cohort based on computerized image features of nuclear morphology and architecture on pre-treatment transurethral resection of bladder tumor (TURBT) tissues.

[0092] Methods: Of the 41 pts, we had H&E images available for 34 pts, of which 23 had PaR and 11 did not have pathological response (PaR) and these were classified as responder (R) and non-responder (NR) groups. A machine learning model (U-net) was developed and invoked for tumor segmentation on the H&E images from theBLASST-1 cohort. A second machine learning model (HoVer-Net) was employed to segment and classify individual nuclei. A total of 408 features relating to the textural and spatial arrangement of individual cancer nuclei were extracted. The 17 most significant features, identified through the least absolute shrinkage and selection operator, were used to train a Cox regression model to predict the risk of death using 361 MIBC pts from the Cancer Genome Atlas (TCGA). This Cox model was then applied to assign a risk score to pts in BLASST-1 , using a threshold learned from TCGA pts, the individual pts in BLASST-1 were assigned as either low-risk or high-risk.

[0093] Results: The top identified prognostic features described the textural appearance of individual nuclei with more texture. This model accurately predicted PaR in BLASST-1 , with an area under a receiver operating characteristic curve of 0.83. Overall, the pts with the same nuclear angle direction within the tumor tissue had a high chance of response to the neoadjuvant chemo-IO combination in the BLASST-1 trial.

[0094] Conclusion: A computerized artificial intelligence (Al) model relying on nuclear morphologic and architecture features demonstrated prognostic capability in MIBC within the TCGA dataset and predictive capability for the PaR in the BLASST-1 trial. These findings support further validation studies.

[0095] Therefore, the present disclosure relates to a method and associated apparatus that makes a medical prediction of a patient’s responsiveness to chemoimmunotherapy using features extracted from a digitized pathology image.

[0096] In some embodiments, the present disclosure relates to a method. The method includes accessing digitized pathology imaging data from a cancer patient, wherein the digitized pathology imaging data identifies cancer nuclei within one or more tumor regions; extracting a plurality of features from the digitized pathology imaging data, the plurality of features are extracted using nuclei angularity measurements of respective ones of the cancer nuclei; and providing the plurality of features to a machine learning model that has been trained to generate a medical prediction corresponding to the cancer patient.

[0097] In other embodiments, the present disclosure relates to a non-transitory computer-readable medium storing computer-executable instructions that, when executed, cause a processor to perform operations, including accessing digitized pathology imaging data from a patient having muscle invasive bladder cancer (MIBC), the digitized pathology imaging data having been segmented to identify cancer nucleiwithin one or more tumor regions; measuring a nuclei angularity of the cancer nuclei; generating a co-occurrence matrix using the nuclei angularity; extracting a plurality of features using the co-occurrence matrix; and providing the plurality of features to a machine learning model that is trained to generate a medical prediction relating to an outcome of the patient being treated with neoadjuvant chemo-immunotherapy.

[0098] In yet other embodiments, the present disclosure relates to an apparatus. The apparatus includes a memory configured to store digitized pathology imaging data from a patient having cancer, the digitized pathology imaging data having been segmented to identify cancer nuclei within one or more tumor regions; a feature extraction tool configured to: measure a nuclei angularity of the cancer nuclei; generate a co-occurrence matrix using the nuclei angularity; extract a plurality of features using the co-occurrence matrix; and a machine learning model that is pre-trained to utilize the plurality of features to generate a medical prediction relating to an outcome of the patient being treated with neoadjuvant chemo-immunotherapy.

[0099] It will be appreciated that the disclosed methods and / or block diagrams may be implemented as computer-executable instructions, in some embodiments. Thus, in one example, a computer-readable storage device (e.g., a non-transitory computer- readable medium) may store computer executable instructions that if executed by a machine (e.g., computer, processor) cause the machine to perform the disclosed methods and / or block diagrams. While executable instructions associated with the disclosed methods and / or block diagrams are described as being stored on a computer- readable storage device, it is to be appreciated that executable instructions associated with other example disclosed methods and / or block diagrams described or claimed herein may also be stored on a computer-readable storage device.[000100] Examples herein can include subject matter such as an apparatus, including a digital whole slide scanner, a CT system, an MRI system, a personalized medicine system, a CADx system, a processor, a system, circuitry, a method, means for performing acts, steps, or blocks of the method, at least one machine-readable medium including executable instructions that, when performed by a machine (e.g., a processor with memory, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like) cause the machine to perform acts of the method or of an apparatus or system, according to embodiments and examples described.[000101] References to “one embodiment”, “an embodiment”, “one example”, and “an example” indicate that the embodiment(s) or example(s) so described may include aparticular feature, structure, characteristic, property, element, or limitation, but that not every embodiment or example necessarily includes that particular feature, structure, characteristic, property, element or limitation. Furthermore, repeated use of the phrase “in one embodiment” does not necessarily refer to the same embodiment, though it may.[000102] “Computer-readable storage device”, as used herein, refers to a device that stores instructions or data. “Computer-readable storage device” does not refer to propagated signals. A computer-readable storage device may take forms, including, but not limited to, non-volatile media, and volatile media. Non-volatile media may include, for example, optical disks, magnetic disks, tapes, and other media. Volatile media may include, for example, semiconductor memories, dynamic memory, and other media. Common forms of a computer-readable storage device may include, but are not limited to, a floppy disk, a flexible disk, a hard disk, a magnetic tape, other magnetic medium, an application specific integrated circuit (ASIC), a compact disk (CD), other optical medium, a random access memory (RAM), a read only memory (ROM), a memory chip or card, a memory stick, and other media from which a computer, a processor or other electronic device can read.[000103] “Circuit”, as used herein, includes but is not limited to hardware, firmware, software in execution on a machine, or combinations of each to perform a function(s) or an action(s), or to cause a function or action from another logic, method, or system. A circuit may include a software controlled microprocessor, a discrete logic (e.g., ASIC), an analog circuit, a digital circuit, a programmed logic device, a memory device containing instructions, and other physical devices. A circuit may include one or more gates, combinations of gates, or other circuit components. Where multiple logical circuits are described, it may be possible to incorporate the multiple logical circuits into one physical circuit. Similarly, where a single logical circuit is described, it may be possible to distribute that single logical circuit between multiple physical circuits. [000104] To the extent that the term “includes” or “including” is employed in the detailed description or the claims, it is intended to be inclusive in a manner similar to the term “comprising” as that term is interpreted when employed as a transitional word in a claim.[000105] Throughout this specification and the claims that follow, unless the context requires otherwise, the words 'comprise' and 'include' and variations such as 'comprising' and 'including' will be understood to be terms of inclusion and notexclusion. For example, when such terms are used to refer to a stated integer or group of integers, such terms do not imply the exclusion of any other integer or group of integers.[000106] To the extent that the term “or” is employed in the detailed description or claims (e.g., A or B) it is intended to mean “A or B or both”. When the applicants intend to indicate “only A or B but not both” then the term “only A or B but not both” will be employed. Thus, use of the term “or” herein is the inclusive, and not the exclusive use. See, Bryan A. Garner, A Dictionary of Modern Legal Usage 624 (2d. Ed. 1995).[000107] While example systems, methods, and other embodiments have been illustrated by describing examples, and while the examples have been described in considerable detail, it is not the intention of the applicants to restrict or in any way limit the scope of the appended claims to such detail. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the systems, methods, and other embodiments described herein. Therefore, the invention is not limited to the specific details, the representative apparatus, and illustrative examples shown and described. Thus, this application is intended to embrace alterations, modifications, and variations that fall within the scope of the appended claims.

Claims

What is claimed is:1 . A method, comprising: accessing digitized pathology imaging data from a cancer patient, wherein the digitized pathology imaging data identifies cancer nuclei within one or more tumor regions; extracting a plurality of features from the digitized pathology imaging data, wherein the plurality of features are extracted using nuclei angularity measurements of respective ones of the cancer nuclei; and providing the plurality of features to a machine learning model that has been trained to generate a medical prediction corresponding to the cancer patient.

2. The method of claim 1 , wherein the machine learning model has been trained to generate the medical prediction relating to an outcome of the cancer patient being treated with neoadjuvant chemo-immunotherapy.

3. The method of claim 1 , wherein the cancer patient has muscle invasive bladder cancer (MIBC) and the digitized pathology imaging data is from tissue taken from the cancer patient prior to the cancer patient undergoing a radical cystectomy.

4. The method of claim 1 , providing the plurality of features to the machine learning model, wherein the machine learning model is configured to generate a risk score corresponding to a risk of death for the cancer patient; and comparing the risk score to a threshold to identify the cancer patient as a responder to neoadjuvant chemo-immunotherapy or as a non-responder to the neoadjuvant chemo-immunotherapy.

5. The method of claim 1 , wherein the plurality of features include angular texture features extracted from a co-occurrence matrix generated from the nuclei angularity measurements.

6. The method of claim 5, wherein the plurality of features further include features relating to spatial arrangements of the cancer nuclei.

7. The method of claim 1 , further comprising: generating a co-occurrence matrix using the nuclei angularity measurements; and extracting the plurality of features using the co-occurrence matrix.

8. The method of claim 1 , wherein the nuclei angularity measurements describe a difference between a major axis extending along a direction of a longest length of one of the cancer nuclei and an x-axis.

9. The method of claim 1 , generating cell subgraphs and cell clusters from the cancer nuclei; measuring a nuclei angularity of the cancer nuclei within the cell clusters; and generating a plurality of co-occurrence matrices using the nuclei angularity measurements taken within respective ones of the cell clusters.

10. The method of claim 1 , wherein the medical prediction indicates a correlation between a responder and a high degree of nuclei angularity within the nuclei angularity measurements.

11. A non-transitory computer-readable medium storing computer-executable instructions that, when executed, cause a processor to perform operations, comprising: accessing digitized pathology imaging data from a patient having muscle invasive bladder cancer (MIBC), wherein the digitized pathology imaging data has been segmented to identify cancer nuclei within one or more tumor regions; measuring a nuclei angularity of the cancer nuclei; generating a co-occurrence matrix using the nuclei angularity; extracting a plurality of features using the co-occurrence matrix; and providing the plurality of features to a machine learning model that is trained to generate a medical prediction relating to an outcome of the patient being treated with neoadjuvant chemo-immunotherapy.

12. The non-transitory computer-readable medium of claim 1 1 , wherein the plurality of features include angular texture features extracted from the co-occurrence matrix.

13. The non-transitory computer-readable medium of claim 12, wherein the plurality of features further include features relating to spatial arrangements of the cancer nuclei.

14. The non-transitory computer-readable medium of claim 11 , wherein the machine learning model is trained to generate a risk score corresponding to a risk of death; and wherein the medical prediction is generated by comparing the risk score to a threshold.

15. The non-transitory computer-readable medium of claim 1 1 , wherein the neoadjuvant chemo-immunotherapy is nivolumab plus gemcitabine and cisplatin.

16. The non-transitory computer-readable medium of claim 1 1 , wherein the digitized pathology imaging data is of tissue taken from the patient that is subsequently expected to undergo a cystectomy.

17. An apparatus, comprising: a memory configured to store digitized pathology imaging data from a patient having cancer, wherein the digitized pathology imaging data has been segmented to identify cancer nuclei within one or more tumor regions; a feature extraction tool configured to: measure a nuclei angularity of the cancer nuclei; generate a co-occurrence matrix using the nuclei angularity; extract a plurality of features using the co-occurrence matrix; and a machine learning model that is pre-trained to utilize the plurality of features to generate a medical prediction relating to an outcome of the patient being treated with neoadjuvant chemo-immunotherapy.

18. The apparatus of claim 17, further comprising: a segmentation tool, comprising: a first segmentation stage configured to segment the digitized pathology imaging data to identify the one or more tumor regions; and a second segmentation stage configured to segment the digitized pathology imaging data to identify the cancer nuclei.

19. The apparatus of claim 18, wherein the first segmentation stage comprises a U-net model; and wherein the second segmentation stage comprises a HoVer-net model.

20. The apparatus of claim 17, wherein the machine learning model is a Cox regression model.

Images