Machine learning identification, classification and quantification of three-level lymphatic structures

Abstract

A method (400) is disclosed that includes receiving an input histological image (110), processing the input histological image using a cell classification model (550) to generate one or more lymphocyte density maps (125) within the input histological image, and performing morphological image processing (130) on the one or more lymphocyte density maps to identify one or more TLS regions (135) within the input histological image. Each TLS region is represented by a corresponding lymphocyte cluster. For each respective TLS region of the one or more TLS regions identified in the input histological image, the method further includes extracting a corresponding set of TLS features (140) from a corresponding lymphocyte cluster, and processing the corresponding set of TLS features using a TLS classification model (350) to classify the corresponding TLS region as one of a first TLS mature state, a second TLS mature state, or a third TLS mature state.

Description

Technical Field

[0001] The present disclosure relates to machine learning identification, classification, and quantification of tertiary lymphoid structures, for example in tumor biopsy specimens. Background Art

[0002] Tertiary lymphoid structures (TLS) (e.g., tertiary lymphoid organs or ectopic lymphoid follicles) are ectopic lymphoid tissues composed of B cells, T cells, and supporting cells, which develop in non-lymphoid organs and are often found in tumors. TLS supports the differentiation of naive T cells into effector T cells and memory T cells, and often develops in chronic inflammatory areas. In clinical pathological settings, TLS has been observed, but diagnostic pathological evaluation or evaluation has not yet been performed to guide treatment. Studies have shown that there is an association between TLS and immuno-oncology (IO) treatment outcomes in multiple indications (e.g., such as Sautes-Fridman et al., 2019, Nat Rev Cancer 19:307 and Vanhersecke et al., "Maturetertiary lymphoid structures predict immune checkpoint inhibitor efficacy insolid tumors independently of PD-L1 expression", Nat Cancer, 2021). The presence of TLS in various tumors shows correlation with outcomes in non-IO environments, and TLS has recently been shown to predict responses to IO treatment in melanoma, osteosarcoma, and RCC. See, e.g., Cabrita et al., 2020, Nature 577:561, Petitprez et al., 2020, Nature 577:556, Helmink et al., 2020, Nature 577:549, Bruno, N&V, 2020, Nature 577:474, Sautes-Fridman et al., 2019, Nat Rev Cancer 19:307. In the research setting, TLS has been evaluated by manual visual methods based on hematoxylin and eosin staining (H&E) and immunohistochemistry (IHC) staining. Image analysis of IHC or immunofluorescence (IF) staining has been used for quantification. These correlations depend on TLS maturity and localization within the tumor microenvironment (TME). Summary of the invention

[0003] One aspect of the present disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations including receiving an input histological image of a patient diagnosed with cancer. The input histological image includes a plurality of image pixels. The operations also include processing the input histological image using a cell classification model to generate one or more lymphocyte density maps within the input histological image, and performing morphological image processing on the one or more lymphocyte density maps to identify one or more TLS regions within the input histological image. Each TLS region is represented by a corresponding lymphocyte cluster. For each corresponding TLS region in the one or more TLS regions identified in the input histological image, the operations also include extracting a set of corresponding TLS features from a corresponding lymphocyte cluster representing the corresponding TLS region, and processing the set of corresponding TLS features using a TLS classification model to classify the corresponding TLS region into one of a first TLS maturity state, a second TLS maturity state, or a third TLS maturity state.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operation also includes processing the input histological image using a tumor detection model to identify a tumor region within the input histological image. Here, processing the input histological image to generate the one or more lymphocyte density maps may include processing the input histological image using the cell classification model by performing single cell imaging analysis on the tumor region identified within the input histological image to generate the one or more lymphocyte density maps. In these implementations, the tumor detection model may be trained by obtaining a plurality of image blocks rasterized from a set of whole slide histopathology images, each image block being manually annotated as including a tumor or a non-tumor; and training the tumor detection model on the plurality of image blocks using a neural network to teach the tumor detection model to learn how to identify tumor regions within a histological image.

[0005] In some examples, a cell classification model is trained by: obtaining a plurality of image slices; and training the cell classification model on the plurality of image slices using a neural network to teach the cell classification model how to classify individual cells in a histological image as tumor cells, lymphocytes, or non-malignant cells. Each image slice includes a corresponding plurality of human cells and manual annotations that label each human cell as a tumor cell, a lymphocyte, or a non-malignant cell.

[0006] In some implementations, a TLS classification model is trained by obtaining a training data set including a plurality of training histological images, wherein each training histological image includes a tumor microenvironment and has manual annotations. The manual annotations identify one or more TLS regions in the training histological images, and for each corresponding TLS region, the true TLS maturity state indicates that the corresponding TLS region includes a first TLS maturity state, a second TLS maturity state, or a third TLS maturity state. Each TLS region is represented by a corresponding lymphocyte cluster. In these implementations, the TLS classification model is further trained by the following operations: for each TLS region, a set of corresponding training TLS features is extracted from the corresponding lymphocyte cluster representing the TLS region; and the TLS classification model is trained on the set of corresponding training TLS features extracted for each TLS region to teach the TLS classification model to learn how to predict the true TLS grade of each corresponding TLS region. Training the TLS classification model may include training the TLS classification model using a classification and regression tree (CART) algorithm.

[0007] The first TLS maturation state may include dense aggregates of at least a threshold number of lymphocytes that do not contain high endothelial cell venules or germinal centers. The second TLS maturation state includes immature TLS, which includes dense aggregates of at least a threshold number of lymphocytes that contain high endothelial cell venules and do not contain any germinal centers. The third TLS maturation state may include mature TLS, which includes dense aggregates of at least a threshold number of lymphocytes that contain high endothelial cell venules and germinal centers. The set of corresponding TLS features extracted from the corresponding lymphocyte clusters may include the area of ​​the corresponding TLS region, the roundness of the corresponding TLS region, and the skewness of the corresponding TLS region.

[0008] In some examples, the operations further include: for each respective TLS region of the one or more TLS regions identified in the input histology image, generating a corresponding pixel mask that highlights at least a perimeter of the corresponding TLS region, generating an output image that enhances the input histology image by superimposing the corresponding pixel mask generated for each TLS region onto the input histology image, and providing the output image for display on a screen in communication with the data processing hardware. In these examples, the corresponding pixel mask generated for each respective TLS region classified as a first maturation state includes a first pixel mask, the corresponding pixel mask generated for each respective TLS region classified as a second maturation state includes a second pixel mask that is visually discernible from the second pixel mask, and the corresponding pixel mask generated for each respective TLS region classified as a third maturation state includes a third pixel mask that is visually discernible from the first pixel mask and the second pixel mask.

[0009] In some implementations, the operation also includes determining a total TLS score for the input histological image based on the TLS maturation status of one or more TLS regions identified in the histological image and TLS features extracted from one or more TLS regions identified in the histological image. In these implementations, the operation may also include determining a treatment recommendation for treating the patient using immunotherapy based on the total TLS score. Here, the immunotherapy may include at least one of a PD-1 inhibitor or a PD-L1 inhibitor. The operation may also include determining a predicted score for the patient's response to immunotherapy based on the TLS maturation status of one or more TLS regions identified in the histological image and TLS features extracted from one or more TLS regions identified in the histological image.

[0010] Another aspect of the present disclosure provides a system comprising data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including receiving an input histological image of a patient diagnosed with cancer. The input histological image comprises a plurality of image pixels. The operations also include processing the input histological image using a cell classification model to generate one or more lymphocyte density maps within the input histological image, and performing morphological image processing on the one or more lymphocyte density maps to identify one or more TLS regions within the input histological image. Each TLS region is represented by a corresponding lymphocyte cluster. For each corresponding TLS region in the one or more TLS regions identified in the input histological image, the operations also include extracting a set of corresponding TLS features from a corresponding lymphocyte cluster representing the corresponding TLS region, and processing a set of corresponding TLS features using a TLS classification model to classify the corresponding TLS region into one of a first TLS maturity state, a second TLS maturity state, or a third TLS maturity state.

[0011] This aspect may include one or more of the following optional features. In some implementations, the operation also includes processing the input histological image using a tumor detection model to identify a tumor region within the input histological image. Here, processing the input histological image to generate the one or more lymphocyte density maps may include processing the input histological image using the cell classification model by performing single cell imaging analysis on the tumor region identified within the input histological image to generate the one or more lymphocyte density maps. In these implementations, the tumor detection model may be trained by obtaining a plurality of image blocks rasterized from a set of whole slide histopathology images, each image block being manually annotated as including a tumor or a non-tumor; and training the tumor detection model on the plurality of image blocks using a neural network to teach the tumor detection model to learn how to identify tumor regions within a histological image.

[0012] In some examples, a cell classification model is trained by: obtaining a plurality of image slices; and training the cell classification model on the plurality of image slices using a neural network to teach the cell classification model how to classify individual cells in a histological image as tumor cells, lymphocytes, or non-malignant cells. Each image slice includes a corresponding plurality of human cells and manual annotations that label each human cell as a tumor cell, a lymphocyte, or a non-malignant cell.

[0013] In some implementations, a TLS classification model is trained by obtaining a training data set including a plurality of training histological images, wherein each training histological image includes a tumor microenvironment and has manual annotations. The manual annotations identify one or more TLS regions in the training histological images, and for each corresponding TLS region, the true TLS maturity state indicates that the corresponding TLS region includes a first TLS maturity state, a second TLS maturity state, or a third TLS maturity state. Each TLS region is represented by a corresponding lymphocyte cluster. In these implementations, the TLS classification model is further trained by the following operations: for each TLS region, a set of corresponding training TLS features is extracted from the corresponding lymphocyte cluster representing the TLS region; and the TLS classification model is trained on the set of corresponding training TLS features extracted for each TLS region to teach the TLS classification model to learn how to predict the true TLS grade of each corresponding TLS region. Training the TLS classification model may include training the TLS classification model using a classification and regression tree (CART) algorithm.

[0014] The first TLS maturation state may include dense aggregates of at least a threshold number of lymphocytes that do not contain high endothelial cell venules or germinal centers. The second TLS maturation state includes immature TLS, which includes dense aggregates of at least a threshold number of lymphocytes that contain high endothelial cell venules and do not contain any germinal centers. The third TLS maturation state may include mature TLS, which includes dense aggregates of at least a threshold number of lymphocytes that contain high endothelial cell venules and germinal centers. The set of corresponding TLS features extracted from the corresponding lymphocyte clusters may include the area of ​​the corresponding TLS region, the roundness of the corresponding TLS region, and the skewness of the corresponding TLS region.

[0015] In some examples, the operations further include: for each respective TLS region of the one or more TLS regions identified in the input histology image, generating a corresponding pixel mask that highlights at least a perimeter of the corresponding TLS region, generating an output image that enhances the input histology image by superimposing the corresponding pixel mask generated for each TLS region onto the input histology image, and providing the output image for display on a screen in communication with the data processing hardware. In these examples, the corresponding pixel mask generated for each respective TLS region classified as a first maturation state includes a first pixel mask, the corresponding pixel mask generated for each respective TLS region classified as a second maturation state includes a second pixel mask that is visually discernible from the second pixel mask, and the corresponding pixel mask generated for each respective TLS region classified as a third maturation state includes a third pixel mask that is visually discernible from the first pixel mask and the second pixel mask.

[0016] In some implementations, the operation also includes determining a total TLS score for the input histological image based on the TLS maturation status of one or more TLS regions identified in the histological image and TLS features extracted from one or more TLS regions identified in the histological image. In these implementations, the operation may also include determining a treatment recommendation for treating the patient using immunotherapy based on the total TLS score. Here, the immunotherapy may include at least one of a PD-1 inhibitor or a PD-L1 inhibitor. The operation may also include determining a predicted score for the patient's response to immunotherapy based on the TLS maturation status of one or more TLS regions identified in the histological image and TLS features extracted from one or more TLS regions identified in the histological image.

[0017] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the following description. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Schematic diagram of an example system for identifying, classifying, and quantifying tertiary lymphoid structures (TLS) in histological images of the tumor microenvironment.

[0019] Figure 2A-2K Several tables listing exemplary TLS features are presented.

[0020] Figure 3A is a diagram of an example training process for training a TLS classification model.

[0021] Figure 3B is a schematic diagram of an example training process for training a tumor detection model.

[0022] Figure 3C is a schematic diagram of an example training process for training a cell classification model.

[0023] Figure 4 is a flow chart of an example arrangement of operations of a method for identifying, classifying, and quantifying TLS in histological images of a tumor microenvironment.

[0024] Figure 5A and Figure 5B is an example confusion matrix comparing the accuracy between the TLS classification model and pathologists.

[0025] Figure 6A-6C is an example plot comparing the performance of the TLS classification model with pathologists.

[0026] Figure 7 Example input histology images and corresponding classified TLS maturation states are shown.

[0027] Figure 8 An example input histology image representing the maturation state of a mature TLS and a corresponding output image including a corresponding pixel mask are shown.

[0028] Fig. 9 Example input histology images representing the maturation status of immature TLS and corresponding output images including corresponding pixel masks are shown.

[0029] Fig.10 Example input histology images representing the maturation status of lymphoid aggregate TLS and corresponding output images including corresponding pixel masks are shown.

[0030] Fig.11 Example output images including TLS regions corresponding to each of a mature TLS maturation state, an immature TLS maturation state, and a lymphoid aggregate TLS maturation state are shown.

[0031] Figure 12-16Example untransformed and transformed output images are shown.

[0032] Fig.17 is a schematic diagram of the process flow chart used to validate the extracted TLS signatures using transcriptome analysis correlation.

[0033] Fig.18 An example table showing a 12-chemokine gene signature.

[0034] Fig.19 A sample feature table is shown.

[0035] Figure 20A-20C shows a sample graphical representation of correlation data.

[0036] Figure 21A-Figure 21D An example graphical representation of a correlation graph verifying the extracted TLS features 140 is shown.

[0037] Fig. 22 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0038] The same reference symbols in different drawings denote the same elements. DETAILED DESCRIPTION

[0039] Tertiary lymphoid structures (TLS) are ectopic lymphoid organs developed in non-lymphoid tissues (such as chronic inflammation and tumor sites). TLS are vascularized lymphoid structures that develop in benign and tumor tissues with chronic inflammation. TLS are highly organized structures that are similar to secondary lymphoid structures (e.g., lymph nodes). TLS can be composed of B cell areas containing active germinal centers, surrounding T cell areas containing various types of dendritic cells (DC), T cells, high endothelial cell venules (HEV) and / or other supporting cells in the structural matrix. Unlike lymph nodes, TLS lack fibrous capsules and are directly exposed to the tumor microenvironment (TME). Compared with the tumor core, TLS are more abundant in the infiltrating margin / stroma. The presence of TLS is associated with favorable outcomes for treating multiple indications (e.g., treating melanoma with nivolumab or nivolumab and ipilimumab). TLS structures can be classified as lymphoid aggregates (LA) (i.e., the first mature state), immature TLS (imTLS) (e.g., grade 1) (i.e., the second mature state), or mature TLS (mTLS) (e.g., grade 2) (i.e., the third mature state) with the presence of germinal centers (GC). In some cases, there is no TLS (e.g., grade 0). Although the biological mechanisms behind its formation are not fully understood, TLS is known to play an important role in anti-tumor immune responses. For example, the presence of TLS is associated with a good prognosis for many cancer types and an improvement in response to immunotherapy.

[0040] The conventional method for TLS detection in patients is the technique of tissue staining for markers of the immune cell lineage by multiplex immunohistochemistry or immunofluorescence techniques. However, multiplex imaging is generally not applicable due to its cost, high complexity, small field of view, and difficulty in scaling, which limits its use in research settings. On the other hand, hematoxylin-eosin (H&E) staining is widely available and remains the clinical standard in histopathology. Evaluation of H&E-stained slides based on pathologist assessment is time- and labor-intensive, and manual and qualitative evaluations performed manually by pathologists are often inaccurate and subject to inter-observer variability.

[0041] The implementation of this article involves the use of machine learning techniques that use deep learning to train a model to learn how to detect the presence of TLS areas in H&E stained histological images and classify each of the TLS areas into one of three TLS maturation states. The first maturation state includes dense aggregates of at least a threshold number of lymphocytes that do not contain high endothelial venules or germinal centers. In some examples, the threshold number is equal to 100. The second maturation state includes immature TLS associated with dense aggregates of at least a threshold number of lymphocytes that contain high endothelial venules and do not contain any germinal centers. The third maturation state includes mature TLS associated with dense aggregates of at least a threshold number of lymphocytes that contain high endothelial venules and germinal centers. More specifically, the implementation includes using a cell classification model to process an input histological image (e.g., an H&E-stained histological image) to generate one or more lymphocyte density maps, performing morphological image processing on the one or more lymphocyte density maps to identify one or more TLS regions within the input histological image, wherein each TLS region is represented by a corresponding lymphocyte cluster, and for each corresponding TLS region, extracting a set of corresponding TLS features from the corresponding lymphocyte cluster representing the corresponding TLS region. Thereafter, the trained TLS classification model receives the set of corresponding TLS features extracted for each corresponding TLS region to classify the corresponding TLS region into one of a first TLS maturity state, a second TLS maturity state, or a third TLS maturity state.

[0042] The implementation of this article also involves calculating the TLS score of the input histological image based on the TLS maturity state output from the TLS classification model and the TLS features of the TLS area identified in the input histological image. The TLS scorer can determine the total area of ​​the tumor area and the corresponding TLS score of each of the three TLS maturity states, and the score is based on the corresponding total TLS area of ​​the TLS area classified for each of the three TLS maturity states. The TLS scorer can then calculate the total TLS score associated with the input histological image of the patient based on the linear weighted sum of each corresponding total TLS area divided by the tumor area. Described in more detail below, the total TLS score can be used to predict various prognostic values ​​of the patient, such as predicting survival outcomes, such as overall survival and progression-free survival. That is, compared with a lower total TLS score, a higher total TLS score indicates significantly improved overall survival and progression-free survival. In this way, the total TLS score can be used to replace the use of tumor stage prediction to predict the prognostic outcome, and / or the prognostic outcome predicted using tumor stage / grading can be further refined by the total TLS score.

[0043] For each corresponding TLS region in one or more TLS regions identified in the input histological image, the image enhancer can generate a corresponding pixel mask that at least highlights the periphery of the corresponding TLS region, and then generate an output image that enhances the input histological image by superimposing the corresponding pixel mask generated for each TLS region onto the input histological image. The output image generated by the image enhancer can be provided for display on a screen for viewing by a healthcare professional (HCP). Here, the image enhancer receives the classification output from the TLS classification model and generates a visually different corresponding pixel mask for each of three different TLS maturation states. For example, the pixel mask generated for the TLS region classified as the first maturation state can include a first color, the pixel mask generated for the TLS region classified as the second maturation state can include a different second color, and the pixel mask generated for the TLS region classified as the third maturation state can include a third color different from the first color and the second color. In some examples, the pixels generated for the TLS region classified as the third maturation state at least highlight the periphery of the corresponding TLS region, and further highlight the pixel area covered by the germinal center.

[0044] The implementation of this article also relates to a training process for training a TLS classification model. Here, the training process obtains a training data set including multiple training histological images, each training histological image containing a tumor microenvironment and including manual annotations from a pathologist. The manual annotation identifies the presence of a TLS region in each training histological image, wherein each TLS region is represented by a corresponding lymphocyte cluster. The manual annotation also identifies the true TLS maturity state of each corresponding TLS region, indicating that the corresponding TLS region includes a first TLS maturity state, a second TLS maturity state, or a third maturity state. Next, the training process extracts a set of corresponding training TLS features from the corresponding lymphocyte cluster representing each TLS region, and the set of training TLS features may include the area of ​​the TLS region, the circularity (i.e., the area of ​​the TLS region multiplied by 4pi and the square of the perimeter of the TLS region), and the skewness of the density of each lymphocyte cluster representing each TLS region. Based on the set of corresponding training TLS features extracted for each TLS region, the training process uses a classification and regression tree (CART) algorithm to train the TLS model to learn how to predict the true TLS grade of each corresponding TLS region.

[0045] It is noteworthy that the cell classification model is trained to learn how to classify individual cells in histological images into tumor cells, lymphocytes, or non-malignant cells. As used herein, lymphocytes may include T cells and B cells. The Mask R-CNN deep learning model can be used to train the cell classification model to learn how to segment and classify individual cell nuclei into tumor cells, lymphocytes, and other non-malignant cells.

[0046] In some examples, the input histology image is processed by performing image preprocessing on the input histology image using a tumor detection model to identify a tumor region within the input histology image, so that the cell classification model is used to perform single cell image analysis on the tumor region identified within the input histology to generate one or more lymphocyte density maps. The tumor detection model can be trained on multiple image patches rasterized from a set of whole slide histopathology images, where each image patch is manually annotated as including a tumor or a non-tumor. More specifically, a deep learning neural network trains the tumor detection model on multiple image patches to teach the tumor detection model to learn how to identify tumor regions within the histology image. The deep learning neural network can include a ResNet18 deep learning model.

[0047] Advantageously, the deep learning-based single cell analysis techniques disclosed herein provide for accurately identifying, classifying, and quantifying the presence of TLS regions in whole slide images from H&E staining without incurring any of the disadvantages of other techniques that employ slice-based or block-based approaches for image analysis. Because TLS vary greatly in size, density, and morphology, there are significant challenges in identifying and interpreting TLS regions using traditional slice-based approaches. As will become clear, the techniques disclosed herein include quantifying the spatial distribution of lymphocytes, thereby providing an accurate and interpretable model for classifying TLS according to their maturation state.

[0048] Similarly, manual and qualitative assessment of TLS performed by pathologists lacks automated counting and quantitative characterization of TLS. Similarly, when evaluated on H&E-stained slides, such manual and qualitative assessment of TLS performed by pathologists was found to be inaccurate and subject to inter-observer variability. See Buisseret L, Desmedt C, Garaud S et al. Reliability of tumor-infiltrating lymphocyte and tertiarylymphoid structure assessment in human breast cancer. Mod Pathol. 2017; 30(9): 1204-1212. doi: 10.1038 / modpathaol.2017.43.

[0049] refer to Figure 1In some implementations, the system 100 includes a client device 111 that inputs a histology image 110 of a patient diagnosed with cancer into a TLS classification model 350 for identifying, classifying, and quantifying the presence of TLS regions within the histology image for use as a predictive biomarker for immune checkpoint inhibitor (ICI) efficacy and prognostic outcomes. The input histology image 110 may optionally include metadata 11 that includes information such as the type of cancer the patient is diagnosed with, the stage / grade of the tumor, and / or patient demographic information. The input histology image 110 may include a whole slide image (WSI) stained with hematoxylin and eosin (H&E). The input histology image 110 includes a plurality of image pixels. The input histology image 110 represents a human tumor biopsy specimen. The input histology image 110 may include a tumor microenvironment for any number of cancers, including, but not limited to, bladder cancer (BLCA), breast cancer (BRCA), gastric adenocarcinoma (STAD), lung adenocarcinoma (LUAD) (e.g., non-small cell lung cancer adenocarcinoma (NSCLC-AD)), and / or lung squamous cell carcinoma (LUSC) (e.g., non-small cell squamous lung cancer (NSCLC-SQ)).

[0050] The client device 111 is associated with a user 10, such as a healthcare professional (HCP), who can communicate with a remote system 141 via a network 132. The remote system 141 can be a distributed system (e.g., a cloud environment) with scalable / elastic resources 142. The resources 142 include computing resources 144 (e.g., data processing hardware) and / or storage resources 146 (e.g., memory hardware). In some implementations, the remote system 141 executes a TLS identification and quantification application 160 (also referred to as "application 160"), which is configured to execute the TLS classification model 450 in addition to other components such as the tumor detection model 450, the cell classification model 550, the lymphocyte aggregator 120, the morphological image processing module 130, the TLS extractor 145, the TLS scorer 150, and the image enhancer 360. Here, the client device 111 may access the application 160 running on the remote system 141 and input the histology input image 110 to the TLS classification model 350 via a graphical user interface (GUI) executed on the client device 111. The GUI may be displayed to the user 10 via the screen 114 of the client device 111. The client device 111 may additionally or alternatively execute the application 160 to enable the ability to run any combination of the TLS classification model 350 and / or other components on the client device 111 for identifying, classifying, and quantifying the presence of TLS regions 135 within the histology image 110.

[0051] The TLS identification and quantification application 160 can determine TLS details 190 and / or treatment recommendations 192 based on the identified TLS areas 135 classified and quantified using the TLS classification model 350. The application 160 can return the TLS details 190 and / or treatment recommendations 192 to the client device 111 so that the client device displays the TLS details 190 and / or treatment recommendations 192 on the screen 114 of the client device 111. The TLS details 190 can include, but are not limited to, the total TLS score 152 of the input histology image 110 and other details, such as the number of TLS areas associated with a first maturation state (e.g., TLS1) classified by the TLS classification model 350, the number of TLS areas associated with a second maturation state (e.g., TLS2) classified by the TLS classification model 350, and the number of TLS areas associated with a third maturation state (e.g., TLS3) classified by the TLS classification model 350. Here, the first maturation state includes dense aggregates of at least a threshold number of lymphocytes that do not contain high endothelial cell venules or germinal centers. In some examples, the threshold number is equal to 100. The second maturity state includes immature TLS associated with dense aggregates of at least a threshold number of lymphocytes containing high endothelial venules and no germinal centers. The third maturity state includes mature TLS associated with dense aggregates of at least a threshold number of lymphocytes containing high endothelial venules and germinal centers. The TLS details 190 provided for display on the screen 114 may also include an output image 110A that enhances the input histological image 110 by superimposing the corresponding pixel mask 112 generated for each TLS area onto the input histological image 110. The treatment recommendation 192 may indicate instructions for applying (or not applying) immunotherapy to the patient to treat the patient. For example, immunotherapy may include PD-1 inhibitors (e.g., anti-PD-1 antibodies) or PD-L1 inhibitors (e.g., anti-PID-L1 antibodies). In one example, the immunotherapy includes the immune checkpoint inhibitor drug nivolumab.

[0052] The treatment recommendation 192 may also include a predicted prognostic outcome for the patient based on the TLS details 190, such as overall survival (OS) (i.e., in months), progression-free survival (PFS) (in months). The treatment recommendation 192 may show the OS and / or PFS prediction for immunotherapy treatment compared to the OS and / or PFS prediction without immunotherapy. The prognostic outcomes predicted by the application 160 may inform the patient, healthcare provider, and / or relatives of the patient to make better testing and treatment decisions for a specific health condition diagnosed, or to conduct risk stratification for a treatment trial.

[0053] In some examples, the input histology image 110 undergoes initial image preprocessing to ensure adequate image quality. The input histology image may include a 40x magnification. However, a WSI slide scanned at a lower magnification (e.g., 20x) may be used. To minimize the effects of image artifacts, image preprocessing may downsample the whole slide image by a factor of 32 and apply an appropriate color factor to remove areas with pen marks, folds, and blur artifacts.

[0054] In the example shown, the tumor detection model 450 processes the input histology image 110 to identify one or more tumor regions 115 within the input histology image 110. Each tumor region 115 may be represented by a corresponding set of pixels in which the tumor region 115 is located in the input histology image 110. Notably, since only TLS within or around the tumor region 115 are relevant, the tumor detection model 450 may segment cancerous tissue from normal tissue, thereby enabling subsequent processing for TLS identification and quantification to focus on the tumor region 115 in the input histology image 110. The tumor detection model 450 may include a pre-trained indication-specific tissue segmentation model configured to process the input histology image 110 to discriminate cancer, cancer-associated stroma, and necrosis from normal tissue. Figure 3BAn example tumor detection model training process 300b that can be used to train a tumor detection model 450 is shown. The training process 300b obtains a plurality of image patches 370 rasterized from a collection of whole slide histopathology images. The histopathology images may include publicly available and previously annotated H&E stained WSIs from patients with colorectal cancer and gastric cancer. Each image patch 370 may include manual annotations 372 indicating the location of tumor regions and non-tumor regions (including adipose tissue, mucus, stroma, or muscle) within the corresponding whole slide histopathology image. The image patches may include 512x512 image patches of 0.5 microns per image pixel. The training process 300b includes training the tumor detection model 450 on the plurality of image patches 370 using a neural network 374 to teach the tumor detection model 450 to learn how to identify tumor regions within histology images. In some examples, the neural network 374 includes a ResNet18 deep learning network, and the loss module 378 calculates a training loss 380 based on the prediction 376 output by the ResNet18 network relative to the true annotation 372. The training process 300b can update the parameters of ResNet 18 based on the training loss 380 until the parameters of ResNet 18 converge to obtain a trained tumor detection model 450. The loss module 378 can adopt a cross entropy loss function and counteract overfitting by applying L2-regularization. The training process 300b can extend the tumor segmentation by 0.5 mm via image expansion to include an invasive margin. The training process can also apply horizontal / vertical flipping and translation to enhance the image block 370 used for training.

[0055] Return to reference Figure 1, after the tumor detection model 450 identifies the tumor region 115, the cell classification model 550 processes the input histology image 110 by performing single cell imaging analysis on the tumor region 115 identified within the input histology image 110 (i.e., on the image pixels corresponding to the tumor region 115) to generate a classified tumor region 115C. That is, the single cell imaging analysis performed by the cell classification model 550 classifies individual cells / nuclei into tumor cells, lymphocytes (i.e., B cells and T cells, dendritic cells (DCs), high endothelial venules (HEVs)), and non-malignant cells. As used herein, the trained cell classification model 550 is used as a lymphocyte mask for classifying which cells in the tumor region 115 include lymphocytes. In this way, the classified tumor region 115C can correspond to a lymphocyte mask that identifies all lymphocytes classified and segmented by the cell classification model 550 in the tumor region 115 within the input histology image 110. Thereafter, the application 160 executes the lymphocyte aggregator 120, which processes the classified tumor regions 115C output by the cell classification model 550 to sort the tumor regions 115C on a predefined grid (e.g., 16 x 16 μm). 2 The number of lymphocytes per unit square is counted by the grid, thereby generating one or more lymphocyte density maps 125 within the input histology image 110.

[0056] Figure 3CAn example cell classification model training process 300c that can be used to train a cell classification model 550 on multiple image slices 382 is shown. Each image slice (i.e., image block) 382 represents a corresponding plurality of human cells and is manually annotated to label each human cell as a tumor cell or a lymphocyte or a non-malignant cell. The multiple image slices may include 1,358 image slices from 66 patients in a publicly available dataset with manual annotations 384, the manual annotations containing 17,582 tumor cells, 22,550 lymphocytes, and 10,675 other non-malignant cells. The training process 300c includes training a cell classification model 550 on multiple image slices 382 using a neural network 386 to teach the cell classification model 550 to learn how to classify individual cells in a histological image as tumor cells, lymphocytes, or non-malignant cells. In some examples, the neural network 386 includes a mask R-CNN deep learning network, and the loss module 392 calculates a training loss 390 based on the prediction 388 output by the mask R-CNN network 386 relative to the true annotation 384. The training process 300b can update the parameters of the mask R-CNN based on the training loss 392 until the parameters of the mask R-CNN converge to obtain a trained cell classification model 550. As used herein, the trained cell classification model 550 is used as a lymphocyte mask for classifying which cells in the tumor region 115 include lymphocytes. The loss module 392 can update the mask R-CNN via the training loss 392 using a stochastic gradient descent technique. The training process can also apply horizontal / vertical flipping and translation to enhance the image slices 382 used for training.

[0057] Return to reference Figure 1 In some implementations, the TLS identification and quantification application 160 performs morphological image processing 130 on one or more lymphocyte density maps 125 to identify one or more TLS regions 135 within the input histology image 110. Notably, each TLS region represents a corresponding lymphocyte cluster. The morphological image processing 130 may indicate pixel locations corresponding to each TLS region 135 identified within the input histology image 110. Each TLS region 135 may correspond to a TLS mask. In some examples, the morphological image processing 130 performed on the lymphocyte density map 125 applies thresholding to exclude lymphocyte clusters having an area less than a predefined threshold from being identified as TLS regions. The predefined threshold area may be equal to 0.0384 mm 2 .

[0058] For each TLS region 135 identified, the application 160 executes a TLS feature extractor 145 configured to extract a corresponding set of TLS features 140 from a corresponding lymphocyte cluster representing the corresponding TLS region 135. The set of TLS features 140 may include human interpretable features (HIFs) associated with the TLS region 135. In some examples, a portion of the TLS features includes sample-level features that include at least one of a summary count, area, shape, or location of the corresponding TLS region 135. The TLS features 140 extracted from the corresponding lymphocyte cluster representing the corresponding TLS region 135 may include the area of ​​the TLS region 135, the circularity of the TLS region 135 (i.e., the ratio of the area of ​​the TLS region 135 multiplied by 4pi to the square of the perimeter of the TLS region 135), and the skewness representing the density of each lymphocyte cluster of the TLS region 135). The TLS feature 140 may additionally or alternatively be at least one of an area of ​​germinal centers within an object in tissue, an area of ​​the object in tissue, a center of mass x of the object in tissue, a center of mass y of the object in tissue, a longest distance of the object from a tumor, a perimeter of the object in tissue, a shortest distance of the object from a tumor, a total germinal centers within the object in tissue, or a ratio of the germinal centers within the object in tissue to the area of ​​the object. Some of the TLS features 140 may include sample level features, including one or more of the following: area of ​​the TLS region 135, total lymphocyte count, area ratio, count ratio, maximum area, maximum longest distance from the tumor, maximum perimeter, maximum shortest distance from the tumor, maximum total area, maximum total count, mean area, mean longest distance from the tumor, mean perimeter, mean shortest distance from the tumor, mean total area, mean total count, median area, median longest distance from the tumor, median perimeter, median shortest distance from the tumor, median total area, median total count, minimum area, minimum longest distance from the tumor, minimum perimeter, minimum shortest distance from the tumor, minimum total area, or minimum total count.

[0059] Figure 2A-2K A plurality of tables are shown that list TLS features 140 that may be extracted by the TLS extractor 145. Each table includes a plurality of columns listing (1) a feature name, (2) a feature type that identifies whether the feature is an identifier, metadata, or a feature, (3) a feature description that describes the extracted feature, and a human interpretable feature (HIF) type that indicates whether the feature is an identifier, metadata, raw feature, minimum feature, maximum feature, median feature, mean feature, attribute feature, feature, and the like.

[0060] Return to reference Figure 1, the TLS classification model 350 can process the set of corresponding TLS features 140 to classify the corresponding TLS region 135 into one of the first TLS maturation state (TLS1), the second TLS maturation state (TLS2), or the third TLS maturation state (TLS3). The first maturation state can be associated with lymphoid aggregates, the second maturation state can be associated with a corresponding lymphocyte cluster with a primary follicle without any germinal center, and the third maturation state can be associated with a corresponding lymphocyte cluster with a primary follicle and a secondary cell with a germinal center. Given that TLS2 and TLS3 tend to have a circular shape and are generally larger than TLS1 and TLS3 has a unique germinal center with a lower lymphocyte density, the above-mentioned TLS features 140 with area, circularity, and skewness can be interpreted by the trained TLS classification model 350 to accurately classify each TLS region 135. Each TLS region classified by the TLS classification model 135 can correspond to a prognostic biomarker. The TLS classification model 350 can output a TLS state 312 indicating the maturation state of each TLS region 135 classified by the TLS classification model 350.

[0061] In some examples, the application 160 executes an image enhancer 360 configured to enhance the input histology image 110 based on the TLS status 312 of one or more TLS regions 135 identified in the input histology image 110 output from the TLS classification model 350. Here, the image enhancer 360 can generate a corresponding pixel mask 112 that highlights at least the perimeter of each corresponding TLS region 135 based on the maturity status (e.g., TLS1, TLS2, or TLS3) of the corresponding TLS region 135. The image enhancer 360 can generate a first pixel mask 112 for the TLS region 135 classified as TLS1, a second pixel mask 112 different from the first pixel mask 112 for the TLS region 135 classified as TLS2, and a third pixel mask 112 different from the first and second pixel masks 112 for the TLS region 135 classified as TLS3. That is, different pixel masks 112 can be visually distinguished from each other. In some examples, different pixel masks 112 are associated with different colors. The image enhancer 360 generates an output image 110A that enhances the input histology image 110 by superimposing the corresponding pixel mask 112 generated for each TLS region 135 onto the input histology image 110. The pixel mask 112 can be superimposed as a graphical feature that highlights at least the periphery of each corresponding TLS region 135, thereby serving as a visual cue indicating the location and corresponding classification (e.g., TLS1, TLS2, or TLS3) of each TLS region 135 identified in the output image 110A. As will become clear, the image enhancer 360 can apply one or more post-processing rules to generate the output image 110A. As described in the previous paragraphs, the application 160 can provide the output image 110 as TLS details 190 to the client device 111 for display on the screen 114.

[0062] In addition to the maturation state, the TLS classification model 350 and / or the TLS feature extractor 145 can also be configured to output / extract topological information associated with the TLS regions 135, such as the coordinates of the TLS regions 135 and their proximity to the tumor bed and location relative to the tumor and / or stromal compartments. In this manner, the image enhancer 360 or image generator can process the topological information and any combination of the input histology image, TLS state 312, TLS regions 135, and TLS features to generate a topological map or heat map as an output image 110A that visually depicts the topological information associated with the TLS regions 135 of potential interest.

[0063] Continue to refer Figure 1The application 160 may also execute a TLS scorer 150 for calculating an overall TLS score 152 for the patient based on the TLS features 140 and the corresponding TLS maturation states 312 of all TLS regions 135 identified in the input histology image 110. The TLS scorer 150 may determine the total area (expressed as “area”) of the tumor region 115. 肿瘤 ”). TLS scorer 150 may also determine a corresponding separate TLS area for each of the three TLS maturity states. For example, TLS scorer 150 may determine a first TLS area (denoted as “area”) based on the total area of ​​TLS regions classified as the first maturity state. TLS1 ”), a second TLS area (expressed as “area TLS2 ”), and determining a third TLS area (expressed as “area TLS2 ”). In some examples, TLS scorer 150 calculates a total TLS score 152 as a linearly weighted sum of the individual TLS areas divided by the tumor area, as shown below. TLS score = (w1 x area TLS1 +w2 x area TLS2 +w3 x area TLS3 )(1)Wherein, w1, w2, and w3 are corresponding weights. The optimal corresponding weights can be selected by performing Cox regression analysis of overall survival for each individual TLS area. In one example, w1 is equal to 0.81, w2 is equal to 0.84, and w3 is equal to 1.0, indicating that the TLS area classified as the third maturity state (e.g., mature TLS) plays the most important role in the anti-tumor immune response.

[0064] It is noteworthy that statistical analysis applied to the total TLS score 152 and the individual TLS scores indicated by the first, second and third TLS areas can be used to predict various prognostic values ​​of the patient, such as predicting survival outcomes, including but not limited to overall survival and progression-free survival. Overall survival can be defined as the time from diagnosis to death or the last follow-up. Progression-free survival can be defined as the time from diagnosis to disease progression, death or the last follow-up. Univariate and multivariate analysis can be performed with the Cox proportional hazard model. Clinical and pathological variables (such as tumor stage and grade) can be included in the multivariate analysis. Kaplan-Meier analysis and log-rank test can be used to evaluate the patient stratification of the risk group. TLS scores associated with tumor status or grade can be further evaluated. Compared with the lower total TLS score, the higher total TLS score indicates significantly improved overall survival and progression-free survival. The overall survival and progression-free survival of patients with low total TLS scores are still better than those of patients without TLS areas. Thus, the total TLS score can be used to predict prognostic outcomes instead of using tumor stage predictions, and / or the prognostic outcomes predicted using tumor stage / grade can be further refined by the total TLS score.

[0065] In some scenarios, the application 160 performs post-processing to adjust the output image 110A based on any combination of the TLS features 140, the (one or more) TLS scores 152, and the TLS states 312. In particular, the application 160 can apply one or more post-processing rules 362 to modify the pixel mask 112 by fixing small and exposed germinal centers, fixing TLS regions 135 without germinal centers that are classified as the third maturation state (mature TLS), fixing mosaics to resolve predictions of multiple classes on the same structure due to confusion of the TLS classification model, applying object-level masking to remove false positive predictions of TLS within cancer and necrotic tissue regions, and / or applying a cutoff value.

[0066] refer to Figure 3AIn some implementations, an example TLS classification model training process 300a trains a TLS classification model 350 to learn how to predict the TLS state of a TLS region identified in a histological image. The training process 300a obtains a training data set 305 including a plurality of training histological images 310, 310a-n. Each training histological image 310 may contain a tumor microenvironment and include manual annotations from a qualified pathologist 312. The manual annotations 312 may identify one or more TLS regions 312a in the training histological image 310, and for each identified TLS region 312a, an indication of a true TLS maturation state 312b of the corresponding TLS region 312a includes a first TLS maturation state, a second TLS maturation state, or a third TLS maturation state. Each TLS region 312a annotated in the training histological image 310 is represented by a corresponding lymphocyte cluster.

[0067] The training process 300a executes a TLS feature extraction module 320 that receives each training histology image 310 and extracts a set of corresponding training TLS features 140 for each TLS region 312a. That is, for each TLS region 312a annotated in the training histology image 310, the TLS feature extraction module 320 can extract a set of corresponding training TLS features 140 from the corresponding lymphocyte cluster representing the TLS region 312a. The TLS feature extraction module 320 can include a pre-trained tumor extraction model 450 and a pre-trained cell classification model 550 to generate a lymphocyte density map. The feature extraction module 320 can also include any other component or combination of components executed by the application 160.

[0068] The training TLS features may include, but are not limited to, the area 140a of the TLS region, the circularity 140b (i.e., the ratio of the area of ​​the TLS region 312a multiplied by 4pi to the square of the perimeter of the TLS region), and the skewness 140c representing the density of the corresponding lymphocyte cluster of the TLS region 312a. Based on the set of corresponding training TLS features 140 extracted for each TLS region 312a, the training process 300a uses a classification and regression tree (CART) algorithm 340 to train a TLS classification model 350 to learn how to predict the true TLS state 312b of each corresponding TLS region 312a. In some examples, the training process 300a uses the scikit-learn package from the Python programming language version 3.6.11 (Python Software Foundation) to train the CART algorithm 340 using default parameter settings (criteria=gini; splitter=best; min_samples_split=2). The maximum depth of the tree is determined to be 4 using 5-fold cross validation in the training data set 305. Given the relative importance of TLS3, the class weights of TLS1, TLS2, and TLS3 can be empirically set to 1, 2, and 3, respectively, during training.

[0069] Figure 4 4 is a flow chart of an example arrangement of operations of a method 400 for identifying, classifying, and quantifying TLS regions 135 within an input histology image 110. The method 400 may be performed on data processing hardware 142 of a remote system 141 and / or on a client device 111. At operation 402, the method 400 includes receiving an input histology image 110 of a patient diagnosed with cancer. The input histology image includes a plurality of image pixels. The input histology image 110 may include an H&E stained image of a sample of a patient's tumor.

[0070] At operation 404, the method 400 includes processing the input histology image 110 using the cell classification model 550 to generate one or more lymphocyte density maps 125 within the input histology image 110. At operation 406, the method 400 includes performing morphological image processing on the one or more lymphocyte density maps 125 to identify one or more TLS regions 135 within the input histology image 110. Here, each TLS region 135 is represented by a corresponding lymphocyte cluster.

[0071] At operation 408, the method 400 includes, for each corresponding TLS region 135, extracting a set of corresponding TLS features 140 from a corresponding lymphocyte cluster representing the corresponding TLS region 135. At operation 410, the method 400 includes: for each corresponding TLS region 135, processing the set of corresponding TLS features using the TLS classification model 350 to classify the corresponding TLS region into one of a first TLS maturation state, a second TLS maturation state, or a third TLS maturation state. The first TLS maturation state includes lymphocyte aggregates that do not contain at least a threshold number of lymphocytes of high endothelial cell venules or germinal centers. The second TLS maturation state includes dense aggregates of at least a threshold number of lymphocytes that contain high endothelial cell venules and do not contain any germinal centers. The third TLS maturation state includes dense aggregates of at least a threshold number of lymphocytes that contain high endothelial cell venules and germinal centers.

[0072] Advantageously, after training the TLS classification model 350, the accuracy of the TLS classification model 350 in identifying and classifying TLS regions 135 within the input histology image 110 is comparable to (or in some scenarios even better than) the accuracy of a pathologist manually classifying TLS. Figure 5A and Figure 5B The confusion matrix 500 shown in FIG. 5 depicts a confusion matrix 500 comparing the accuracy 510 between the pathologist and the trained TLS classification model 350. In particular, the first confusion matrix 500, 500a ( Figure 5A ) shows the normalized accuracy 510 of the TLS classification model 350, and the second confusion matrix 500, 500b ( Figure 5B ) shows the normalized accuracy of the pathologist annotators. Here, the confusion matrix 500 shows the accuracy 510 for each TLS maturation state 312 (e.g., mature TLS, immature TLS, germinal center, lymphoid aggregates, etc.). The true maturation state of these input histology images 110 was generated by the majority consensus of 5 expert pathologists. However, Figure 6A-6C Further shown are plots 600 comparing TLS identification and classification performance between the TLS classification model 350 and the pathologist annotator. In particular, the first plot 600, 600a ( Fig. 6A ) depicts a comparison of the accuracy scores 610, the second plot 600, 600b ( Figure 6B ) depicts a comparison of the F1-scores 620, and the third plot 600, 600c ( Figure 6C ) depicts a comparison of recall scores 630. Here, each corresponding plot 600 graphically represents the score for each of the different TLS maturity states 312.

[0073] Figure 7 Depicted are input histology images 700 that each correspond to a TLS maturity state 312 classified by the TLS classification model 350. Thus, the input histology image 110 ( Figure 1 ) can be interchangeably referred to as Figure 7 4. The input histology image 700 of FIG. 4 is an input histology image 700. It is noteworthy that the input histology image 700 includes the classified TLS maturation state 312 but is not annotated as the output image 110A. In some examples, the input histology image 700 corresponds to the entire region of the input histology image 700. In other examples, the input histology image only corresponds to the tumor region 115 detected by the tumor detection model 450 or detected by the morphological image processor 130 within the input histology image 700. Figure 1 ) identifies the TLS zone 135.

[0074] In the example shown, the first input histology image 700, 700a corresponds to a first TLS maturation state 312, 312a indicating a lymphoid aggregate maturation state. In particular, the input histology image 700 corresponding to the first TLS maturation state 312a may include dense aggregates of at least a threshold number of lymphocytes (e.g., 100 lymphocytes) that contain neither high endothelial cell venules nor germinal centers. The second input histology image 700, 700b corresponds to a second TLS maturation state 312, 312b indicating an immature TLS maturation state. The input histology image 700 corresponding to the second TLS maturation state 312b may include dense aggregates of at least a threshold number of lymphocytes (e.g., 100 lymphocytes) that contain high endothelial cell venules (in contrast to the first TLS maturation state 312a) but do not contain any germinal centers. The third input histology image 700, 700c corresponds to a third TLS maturation state 312, 312c indicating a mature TLS maturation state. The input histology image 700 corresponding to the third TLS maturation state 312c may include dense aggregates of at least a threshold number of lymphocytes (eg, 100 lymphocytes) containing high endothelial venules and germinal centers 313 (in contrast to the first and second TLS maturation states 312a, 312b).

[0075] Continue to refer Figure 7, the fourth input histology image 700, 700d exhibits a germinal center 313. In some implementations, the germinal center 313 is not a distinguishable TLS maturation state 312, and the germinal center 313 is a feature of the mature TLS maturation state 312c. In other implementations, the TLS classification model 350 classifies the germinal center 313 as a distinguishable TLS maturation state 312 independent of other TLS maturation states 312. The input histology image 313 with the germinal center 700 includes a lighter, less dense area at the center of a mature TLS (e.g., the third TLS maturation state 312c) surrounded by a dense lymphocyte area. Although not in Figure 7 , but the TLS classification model 350 can also classify a fourth TLS maturation state (not shown) indicating a non-TLS region (e.g., a zero TLS region present in the input histology image 700) or other region. As used herein, the first TLS maturation state 312a, the second TLS maturation state 312b, and the third TLS maturation state 312c can be interchangeably referred to as a lymphoid aggregate TLS maturation state 312a, an immature TLS maturation state 312b, and a mature TLS maturation state 312c, respectively.

[0076] Figure 8-Figure 10 Depicted is the image intensifier 360 ( Figure 1 ) and the corresponding output image (eg, TLS enhanced histology image) 110A. In other words, the application 160 receives Figure 8-Figure 10 135 as an input, and generates an output image 110A (left) as an output. In some examples, the image enhancer 360 generates a corresponding pixel mask 112 that highlights at least the perimeter of the corresponding TLS region 135. In other examples, the corresponding pixel mask 112 highlights the entire area of ​​the corresponding TLS region 135. Thereafter, the image enhancer 360 can generate an output image 110A that enhances the input histology image 110 by superimposing the corresponding pixel mask 112 generated for each TLS region 135 onto the input histology image 110.

[0077] In addition, the image enhancer 360 generates a first pixel mask 112, 112a for each corresponding TLS region 135 classified as the first TLS maturity state 312a, a second pixel mask 112, 112b for each corresponding TLS region 135 classified as the second maturity state 312b, and a third pixel mask 112, 112c for each corresponding TLS region 135 classified as the third maturity state 312c. It is worth noting that each pixel mask 112 is visually distinguishable from the other pixel masks 112, so that the output image 110A visually depicts different maturity states 312 using the visually distinguishable pixel masks 112. In this way, the output image 110A is displayed on the screen 114 of the user device 111 so that the user 10 ( Figure 1 ) can easily visualize the different TLS maturity states 312 included in the output image 110A. Optionally, the image enhancer 360 can generate a fourth pixel mask 112, 112d for each respective TLS region 135 classified as a non-TLS region.

[0078] For example, Figure 8 A graphical representation 800 of an input histology image 110 (right) representing tissue of a mature TLS maturation state 312c and a corresponding output image 110A (left) including a third pixel mask 112c highlighting the area of ​​a TLS region 135 classified as the mature TLS maturation state 312c is shown. However, the mature TLS maturation state 312c includes a germinal center 313 encompassed by the TLS region 135 corresponding to the mature TLS maturation state 312c. To this end, the third pixel mask 112c includes an inner third pixel mask 112c1 highlighting the area of ​​the germinal center 313 and an outer third pixel mask 112c2 highlighting the area of ​​the mature TLS maturation state 312c.

[0079] Fig. 9 A graphical representation 900 of an input histology image 110 (right) of tissue representing an immature TLS maturation state 312b and a corresponding output image 110A (left) including a second pixel mask 112b highlighting the area of ​​the TLS region 135 classified as the immature TLS maturation state 312b is shown. In yet another example, Fig.10A graphical representation 1000 is shown of an input histology image 110 (right) of tissue representing a lymphoid aggregate TLS maturation state 312a and a corresponding output image 110A (left) including a first pixel mask 112a highlighting the area of ​​the TLS region 135 classified as the lymphoid aggregate TLS maturation state 312a. In addition, the output image 110A shown in each of the graphical representations 800, 900, 1000 further includes a fourth pixel mask 112d that highlights the area of ​​the output image 110A corresponding to the non-TLS mature TLS region (e.g., non-TLS maturation state).

[0080] Reference now Fig.11 In some implementations, the input histology image 110 includes several classified TLS maturity states 312. For example, the graphical representation 1100 shows an output image 110A that includes three TLS regions 135 corresponding to each of the first, second, and third TLS maturity states 312a-c. Here, each corresponding pixel mask 112 superimposed on the input histology image easily indicates to the user the different identified TLS regions 135 and the corresponding classified TLS maturity states 312. Fig.11 As shown, the output image 110A includes a first TLS region 135, 135a classified as a lymphatic aggregate TLS maturation state 312a, a second TLS region 135, 135b classified as an immature TLS maturation state 312b, and a third TLS region 135, 135c classified as a mature TLS classification state 312c including a germinal center 313. In addition, below the output image 110A, an expanded view of the identified TLS regions 135 is shown adjacent to the corresponding input histology image 110. For example, the first TLS region 135a includes a first pixel mask 112a highlighting an area of ​​the first TLS region 135a as the lymphatic aggregate TLS maturation state 312a, the second TLS region 135b includes a second pixel mask 112b highlighting an area of ​​the second TLS region 135b as the immature TLS maturation state 312b, and the third TLS region 135c includes a third pixel mask 112c highlighting an area of ​​the third TLS region 135c as the mature TLS maturation state 312c. Immediately adjacent to each expanded TLS region 135 is a corresponding portion of the input histology image 110 input to the application 160 that corresponds to the TLS region 135 .

[0081] Return to reference Figure 1In some implementations, the image enhancer 360 applies one or more post-processing rules 362 before generating the output image 110A. That is, in some scenarios, the TLS classification model 350 classifies the TLS region 135 as a particular TLS maturity state 312 that does not meet a threshold (e.g., a post-processing threshold). Therefore, the post-processing rules 362 are applied to filter out the classified TLS maturity states 312 that fail to meet one or more post-processing rules. In this way, the image enhancer 360 applies the post-processing rules 362 to correct any false positive or otherwise incorrect classifications generated by the TLS classification model 350. For example, the post-processing rules 362 may include, but are not limited to: fixing small and exposed germinal centers, fixing mature TLS without germinal centers, fixing stitching regions to resolve predictions of multiple classes on the same structure due to model confusion, object-level masking to remove false positive predictions of TLS within cancer and necrotic tissue regions, and / or applying a cutoff value.

[0082] Figure 12-Figure 15 The output image 110A generated by the image enhancer 360 with and without applying the post-processing rule 362 is shown. When the image enhancer 360 does not apply the post-processing rule 362, the output image 110A may be referred to as an untransformed output image 110A, 110A1. On the other hand, when the image enhancer 360 applies the post-processing rule 362, the output image 110A may be referred to as a transformed output image 110A, 110A2. For example, Fig.12 A graphical representation 1200 of the output image 110A is shown when the post-processing rules 362 are applied to fix (i.e., filter) small and exposed germinal centers. Fig.12 As shown, the untransformed output image 110A1 includes a germinal center 313 partially surrounded by a TLS region 135, which is classified as a mature TLS maturity state 312c and a non-TLS maturity state represented by its corresponding pixel mask 112. Here, the post-processing rule 362 defines that for a germinal center 313 that fails to meet a threshold amount of TLS region 135 surrounding the germinal center 313 to be classified as a mature TLS maturity state 312c (e.g., 70% of the germinal center 313 is surrounded by mature TLS), the image enhancer 360 reclassifies the germinal center 313 as a TLS maturity state 312 surrounding a majority of the germinal center 313.

[0083] For example, Fig.12As shown, the untransformed output image 110A1 includes an outer third pixel mask 212c2 (e.g., indicating a mature TLS maturation state 312c) that only partially surrounds the inner third pixel mask 212c1 (e.g., indicating a germinal center 313), thereby failing to meet the threshold amount. Therefore, the image enhancer 360 reclassifies the germinal center 313 as a non-TLS maturation state because most of the periphery of the germinal center 313 is surrounded by the non-TLS area. As a result, the transformed output image 110A2 removes (i.e., filters) the germinal center 313, so that the transformed output image 110A2 only includes the fourth pixel mask 112d. Alternatively, the post-processing rule 362 can define a rule that removes (i.e., filters) the germinal center 313 for images having an area that fails to meet the threshold (e.g., 4480 μm 2 ) of an area of ​​a germinal center 313, the image intensifier 360 reclassifies the germinal center 313 into a TLS maturity state 312 that is mostly around the periphery of the germinal center 313.

[0084] Reference now Fig.13 In some implementations, the post-processing rule 362 is configured to repair the classified mature TLS maturation state 312c without the germinal center 313. Here, the image enhancer 360 reclassifies the TLS region classified as the mature TLS maturation state 312c that is not connected to the germinal center 313 as the immature TLS maturation state 312b. For example, the mature TLS maturation state 312c region may need to fully cover the germinal center 313, or partially cover the germinal center 313 to meet the threshold. Fig.13 As shown, the graphical representation 1300 includes an untransformed output image 110A1, which includes a second pixel mask 112b (e.g., indicating an immature TLS maturation state 312b), an inner third pixel mask 112c1 (e.g., indicating a germinal center 313), and an outer third pixel mask 112c2 (e.g., indicating a mature TLS maturation state 312c). In this example, the outer third pixel mask 112c2 fails to cover the germinal center 313 by the threshold. That is, the outer third pixel mask 112c2 only partially covers the germinal center 313, but not enough to meet the threshold. Therefore, in this scenario, the image enhancer 360 reclassifies the mature TLS maturation state 312c and the germinal center 313 as the immature TLS maturation state 312b, as shown in the transformed output image 110A2 with the second pixel mask 112b. The output image 110A also includes a fourth pixel mask 112d corresponding to a non-TLS area of ​​the output image 110.

[0085] Reference now Fig.14In some examples, the post-processing rule 362 is configured to repair the stitched region 1402 included in the output image 110A. Here, the stitched region 1402 refers to a single TLS region 135 including a plurality of classified TLS maturation states 312. In some configurations, when the stitched region 1402 includes at least an immature TLS maturation state 312b and a lymphatic aggregate TLS maturation state 312a, the image enhancer 360 reclassifies the entire stitched region 1402 as the immature TLS maturation state 312b based on a threshold ratio (e.g., 70%) of determining that the stitched region 1402 includes the immature TLS maturation state 312b. Otherwise, the image enhancer 360 reclassifies the entire stitched region 1402 as the lymphatic aggregate TLS maturation state 312a. In other configurations where the stitched region 1402 includes at least an immature TLS maturity state 312b and a mature TLS maturity state 312c, the image enhancer 360 reclassifies the entire stitched region 1402 as the mature TLS maturity state 312c based on determining that the stitched region includes a threshold ratio (e.g., 70%) of mature TLS maturity states 312c. Otherwise, the image enhancer 360 reclassifies the entire stitched region 1402 as the immature TLS maturity state 312b. In yet another configuration where the stitched region 1402 includes at least a lymphatic aggregate TLS maturity state 312a and a mature TLS maturity state 312c, the image enhancer 360 reclassifies the entire stitched region 1402 as the mature TLS maturity state 312c based on determining that the stitched region 1402 includes a threshold ratio (e.g., 70%) of mature TLS maturity states 312c. Otherwise, the image enhancer 360 reclassifies the entire stitched region 1402 as the lymphatic aggregate TLS maturity state 312a.

[0086] like Fig.14 As shown, the graphical representation 1400 includes an untransformed output image 110A1 depicting a stitched region 1402 including a first pixel mask 112a (e.g., indicating a lymphatic aggregate TLS maturation state 312a) and a second pixel mask 112b (e.g., indicating an immature TLS maturation state 312b). Here, the stitched region 1402 does not meet the threshold ratio for the immature TLS maturation state 312b. As such, the image enhancer 360 reclassifies the entire area of ​​the stitched region 1402 as a lymphatic aggregate maturation state 312a, as shown in the transformed output 110A2 including the first pixel mask 112a. Notably, the transformed output 110A2 eliminates the stitched region 1402 because the TLS region includes only a single TLS maturation state 312. The output image 110A also includes a fourth pixel mask 112d corresponding to a non-TLS region of the output image 110.

[0087] Reference now Fig.15In some implementations, the post-processing rules 362 are configured to remove false positive predictions of TLS maturity states within the cancerous and necrotic tissue regions 312. In particular, the image enhancer 360 determines whether a ratio of cancer and necrosis in an object or TLS region classified as a first, second, or third TLS maturity state 312a, 312b, 312c satisfies a threshold ratio (e.g., 20%) for the object or TLS region. In response to determining that the ratio of cancer and necrosis satisfies the threshold ratio, the image enhancer 360 reclassifies the TLS maturity state 312 as a non-TLS maturity state. For example, if Fig.15 As shown, graphical representation 1500 includes an untransformed output image 110A1 that includes a cancer pixel mask 1502 and a necrotic pixel mask 1504. In this example, cancer pixel mask 1502 and necrotic pixel mask 1504 meet the threshold ratio of tissue, and therefore, image enhancer 360 reclassifies cancer pixel mask 1502 and necrotic pixel mask 1504 as non-TLS maturity state 312d. Therefore, the transformed output image 110A2 includes only a fourth pixel mask 112d corresponding to the non-TLS region of the transformed output image 110A.

[0088] Reference now Fig.16 In some examples, the post-processing rules 362 are configured to apply a cutoff value that filters the classified TLS maturation states 312 that fail to meet a minimum threshold area, a maximum threshold area, and / or a maximum number of germinal centers 313. TLS maturation states 312 that fail to meet the threshold are reclassified as non-TLS areas. In particular, the lymphoid aggregate TLS maturation state 312a may have a minimum threshold area (e.g., 0.0008 mm 2 ) and no maximum threshold area. On the other hand, the immature TLS maturity state 312b may include a minimum threshold area (e.g., 0.018 mm 2 ) and the maximum threshold area (e.g., 2.0 mm 2 ). In addition, the mature TLS maturity state 312c may have a maximum threshold number of germinal centers 313 (e.g., 8 germinal centers 313). For example, if the mature TLS maturity state 312c includes a number of germinal centers 313 exceeding the maximum threshold number, the image enhancer 360 reclassifies the mature TLS maturity state 312c as a non-TLS region. Fig.16As shown, the graphical representation 1600 includes an untransformed output image 110A1, which includes a first pixel mask 112a, a second pixel mask 112b, an inner third pixel mask 112c1, an outer third pixel mask 112c2, and a fourth pixel mask 112d. However, none of the pixel masks 112 meet the cutoff threshold, and therefore, the image enhancer 360 reclassifies each of the first, second, and third TLS maturity states 312a-c into the fourth TLS classification state 312d, as shown in the transformed output image 110A2. That is, the transformed output image 110A2 only includes the fourth TLS classification state 312d.

[0089] Fig.17 A process flow chart 1700 is shown for validating the extracted TLS signature 140 using RNA sequence analysis or transcriptome analysis correlation. That is, various gene markers of TLS associated with chemokines or cell populations have been studied. For example, Fig.18 Table 1800 of 12-chemokine gene signatures is shown, which are obtained by associating metagenes associated with inflammation and with enhanced patient survival in colorectal cancer, melanoma, and breast cancer. The 12-chemokine gene signatures of Table 1800 include CCL2, CCL3, CCL4, CCL5, CCL8, CCL18, CCL19, CCL21, CXCL9, CXCL10, CXCL11, and CXCL13. In another example, an 8-gene signature representing follicular helper T (TFH) cells characterizes breast cancer, and the 8-gene signature particularly includes CXCL13. In yet another example, a 19-gene signature associated with type 1 helper T (Th1) cells and B cells indicates the presence of TLS. Although there are various gene signatures associated with the presence of TLS, the number of studies that have recently studied the most accurate TLS gene signatures is limited.

[0090] Return to reference Fig.17 , while TLS detection by immunohistochemistry in tissue sections is a robust and specific method, the process flow chart 1700 is intended to compare several gene signatures extracted from TLS positive cancer tissues. As will become clear, the heterogeneity of gene expression among different cancer types leads to a better understanding of gene signatures associated with the presence of TLS. In particular, the process flow chart 1700 includes a TLS feature extraction module 320, a transcriptomics module 1710, a feature selector 1720, and a clustering module 1730. The TLS feature extraction module 320 is configured to receive an input histology image 110 as input and extract TLS features 140 corresponding to each corresponding input histology image 110. For example, the TLS feature extraction module 320 can use a TLS feature extractor 145 ( Figure 1) to extract TLS features 140.

[0091] The transcriptomics module 1710 is configured to receive an input histology image 110 as an input and generate a gene expression signature (GES) 1712 as an output for each corresponding input histology image 110. Here, the transcriptomics module 1710 can generate the GES 1712 by extracting RNA sequences from the corresponding input histology image 110. Using the TLS features 140 and the GES 1712 generated for each input histology image 110, the feature selector 1720 generates a feature table 1722. That is, for each corresponding input histology image 110, the feature extractor 1720 pairs the TLS features 140 with the GES 172 derived from the corresponding input histology image 110 in the feature table 1722. The feature table 1722 includes the pairings of all received input histology images 110. In this way, the feature table 1722 constructs the TLS features 140 and the GES 1712 so that the clustering module 1730 can determine the correlation between the TLS features and the GES 1712. In some examples, the feature table 1722 includes other TLS features 140 and a corresponding number of annotations for each TLS feature 140 in the input histology image set, such as table 1900 ( Fig.19 ). In some implementations, the feature selector 1720 may filter the feature table 1722 to include only specific TLS features 140. For example, the feature selector 1720 may apply a linear regression lasso penalty to generate the feature table 1722.

[0092] Continue to refer Fig.17 , the clustering module 1730 is configured to receive the clustering table 1722 as input and generate correlation data 1732 as output. Notably, using the gene marker data indicating the presence and classification of TLS, the clustering module 1730 can verify that the extracted TLS features 140 are correlated with the presence and classification of TLS in the input histology image 110. In addition, the clustering module 1730 can utilize the extracted TLS features 140 to further determine the gene markers indicating the presence and classification of TLS in the tissue. That is, the clustering module 1730 can further determine the gene markers that can identify the TLS that is not yet known.

[0093] For example, Figure 20A-20C Example correlation data 1732 is shown that verifies that TLS signature 140 is strongly correlated with GES in an example breast cancer gene (BRCA) analysis ( Fig.17 ) of the graphical representation 200. In particular, the graphical representation 2000a ( Fig. 20A) shows a correlation graph 2002 showing that the TLS maturation states 312 and the TLS signatures 140 correspond to the TLS induced genes shown in table 2004 in the BRCA analysis. In addition, the correlation graph 2002 depicts the TLS induced genes that appear in each of the first, second, and third TLS maturation states 312a-c and the TLS induced genes associated with the individual TLS maturation states 312. Fig.17 ) Further processing of the correlation graph 2002 can generate signatures for certain cancers. In short, the correlation graph 2002 highlights that the TLS features 140 are strongly correlated with the GES of the input histology image 110 for BRCA.

[0094] Fig. 20B A graphical representation of a hierarchical clustering plot 2000b is shown. Here, the plot includes cluster 1 corresponding to low expressing breast cancer samples, cluster 2 corresponding to medium expressing breast cancer samples, and cluster 3 corresponding to high expressing breast cancer samples. Along the y-axis, the first, second and third TLS maturation states 312a-c and TLS induced genes are plotted for each breast sample cluster. Fig. 20C A graphical representation 2000c is shown depicting a plot of overall survival rates of breast cancer samples from patients with the x-axis being a timeline in months and the y-axis being 10. Thus, the graphical representation 2000c shows that breast cancer samples in clusters with upregulated chemokines have higher long-term overall survival rates.

[0095] Figure 21A-Figure 21D Graphical representations 2100 of correlation plots validating the extracted TLS signatures 140 with gene signatures are shown. The graphical representations 2100 correlate the TLS signatures 140 and gene signatures between different cancer types (x-axis), including BRCA, bladder cancer (BLCA), lung adenocarcinoma (LUAD), lung squamous cell carcinoma (LUSC), and gastric adenocarcinoma (STAD). In addition, each graphical representation 2100 plots TLS-induced genes along the y-axis. For example, graphical representation 2100a ( Fig.21A ) includes a TLS feature 140 of a proportional area of ​​a mature TLS maturity state 312c, graphically represented 2100b ( Fig.21B ) includes a TLS feature 140 of a proportional area of ​​an immature TLS maturity state 312b, and a graphical representation 2100c ( Fig. 21C ) includes a TLS feature 140 of the proportional area of ​​the lymphoid aggregate TLS maturation state 312c. Figure 2DC shows a graphical representation 2100d depicting a plot of the overall survival of patients from LUAD cancer samples and BRCA cancer samples as a timeline in months on the x-axis and the y-axis. Thus, the graphical representation 2100d shows that the proportional areas 312 of different TLS maturation states are associated with a subset of TLS-induced genes, and in particular, the proportional areas of mature TLS maturation states 312c demonstrate prognostic value in LUAD and BRCA samples. Thus, the graphical representation 2000c shows that breast cancer samples in clusters with upregulated chemokines have higher long-term overall survival.

[0096] Anti-PD-1 antibodies known in the art can be used in the compositions and methods described herein. A variety of human monoclonal antibodies that specifically bind to PD-1 with high affinity have been disclosed in U.S. Pat. No. 8,008,449. The anti-PD-1 human antibodies disclosed in U.S. Pat. No. 8,008,449 have been shown to exhibit one or more of the following characteristics: (a) a 1 x 10 -7 M or smaller K D Binds to human PD-1 as determined by surface plasmon resonance using a Biacore biosensor system; (b) does not substantially bind to human CD28, CTLA-4, or ICOS; (c) increases T cell proliferation in a mixed lymphocyte reaction (MLR) assay; (d) increases interferon-γ production in an MLR assay; (e) increases IL-2 secretion in an MLR assay; (f) binds to human PD-1 and cynomolgus monkey PD-1; (g) inhibits the binding of PD-L1 and / or PD-L2 to PD-1; (h) stimulates an antigen-specific memory response; (i) stimulates an antibody response; and (j) inhibits tumor cell growth in vivo. Anti-PD-1 antibodies useful in the present disclosure include monoclonal antibodies that specifically bind to human PD-1 and exhibit at least one, and in some embodiments at least five, of the foregoing characteristics.

[0097] Other anti-PD-1 monoclonal antibodies have been described, for example, in U.S. Patent Nos. 6,808,710, 7,488,802, 8,168,757, and 8,354,509, U.S. Publication No. 2016 / 0272708, and PCT Publication Nos. WO 2012 / 145493, WO 2008 / 156712, WO 2015 / 112900, WO 2012 / 145493, WO 2015 / 112800, WO 2014 / 206107, WO 2015 / 35606, WO 2015 / 085847, WO 2014 / 179664, WO 2017 / 020291, WO 2017 / 020858, WO 2016 / 197367, WO 2017 / 024515, WO 2017 / 025051, WO 2017 / 123557, WO 2016 / 106159, WO 2014 / 194302, WO 2017 / 040790, WO 2017 / 133540, WO 2017 / 132827, WO 2017 / 024465, WO 2017 / 025016, WO 2017 / 106061, WO 2017 / 19846, WO 2017 / 024465, WO 2017 / 025016, WO 2017 / 132825 and WO 2017 / 133540, each of which is incorporated by reference in its entirety.

[0098] In some implementations, the anti-PD-1 antibody is selected from nivolumab (also known as 5C4, BMS-936558, MDX-1106, and ONO-4538), pembrolizumab (Merck; also known as pembrolizumab and MK-3475; see WO 2008 / 156712), PDR001 (Novartis; see WO 2015 / 112900), MEDI-0680 (AstraZeneca; also known as AMP-514; see WO 2012 / 145493), cemiplizumab (Regeneron; also known as REGN-2810; see WO 2015 / 112800), JS001 (TAIZHOU JUNSHIPHARMA; also known as toripalimab; see Si-Yang Liu et al., J. Hematol. Oncol. 10:136 (2017)), BGB-A317 (Beigene; also known as tislelizumab; see WO 2015 / 35606 and US 2015 / 0079109), INCSHR1210 (Jiangsu Hengrui Medicine; also known as SHR-1210; see WO 2015 / 085847; Si-Yang Liu et al., J. Hematol. Oncol. 10:136 (2017)), TSR-042 (Tesaro Biopharmaceutical; also known as ANB011; see WO 2014 / 179664), GLS-010 (Wuxi / Harbin Gloria Pharmaceuticals; also known as WBP3055; see Si-Yang Liu et al., J. Hematol. Oncol. 10:136 (2017)), AM-0001 (Armo), STI-1110 (Sorrento Therapeutics; see WO 2014 / 194302), AGEN2034 (Agenus; see WO 2017 / 040790), MGA012 (Macrogenics, see WO 2017 / 19846), BCD-100 (Biocad; Kaplon et al., mAbs 10(2):183-203 (2018), and IBI308 (Innovent; see WO 2017 / 024465, WO 2017 / 025016, WO 2017 / 132825 and WO 2017 / 133540).

[0099] Nivolumab is a fully human IgG4 (S228P) PD-1 immune checkpoint inhibitor antibody that selectively blocks interaction with PD-1 ligands (PD-L1 and PD-L2), thereby blocking downregulation of anti-tumor T cell function (U.S. Pat. No. 8,008,449; Wang et al., 2014 Cancer Immunol Res. 2 (9): 846-56). Pembrolizumab is a humanized monoclonal IgG4 (S228P) antibody against the human cell surface receptor PD-1 (programmed death protein-1 or programmed cell death protein-1). Pembrolizumab is described in, for example, U.S. Pat. Nos. 8,354,509 and 8,900,587.

[0100] Anti-PD-1 antibodies that can be used in the disclosed compositions and methods also include isolated antibodies that specifically bind to human PD-1 and cross-compete with any anti-PD-1 antibody disclosed herein (e.g., nivolumab) for binding to human PD-1 (see, e.g., U.S. Pat. Nos. 8,008,449 and 8,779,105; WO 2013 / 173223). In some embodiments, the anti-PD-1 antibody binds to the same epitope as any anti-PD-1 antibody described herein (e.g., nivolumab). The ability of antibodies to cross-compete for binding to antigens indicates that these monoclonal antibodies bind to the same epitope region of the antigen and sterically hinder the binding of other cross-competing antibodies to that particular epitope region. It is expected that these cross-competing antibodies have very similar functional properties to the reference antibody (e.g., nivolumab) because they bind to the same epitope region of PD-1. Cross-competing antibodies can be readily identified based on their ability to cross-compete with nivolumab in standard PD-1 binding assays such as Biacore analysis, ELISA assays, or flow cytometry (see, e.g., WO 2013 / 173223).

[0101] In some implementations, the antibody that cross-competes with nivolumab for binding to human PD-1 or binds to the same epitope region of the human PD-1 antibody as nivolumab is a monoclonal antibody. For administration to human subjects, these cross-competing antibodies are chimeric antibodies, engineered antibodies, or humanized antibodies or human antibodies. Such chimeric, engineered, humanized or human monoclonal antibodies can be prepared and isolated by methods well known in the art.

[0102] Anti-PD-1 antibodies useful in the compositions and methods of the present disclosure also include antigen-binding portions of the above antibodies. It has been well documented that the antigen-binding function of an antibody can be performed by fragments of a full-length antibody.

[0103] Anti-PD-1 antibodies suitable for use in the disclosed compositions and methods are antibodies that bind to PD-1 with high specificity and affinity, block the binding of PD-L1 and / or PD-L2, and inhibit the immunosuppressive effects of the PD-1 signaling pathway. In any composition or method disclosed herein, the anti-PD-1 "antibody" includes an antigen binding portion or fragment that binds to the PD-1 receptor and exhibits functional properties similar to those of the intact antibody in terms of inhibiting ligand binding and upregulating the immune system. In certain embodiments, the anti-PD-1 antibody or antigen binding portion thereof cross-competes with nivolumab for binding to human PD-1.

[0104] In some examples, the anti-PD-1 antibody is administered once every 2, 3, 4, 5, 6, 7 or 8 weeks at a dose ranging from 0.1 mg / kg to 20.0 mg / kg body weight, for example, once every 2, 3 or 4 weeks at 0.1 mg / kg to 10.0 mg / kg body weight. In other embodiments, the anti-PD-1 antibody is administered once every 2 weeks at a dose of about 2 mg / kg, about 3 mg / kg, about 4 mg / kg, about 5 mg / kg, about 6 mg / kg, about 7 mg / kg, about 8 mg / kg, about 9 mg / kg or 10 mg / kg body weight. In other embodiments, the anti-PD-1 antibody is administered once every 3 weeks at a dose of about 2 mg / kg, about 3 mg / kg, about 4 mg / kg, about 5 mg / kg, about 6 mg / kg, about 7 mg / kg, about 8 mg / kg, about 9 mg / kg or 10 mg / kg body weight. In one embodiment, the anti-PD-1 antibody is administered approximately once every 3 weeks at a dose of about 5 mg / kg body weight. In another embodiment, the anti-PD-1 antibody (e.g., nivolumab) is administered approximately once every 2 weeks at a dose of about 3 mg / kg body weight. In other embodiments, the anti-PD-1 antibody (e.g., pembrolizumab) is administered approximately once every 3 weeks at a dose of about 2 mg / kg body weight.

[0105] The anti-PD-1 antibodies useful in the present disclosure may be administered in a flat dose. In some embodiments, the anti-PD-1 antibody is administered in a flat dose of from about 100 to about 1000 mg, from about 100 mg to about 900 mg, from about 100 mg to about 800 mg, from about 100 mg to about 700 mg, from about 100 mg to about 600 mg, from about 100 mg to about 500 mg, from about 200 mg to about 1000 mg, from about 200 mg to about 900 mg, from about 200 mg to about 800 mg, from about 200 mg to about 700 mg, from about 200 mg to about 600 mg, from about 200 mg to about 500 mg, from about 200 mg to about 480 mg, or from about 240 mg to about 480 mg. In one embodiment, the anti-PD-1 antibody is administered at a flat dose of at least about 200 mg, at least about 220 mg, at least about 240 mg, at least about 260 mg, at least about 280 mg, at least about 300 mg, at least about 320 mg, at least about 340 mg, at least about 360 mg, at least about 380 mg, at least about 400 mg, at least about 420 mg, at least about 440 mg, at least about 460 mg, at least about 480 mg, at least about 500 mg, at least about 520 mg, at least about 540 mg, at least about 550 mg, at least about 560 mg, at least about 580 mg, at least about 600 mg, at least about 620 mg, at least about 640 mg, at least about 660 mg, at least about 680 mg, at least about 700 mg, or at least about 720 mg at a dosing interval of about 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10 weeks. In another embodiment, the anti-PD-1 antibody is administered at a dosing interval of about 1, 2, 3, or 4 weeks at a flat dose of about 200 mg to about 800 mg, about 200 mg to about 700 mg, about 200 mg to about 600 mg, about 200 mg to about 500 mg.

[0106] In some implementations, the anti-PD-1 antibody is administered approximately once every 3 weeks at a flat dose of about 200 mg. In other embodiments, the anti-PD-1 antibody is administered approximately once every 2 weeks at a flat dose of about 200 mg. In other embodiments, the anti-PD-1 antibody is administered approximately once every 2 weeks at a flat dose of about 240 mg. In certain embodiments, the anti-PD-1 antibody is administered approximately once every 4 weeks at a flat dose of about 480 mg.

[0107] In some additional implementations, nivolumab is administered approximately once every 2 weeks at a flat dose of about 240 mg. In some embodiments, nivolumab is administered approximately once every 3 weeks at a flat dose of about 240 mg. In some embodiments, nivolumab is administered approximately once every 3 weeks at a flat dose of about 360 mg. In some embodiments, nivolumab is administered approximately once every 4 weeks at a flat dose of about 480 mg.

[0108] Alternatively, pembrolizumab can be administered approximately once every 2 weeks at a flat dose of about 200 mg. In some embodiments, pembrolizumab is administered approximately once every 3 weeks at a flat dose of about 200 mg. In some embodiments, pembrolizumab is administered approximately once every 4 weeks at a flat dose of about 400 mg.

[0109] In some aspects, the PD-1 inhibitor is a small molecule. In some aspects, the PD-1 inhibitor includes a millamolecule. In some aspects, the PD-1 inhibitor includes a macrocyclic peptide. The PD-1 inhibitor may include BMS-986189. In some other aspects, the PD-1 inhibitor includes an inhibitor disclosed in International Publication No. WO 2014 / 151634, which is incorporated herein by reference in its entirety. In some aspects, the PD-1 inhibitor includes INCMGA00012 (Incyte Corporation). In some aspects, the PD-1 inhibitor includes a combination of an anti-PD-1 antibody disclosed herein and a PD-1 small molecule inhibitor.

[0110] In some implementations, in any of the methods disclosed herein, an anti-PD-L1 antibody is substituted for an anti-PD-1 antibody. Anti-PD-L1 antibodies known in the art can be used in the compositions and methods of the present disclosure. Examples of anti-PD-L1 antibodies that can be used in the compositions and methods of the present disclosure include antibodies disclosed in U.S. Pat. No. 9,580,507. The anti-PD-L1 human monoclonal antibodies disclosed in U.S. Pat. No. 9,580,507 have been shown to exhibit one or more of the following characteristics: (a) a 1 x 10 -7 M or smaller K DBind to human PD-L1 as determined by surface plasmon resonance using a Biacore biosensor system; (b) increase T cell proliferation in a mixed lymphocyte reaction (MLR) assay; (c) increase interferon-γ production in an MLR assay; (d) increase IL-2 secretion in an MLR assay; (e) stimulate antibody responses; and (f) reverse the effects of T regulatory cells on T cell effector cells and / or dendritic cells. Anti-PD-L1 antibodies useful in the present disclosure include monoclonal antibodies that specifically bind to human PD-L1 and exhibit at least one, in some embodiments at least five of the foregoing characteristics.

[0111] The anti-PD-L1 antibody may be selected from: BMS-936559 (also known as 12A4, MDX-1105; see, e.g., U.S. Pat. No. 7,943,743 and WO 2013 / 173223), atezolizumab (Roche; also known as MPDL3280A, RG7446; see US 8,217,149; see also Herbst et al. (2013) J Clin Oncol 31(Suppl):3000), durvalumab (AstraZeneca; also known as IMFINZI TM , MEDI-4736; see WO 2011 / 066389), Avelumab (Pfizer; also known as MSB-0010718C; see WO 2013 / 079174), STI-1014 (Sorrento; see WO 2013 / 181634), CX-072 (Cytomx; see WO 2016 / 149201), KN035 (3DMed / Alphamab; see Zhang et al., Cell Discov. 7:3 (March 2017)), LY3300054 (Eli Lilly Co.; see, e.g., WO 2017 / 034916), BGB-A333 (BeiGene; see Desai et al., JCO 36(15 Suppl):TPS3113 (2018)), and CK-301 (Checkpoint Therapeutics; see Gorelik et al., AACR: Abstract 4606 (April 2016)).

[0112] Atezolizumab is a fully humanized IgG1 monoclonal anti-PD-L1 antibody. Durvalumab is a human IgG1κ monoclonal anti-PD-L1 antibody. Avelumab is a human IgG1λ monoclonal anti-PD-L1 antibody. Anti-PD-L1 antibodies that can be used in the disclosed compositions and methods also include isolated antibodies that specifically bind to human PD-L1 and cross-compete with any anti-PD-L1 antibody disclosed herein (e.g., atezolizumab, durvalumab and / or avelumab) for binding to human PD-L1. In some embodiments, the anti-PD-L1 antibody binds to the same epitope as any anti-PD-L1 antibody described herein (e.g., atezolizumab, durvalumab and / or avelumab). The ability of antibodies to cross-compete for binding to antigens indicates that these antibodies bind to the same epitope region of the antigen and sterically hinder the binding of other cross-competing antibodies to that particular epitope region. These cross-competing antibodies are expected to have very similar functional properties to the reference antibodies (e.g., atezolizumab and / or avelumab) because they bind to the same epitope region of PD-L1. Cross-competing antibodies can be easily identified based on their ability to cross-compete with atezolizumab and / or avelumab in standard PD-L1 binding assays (such as Biacore analysis, ELISA assays, or flow cytometry) (see, e.g., WO 2013 / 173223).

[0113] Antibodies that cross-compete with atezolizumab, durvalumab and / or avelumab for binding to human PD-L1 or bind to the same epitope region of human PD-L1 antibodies as atezolizumab, durvalumab and / or avelumab are monoclonal antibodies. For administration to human subjects, these cross-competing antibodies are chimeric, engineered, or humanized or human antibodies. Such chimeric, engineered, humanized or human monoclonal antibodies can be prepared and isolated by methods well known in the art.

[0114] Anti-PD-L1 antibodies useful in the compositions and methods of the disclosed disclosure also include antigen-binding portions of the above antibodies. It has been well documented that the antigen-binding function of an antibody can be performed by fragments of a full-length antibody.

[0115] Anti-PD-L1 antibodies suitable for use in the disclosed compositions and methods are antibodies that bind to PD-L1 with high specificity and affinity, block the binding of PD-1, and inhibit the immunosuppressive effects of the PD-1 signaling pathway. In any composition or method disclosed herein, an anti-PD-L1 "antibody" includes an antigen binding portion or fragment that binds to PD-L1 and exhibits functional properties similar to those of a complete antibody in terms of inhibiting receptor binding and upregulating the immune system. In certain embodiments, the anti-PD-L1 antibody or antigen binding portion thereof cross-competes with atezolizumab, durvalumab, and / or avelumab for binding to human PD-L1.

[0116] The anti-PD-L1 antibodies useful in the present disclosure can be any PD-L1 antibody that specifically binds to PD-L1, such as an antibody that cross-competes with durvalumab, avelumab, or atezolizumab for binding to human PD-1, such as an antibody that binds to the same epitope as durvalumab, avelumab, or atezolizumab. In certain embodiments, the anti-PD-L1 antibody is durvalumab. In other embodiments, the anti-PD-L1 antibody is avelumab. In some embodiments, the anti-PD-L1 antibody is atezolizumab.

[0117] In some embodiments, the anti-PD-L1 antibody is administered about once every 2, 3, 4, 5, 6, 7 or 8 weeks at a dose ranging from about 0.1 mg / kg to about 20.0 mg / kg body weight, about 2 mg / kg, about 3 mg / kg, about 4 mg / kg, about 5 mg / kg, about 6 mg / kg, about 7 mg / kg, about 8 mg / kg, about 9 mg / kg, about 10 mg / kg, about 11 mg / kg, about 12 mg / kg, about 13 mg / kg, about 14 mg / kg, about 15 mg / kg, about 16 mg / kg, about 17 mg / kg, about 18 mg / kg, about 19 mg / kg or about 20 mg / kg.

[0118] The anti-PD-L1 antibody can be administered approximately once every 3 weeks at a dose of about 15 mg / kg body weight. In other embodiments, the anti-PD-L1 antibody is administered approximately once every 2 weeks at a dose of about 10 mg / kg body weight.

[0119] In some scenarios, the anti-PD-L1 antibodies that can be used in the present disclosure are flat doses. In some embodiments, the anti-PD-L1 antibody is administered in a flat dose of about 200 mg to about 1600 mg, about 200 mg to about 1500 mg, about 200 mg to about 1400 mg, about 200 mg to about 1300 mg, about 200 mg to about 1200 mg, about 200 mg to about 1100 mg, about 200 mg to about 1000 mg, about 200 mg to about 900 mg, about 200 mg to about 800 mg, about 200 mg to about 700 mg, about 200 mg to about 600 mg, about 700 mg to about 1300 mg, about 800 mg to about 1200 mg, about 700 mg to about 900 mg, or about 1100 mg to about 1300 mg. In some embodiments, the anti-PD-L1 antibody is administered at a flat dose of at least about 240 mg, at least about 300 mg, at least about 320 mg, at least about 400 mg, at least about 480 mg, at least about 500 mg, at least about 560 mg, at least about 600 mg, at least about 640 mg, at least about 700 mg, at least about 720 mg, at least about 800 mg, at least about 840 mg, at least about 880 mg, at least about 900 mg, at least 960 mg, at least about 1000 mg, at least about 1040 mg, at least about 1100 mg, at least about 1120 mg, at least about 1200 mg, at least about 1280 mg, at least about 1300 mg, at least about 1360 mg, or at least about 1400 mg at a dosing interval of about 1, 2, 3, or 4 weeks. In some embodiments, the anti-PD-L1 antibody is administered at a flat dose of about 1200 mg approximately once every 3 weeks. In other embodiments, the anti-PD-L1 antibody is administered approximately once every 2 weeks at a flat dose of about 800 mg. In other embodiments, the anti-PD-L1 antibody is administered approximately once every 2 weeks at a flat dose of about 840 mg.

[0120] Atezolizumab is administered approximately once every 3 weeks at a flat dose of about 1200 mg. In some examples, atezolizumab is administered approximately once every 2 weeks at a flat dose of about 800 mg. In other examples, atezolizumab is administered approximately once every 2 weeks at a flat dose of about 840 mg. Optionally, avelumab can be administered approximately once every 2 weeks at a flat dose of about 800 mg.

[0121] In some examples, durvalumab is administered approximately once every 2 weeks at a dose of about 10 mg / kg. In other examples, durvalumab is administered approximately once every 2 weeks at a flat dose of about 800 mg / kg. Durvalumab may optionally be administered approximately once every 3 weeks at a flat dose of about 1200 mg / kg.

[0122] The PD-L1 inhibitor may include a small molecule or a Mira molecule. The PD-L1 inhibitor may include a macrocyclic peptide. In some implementations, the PD-L1 inhibitor includes BMS-986189. The PD-L1 inhibitor may include a Mira molecule having the following formula:

[0123] Where R1-R13 are amino acid side chains, R a -R n is hydrogen, methyl or forms a ring with an adjacent R group, and R14 is -C(O)NHR15, wherein R15 is hydrogen or a glycine residue, and the glycine residue is optionally substituted with another glycine residue and / or tail that can improve the pharmacokinetic properties. In some aspects, the PD-L1 inhibitor includes compounds disclosed in International Publication No. WO2014 / 151634, which is incorporated herein by reference in its entirety. In some aspects, the PD-L1 inhibitor includes compounds disclosed in International Publication Nos. WO 2016 / 039749, WO 2016 / 149351, WO 2016 / 077518, WO 2016 / 100285, WO 2016 / 100608, WO 2016 / 126646, WO 2016 / 057624, WO 2017 / 151830, WO 2017 / 176608, WO 2018 / 085750, WO 2018 / 237153, or WO 2019 / 070643, each of which is incorporated herein by reference in its entirety.

[0124] The PD-L1 inhibitor includes a small molecule PD-L1 inhibitor disclosed in: International Publication Nos. WO 2015 / 034820, WO 2015 / 160641, WO 2018 / 044963, WO 2017 / 066227, WO 2018 / 009505, WO 2018 / 183171, WO 2018 / 118848, WO 2019 / 147662 or WO 2019 / 169123, each of which is incorporated herein by reference in its entirety. In some implementations, the PD-L1 inhibitor includes a combination of an anti-PD-L1 antibody disclosed herein and a PD-L1 small molecule inhibitor disclosed herein.

[0125] A software application (i.e., a software resource) may refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0126] Non-transitory memory can be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0127] Fig. 22 2200 that can be used to implement the systems and methods described in this document. Computing device 2200 is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are exemplary only and are not meant to limit implementations of the inventions described and / or claimed in this document.

[0128] The computing device 2200 includes a processor 2210, a memory 2220, a storage device 2230, a high-speed interface / controller 2240 connected to the memory 2220 and a high-speed expansion port 2250, and a low-speed interface / controller 2260 connected to a low-speed bus 2270 and the storage device 2230. Each of the components 2210, 2220, 2230, 2240, 2250, and 2260 is interconnected using various buses and can be mounted on a common motherboard or in other appropriate ways. The processor 2210 can process instructions for execution within the computing device 2200, including instructions stored in the memory 2220 or on the storage device 2230, to display graphical information of a graphical user interface (GUI) on an external input / output device (such as a display 2280 coupled to the high-speed interface 2240). In other implementations, multiple processors and / or multiple buses and multiple memories and memory types can be used as appropriate. Furthermore, multiple computing devices 2200 may be connected, with each device providing portions of the necessary operations (eg, as a server bank, a group of blade servers, or a multi-processor system).

[0129] The memory 2220 stores information non-temporarily within the computing device 2200. The memory 2220 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-temporary memory 2220 may be a physical device for storing programs (e.g., instruction sequences) or data (e.g., program state information) on a temporary or permanent basis for use by the computing device 2200. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0130] Storage device 2230 can provide mass storage for computing device 2200. In some implementations, storage device 2230 is a computer readable medium. In various implementations, storage device 2230 can be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a device array, including a device in a storage area network or other configuration. In another implementation, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine readable medium, such as a memory on memory 2220, storage device 2230, or processor 2210.

[0131] The high-speed controller 2240 manages bandwidth-intensive operations of the computing device 2200, while the low-speed controller 2260 manages less bandwidth-intensive operations. This division of responsibilities is exemplary only. In some implementations, the high-speed controller 2240 is coupled to the memory 2220, the display 2280 (e.g., through a graphics processor or accelerator), and to the high-speed expansion port 2250 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 2260 is coupled to the storage device 2230 and the low-speed expansion port 2290. The low-speed expansion port 2290, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device (such as a switch or router), for example, through a network adapter.

[0132] As shown, computing device 2200 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 2200a or multiple times in a group of such servers 2200a as a laptop computer 2200b or part of a rack server system 2200c.

[0133] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include being implemented in one or more computer programs executable and / or readable on a programmable system comprising at least one programmable processor, at least one input device, and at least one output device, the programmable processor being either special-purpose or general-purpose, coupled to receive data and instructions from a storage system and to transmit data and instructions to the storage system.

[0134] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0135] The process and logic flow described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), which execute one or more computer programs to perform functions by operating input data and generating output. The process and logic flow can also be performed by a dedicated logic circuit system (for example, FPGA (field programmable gate array) or ASIC (application-specific integrated circuit)). Processors suitable for executing computer programs include, for example, general-purpose microprocessors and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer include a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (for example, disks, magneto-optical disks, or optical disks) for storing data, or be operably coupled to receive data from a mass storage device or transmit data to a mass storage device, or both. However, the computer need not have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, by way of example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated in a special purpose logic circuit system.

[0136] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen) for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, speech, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user (e.g., by sending a web page to a web browser in response to a request received from a web browser on a client device of the user).

[0137] Many implementations have been described. However, it will be appreciated that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other implementations are within the scope of the following claims.

Claims

1. A computer-implemented method (400) executed on data processing hardware (142), the method causing the data processing hardware (142) to perform operations, the operations comprising: receiving an input histology image (110) for a patient diagnosed with cancer, the input histology image (110) comprising a plurality of image pixels; processing the input histology image (110) using a cell classification model (550) to generate one or more lymphocyte density maps (125) within the input histology image (110); performing morphological image processing (130) on the one or more lymphocyte density maps (125) to identify one or more TLS regions (135) within the input histology image (110), each TLS region (135) being represented by a corresponding lymphocyte cluster; as well as For each corresponding TLS region (110) of the one or more TLS regions (135) identified in the input histology image (135): extracting a corresponding set of TLS features (140) from corresponding lymphocyte clusters representing said corresponding TLS regions (135); as well as The set of corresponding TLS features (140) is processed using a TLS classification model (350) to classify a corresponding TLS region (135) into one of a first TLS maturity state (312), a second TLS maturity state (312), or a third TLS maturity state (312).

2. The computer-implemented method (400) of claim 1, wherein: The operations further include: processing the input histology image (110) using a tumor detection model (450) to identify a tumor region (115) within the input histology image (110), Wherein, processing the input histological image (110) to generate the one or more lymphocyte density maps (125) includes using the cell classification model (550) to process the input histological image (110) by performing single cell imaging analysis on the tumor region (115) identified within the input histological image (110) to generate the one or more lymphocyte density maps (125).

3. The computer-implemented method (400) of claim 2, wherein: The tumor detection model (450) is trained in the following manner: obtaining a plurality of image patches (370) rasterized from a set of whole slide histopathology images, each image patch (370) manually annotated as including tumor or non-tumor; and The tumor detection model (450) is trained on the plurality of image patches (370) using a neural network (374) to teach the tumor detection model (450) how to identify tumor regions (115) within a histology image.

4. The computer-implemented method (400) of any one of claims 1-3, wherein: The cell classification model (550) is trained by: obtaining a plurality of image slices (382), each image slice comprising a corresponding plurality of human cells and manual annotations that label each human cell as a tumor cell, a lymphocyte, or a non-malignant cell; and The cell classification model (550) is trained on the plurality of image slices (382) using a neural network (386) to teach the cell classification model (550) to learn how to classify individual cells in the histology images as tumor cells, lymphocytes, or non-malignant cells.

5. The computer-implemented method (400) of any one of claims 1-4, wherein: The TLS classification model (350) is trained in the following manner: A training data set (305) is obtained that includes a plurality of training histology images (310), each training histology image (310) containing a tumor microenvironment and including manual annotations (312), the manual annotations identifying: One or more TLS regions (135) in the training histology image (310), each TLS region (135) being represented by a corresponding lymphocyte cluster; as well as For each corresponding TLS zone (135), the actual TLS maturity state (312) indicates that the corresponding TLS zone (135) includes a first TLS maturity state (312), a second TLS maturity state (312), or a third TLS maturity state (312); For each TLS region (135), extracting a corresponding set of training TLS features (140) from a corresponding lymphocyte cluster representing said TLS region (135); as well as The TLS classification model (350) is trained on a set of corresponding training TLS features (140) extracted for each TLS region (135) to teach the TLS classification model (350) to learn how to predict the true TLS classification of each corresponding TLS region (135).

6. The computer-implemented method (400) of claim 5, wherein: Training the TLS classification model (350) includes using a classification and regression tree (CART) algorithm to train the TLS classification model (350).

7. The computer-implemented method (400) of any one of claims 1-6, wherein: The first TLS maturation state (312) includes dense aggregates of at least a threshold number of lymphocytes that do not contain high endothelial venules or germinal centers (313); The second TLS maturation state (312) includes immature TLS comprising dense aggregates of at least a threshold number of lymphocytes containing high endothelial venules and not containing any germinal centers (313); and The third TLS maturation state (312) includes mature TLS comprising dense aggregates of at least a threshold number of lymphocytes including high endothelial venules and germinal centers (313).

8. The computer-implemented method (400) of any one of claims 1-7, wherein: The operations further include: For each respective TLS region (110) of the one or more TLS regions (135) identified in the input histology image (135), generating a corresponding pixel mask (112) that highlights at least a perimeter of the respective TLS region (135); generating an output image (110A) that enhances the input histology image (110) by superimposing a corresponding pixel mask (112) generated for each of the TLS regions (135) onto the input histology image (110); and The output image (110A) is provided for display on a screen in communication with the data processing hardware (142).

9. The computer-implemented method (400) of claim 8, wherein: The corresponding pixel mask (112) generated for each respective TLS region (135) classified as the first maturity state comprises a first pixel mask (112); The corresponding pixel mask (112) generated for each respective TLS region (135) classified as the second maturity state includes a second pixel mask (112) visually discernible from the second pixel mask (112); as well as The corresponding pixel mask (112) generated for each respective TLS region (135) classified as the third maturity state includes a third pixel mask (112) visually distinguishable from the first pixel mask (112) and the second pixel mask (112).

10. The computer-implemented method (400) of any one of claims 1-9, wherein: The set of corresponding TLS features (140) extracted from the corresponding lymphocyte cluster includes the area of ​​the corresponding TLS region (135), the circularity of the corresponding TLS region (135), and the skewness of the corresponding TLS region (135).

11. The computer-implemented method (400) of any one of claims 1-10, wherein: The operations further include determining an overall TLS score (152) for the input histology image (110) based on the TLS maturation status (312) of the one or more TLS regions (135) identified in the histology image and the TLS features (140) extracted from the one or more TLS regions (135) identified in the histology image.

12. The computer-implemented method (400) of claim 11, wherein: The operations further include determining a treatment recommendation (192) to treat the patient using immunotherapy based on the total TLS score (152).

13. The computer-implemented method (400) of claim 12, wherein: The immunotherapy includes at least one of a PD-1 inhibitor or a PD-L1 inhibitor.

14. The computer-implemented method (400) of any one of claims 11-13, wherein: The operation further includes determining a prediction score for the patient's response to immunotherapy based on the TLS maturation state (312) of the one or more TLS regions (135) identified in the histology image and the TLS features (140) extracted from the one or more TLS regions (135) identified in the histology image.

15. A system (100), comprising: Data processing hardware (142); as well as Memory hardware (144) in communication with the data processing hardware (142), the memory hardware (144) storing instructions that, when executed on the data processing hardware (142), cause the data processing hardware (142) to perform operations, the operations comprising: receiving an input histology image (110) for a patient diagnosed with cancer, the input histology image (110) comprising a plurality of image pixels; processing the input histology image (110) using a cell classification model (550) to generate one or more lymphocyte density maps (125) within the input histology image (110); performing morphological image processing (130) on the one or more lymphocyte density maps (125) to identify one or more TLS regions (135) within the input histology image (110), each TLS region (135) being represented by a corresponding lymphocyte cluster; and For each corresponding TLS region (110) of the one or more TLS regions (135) identified in the input histology image (135): extracting a corresponding set of TLS features (140) from corresponding lymphocyte clusters representing said corresponding TLS regions (135); and The set of corresponding TLS features (140) is processed using a TLS classification model (350) to classify a corresponding TLS region (135) into one of a first TLS maturity state (312), a second TLS maturity state (312), or a third TLS maturity state (312).

16. The system (100) of claim 15, wherein: The operations further include: processing the input histology image (110) using a tumor detection model (450) to identify a tumor region (115) within the input histology image (110), Wherein, processing the input histological image (110) to generate the one or more lymphocyte density maps (125) includes using the cell classification model (550) to process the input histological image (110) by performing single cell imaging analysis on the tumor region (115) identified within the input histological image (110) to generate the one or more lymphocyte density maps (125).

17. The system (100) of claim 16, wherein: The tumor detection model (450) is trained in the following manner: obtaining a plurality of image patches (370) rasterized from a set of whole slide histopathology images, each image patch (370) manually annotated as including tumor or non-tumor; and The tumor detection model (450) is trained on the plurality of image patches (370) using a neural network (374) to teach the tumor detection model (450) how to identify tumor regions (115) within a histology image.

18. The system (100) according to any one of claims 15 to 17, wherein: The cell classification model (550) is trained by: obtaining a plurality of image slices (382), each image slice comprising a corresponding plurality of human cells and manual annotations that label each human cell as a tumor cell, a lymphocyte, or a non-malignant cell; and The cell classification model (550) is trained on the plurality of image slices (382) using a neural network (386) to teach the cell classification model (550) to learn how to classify individual cells in the histology images as tumor cells, lymphocytes, or non-malignant cells.

19. The system (100) according to any one of claims 15 to 18, wherein: The TLS classification model (350) is trained in the following manner: A training data set (305) is obtained that includes a plurality of training histology images (310), each training histology image (310) containing a tumor microenvironment and including manual annotations (312), the manual annotations identifying: One or more TLS regions (135) in the training histology image (310), each TLS region (135) being represented by a corresponding lymphocyte cluster; as well as For each corresponding TLS zone (135), the actual TLS maturity state (312) indicates that the corresponding TLS zone (135) includes a first TLS maturity state (312), a second TLS maturity state (312), or a third TLS maturity state (312); For each TLS region (135), extracting a corresponding set of training TLS features (140) from a corresponding lymphocyte cluster representing said TLS region (135); as well as The TLS classification model (350) is trained on a set of corresponding training TLS features (140) extracted for each TLS region (135) to teach the TLS classification model (350) to learn how to predict the true TLS classification of each corresponding TLS region (135).

20. The system (100) of claim 19, wherein: Training the TLS classification model (350) includes using a classification and regression tree (CART) algorithm to train the TLS classification model (350).

21. The system (100) according to any one of claims 15 to 20, wherein: The first TLS maturation state (312) includes dense aggregates of at least a threshold number of lymphocytes that do not contain high endothelial venules or germinal centers (313); The second TLS maturation state (312) includes immature TLS comprising dense aggregates of at least a threshold number of lymphocytes containing high endothelial venules and not containing any germinal centers (313); and The third TLS maturation state (312) includes mature TLS comprising dense aggregates of at least a threshold number of lymphocytes including high endothelial venules and germinal centers (313).

22. The system (100) according to any one of claims 15 to 21, wherein: The operations further include: For each respective TLS region (110) of the one or more TLS regions (135) identified in the input histology image (135), generating a corresponding pixel mask (112) that highlights at least a perimeter of the respective TLS region (135); generating an output image (110A) that enhances the input histology image (110) by superimposing a corresponding pixel mask (112) generated for each of the TLS regions (135) onto the input histology image (110); and The output image (110A) is provided for display on a screen in communication with the data processing hardware (142).

23. The system (100) of claim 22, wherein: The corresponding pixel mask (112) generated for each respective TLS region (135) classified as the first maturity state comprises a first pixel mask (112); The corresponding pixel mask (112) generated for each respective TLS region (135) classified as the second maturity state includes a second pixel mask (112) visually discernible from the second pixel mask (112); as well as The corresponding pixel mask (112) generated for each respective TLS region (135) classified as the third maturity state includes a third pixel mask (112) visually distinguishable from the first pixel mask (112) and the second pixel mask (112).

24. The system (100) according to any one of claims 15 to 23, wherein: The set of corresponding TLS features (140) extracted from the corresponding lymphocyte cluster includes the area of ​​the corresponding TLS region (135), the circularity of the corresponding TLS region (135), and the skewness of the corresponding TLS region (135).

25. The system (100) according to any one of claims 15 to 24, wherein: The operations further include determining an overall TLS score (152) for the input histology image (110) based on the TLS maturation status (312) of the one or more TLS regions (135) identified in the histology image and the TLS features (140) extracted from the one or more TLS regions (135) identified in the histology image.

26. The system (100) of claim 25, wherein: The operations further include determining a treatment recommendation (192) to treat the patient using immunotherapy based on the total TLS score (152).

27. The system (100) of claim 26, wherein: The immunotherapy includes at least one of a PD-1 inhibitor or a PD-L1 inhibitor.

28. The system (100) according to any one of claims 25-27, wherein: The operation further includes determining a prediction score for the patient's response to immunotherapy based on the TLS maturation state (312) of the one or more TLS regions (135) identified in the histology image and the TLS features (140) extracted from the one or more TLS regions (135) identified in the histology image.

Images