Machine learning identification, classification, and quantification of tertiary lymphoid tissue-like structures

Abstract

The method (400) includes receiving an input tissue structure image (110), generating one or more lymphocyte density maps (125) inside the input tissue structure image by processing the input tissue structure image using a cell classification model (550), and identifying one or more TLS regions (135) inside the input tissue structure image by performing morphological image processing (130) on the one or more lymphocyte density maps. Each TLS region is represented by each cluster of lymphocyte cells. For each corresponding TLS region of the one or more TLS regions identified in the input tissue structure image, the method includes extracting each set of TLS features (140) from each cluster of lymphocyte cells, and classifying the corresponding TLS region as one of a first TLS maturation state, a second TLS maturation state, and a third TLS maturation state by processing each set of TLS features using a TLS classification model (350).

Description

Technical Field

[0001] The present disclosure relates to the machine learning identification, classification, and quantification of tertiary lymphoid structure-like structures, for example, in tumor biopsy specimens.

Background Art

[0002] Tertiary lymphoid structure-like structures (TLS) (e.g., tertiary lymphoid organs or ectopic lymphoid follicles) are ectopic lymphoid tissues composed of B cells, T cells, and supporting cells that develop in non-lymphoid organs and are often found in tumors. TLS supports the differentiation of naive T cells into effector and memory T cells and frequently occurs in areas of chronic inflammation. Although TLS has been observed in the clinicopathological setting, it is currently not evaluated for diagnostic pathology or to guide treatment. Multiple studies have shown an association between TLS across multiple signs and the treatment outcomes of immuno-oncology (IO). (See, for example, Non-Patent Document 1). The presence of TLS in various tumors has shown an association with outcomes in non-IO settings, and more recently, it has been found that TLS can predict the response to IO treatment in melanoma, osteosarcoma, and RCC. See, for example, Non-Patent Document 2. In the research setting, TLS has been evaluated by manual visual methods based on hematoxylin and eosin stain (H&E) and immunohistochemistry (IHC) stain. For quantification, image analysis of IHC or immunofluorescent (IF) stain has been used. These correlations depend on the maturation and localization of TLS within the tumor microenvironment (TME).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 61

Patent Document 62

Patent Document 63

Patent Document 64

Non-Patent Document

[0004]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

Non-Patent Document 4

Non-Patent Document 5

Non-Patent Document 6

Non-Patent Document 7

Non-Patent Document 8

Non-Patent Document 9

Non-Patent Document 10

Summary of the Invention

[0005] One aspect of the present disclosure provides a computer-implemented method that causes data processing hardware to perform operations including receiving an input tissue structure image regarding a patient diagnosed with cancer when executed on the data processing hardware. The input tissue structure image includes a plurality of image pixels. The operations also include generating one or more lymphocyte density maps within the input tissue structure image by processing the input tissue structure image using a cell classification model, and identifying one or more TLS regions within the input tissue structure image by performing morphological image processing on the one or more lymphocyte density maps. Each TLS region is represented by each cluster of lymphocyte cells. For each corresponding TLS region of the one or more TLS regions identified in the input tissue structure image, the operations also include extracting each set of TLS features from each cluster of the lymphocyte cells representing the corresponding TLS region, and classifying the corresponding TLS region as one of a first TLS maturation state, a second TLS maturation state, and a third TLS maturation state by processing each set of TLS features using a TLS classification model.

[0006] Embodiments of the present disclosure may include one or more of the following optional features. In some embodiments, the operation also includes identifying a tumor region within the input tissue structure image by processing the input tissue structure image using a tumor detection model. Here, generating one or more lymphocyte density maps by processing the input tissue structure image may include generating one or more lymphocyte density maps by performing single-cell imaging analysis on the tumor region identified within the input tissue structure image and by processing the input tissue structure image using a cell classification model. In these embodiments, the tumor detection model may be learned by obtaining a plurality of image tiles rasterized from a set of histopathological images of the entire slide, each manually annotated as containing a tumor or non-tumor, and causing the tumor detection model to learn the plurality of image tiles using a neural network so as to teach the tumor detection model a method of identifying a tumor region within the tissue structure image.

[0007] In some embodiments, the cell classification model is learned by obtaining a plurality of image patches and causing the cell classification model to learn the plurality of image patches using a neural network so as to teach the cell classification model a method of classifying individual cells in the tissue structure image as tumor cells, lymphocyte cells, or non-malignant cells. Each image patch includes a corresponding plurality of human cells and a manual annotation labeling each human cell as a tumor cell, a lymphocyte cell, or a non-malignant cell.

[0008] In some embodiments, the TLS classification model is trained by obtaining a training dataset that includes a plurality of histological structure images for training, each including a tumor microenvironment and each having a manual annotation. The manual annotation identifies one or more TLS regions in the histological structure image for training and, for each corresponding TLS region, a ground truth TLS maturity state indicating that the corresponding TLS region includes a first TLS maturity state, a second TLS maturity state, or a third TLS maturity state. Each TLS region is represented by each cluster of lymphocytes. In these embodiments, the TLS classification model is further trained by, for each TLS region, extracting each set of TLS features from each cluster of lymphocytes representing the TLS region and training the TLS classification model to learn a method for predicting the ground truth TLS grade for each corresponding TLS region, by causing the TLS classification model to learn each set of training TLS features extracted for each TLS region. Causing the TLS classification model to learn may include causing the TLS classification model to learn using a CART (classification and regression trees) algorithm.

[0009] The first TLS maturity state may include a dense aggregate of at least a threshold number of lymphocytes that does not include high endothelial venules or germinal centers. The second TLS maturity state may include an immature TLS that includes a dense aggregate of at least the above threshold number of lymphocytes that includes high endothelial venules and does not include germinal centers. The third TLS maturity state may include a mature TLS that includes a dense aggregate of at least a threshold number of lymphocytes that includes high endothelial venules and germinal centers. Each set of TLS features extracted from each cluster of lymphocytes may include the area of the corresponding TLS region, the roundness of the corresponding TLS region, and the distortion of the corresponding TLS region.

[0010] In some embodiments, the operation further includes generating, for each corresponding TLS region of one or more TLS regions identified in the input tissue structure image, a pixel mask that emphasizes at least the periphery of the corresponding TLS region; generating an output image that enhances the input tissue structure image by superimposing each pixel mask generated for each TLS region over the input tissue structure image; and providing the output image for display on a screen that communicates with the data processing hardware. In these embodiments, each pixel mask generated for each corresponding TLS region classified as a first maturation state includes a first pixel mask, each pixel mask generated for each corresponding TLS region classified as a second maturation state includes a second pixel mask that is visually distinguishable from the first pixel mask, and each pixel mask generated for each corresponding TLS region classified as a third maturation state includes a third pixel mask that is visually distinguishable from the first pixel mask and the second pixel mask.

[0011] In some embodiments, the operation also includes determining an overall TLS score for the input tissue structure image based on the TLS maturation state for one or more TLS regions identified in the tissue structure image and the TLS features extracted from one or more TLS regions identified in the tissue structure image. In these embodiments, the operation may also include determining a treatment recommendation for treating a patient with immunotherapy based on the overall TLS score. Here, the immunotherapy may include at least one of a PD-1 inhibitor or a PD-L1 inhibitor. The operation may also include determining a predicted score for the patient's response to immunotherapy based on the TLS maturation state for one or more TLS regions identified in the tissue structure image and the TLS features extracted from one or more TLS regions identified in the tissue structure image.

[0012] Another aspect of the present disclosure provides a system that includes data processing hardware and memory hardware that communicates with the data processing hardware. The memory hardware stores instructions that cause the data processing hardware to perform operations including receiving an input tissue structure image regarding a patient diagnosed with cancer when executed on the data processing hardware. The input tissue structure image includes a plurality of image pixels. The operations also include generating one or more lymphocyte density maps within the input tissue structure image by processing the input tissue structure image using a cell classification model, and identifying one or more TLS regions within the input tissue structure image by performing morphological image processing on the one or more lymphocyte density maps. Each TLS region (135) is represented by each cluster of lymphocyte cells. For each corresponding TLS region of the one or more TLS regions identified in the input tissue structure image, the operations also include extracting each set of TLS features from each cluster of the lymphocyte cells representing the corresponding TLS region, and classifying the corresponding TLS region as one of a first TLS maturation state, a second TLS maturation state, and a third TLS maturation state by processing each set of TLS features using a TLS classification model.

[0013] This aspect may include one or more of the following optional features. In some embodiments, this operation also includes identifying a tumor region within the input tissue structure image by processing the input tissue structure image using a tumor detection model. Here, generating one or more lymphocyte density maps by processing the input tissue structure image may include performing single-cell imaging analysis on the tumor region identified within the input tissue structure image and generating one or more lymphocyte density maps by processing the input tissue structure image using a cell classification model. In these embodiments, the tumor detection model may be learned by obtaining a plurality of image tiles rasterized from a set of histopathological images of the entire slide, each manually annotated as containing a tumor or non-tumor, and teaching the tumor detection model how to identify a tumor region within the tissue structure image by using a neural network to learn the tumor detection model with the plurality of image tiles.

[0014] In some embodiments, the cell classification model is learned by obtaining a plurality of image patches and teaching the cell classification model how to classify individual cells in the tissue structure image as tumor cells, lymphocyte cells, or non-malignant cells by using a neural network to learn the cell classification model with the plurality of image patches. Each image patch includes a corresponding plurality of human cells and manual annotation labeling each human cell as a tumor cell, lymphocyte cell, or non-malignant cell.

[0015] In some embodiments, the TLS classification model is trained by obtaining a training dataset that includes a plurality of histological structure images for training, each of which includes a tumor microenvironment and has manual annotation. The manual annotation identifies one or more TLS regions in the histological structure image for training and, for each corresponding TLS region, a ground truth TLS maturation state indicating that the corresponding TLS region includes a first TLS maturation state, a second TLS maturation state, or a third TLS maturation state. Each TLS region is represented by each cluster of lymphocytes. In these embodiments, the TLS classification model is further trained by, for each TLS region, extracting each set of TLS features from each cluster of lymphocytes representing the TLS region and training the TLS classification model to learn a method for predicting the ground truth TLS grade for each corresponding TLS region, by causing the TLS classification model to learn each set of training TLS features extracted for each TLS region. Causing the TLS classification model to learn may include causing the TLS classification model to learn using a CART (classification and regression trees) algorithm.

[0016] The first TLS maturation state may include a dense aggregate of at least a threshold number of lymphocytes that does not include high endothelial venules or germinal centers. The second TLS maturation state may include an immature TLS that includes a dense aggregate of at least the above threshold number of lymphocytes that includes high endothelial venules and does not include germinal centers. The third TLS maturation state may include a mature TLS that includes a dense aggregate of at least a threshold number of lymphocytes that includes high endothelial venules and germinal centers. Each set of TLS features extracted from each cluster of lymphocytes may include the area of the corresponding TLS region, the roundness of the corresponding TLS region, and the distortion of the corresponding TLS region.

[0017] In some embodiments, the operation further includes generating, for each corresponding TLS region of one or more TLS regions identified in the input tissue structure image, a pixel mask that emphasizes at least the periphery of the corresponding TLS region; generating an output image that enhances the input tissue structure image by superimposing each pixel mask generated for each TLS region over the input tissue structure image; and providing the output image for display on a screen that communicates with data processing hardware. In these embodiments, each pixel mask generated for each corresponding TLS region classified as a first maturation state includes a first pixel mask, each pixel mask generated for each corresponding TLS region classified as a second maturation state includes a second pixel mask that is visually distinguishable from the first pixel mask, and each pixel mask generated for each corresponding TLS region classified as a third maturation state includes a third pixel mask that is visually distinguishable from the first pixel mask and the second pixel mask.

[0018] In some embodiments, the operation also includes determining an overall TLS score for the input tissue structure image based on the TLS maturation state for one or more TLS regions identified in the tissue structure image and the TLS features extracted from one or more TLS regions identified in the tissue structure image. In these embodiments, the operation may also include determining a treatment recommendation for treating a patient with immunotherapy based on the overall TLS score. Here, the immunotherapy may include at least one of a PD-1 inhibitor or a PD-L1 inhibitor. The operation may also include determining a prediction score for a patient's response to immunotherapy based on the TLS maturation state for one or more TLS regions identified in the tissue structure image and the TLS features extracted from one or more TLS regions identified in the tissue structure image.

[0019] Details of one or more embodiments of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims.

Brief Description of the Drawings

[0020]

Figure 1

Figure 2A

Figure 2B

Figure 2C

Figure 2D

Figure 2E

Figure 2F

Figure 2G

Figure 2H

Figure 2I

Figure 2J

Figure 2K

Figure 3A

Figure 3B

Figure 3C

Figure 4

Figure 5A

Figure 5B

Figure 6A

Figure 6B

Figure 6C

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20A

Figure 20B

Figure 20C

Figure 21A

Figure 21B

Figure 21C

Figure 21D

Figure 22

DETAILED DESCRIPTION OF THE INVENTION

[0021] Similar reference numerals in the various drawings indicate similar components.

[0022] Tertiary lymphoid tissue-like structures (TLS) are ectopic lymphoid organs that develop in non-lymphoid tissues, such as sites of chronic inflammation and tumors. TLS are lymphoid structures with a vascular pattern that develop in benign and tumor tissues with chronic inflammation. TLS are highly organized structures that resemble secondary lymphoid structures (e.g., lymph nodes). TLS can consist of a B cell zone containing an active germinal center, surrounded by a T cell zone that contains various types of dendritic cells (DC), T cells, high endothelial venules (HEV), and / or other supporting cells within the structural matrix. Unlike lymph nodes, TLS lack a fibrous capsule and are directly exposed to the tumor microenvironment (TME). TLS are more abundant in the invasive margin / stroma compared to the tumor core. The presence of TLS has been associated with favorable outcomes in the treatment of multiple indications (e.g., treatment of melanoma with nivolumab or with nivolumab and ipilimumab). TLS structures can be classified as lymphoid aggregates (LA) (i.e., the first state of maturation), immature TLS (imTLS) (e.g., grade 1) (i.e., the second state of maturation), or mature TLS (mTLS) with a germinal center (GC) (e.g., grade 2) (i.e., the third state of maturation). In some cases, TLS are absent (e.g., grade 0). While the biological mechanisms underlying their formation are incompletely understood, TLS are known to play an important role in the anti-tumor immune response. For example, the presence of TLS has been associated with a favorable prognosis and an improved response to immunotherapy across a number of cancer types.

[0023] Conventional approaches for TLS detection in patients involve using tissue staining techniques for markers of immune cell lineages by means of multiple immunohistochemistry or immunofluorescence techniques. However, multiplex imaging is not applicable on a daily basis considering its cost, high complexity, small field of view, and difficulty in scaling, which limits its use in a research environment. On the other hand, hematoxylin and eosin (H&E) staining is widely available and remains a clinical standard in histopathology. Evaluating slides stained with H&E based on the assessment of pathologists is time-consuming and labor-intensive, and the manual and qualitative assessment performed manually by pathologists is often inaccurate and subject to observer variability.

[0024] Embodiments of the present application relate to a machine learning technique that uses deep learning to train a model to learn a method for detecting the presence of TLS regions in H&E-stained tissue structure images and classifying each of the TLS regions into one of three TLS maturation states. The first TLS maturation state includes a dense aggregate of at least a threshold number of lymphocytes that does not include high endothelial venules or germinal centers. In some embodiments, the threshold number is equal to 100. The second TLS maturation state includes immature TLS associated with a dense aggregate of at least the threshold number of lymphocytes that includes high endothelial venules and does not include germinal centers. The third TLS maturation state includes mature TLS associated with a dense aggregate of at least a threshold number of lymphocytes that includes high endothelial venules and germinal centers. More specifically, the embodiments include generating one or more lymphocyte density maps by processing an input tissue structure image (e.g., an H&E-stained tissue structure image) using a cell classification model, and performing morphological image processing on the one or more lymphocyte density maps to identify one or more TLS regions within the input tissue structure image, the one or more TLS regions being represented by respective clusters of lymphocyte cells, and extracting each set of TLS features from each cluster of the lymphocyte cells representing the corresponding TLS region for each corresponding TLS region. Thereafter, the trained TLS classification model receives each set of TLS features extracted for each corresponding TLS region and classifies the corresponding TLS region as one of the first TLS maturation state, the second TLS maturation state, and the third TLS maturation state.

[0025] Embodiments of the present application further relate to calculating a TLS score for an input tissue structure image based on the TLS maturity state output from the TLS classification model and the TLS features related to the TLS regions identified in the input tissue structure image. The TLS score determiner may determine each TLS score for each of the three TLS maturity states based on the total tumor area and, further, the total TLS area of each of the TLS regions classified for each of the three TLS maturity states. The TLS score determiner may then calculate an overall TLS score for a patient associated with the input tissue structure image based on the linearly weighted sum of the total TLS areas divided by the tumor area. As will be described in detail later, the overall TLS score may be used to predict various prognostic values for a patient, such as predicting survival outcomes such as overall survival and progression-free survival. That is, a higher overall TLS score indicates a significantly improved overall survival and progression-free survival compared to a lower overall TLS score. Thus, the overall TLS score may be used to predict prognostic results instead of using tumor stage prediction, and / or the prognostic results predicted using tumor stage / grade may be further refined by the overall TLS score.

[0026] For each corresponding TLS region among one or more TLS regions identified in the input tissue structure image, the image enhancer may generate each pixel mask that emphasizes at least the periphery of the corresponding TLS region, and then generate an output image that enhances the input tissue structure image by superimposing each pixel mask generated for each TLS region on the input tissue structure image. The output image generated by the image enhancer may be provided for display on a screen for a healthcare professional (HCP) to view. Here, the image enhancer receives a classification output from the TLS classification model and generates each visually different pixel mask for each of the three different TLS maturation states. For example, the pixel mask generated for a TLS region classified as the first maturation state may include a first color, the pixel mask generated for a TLS region classified as the second maturation state may include a different second color, and the pixel mask generated for a TLS region classified as the third maturation state may include a third color different from the first and second colors. In some embodiments, the pixels generated for a TLS region classified as the third maturation state emphasize at least the periphery of the corresponding TLS region and further emphasize the area of pixels encompassed by the germinal center.

[0027] Embodiments of the present application further relate to a learning process for training a TLS classification model. Here, the learning process obtains a training dataset including a plurality of training histology images. Each of the plurality of training histology images includes a tumor microenvironment and manual annotation from a pathologist. The manual annotation identifies the presence of TLS regions in each training histology image, where each TLS region is represented by a respective cluster of lymphocytes. The manual annotation further identifies a ground truth TLS maturation state for each corresponding TLS region indicating that the corresponding TLS region includes a first TLS maturation state, a second TLS maturation state, or a third maturation state. Next, the learning process extracts each set of training TLS features from each cluster of lymphocytes representing each TLS region. Each set of training TLS features may include the area of the TLS region, the roundness (i.e., the ratio of the area of the TLS region multiplied by 4π to the square of the perimeter of the TLS region), and the density distortion of each cluster of lymphocytes representing each TLS region. Based on each set of training TLS features extracted for each TLS region, the learning process trains a TLS model using a CART (classification and regression trees) algorithm to learn a method for predicting the ground truth TLS grade for each corresponding TLS region.

[0028] In particular, the cell classification model is trained to learn a method for classifying individual cells in a histology image as tumor cells, lymphocytes, or non-malignant cells. As used in the present application, lymphocytes may include T cells and B cells. The cell classification model may be trained using a Mask R-CNN deep learning model to learn a method for segmenting and classifying individual nuclei into tumor cells, lymphocytes, and other non-malignant cells.

[0029] In some embodiments, the input tissue structure image is processed using a tumor detection model to identify tumor regions within the input tissue structure image, and thereby, one or more lymphocyte density maps are generated by performing single-cell image analysis on the identified tumor regions within the input tissue structure using a cell classification model, whereby image preprocessing is performed on the input tissue structure image. The tumor detection model may be trained with a plurality of image tiles rasterized from a set of histopathological images of the entire slide, each manually annotated as containing a tumor or non-tumor. More specifically, the tumor detection model is trained with a plurality of image tiles such that a deep learning neural network teaches the tumor detection model how to identify tumor regions within the tissue structure image. The deep learning neural network may include a ResNet18 deep learning model.

[0030] As an advantage, the deep learning-based single-cell analysis technique disclosed in the present application does not suffer from any drawbacks of other techniques that employ a patch- or tile-based approach for image analysis, and provides the ability to accurately identify, classify, and quantify the presence of TLS regions from an image of an entire H&E-stained slide. Since TLS can vary significantly in size, density, and morphology, there are significant challenges in using conventional patch-based approaches to identify and interpret TLS regions. As will become apparent, the techniques disclosed therein include quantifying the spatial distribution of lymphocytes and thereby providing an accurate and interpretable model for classifying TLS according to their state of maturation.

[0031] Similarly, the manual and qualitative assessment of TLS performed by pathologists lacks the automated enumeration and quantitative characterization of TLS. By the same concept, such manual and qualitative assessment of TLS performed by pathologists is found to be inaccurate and subject to inter-observer variability when evaluated against H&E-stained slides. See Non-Patent Document 3.

[0032] Referring to FIG. 1, in some embodiments, the system 100 inputs a tissue structure image 110 of a patient diagnosed with cancer into a TLS classification model 350 to identify, classify, and quantify the presence of TLS regions within the tissue structure image for use as a predictive biomarker of the efficacy and prognostic outcome of immune-checkpoint inhibitors (ICIs). The input tissue structure image 110 may optionally include metadata 11 that includes information such as the type of cancer diagnosed in the patient, the tumor stage / grade, and / or the patient's demographic information. The input tissue structure image 110 may include an image of an entire hematoxylin and eosin (H&E)-stained slide (whole slide image WSI). The input tissue structure image 110 includes a plurality of image pixels. The input tissue structure image 110 characterizes a human tumor biopsy specimen. The input tissue structure image 110 may include a tumor microenvironment for any number of cancers including, but not limited to, bladder cancer (BLCA), breast cancer (BRCA), stomach adenocarcinoma (STAD), lung adenocarcinoma (LUAD) (e.g., non-small cell lung cancer adenocarcinoma (NSCLC-AD)), and / or lung squamous cell carcinoma (LUSC) (e.g., non-small cell lung cancer squamous (NSCLC-SQ)).

[0033] The client device 111 is associated with a user 10, such as a healthcare provider (HCP), who may communicate with a remote system 141 via a network 132. The remote system 141 may be a distributed system (e.g., a cloud environment) having scalable / elastic resources 142. The resources 142 may include computing resources 144 (e.g., data processing hardware) and / or storage resources 146 (e.g., memory hardware). In some embodiments, the remote system 141 executes a TLS classification model 450 and is configured to execute other components such as a tumor detection model 450, a cell classification model 550, a lymphocyte aggregator 120, a morphological image processing module 130, a TLS extractor 145, a TLS score determiner 150, and an image enhancer 360, and executes a TLS identification and quantification application 160 (also simply referred to as "application 160"). Here, the client device 111 may access the application 160 operating in the remote system 141, and may also input the tissue structure input image 110 into the TLS classification model 350 via a GUI (graphical user interface) executed in the client device 111. The GUI may be displayed to the user 10 via the screen 114 of the client device 111. The client device 111 may, additionally or alternatively, execute an application 160 that implements the ability to execute any combination of the TLS classification model 350 and / or other components in the client device 111 to identify, classify, and quantify the presence of TLS regions 135 within the tissue structure image 110.

[0034] The TLS identification and quantification application 160 may identify TLS details 190 and / or treatment recommendations 192 based on the identified TLS regions 135 that have been classified and quantified using the TLS classification model 350. The application 160 may return the TLS details 190 and / or treatment recommendations 192 to the client device 111, causing the client device to display the TLS details 190 and / or treatment recommendations 192 on the screen 114 of the client device 111. The TLS details 190 may include an overall TLS score 152 for the input tissue structure image 110, and may also include other details such as the number of TLS regions associated with a first maturation state (e.g., TLS1) classified by the TLS classification model 350, the number of TLS regions associated with a second maturation state (e.g., TLS2) classified by the TLS classification model 350, and the number of TLS regions associated with a third maturation state (e.g., TLS3) classified by the TLS classification model 350, but are not limited thereto. Here, the first TLS maturation state includes a dense aggregate of at least a threshold number of lymphocytes that does not include high endothelial venules or germinal centers. In some embodiments, the threshold number is equal to 100. The second TLS maturation state includes immature TLS associated with a dense aggregate of at least the above threshold number of lymphocytes that includes high endothelial venules and does not include germinal centers. The third TLS maturation state includes mature TLS associated with a dense aggregate of at least a threshold number of lymphocytes that includes high endothelial venules and germinal centers. The TLS details 190 provided for display on the screen 114 may further include an output image 110A that enhances the input tissue structure image 110 by superimposing each pixel mask 112 generated for each of the TLS regions over the input tissue structure image 110. The treatment recommendation 192 may indicate an instruction to apply (or not apply) immunotherapy to the patient to treat the patient. For example, the immunotherapy may include a PD-1 inhibitor (e.g., an anti-PD-1 antibody) or a PD-L1 inhibitor (e.g., an anti-PID-L1 antibody). In one embodiment, the immunotherapy includes nivolumab, a drug that is an immune checkpoint inhibitor.

[0035] Treatment advice 192 may further include prognostic outcomes predicted for a patient based on TLS details 190, such as overall survival (OS) (i.e., number of months) and progression-free survival (PFS) (number of months). Treatment advice 192 may show OS and / or PFS predictions regarding immunotherapy, contrasted with OS and / or PFS predictions without immunotherapy. The prognostic outcomes predicted by application 160 may be notified to the patient, healthcare provider, and / or the patient's relatives to make better examination and treatment decisions regarding the specific health condition diagnosed, or to perform risk stratification for clinical trials.

[0036] In some embodiments, the input tissue structure image 110 undergoes initial image preprocessing to ensure sufficient image quality. The input tissue structure image may include a magnification of 40x. However, WSI slides scanned at a lower magnification (e.g., 20x) may be used. To minimize the impact of image artifacts, the image preprocessing may downsample the image of the entire slide by a factor of 32 and apply appropriate color factors to remove regions with writing, folding, and blur artifacts.

[0037] In the illustrated embodiment, the tumor detection model 450 processes the input tissue structure image 110 to identify one or more tumor regions 115 within the input tissue structure image 110. Each tumor region 115 may be represented by a corresponding group of pixels in the input tissue structure image 110 where the tumor region 115 is located. In particular, since only the TLSs inside or around the tumor region 115 are relevant, the tumor detection model 450 may separate cancerous tissue from normal tissue, enabling post-processing for TLS identification and quantification to focus on the tumor regions 115 in the input tissue structure image 110. The tumor detection model 450 may include a pre-trained symptom-specific tissue segmentation model configured to distinguish cancer, stroma associated with the cancer, and necrosis from normal tissue by processing the input tissue structure image 110. FIG. 3B shows an exemplary tumor detection model learning process 300b that can be used to train the tumor detection model 450. The learning process 300b obtains a plurality of image tiles 370 rasterized from a set of histopathological images of the entire slide. The histopathological images may include publicly available and already annotated H&E-stained WSIs from patients with colorectal and gastric cancers. Each image tile 370 may include manual annotations 372 indicating the locations of tumor regions and non-tumor regions (including adipose tissue, mucus, stroma, or muscle) within the corresponding histopathological image of the entire slide. The image tiles may include 512×512 image tiles at 0.5 micrometers per image pixel. The learning process 300b includes training the tumor detection model 450 with a plurality of image tiles 370 so as to teach the tumor detection model 450 how to identify tumor regions within the tissue structure image using the neural network 374. In some embodiments, the neural network 374 includes a ResNet18 deep learning network, and the loss module 378 calculates a learning loss 380 based on the predictions 376 output by the ResNet 18 network against the ground truth annotations 372.The learning process 300b may update the parameters of REsNet 18 based on the learning loss 380 until a tumor detection model 450 with converged and learned parameters of ResNet 18 is obtained. The loss module 378 may use a cross-entropy loss function and apply L2 regularization to cancel overfitting. The learning process 300b may expand the tumor segmentation through image expansion by 0.5 mm so as to include an infiltration margin. The learning process may further enhance the image tiles 370 used for learning by applying horizontal / vertical flipping and translation.

[0038] Referring back to FIG. 1, after the tumor detection model 450 identifies the tumor region 115, the cell classification model 550 processes the input tissue structure image 110 by performing single-cell imaging analysis on the identified tumor region 115 within the input tissue structure image 110 (i.e., on the image pixels corresponding to the tumor region 115) to generate a classified tumor region 115C. That is, the single-cell imaging analysis performed by the cell classification model 550 classifies individual cells / nuclei as tumor cells, lymphocyte cells (i.e., B cells and T cells, dendritic cells (DC), high endothelial venules (HEV)), and non-malignant cells. As used in the present application, the learned cell classification model 550 functions as a lymphocyte mask for classifying which cells in the tumor region 115 contain lymphocytes. Thus, the classified tumor region 115C may correspond to a lymphocyte mask that identifies all lymphocyte cells classified and segmented by the cell classification model 550 in the tumor region 115 within the input tissue structure image 110. Thereafter, the application 160 executes a lymphocyte aggregator 120, which processes the classified tumor region 115C output by the cell classification model 550 to count the number of lymphocytes for each unit square in a predefined grid (e.g., a 16×16 μm 2 grid). Thereby, one or more lymphocyte density maps 125 within the input tissue structure image 110 are generated.

[0039] FIG. 3C shows an exemplary cell classification model learning process 300c that can be used to cause a cell classification model 550 to learn a plurality of image patches 382. Each image patch (i.e., image tile) 382 is manually annotated to characterize a corresponding plurality of human cells and label each human cell as a tumor cell, lymphocyte cell, or non-malignant cell. The plurality of image patches may include 1358 image patches from 66 patients in a publicly available dataset having manual annotation 384 including 17582 tumor cells, 22550 lymphocyte cells, and 10675 other non-malignant cells. The learning process 300c includes causing the cell classification model 550 to learn a plurality of image patches 382 using a neural network 386 so as to teach the cell classification model 550 a method of classifying individual cells in a tissue structure image as tumor cells, lymphocyte cells, or non-malignant cells. In some embodiments, the neural network 386 includes a Mask R-CNN deep learning network, and the loss module 392 calculates a learning loss 390 based on predictions 388 output by the Mask R-CNN network 386 with respect to the ground truth annotation 384. The learning process 300b may update the parameters of the Mask R-CNN based on the learning loss 392 until the parameters of the Mask R-CNN converge to obtain the learned cell classification model 550. As used in the present application, the learned cell classification model 550 functions as a lymphocyte mask for classifying which cells in the tumor region 115 contain lymphocytes. The loss module 392 may update the Mask R-CNN via the learning loss 392 using a stochastic gradient descent technique. The learning process may further enhance the image patches 382 used for learning by applying horizontal / vertical flipping and translation.

[0040] Referring again to FIG. 1, in some embodiments, the TLS identification and quantification application 160 identifies one or more TLS regions 135 within the input tissue structure image 110 by performing morphological image processing 130 on one or more lymphocyte density maps 125. In particular, each TLS region represents each cluster of lymphocyte cells. The morphological image processing 130 may indicate the pixel positions corresponding to each TLS region 135 identified within the input tissue structure image 110. Each TLS region 135 may correspond to a TLS mask. In some embodiments, the morphological image processing 130 performed on the lymphocyte density map 125 is applied to set a threshold such that lymphocyte clusters having an area smaller than a predefined threshold area are excluded from being identified as TLS regions. The predefined threshold area may be equal to 0.0384 mm 2 2.

[0041] For each identified TLS region 135, the application 160 executes a TLS feature extractor 145 configured to extract each set of TLS features 140 from each cluster of lymphocytes representing the corresponding TLS region 135. The set of TLS features 140 may include human interpretable features (HIFs) associated with the TLS region 135. In some embodiments, some of the TLS features include sample-level features that include at least one of the summary count, area, shape, or location of the corresponding TLS region 135. The TLS features 140 extracted from each cluster of lymphocytes representing the corresponding TLS region 135 may include the area of the TLS region 135, the roundness of the TLS region 135 (i.e., the ratio of the area of the TLS region 135 multiplied by 4π to the square of the perimeter of the TLS region 135), and the density distortion of each cluster of lymphocytes representing the TLS region 135. The TLS features 140 may additionally or alternatively include at least one of the area of the germinal center inside the object in the tissue, the area of the object in the tissue, the centroid x of the object in the tissue, the centroid y of the object in the tissue, the longest distance of the object from the tumor, the perimeter of the object in the tissue, the shortest distance of the object from the tumor, the total germinal center inside the object in the tissue, or the ratio of the area of the germinal center inside the object to the object in the tissue. Some of the TLS features 140 may include sample-level features that include one or more of the area of the TLS region 135, the total count of lymphocytes, the area ratio, the count ratio, the maximum value of the area, the maximum value of the longest distance from the tumor, the maximum value of the perimeter, the maximum value of the shortest distance from the tumor, the maximum value of the total area, the maximum value of the total count, the average value of the area, the average value of the longest distance from the tumor, the average value of the perimeter, the average value of the shortest distance from the tumor, the average value of the total area, the average value of the total count, the median value of the area, the median value of the longest distance from the tumor, the median value of the perimeter, the median value of the shortest distance from the tumor, the median value of the total area, the median value of the total count, the minimum value of the area, the minimum value of the longest distance from the tumor, the minimum value of the perimeter, the minimum value of the shortest distance from the tumor, the minimum value of the total area, or the minimum value of the total count.

[0042] Figures 2A - 2K show a plurality of tables listing TLS features 140 that can be extracted by TLS extractor 145. Each table includes multiple columns that enumerate (1) the feature name, (2) the feature type that identifies whether the feature is identification information, metadata, or a feature, and (3) the description of the feature that explains the extracted feature, and a human - interpretable feature (HIF) type that indicates whether the feature is identification information, metadata, an unprocessed feature, a minimum - value feature, a maximum - value feature, a median - value feature, an average - value feature, a ratio - feature, or a sum - feature.

[0043] Referring back to FIG. 1, the TLS classification model 350 may classify each set of TLS features 140 to determine the corresponding TLS region 135 as one of a first TLS maturation state (TLS1), a second TLS maturation state (TLS2), or a third TLS maturation state (TLS3). The first maturation state may be associated with lymphocyte aggregates, the second maturation state may be associated with each cluster of lymphocyte cells having primary follicles without germinal centers, and the third maturation state may be associated with each cluster of lymphocyte cells having primary follicles and secondary cells having germinal centers. Assuming that TLS2 and TLS3 tend to have a circular shape and are typically larger than TLS1, and that TLS3 has a unique germinal center with a lower lymphocyte density, the aforementioned TLS features 140 of area, roundness, and distortion can be interpreted by the TLS classification model 350 learned to accurately classify each TLS region 135. Each TLS region 135 classified by the TLS classification model may correspond to a prognostic biomarker. The TLS classification model 350 may output a TLS state 312 indicating the maturation state of each TLS region 135 classified by the TLS classification model 350.

[0044] In some embodiments, the application 160 executes an image enhancer 360 configured to augment the input organizational structure image 110 based on the TLS status 312 output from the TLS classification model 350 for one or more TLS regions 135 identified in the input organizational structure image 110. Here, the image enhancer 360 may generate each pixel mask 112 that emphasizes at least the surroundings of each corresponding TLS region 135 based on the maturity state of the corresponding TLS region 135 (e.g., TLS1, TLS2, or TLS3). The image enhancer 360 may generate a first pixel mask 112 for the TLS region 135 classified as TLS1, a second pixel mask 112 different from the first pixel mask 112 for the TLS region 135 classified as TLS2, and a third pixel mask 112 different from the first and second pixel masks 112 for the TLS region 135 classified as TLS3. That is, the different pixel masks 112 may be visually distinguishable from each other. In some embodiments, the different pixel masks 112 are associated with different colors. The image enhancer 360 generates an output image 110A that augments the input organizational structure image 110 by superimposing each pixel mask 112 generated for each of the TLS regions 135 over the input organizational structure image 110. The pixel mask 112 is superimposed as a graphical feature that emphasizes at least the surroundings of each corresponding TLS region 135, thereby acting as a visual cue indicating the position and corresponding classification (e.g., TLS1, TLS2, or TLS3) of each TLS region 135 identified in the output image 110A. As will become apparent, the image enhancer 360 may generate the output image 110A by applying one or more post-processing rules. As described in the previous paragraph, the application 160 may provide the output image 110 to the client device 111 for display on the screen 114 as TLS details 190.

[0045] In addition to the maturity state, the TLS classification model 350 and / or the TLS feature extractor 145 may be further configured to output / extract topological information associated with the TLS region 135, such as the coordinates of the TLS region 135, their proximity to the tumor bed, and their position relative to the tumor and / or stromal compartments. Thus, the image enhancer 360 or the image generator may process the topological information and any combination of the input tissue structure image, the TLS state 312, the TLS region 135, and the TLS features to generate a topology or heat map as an output image 110A that visually shows the topological information associated with the TLS region 135 of interest.

[0046] Continuing to refer to FIG. 1, the application 160 may further execute a TLS score determiner 150 for calculating an overall TLS score 152 for the patient based on the TLS features 140 and the corresponding TLS maturity states 312 for all TLS regions 135 identified in the input tissue structure image 110. The TLS score determiner 150 may determine the total area of the tumor region 115 (denoted as "area" tumor ). The TLS score determiner 150 may further determine each individual TLS area for each of the three TLS maturity states. For example, the TLS score determiner 150 may determine a first TLS area (denoted as "area" TLS1 ) based on the total area of the TLS regions classified as the first maturity state, a second TLS area (denoted as "area" TLS2 ) based on the total area of the TLS regions classified as the second maturity state, and a third TLS area (denoted as "area" TLS2 ) based on the total area of the TLS regions classified as the second maturity state. In some embodiments, the TLS score determiner 150 calculates the overall TLS score 152 as a linearly weighted sum of the individual TLS areas divided by the tumor area, as follows.

[0047] TLS score = (w1 × area TLS1 + w2 × area TLS2 + w3 × area TLS3) (1)

[0048] Here, w1, w2, and w3 are the corresponding weights. The optimal corresponding weights may be selected by performing a Cox regression analysis over the entire survival period for each of the individual TLS areas. In one embodiment, w1 is equal to 0.81, w2 is equal to 0.84, and w3 is equal to 1.0. This suggests that the TLS regions classified as the third maturation state (e.g., mature TLS) play the most important role in the anti-tumor immune response.

[0049] In particular, the statistical analysis applied to the overall TLS score 152 and the individual TLS scores represented by the first, second, and third TLS areas may be used to predict various prognostic values for the patient, such as, but not limited to, predicted survival outcomes including overall survival and progression-free survival. The overall survival may be defined as the time from diagnosis to death or the last follow-up observation. The progression-free survival may be defined as the time from diagnosis to disease progression, death, or the last follow-up observation. Univariate and multivariate analyses may be performed with the Cox proportional hazards model. The multivariate analysis may include clinical and pathological variables such as tumor stage and grade. The Kaplan-Meier analysis and log-rank test may be used to evaluate the stratification of patients by risk group. The TLS score may be evaluated in association with the tumor state or grade. A higher overall TLS score indicates significantly improved overall survival and progression-free survival compared to a lower overall TLS score. For patients with a low overall TLS score, the overall survival and progression-free survival are even better than those in the absence of the identified TLS regions. Thus, the overall TLS score may be used to predict prognostic outcomes instead of using tumor stage prediction, and / or the prognostic outcomes predicted using tumor stage / grade may be further refined by the overall TLS score.

[0050] In some scenarios, application 160 performs post - processing to adjust output image 110A based on any combination of TLS features 140, one or more TLS scores 152, and TLS states 312. In particular, application 160 may apply one or more post - processing rules 362 to modify pixel mask 112 by fixing small unmasked germinal centers, fixing TLS regions 135 without germinal centers classified as a third maturation state (mature TLS), fixing mosaics to address multiple - class predictions in the same structure due to confusion by the TLS classification model, applying object - level masking to remove false - positive predictions of TLS inside cancer and necrotic tissue regions, and / or applying a cutoff.

[0051] Referring to FIG. 3A, in some embodiments, an exemplary TLS classification model learning process 300a causes TLS classification model 350 to learn a method for predicting the TLS state regarding the TLS regions identified in the tissue structure image. The learning process 300a obtains a training dataset 305 that includes a plurality of training tissue structure images 310, 310a - n. Each training tissue structure image 310 may include a tumor microenvironment and may include manual annotation 312 from a qualified pathologist. The manual annotation 312 identifies one or more TLS regions 312a in the training tissue structure image 310 and, for each identified TLS region 312a, a ground - truth TLS maturation state 312b indicating that the corresponding TLS region 312a includes a first TLS maturation state, a second TLS maturation state, or a third TLS maturation state. Each TLS region 312a annotated in the training tissue structure image 310 is represented by each cluster of lymphocytes.

[0052] The learning process 300a executes a TLS feature extraction module 320, which receives each training organizational structure image 310 and extracts each set of training TLS features 140 for each TLS region 312a. That is, for each TLS region 312a annotated in the training organizational structure image 310, the TLS feature extraction module 320 may extract each set of training TLS features 140 from each cluster of lymphocytes representing the TLS region 312a. The TLS feature extraction module 320 may include a pre-trained tumor extraction model 450 and a pre-trained cell classification model 550 for generating a lymphocyte density map. The feature extraction module 320 may also include any other optional component, or combination of components, executed by the application 160.

[0053] The training TLS features may include, but are not limited to, the area 140a of the TLS region, the roundness 140b (i.e., the ratio of the area of the TLS region 312a multiplied by 4π to the square of the perimeter of the TLS region), and the density distortion 140c of each cluster of lymphocytes representing the TLS region 312a. Based on each set of training TLS features 140 extracted for each TLS region 312a, the learning process 300a uses a CART (classification and regression trees) algorithm 340 to train a TLS classification model 350 to learn a method for predicting the ground truth TLS state 312b for each corresponding TLS region 312a. In some embodiments, the training process 300a uses default parameter settings (criterion = gini; splitter = best; min_samples_split = 2) and uses the scikit-learn package from Python programming language version 3.6.11 (Python Software Foundation) to train the CART algorithm 340. The deepest depth of the tree was determined to be 4 using 5-fold cross-validation in the training dataset 305. Assuming the relative importance of TLS3, the class weights for TLS1, TLS2, and TLS3 may be empirically set to 1, 2, and 3, respectively, during training.

[0054] Figure 4 is a flowchart of an exemplary configuration of operations related to a method 400 for identifying, classifying, and quantifying TLS regions 135 within an input tissue structure image 110. Method 400 may be executed in the data processing hardware 142 of a remote system 141 and / or in a client device 111. In operation 402, method 400 includes receiving an input tissue structure image 110 related to a patient diagnosed with cancer. The input tissue structure image includes a plurality of image pixels. The input tissue structure image 110 may include a H&E stained image of a sample of the patient's tumor.

[0055] In operation 404, method 400 includes generating one or more lymphocyte density maps 125 within the input tissue structure image 110 by processing the input tissue structure image 110 using a cell classification model 550. In operation 406, method 400 includes identifying one or more TLS regions 135 within the input tissue structure image 110 by performing morphological image processing on the one or more lymphocyte density maps 125. Here, each TLS region 135 is represented by each cluster of lymphocyte cells.

[0056] In operation 408, method 400 includes extracting each set of TLS features 140 from each cluster of lymphocyte cells representing the corresponding TLS region 135 for each corresponding TLS region 135. In operation 410, method 400 includes classifying each corresponding TLS region as one of a first TLS maturation state, a second TLS maturation state, and a third TLS maturation state by processing each set of TLS features using a TLS classification model 350 for each corresponding TLS region 135. The first TLS maturation state includes lymphocyte aggregates of at least a threshold number of lymphocytes that do not include high endothelial venules or germinal centers. The second TLS maturation state includes dense aggregates of at least the above-mentioned threshold number of lymphocytes that include high endothelial venules and do not include germinal centers. The third TLS maturation state includes dense aggregates of at least a threshold number of lymphocytes that include high endothelial venules and germinal centers.

[0057] As a superior feature, after training the TLS classification model 350, the accuracy of the TLS classification model 350 for identifying and classifying the TLS regions 135 within the input tissue structure image 110 is comparable to (or, in some scenarios, even better than) the accuracy of a pathologist manually classifying TLS. For example, the confusion matrix 500 shown in FIGS. 5A and 5B shows the confusion matrix 500 comparing the accuracy 510 between the pathologist and the trained TLS classification model 350. In particular, the first confusion matrix 500, 500a (FIG. 5A) shows the normalized accuracy 510 of the TLS classification model 350, and the second confusion matrix 500, 500b (FIG. 5B) shows the normalized accuracy of the pathologist annotator. Here, the confusion matrix 500 shows the accuracy 510 for each TLS maturation state 312 (e.g., mature TLS, immature TLS, germinal center, lymphocyte aggregates, and others). These ground truth maturation states of the input tissue structure images 110 were generated by a majority consensus of five experienced pathologists. Further, FIGS. 6A-6C show plots 600 comparing the TLS identification and classification performance between the TLS classification model 350 and the pathologist annotator. In particular, the first plot 600, 600a (FIG. 6A) shows a comparison of the accuracy scores 610, the second plot 600, 600b (FIG. 6B) shows a comparison of the F1 scores 620, and the third plot 600, 600c (FIG. 6C) shows a comparison of the recall scores 630. Here, each plot 600 graphically represents the scores for each of the different TLS maturation states 312.

[0058] FIG. 7 shows input tissue structure images 700 each corresponding to a TLS maturation state 312 classified by the TLS classification model 350. Thus, the input tissue structure image 110 (FIG. 1) may also be interchangeably referred to as the input tissue structure image 700 with respect to FIG. 7. In particular, the input tissue structure image 700 includes the classified TLS maturation state 312 but is not annotated as the output image 110A. In some embodiments, the input tissue structure image 700 corresponds to the entire area of the input tissue structure image 700. In other embodiments, the input tissue structure image corresponds only to the tumor region 115 detected by the tumor detection model 450 within the input tissue structure image 700, or only to the TLS region 135 identified by the morphological image processor 130 (FIG. 1).

[0059] In the illustrated embodiment, the first input tissue structure images 700, 700a correspond to the first TLS maturation states 312, 312a indicating the lymphocyte aggregate maturation state. In particular, the input tissue structure image 700 corresponding to the first TLS maturation state 312a may include a dense aggregate of at least a threshold number of lymphocytes (e.g., 100 lymphocytes) that does not include high endothelial venules and germinal centers. The second input tissue structure images 700, 700b correspond to the second TLS maturation states 312, 312b indicating the immature TLS maturation state. The input tissue structure image 700 corresponding to the second TLS maturation state 312b may include a dense aggregate of at least a threshold number of lymphocytes (e.g., 100 lymphocytes) that includes high endothelial venules (in contrast to the first TLS maturation state 312a) but does not include germinal centers. The third input tissue structure images 700, 700c correspond to the third TLS maturation states 312, 312c indicating the mature TLS maturation state. The input tissue structure image 700 corresponding to the third TLS maturation state 312c may include a dense aggregate of at least a threshold number of lymphocytes (e.g., 100 lymphocytes) that includes high endothelial venules and germinal centers 313 (in contrast to the first and second TLS maturation states 312a, 312b).

[0060] Continuing to refer to FIG. 7, the fourth input tissue structure images 700, 700d show germinal centers 313. In some embodiments, the germinal centers 313 are not distinct TLS maturation states 312; rather, the germinal centers 313 are characteristic of the mature TLS maturation state 312c. In other embodiments, the TLS classification model 350 classifies the germinal centers 313 as distinct TLS maturation states 312 independent of the other TLS maturation states 312. The input tissue structure image 700 including the germinal centers 313 includes a lighter-colored, less dense region at the center of a mature TLS (e.g., the third TLS maturation state 312c) surrounded by a dense lymphocyte region. Although not shown in FIG. 7, the TLS classification model 350 may also classify a fourth TLS maturation state (not shown) indicating a non-TLS region (e.g., a zero TLS region present in the input tissue structure image 700) or other regions. As used herein, the first TLS maturation state 312a, the second TLS maturation state 312b, and the third TLS maturation state 312c may each be interchangeably referred to as a lymphocyte aggregate TLS maturation state 312a, an immature TLS maturation state 312b, and a mature TLS maturation state 312c, respectively.

[0061] FIGS. 8-10 show an exemplary input tissue structure image 110 and a corresponding output image (e.g., a TLS-enhanced tissue structure image) 110A generated by the image enhancer 360 (FIG. 1). In other words, the application 160 receives the exemplary input tissue structure image 110 (right side) shown in FIGS. 8-10 as an input and generates the output image 110A (left side) as an output. In some embodiments, the image enhancer 360 generates each pixel mask 112 that emphasizes at least the perimeter of the corresponding TLS region 135. In other embodiments, each pixel mask 112 emphasizes the entire area of the corresponding TLS region 135. Accordingly, the image enhancer 360 may generate an output image 110A that enhances the input tissue structure image 110 by superimposing each pixel mask 112 generated for each of the TLS regions 135 over the input tissue structure image 110.

[0062] In addition, the image enhancer 360 generates a first pixel mask 112, 112a for each corresponding TLS region 135 classified as a first TLS maturity state 312a, a second pixel mask 112, 112b for each corresponding TLS region 135 classified as a second maturity state 312b, and a third pixel mask 112, 112c for each corresponding TLS region 135 classified as a third maturity state 312c. In particular, each pixel mask 112 is visually distinguishable from other pixel masks 112 such that the output image 110A visually indicates different maturity states 312 using visually distinct pixel masks 112. Thus, the output image 110A is displayed on the screen 114 of the user device 111 such that the user 10 (FIG. 1) can easily visualize the different TLS maturity states 312 included in the output image 110A. Optionally, the image enhancer 360 may generate a fourth pixel mask 112, 112d for each corresponding TLS region 135 classified as a non-TLS region.

[0063] For example, FIG. 8 shows a graphical representation 800 of an input tissue structure image 110 (right side) representing tissue in a mature TLS maturity state 312c and a corresponding output image 110A (left side) including a third pixel mask 112c that emphasizes the area of the TLS region 135 classified as the mature TLS maturity state 312c. Further, the mature TLS maturity state 312c includes a germinal center 313 encompassed by the TLS region 135 corresponding to the mature TLS maturity state 312c. For that purpose, the third pixel mask 112c includes an inner third pixel mask 112c1 that emphasizes the area of the germinal center 313 and an outer third pixel mask 112c2 that emphasizes the area of the mature TLS maturity state 312c.

[0064] FIG. 9 shows a graphical representation 900 of an input tissue structure image 110 (right side) representing the tissue of the immature TLS maturation state 312b and a corresponding output image 110A (left side) including a second pixel mask 112b that emphasizes the area of the TLS region 135 classified as the immature TLS maturation state 312b. In yet another example, FIG. 10 shows a graphical representation 1000 of an input tissue structure image 110 (right side) representing the tissue of the lymphocyte aggregate TLS maturation state 312a and a corresponding output image 110A (left side) including a first pixel mask 112a that emphasizes the area of the TLS region 135 classified as the lymphocyte aggregate TLS maturation state 312a. Also, each of the output images 110A shown in the graphical representations 800, 900, 1000 further includes a fourth pixel mask 112d that emphasizes the area of the output image 110A corresponding to the non-TLS mature TLS region (e.g., non-TLS mature state).

[0065] Referring now to FIG. 11, in some embodiments, the input tissue structure image 110 includes several classified TLS maturation states 312. For example, the graphical representation 1100 shows an output image 110A that includes three TLS regions 135 corresponding to each of the first, second, and third TLS maturation states 312a - c. Here, each pixel mask 112 superimposed on the input tissue structure image easily shows the user the different identified TLS regions 135 and the corresponding classified TLS maturation states 312. As shown in FIG. 11, the output image 110A includes a first TLS region 135, 135a classified as a lymphocyte aggregate TLS maturation state 312a, a second TLS region 135, 135b classified as an immature TLS maturation state 312b, and a third TLS region 135, 135c classified as a mature TLS classification state 312c that includes a germinal center 313. Also, below the output image 110A, an enlarged view of the identified TLS region 135 is shown adjacent to the corresponding input tissue structure image 110. For example, the first TLS region 135a includes a first pixel mask 112a that emphasizes the area of the first TLS region 135a as a lymphocyte aggregate TLS maturation state 312a, the second TLS region 135b includes a second pixel mask 112b that emphasizes the area of the second TLS region 135b as an immature TLS maturation state 312b, and the third TLS region 135c includes a third pixel mask 112c that emphasizes the area of the third TLS region 135c as a mature TLS maturation state 312c. Adjacent to each enlarged TLS region 135 is the corresponding portion of the input tissue structure image 110 input into the application 160 that corresponds to the TLS region 135.

[0066] Referring back to FIG. 1, in some embodiments, the image enhancer 360 applies one or more post - processing rules 362 before generating the output image 110A. That is, in some scenarios, the TLS classification model 350 classifies the TLS region 135 as a particular TLS maturation state 312 that does not meet a threshold (e.g., a post - processing threshold). Thus, applying the post - processing rules 362 filters out and removes the classified TLS maturation states 312 that do not meet one or more post - processing rules. In this way, the image enhancer 360 corrects any false positives or otherwise incorrect classifications generated by the TLS classification model 350 by applying the post - processing rules 362. For example, the post - processing rules 362 may include, but are not limited to, fixing small exposed germinal centers, fixing mature TLS without germinal centers, fixing mosaics to address predictions of multiple classes in the same structure due to model confusion, performing object - level masking to remove false - positive predictions of TLS inside cancer and necrotic tissue regions, and / or applying a cutoff.

[0067] Figures 12 to 15 show the output image 110A generated by the image enhancer 360 both when the post - processing rule 362 is applied and when it is not applied. When the image enhancer 360 does not apply the post - processing rule 362, the output image 110A may be referred to as the un - converted output images 110A, 110A1. On the other hand, when the image enhancer 360 applies the post - processing rule 362, the output image 110A may be referred to as the converted output images 110A, 110A2. For example, FIG. 12 shows a graphical representation 1200 of the output image 110A when applying a post - processing rule 362 that fixes (i.e., filters) the small exposed germ center. As shown in FIG. 12, the un - converted output image 110A1 includes the germ center 313, and the germ center 313 is classified as the mature TLS mature state 312c and the non - TLS mature state, and is partially surrounded by the TLS region 135 indicated by each of those pixel masks 112. Here, the post - processing rule 362 defines that for a germ center 313 that does not meet a threshold amount (e.g., 70% of the germ center 313 surrounded by the mature TLS) of the TLS region 135 classified as the mature TLS mature state 312c surrounding the germ center 313, the image enhancer 360 re - classifies the germ center 313 as the TLS mature state 312 that surrounds most of the germ center 313.

[0068] For example, as shown in FIG. 12, the untransformed output image 110A1 includes an outer third pixel mask 212c2 (e.g., indicating a mature TLS mature state 312c), which only partially surrounds an inner third pixel mask 212c1 (e.g., indicating a germinal center 313), thereby not meeting the threshold amount. Thus, since most of the area around the germinal center 313 is surrounded by non-TLS regions, the image enhancer 360 reclassifies the germinal center 313 as a non-TLS mature state. As a result, the transformed output image 110A2 removes (i.e., filters) the germinal center 313 such that the transformed output image 110A2 only includes a fourth pixel mask 112d. Alternatively, the post-processing rule 362 may define that for a germinal center 313 having an area that does not meet a threshold area (e.g., 4480 μm2), the image enhancer 360 reclassifies the germinal center 313 as a TLS mature state 312 that surrounds most of the area around the germinal center 313.

[0069] Referring now to FIG. 13, in some embodiments, the post - processing rule 362 is configured to fix a classified mature TLS maturation state 312c that does not have a germinal center 313. Here, the image enhancer 360 re - classifies a TLS region classified as a mature TLS maturation state 312c that is not connected to the germinal center 313 as an immature TLS maturation state 312b. For example, the region of the mature TLS maturation state 312c may need to completely enclose the germinal center 313 or partially enclose the germinal center 313 that meets a threshold. As shown in FIG. 13, the graphical representation 1300 includes the un - converted output image 110A1, which includes a second pixel mask 112b (e.g., indicating an immature TLS maturation state 312b), an inner third pixel mask 112c1 (e.g., indicating the germinal center 313), and an outer third pixel mask 112c2 (e.g., indicating the mature TLS maturation state 312c). In this embodiment, the outer third pixel mask 112c2 does not enclose the germinal center 313 by a threshold. That is, the outer third pixel mask 112c2 only partially encloses the germinal center 313 and is not sufficient to meet the threshold. Thus, in this scenario, the image enhancer 360 re - classifies the mature TLS maturation state 312c and the germinal center 313 as an immature TLS maturation state 312b, as shown in the converted output image 110A2 having the second pixel mask 112b. The output image 110A also includes a fourth pixel mask 112d corresponding to the non - TLS region of the output image 110.

[0070] Referring now to FIG. 14, in some embodiments, the post - processing rule 362 is configured to fix the mosaic 1402 included in the output image 110A. Here, the mosaic 1402 represents a single TLS region 135 that includes a plurality of classified TLS maturation states 312. In some configurations, when the mosaic 1402 includes at least the immature TLS maturation state 312b and the lymphocyte aggregate TLS maturation state 312a, the image enhancer 360 re - classifies the entire mosaic 1402 as the immature TLS maturation state 312b based on determining that the mosaic 1402 includes a threshold ratio of the immature TLS maturation state 312b (e.g., 70 percent). Otherwise, the image enhancer 360 re - classifies the entire mosaic 1402 as the lymphocyte aggregate TLS maturation state 312a. In other configurations, when the mosaic 1402 includes at least the immature TLS maturation state 312b and the mature TLS maturation state 312c, the image enhancer 360 re - classifies the entire mosaic 1402 as the mature TLS maturation state 312c based on determining that the mosaic includes a threshold ratio of the mature TLS maturation state 312c (e.g., 70 percent). Otherwise, the image enhancer 360 re - classifies the entire mosaic 1402 as the immature TLS maturation state 312b. In still other configurations, when the mosaic 1402 includes at least the lymphocyte aggregate TLS maturation state 312a and the mature TLS maturation state 312c, the image enhancer 360 re - classifies the entire mosaic 1402 as the mature TLS maturation state 312c based on determining that the mosaic 1402 includes a threshold ratio of the mature TLS maturation state 312c (e.g., 70 percent). Otherwise, the image enhancer 360 re - classifies the entire mosaic 1402 as the lymphocyte aggregate TLS maturation state 312a.

[0071] As shown in FIG. 14, the graphical representation 1400 includes an untransformed output image 110A1 showing a mosaic 1402 that includes a first pixel mask 112a (e.g., indicating a lymphocyte aggregate TLS maturation state 312a) and a second pixel mask 112b (e.g., indicating an immature TLS maturation state 312b). Here, the mosaic 1402 does not meet the threshold ratio of the immature TLS maturation state 312b. Thus, the image enhancer 360 reclassifies the entire area of the mosaic 1402 as a lymphocyte aggregate maturation state 312a, as shown in the transformed output 110A2 that includes the first pixel mask 112a. In particular, since the TLS region includes only a single TLS maturation state 312, the transformed output 110A2 removes the mosaic 1402. The output image 110A also includes a fourth pixel mask 112d corresponding to the non-TLS region of the output image 110.

[0072] Referring now to FIG. 15, in some embodiments, the post-processing rule 362 is configured to remove false positive predictions of the TLS maturation state 312 inside the tissue regions of cancer and necrosis. In particular, the image enhancer 360 determines whether the ratio of cancer and necrosis in an object or TLS region classified as a first, second, or third TLS maturation state 312a, 312b, 312c meets the threshold ratio (e.g., 20 percent) of the object or TLS region. In response to determining that the ratio of cancer and necrosis meets the threshold ratio, the image enhancer 360 reclassifies the TLS maturation state 312 as a non-TLS maturation state. For example, as shown in FIG. 15, the graphical representation 1500 includes an untransformed output image 110A1 that includes a cancer pixel mask 1502 and a necrosis pixel mask 1504. In this embodiment, the cancer pixel mask 1502 and the necrosis pixel mask 1504 meet the threshold ratio of the tissue, and thus, the image enhancer 360 reclassifies the cancer pixel mask 1502 and the necrosis pixel mask 1504 as a non-TLS maturation state 312d. Thus, the transformed output image 110A2 includes only a fourth pixel mask 112d corresponding to the non-TLS region of the transformed output image 110A.

[0073] Referring now to FIG. 16, in some embodiments, the post - processing rule 362 is configured to apply a cutoff that filters the classified TLS maturation state 312 that does not meet any of a minimum threshold area, a maximum threshold area, and / or a maximum number of germinal centers 313. The TLS maturation state 312 that does not meet the threshold is re - classified as a non - TLS region. In particular, the lymphocyte aggregate TLS maturation state 312a may have a minimum threshold area (e.g., 0.0008 mm^2) and may not have a maximum threshold area. On the other hand, the immature TLS maturation state 312b may include a minimum threshold area (e.g., 0.018 mm^2) and a maximum threshold area (e.g., 2.0 mm^2). Also, the mature TLS maturation state 312c may have a maximum number of germinal centers 313 (e.g., 8 germinal centers 313). For example, if the mature TLS maturation state 312c includes a large number of germinal centers 313 that exceed the maximum number threshold, the image enhancer 360 re - classifies the mature TLS maturation state 312c as a non - TLS region. As shown in FIG. 16, the graphical representation 1600 includes the untransformed output image 110A1, which includes a first pixel mask 112a, a second pixel mask 112b, an inner third pixel mask 112c1, an outer third pixel mask 112c2, and a fourth pixel mask 112d. However, none of the pixel masks 112 meet the cutoff threshold, and thus, the image enhancer 360 re - classifies each of the first, second, and third TLS maturation states 312a - c as a fourth TLS classification state 312d, as shown in the transformed output image 110A2. That is, the transformed output image 110A2 includes only the fourth TLS classification state 312d.

[0074] FIG. 17 shows a processing flowchart 1700 for verifying the extracted TLS feature 140 using ribonucleic acid (RNA) sequence analysis or transcriptome analysis correlation. That is, various gene signatures of TLS related to either chemokines or cell populations were studied. For example, FIG. 18 shows a table 1800 of 12 chemokine gene signatures derived by correlating a meta-gene related to combustion, which was associated with improved patient survival in colorectal cancer, melanoma, and breast cancer. The 12 chemokine gene signatures in table 1800 include CCL2, CCL3, CCL4, CCL5, CCL8, CCL18, CCL19, CCL21, CXCL9, CXCL10, CXCL11, and CXCL13. In another example, an 8-gene signature representing T follicular helper (TFH) cells, particularly including CXCL13, characterizes breast cancer. In yet another example, a 19-gene signature related to T helper type 1 (Th1) cells and B cells indicates the presence of TLS. Despite various gene signatures correlating with the presence of TLS, recently, a limited number of studies have investigated the most accurate TLS gene signatures.

[0075] Referring again to FIG. 17, while immunohistochemical detection of TLS in tissue sections is a robust and specific approach, the process flow diagram 1700 aims to compare several gene signatures extracted from TLS-positive cancer tissues. As will become apparent, the diversity of gene expression among different cancer types leads to a better understanding of gene signatures correlated with the presence of TLS. In particular, the process flow diagram 1700 includes a TLS feature extraction module 320, a transcriptome module 1710, a feature selector 1720, and a clustering module 1730. The TLS feature extraction module 320 is configured to receive the input tissue structure image 110 as input and extract the TLS feature 140 corresponding to each input tissue structure image 110. For example, the TLS feature extraction module 320 may extract the TLS feature 140 using the TLS feature extractor 145 (FIG. 1).

[0076] The transcriptome module 1710 is configured to receive an input tissue structure image 110 as input and generate, as output, a gene expression signature (GES) 1712 for each input tissue structure image 110. Here, the transcriptome module 1710 may generate the GES 1712 by extracting an RNA sequence from each input tissue structure image 110. Using the TLS features 140 and the GES 1712 generated for each of the input tissue structure images 110, the feature selector 1720 generates a feature table 1722. That is, for each input tissue structure image 110, the feature extractor 1720 pairs, in the feature table 1722, the TLS features 140 and the GES 172 derived from each input tissue structure image 110. The feature table 1722 includes pairings for all of the received input tissue structure images 110. Thus, the feature table 1722 structures the TLS features 140 and the GES 1712 so that the clustering module 1730 can determine the correlation between the TLS features and the GES 1712. In some embodiments, the feature table 1722 includes, as shown in table 1900 (FIG. 19), the number of other TLS features 140 and corresponding annotations for each TLS feature 140 in a set of input tissue structure images. In some embodiments, the feature selector 1720 may filter the feature table 1722 to include only specific TLS features 140. For example, the feature selector 1720 may apply a linear regression lasso penalty to generate the feature table 1722.

[0077] Continuing to refer to FIG. 17, the clustering module 1730 is configured to receive the clustering table 1722 as an input and generate the correlation data 1732 as an output. In particular, using the gene signature data indicating the presence and classification of TLS, the clustering module 1730 may verify that the extracted TLS feature 140 correlates with the presence and classification of TLS in the input tissue structure image 110. Also, the clustering module 1730 may further determine a gene signature indicating the presence and classification of TLS in the tissue by binning with the extracted TLS feature 140. That is, the clustering module 1730 may further determine a gene signature that can identify hitherto unknown TLS.

[0078] For example, FIGS. 20A to 20C show a graphical representation 200 of exemplary correlation data 1732 (FIG. 17) verifying that the TLS feature 140 strongly correlates with GES in an exemplary breast cancer gene (BRCA) analysis. In particular, the graphical representation 2000a (FIG. 20A) shows a correlation diagram 2002 indicating that the TLS maturation state 312 and the TLS feature 140 correspond to the TLS-inducing genes shown in table 2004 in the BRCA analysis. Also, the correlation diagram 2002 shows the TLS-inducing genes occurring in each of the first, second, and third TLS maturation states 312a to c and the TLS-inducing genes correlated with the individual TLS maturation states 312. Further processing of the correlation diagram 2002 by the clustering module 1730 (FIG. 17) may generate a signature for a given cancer. Briefly, the correlation diagram 2002 emphasizes that the TLS feature 140 strongly correlates with GES for the input tissue structure image 110 of BRCA.

[0079] FIG. 20B shows a graphical representation 2000b of a hierarchical clustering plot. Here, the plot includes cluster 1 corresponding to low-expression breast cancer samples, cluster 2 corresponding to medium-expression breast cancer samples, and cluster 3 corresponding to high-expression breast cancer samples. Along the Y-axis, the first, second, and third TLS maturation states 312a - c and TLS-inducing genes are plotted for each of the breast sample clusters. FIG. 20C shows a graphical representation 2000c of a plot with the X-axis as the timeline in months and the Y-axis as the overall survival rate of patients from breast cancer samples. Thus, the graphical representation 2000c shows that breast cancer samples in the cluster containing upregulated chemokines have a higher long-term overall survival rate.

[0080] FIGS. 21A - 21D show exemplary graphical representations 2100 of correlation diagrams for verifying TLS features 140 extracted by gene signatures. The graphical representations 2100 correlate the TLS features 140 and gene signatures among different cancer types (X-axis), including BRCA, bladder cancer (BLCA), lung adenocarcinoma (LUAD), lung squamous cell carcinoma (LUSC), and stomach adenocarcinoma (STAD). Also, each graphical representation 2100 plots the TLS-inducing genes along the Y-axis. For example, graphical representation 2100a (FIG. 21A) includes the TLS features 140 of the proportional area of the mature TLS maturation state 312c, graphical representation 2100b (FIG. 21B) includes the TLS features 140 regarding the proportional area of the immature TLS maturation state 312b, and graphical representation 2100c (FIG. 21C) includes the TLS features 140 regarding the proportional area of the lymphocyte aggregate TLS maturation state 312c. FIG. 2DC shows a graphical representation 2000d of a plot with the X-axis as the timeline in months and the Y-axis as the overall survival rate of patients from LUAD cancer samples and BRCA cancer samples. Thus, the graphical representation 2100d shows that the proportional areas of different TLS maturation states 312 correlate with a subset of TLS-inducing genes, and in particular, the proportional area of the mature TLS maturation state 312c demonstrates prognostic value in LUAD and BRCA samples. Thus, the graphical representation 2000c shows that breast cancer samples in the cluster containing upregulated chemokines have a higher long-term overall survival rate.

[0081] Anti-PD-1 antibodies known in the art can be used in the configurations and methods described herein. Various human monoclonal antibodies having high affinity and specifically binding to PD-1 are disclosed in Patent Document 1. The anti-PD-1 human antibodies disclosed in Patent Document 1 have been demonstrated to exhibit one or more of the following characteristics: (a) having a K -7 of 1×10 D or less and binding to human PD-1 as determined by surface plasmon resonance using a Biacore biosensor system; (b) not substantially binding to human CD28, CTLA-4, or ICOS; (c) increasing T cell proliferation in a mixed lymphocyte reaction (MLR) assay; (d) increasing interferon-γ production in an MLR assay; (e) increasing IL-2 secretion in an MLR assay; (f) binding to human PD-1 and cynomolgus monkey PD-1; (g) inhibiting the binding of PD-L1 and / or PD-L2 to PD-1; (h) stimulating an antigen-specific memory response; (i) stimulating an antibody response; and (j) inhibiting the growth of tumor cells in vivo. Anti-PD-1 antibodies that can be used in the present disclosure include monoclonal antibodies that specifically bind to human PD-1 and exhibit at least one of the above-described characteristics, and in some embodiments, at least five of them.

[0082] Other anti-PD-1 monoclonal antibodies are described, for example, in Patent Documents 2 to 30, the entire contents of each of which are incorporated by reference.

[0083] In some embodiments, the anti-PD-1 antibody is selected from the group consisting of nivolumab (also known as OPDIVO®, 5C4, BMS-936558, MDX-1106, and ONO-4538), pembrolizumab (Merck; also known as KEYTRUDA®, lambrolizumab, and MK-3475; see Patent Document 8), PDR001 (Novartis; see Patent Document 9), MEDI-0680 (AstraZeneca; also known as AMP-514; see Patent Document 7), semaprimab (Regeneron; also known as REGN-2810; see Patent Document 10), JS001 (TAIZHOU JUNSHI PHARMA; also known as toripalimab; see Non-Patent Document 4), BGB-A317 (Beigene; also known as Tislelizumab; see Patent Document 12 and Patent Document 31), INCSHR1210 (Jiangsu Hengrui Medicine; also known as SHR-1210; see Patent Document 13; Non-Patent Document 4), TSR-042 (Tesaro Biopharmaceutical; also known as ANB011; see Patent Document 14), GLS-010 (Wuxi / Harbin Gloria Pharmaceuticals; also known as WBP3055; see Non-Patent Document 4), AM-0001 (Armo), STI-1110 (Sorrento Therapeutics; see Patent Document 22), AGEN2034 (Agenus; see Patent Document 23), MGA012 (Macrogenics, see Patent Document 29), BCD-100 (Biocad; Non-Patent Document 5), and IBI308 (Innovent; see Patent Document 26, Patent Document 27, Patent Document 30, and Patent Document 24).

[0084] Nivolumab is a fully human IgG4 (S228P) PD-1 immune checkpoint inhibitor antibody that selectively prevents interaction with PD-1 ligands (PD-L1 and PD-L2), thereby blocking the downregulation of anti-tumor T cell function (Patent Document 1, Non-Patent Document 6). Pembrolizumab is a humanized monoclonal IgG4 (S228P) antibody that targets the human cell surface receptor PD-1 (programmed death-1 or programmed cell death-1). Pembrolizumab is described, for example, in Patent Document 5 and Patent Document 32.

[0085] Anti-PD-1 antibodies that can be used in the disclosed configurations and methods are also isolated antibodies that specifically bind to human PD-1 and, when binding to human PD-1, include antibodies that cross-compete with any anti-PD-1 antibody disclosed herein, such as nivolumab (see, for example, Patent Document 1 and Patent Document 33, Patent Document 34). In some embodiments, the anti-PD-1 antibody binds to the same epitope as any of the anti-PD-1 antibodies described herein, such as nivolumab. The ability of antibodies to cross-compete when binding to an antigen indicates that these monoclonal antibodies bind to the same epitope region of the antigen and sterically hinder the binding of other cross-competing antibodies to that specific epitope region. These cross-competing antibodies are expected to have functional properties very similar to those of a reference antibody, such as nivolumab, due to their binding to the same epitope region of PD-1. Cross-competing antibodies can be readily identified based on their ability to cross-compete with nivolumab in standard PD-1 binding assays such as Biacore analysis, ELISA analysis, or flow cytometry (see, for example, Patent Document 34).

[0086] In some embodiments, nivolumab, an antibody that cross-competes when binding to human PD-1 or binds to the same epitope region of a human PD-1 antibody, is a monoclonal antibody. For administration to human subjects, these cross-competing antibodies are chimeric antibodies, modified antibodies, or humanized or human antibodies. Such chimeric, modified, humanized, or human monoclonal antibodies can be prepared and isolated by methods well known in the art.

[0087] Anti-PD-1 antibodies useful in the compositions and methods of the present disclosure also include the antigen-binding portions of the antibodies described above. It has been well demonstrated that the antigen-binding function of an antibody can be performed by a fragment of the full-length antibody.

[0088] Anti-PD-1 antibodies suitable for use in the disclosed compositions and methods are antibodies that bind to PD-1 with high specificity and affinity, block the binding of PD-L1 or PD-L2, and inhibit the immunosuppressive effect of the PD-1 signaling pathway. In any of the compositions or methods disclosed herein, an anti-PD-1 "antibody" is an antigen-binding portion or fragment that binds to the PD-1 receptor and includes an antigen-binding portion or fragment that exhibits functional properties similar to those of the whole antibody with respect to inhibiting ligand binding and upregulating the immune system. In certain embodiments, the anti-PD-1 antibody or its antigen-binding portion cross-competes with nivolumab when binding to human PD-1.

[0089] In some embodiments, the anti-PD-1 antibody is administered at a dosage per body weight in the range of from 0.1 mg / kg to 20.0 mg / kg, once every 2, 3, 4, 5, 6, 7, or 8 weeks, for example, at a dosage per body weight of from 0.1 mg / kg to 10.0 mg / kg, once every 2, 3, or 4 weeks. In other embodiments, the anti-PD-1 antibody is administered at a dosage per body weight of about 2 mg / kg, about 3 mg / kg, about 4 mg / kg, about 5 mg / kg, about 6 mg / kg, about 7 mg / kg, about 8 mg / kg, about 9 mg / kg, or 10 mg / kg, once every 2 weeks. In other embodiments, the anti-PD-1 antibody is administered at a dosage per body weight of about 2 mg / kg, about 3 mg / kg, about 4 mg / kg, about 5 mg / kg, about 6 mg / kg, about 7 mg / kg, about 8 mg / kg, about 9 mg / kg, or 10 mg / kg, once every 3 weeks. In one embodiment, the anti-PD-1 antibody is administered at a dosage per body weight of about 5 mg / kg, once every 3 weeks. In another embodiment, the anti-PD-1 antibody, such as nivolumab, is administered at a dosage per body weight of about 3 mg / kg, once every 2 weeks. In other embodiments, the anti-PD-1 antibody, such as pembrolizumab, is administered at a dosage per body weight of about 2 mg / kg, once every 3 weeks.

[0090] For the present disclosure, the anti-PD-1 antibody may be administered at a uniform dosage. In some embodiments, the anti-PD-1 antibody is administered at a uniform dosage of from about 100 to about 1000 mg, from about 100 mg to about 900 mg, from about 100 mg to about 800 mg, from about 100 mg to about 700 mg, from about 100 mg to about 600 mg, from about 100 mg to about 500 mg, from about 200 mg to about 1000 mg, from about 200 mg to about 900 mg, from about 200 mg to about 800 mg, from about 200 mg to about 700 mg, from about 200 mg to about 600 mg, from about 200 mg to about 500 mg, from about 200 mg to about 480 mg, or from about 240 mg to about 480 mg. In one embodiment, the anti-PD-1 antibody is administered at a uniform dosage of at least about 200 mg, at least about 220 mg, at least about 240 mg, at least about 260 mg, at least about 280 mg, at least about 300 mg, at least about 320 mg, at least about 340 mg, at least about 360 mg, at least about 380 mg, at least about 400 mg, at least about 420 mg, at least about 440 mg, at least about 460 mg, at least about 480 mg, at least about 500 mg, at least about 520 mg, at least about 540 mg, at least about 550 mg, at least about 560 mg, at least about 580 mg, at least about 600 mg, at least about 620 mg, at least about 640 mg, at least about 660 mg, at least about 680 mg, at least about 700 mg, or at least about 720 mg at dosing intervals of about 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10 weeks. In another embodiment, the anti-PD-1 antibody is administered at a uniform dosage of from about 200 mg to about 800 mg, from about 200 mg to about 700 mg, from about 200 mg to about 600 mg, from about 200 mg to about 500 mg at dosing intervals of about 1, 2, 3, or 4 weeks.

[0091] In some embodiments, the anti-PD-1 antibody is administered at a uniform dosage of about 200 mg once every three weeks. In other embodiments, the anti-PD-1 antibody is administered at a uniform dosage of about 200 mg once every two weeks. In other embodiments, the anti-PD-1 antibody is administered at a uniform dosage of about 240 mg once every two weeks. In certain embodiments, the anti-PD-1 antibody is administered at a uniform dosage of about 480 mg once every four weeks.

[0092] In some additional embodiments, nivolumab is administered at a uniform dosage of about 240 mg once every two weeks. In some embodiments, nivolumab is administered at a uniform dosage of about 240 mg once every three weeks. In some embodiments, nivolumab is administered at a uniform dosage of about 360 mg once every three weeks. In some embodiments, nivolumab is administered at a uniform dosage of about 480 mg once every four weeks.

[0093] Alternatively, pembrolizumab is administered at a uniform dosage of about 200 mg once every two weeks. In some embodiments, pembrolizumab is administered at a uniform dosage of about 200 mg once every three weeks. In some embodiments, pembrolizumab is administered at a uniform dosage of about 400 mg once every four weeks.

[0094] In some aspects, the PD-1 inhibitor is a small molecule. In some aspects, the PD-1 inhibitor comprises a millamolecule. In some aspects, the PD-1 inhibitor comprises a macrocyclic peptide. The PD-1 inhibitor may comprise BMS-986189. In some additional aspects, the PD-1 inhibitor comprises the inhibitor disclosed in Patent Document 35, which is hereby incorporated by reference in its entirety. In some aspects, the PD-1 inhibitor comprises INCMGA00012 (Incyte Corporation). In some aspects, the PD-1 inhibitor comprises a combination of the anti-PD-1 antibody disclosed in the present application and a PD-1 small molecule inhibitor.

[0095] In some embodiments, in any of the methods disclosed herein, the anti-PD-1 antibody is replaced with an anti-PD-L1 antibody. Anti-PD-L1 antibodies known in the art can be used in the compositions and methods of the present disclosure. Examples of anti-PD-L1 antibodies useful in the compositions and methods of the present disclosure include the antibodies disclosed in Patent Document 36. The anti-PD-L1 human monoclonal antibody disclosed in Patent Document 36 has been demonstrated to exhibit one or more of the following characteristics: (a) binding to human PD-L1 with a K -7 of 1×10 D M or less as determined by surface plasmon resonance using a Biacore biosensor system; (b) increasing T cell proliferation in a mixed lymphocyte reaction (MLR) assay; (c) increasing interferon-γ production in an MLR assay; (d) increasing IL-2 secretion in an MLR assay; (e) stimulating an antibody response; and (f) reversing the effect of regulatory T cells on T cell effector cells and / or dendritic cells. Anti-PD-L1 antibodies that can be used in the present disclosure specifically bind to human PD-L1 and include monoclonal antibodies that exhibit at least one of the characteristics described above, and in some embodiments at least five of them.

[0096] The anti-PD-L1 antibody may be selected from the group consisting of: BMS-936559 (also known as 12A4 and MDX-1105; see, for example, Patent Document 37 and Patent Document 34), atezolizumab (Roche; also known as TECENTRIQ®; MPDL3280A, RG7446; see Patent Document 38; also see Non-Patent Document 7), durvalumab (AstraZeneca; also known as IMFINZI® and MEDI-4736; see Patent Document 39), avelumab (Pfizer; also known as BAVENCIO® and MSB-0010718C; see Patent Document 40), STI-1014 (Sorrento; see Patent Document 41), CX-072 (Cytomx; see Patent Document 42), KN035 (3D Med / Alphamab; see Non-Patent Document 8), LY3300054 (Eli Lilly Co.; see, for example, Patent Document 43), BGB-A333 (BeiGene; see Non-Patent Document 9), and CK-301 (Checkpoint Therapeutics; see Non-Patent Document 10).

[0097] Atezolizumab is a fully humanized IgG1 monoclonal anti-PD-L1 antibody. Durvalumab is a human IgG1 kappa monoclonal anti-PD-L1 antibody. Avelumab is a human IgG1 lambda monoclonal anti-PD-L1 antibody. Anti-PD-L1 antibodies that can be used in the disclosed compositions and methods are also isolated antibodies that specifically bind to human PD-L1 and that cross-compete with any of the anti-PD-L1 antibodies disclosed herein, such as atezolizumab, durvalumab, and / or avelumab, when binding to human PD-L1. In some embodiments, the anti-PD-L1 antibody binds to the same epitope as any of the anti-PD-L1 antibodies described herein, such as atezolizumab, durvalumab, and / or avelumab. The ability of antibodies to cross-compete when binding to an antigen indicates that these antibodies bind to the same epitope region of the antigen and sterically hinder the binding of other cross-competing antibodies to that particular epitope region. It is expected that these cross-competing antibodies will have functional properties very similar to those of a reference antibody, such as atezolizumab and / or avelumab, due to their binding to the same epitope region of PD-L1. Cross-competing antibodies can be readily identified based on their ability to cross-compete with atezolizumab and / or avelumab in standard PD-L1 binding assays, such as Biacore assays, ELISA assays, or flow cytometry (see, for example, Patent Document 34).

[0098] Antibodies that cross-compete when binding to human PD-L1 or that bind to the same epitope region of a human PD-L1 antibody, such as atezolizumab, durvalumab, and / or avelumab, are monoclonal antibodies. For administration to a human subject, these cross-competing antibodies are chimeric antibodies, modified antibodies, or humanized or human antibodies. Such chimeric, modified, humanized, or human monoclonal antibodies can be prepared and isolated by methods well known in the art.

[0099] Anti-PD-L1 antibodies that can be used in the configurations and methods of the present disclosure also include the antigen-binding portions of the antibodies described above. It has been well demonstrated that the antigen-binding function of an antibody can be carried out by fragments of the full-length antibody.

[0100] Anti-PD-L1 antibodies suitable for use in the disclosed configurations and methods are antibodies that bind to PD-L1 with high specificity and affinity, block the binding of PD-1, and inhibit the immunosuppressive effect of the PD-1 signaling pathway. In any of the configurations or methods disclosed herein, an anti-PD-L1 "antibody" is an antigen-binding portion or fragment that binds to PD-L1 and exhibits functional properties similar to those of the whole antibody with respect to inhibiting receptor binding and upregulating the immune system. In certain embodiments, the anti-PD-L1 antibody or its antigen-binding portion cross-competes with atezolizumab, durvalumab, and / or avelumab when binding to human PD-L1.

[0101] Anti-PD-L1 antibodies useful for the present disclosure can be any PD-L1 antibody that specifically binds to PD-L1, for example, an antibody that cross-competes with durvalumab, avelumab, or atezolizumab when binding to human PD-1, such as an antibody that binds to the same epitope as durvalumab, avelumab, or atezolizumab. In certain embodiments, the anti-PD-L1 antibody is durvalumab. In other embodiments, the anti-PD-L1 antibody is avelumab. In some embodiments, the anti-PD-L1 antibody is atezolizumab.

[0102] In some embodiments, the anti-PD-L1 antibody is administered once every 2, 3, 4, 5, 6, 7, or 8 weeks, at a dosage ranging from about 0.1 mg / kg to about 20.0 mg / kg, about 2 mg / kg, about 3 mg / kg, about 4 mg / kg, about 5 mg / kg, about 6 mg / kg, about 7 mg / kg, about 8 mg / kg, about 9 mg / kg, about 10 mg / kg, about 11 mg / kg, about 12 mg / kg, about 13 mg / kg, about 14 mg / kg, about 15 mg / kg, about 16 mg / kg, about 17 mg / kg, about 18 mg / kg, about 19 mg / kg, or about 20 mg / kg per body weight.

[0103] The anti-PD-L1 antibody may be administered once every 3 weeks, at a dosage of about 15 mg / kg per body weight. In other embodiments, the anti-PD-L1 antibody is administered once every 2 weeks, at a dosage of about 10 mg / kg per body weight.

[0104] In some scenarios, the anti-PD-L1 antibodies useful for the present disclosure are of a uniform dosage. In some embodiments, the anti-PD-L1 antibody is administered at a uniform dosage of from about 200 mg to about 1600 mg, from about 200 mg to about 1500 mg, from about 200 mg to about 1400 mg, from about 200 mg to about 1300 mg, from about 200 mg to about 1200 mg, from about 200 mg to about 1100 mg, from about 200 mg to about 1000 mg, from about 200 mg to about 900 mg, from about 200 mg to about 800 mg, from about 200 mg to about 700 mg, from about 200 mg to about 600 mg, from about 700 mg to about 1300 mg, from about 800 mg to about 1200 mg, from about 700 mg to about 900 mg, or from about 1100 mg to about 1300 mg. In some embodiments, the anti-PD-L1 antibody is administered at a uniform dosage of at least about 240 mg, at least about 300 mg, at least about 320 mg, at least about 400 mg, at least about 480 mg, at least about 500 mg, at least about 560 mg, at least about 600 mg, at least about 640 mg, at least about 700 mg, at least 720 mg, at least about 800 mg, at least about 840 mg, at least about 880 mg, at least about 900 mg, at least 960 mg, at least about 1000 mg, at least about 1040 mg, at least about 1100 mg, at least about 1120 mg, at least about 1200 mg, at least about 1280 mg, at least about 1300 mg, at least about 1360 mg, or at least about 1400 mg, with a dosing interval of about 1, 2, 3, or 4 weeks. In some embodiments, the anti-PD-L1 antibody is administered at a uniform dosage of about 1200 mg once every 3 weeks. In other embodiments, the anti-PD-L1 antibody is administered at a uniform dosage of about 800 mg once every 2 weeks. In other embodiments, the anti-PD-L1 antibody is administered at a uniform dosage of about 840 mg once every 2 weeks.

[0105] Atezolizumab is administered at a uniform dose of approximately 1200 mg once every three weeks. In some embodiments, atezolizumab is administered at a uniform dose of approximately 800 mg once every two weeks. In other embodiments, atezolizumab is administered at a uniform dose of approximately 840 mg once every two weeks. Optionally, avelumab may be administered at a uniform dose of approximately 800 mg once every two weeks.

[0106] In some embodiments, durvalumab is administered at a dose of approximately 10 mg / kg once every two weeks. In other embodiments, durvalumab is administered at a uniform dose of approximately 800 mg / kg once every two weeks. Optionally, durvalumab may be administered at a uniform dose of approximately 1200 mg / kg once every three weeks.

[0107] The PD-L1 inhibitor may comprise a small molecule or a miramolecule. The PD-L1 inhibitor may comprise a macrocyclic peptide. In some embodiments, the PD-L1 inhibitor comprises BMS-986189. The PD-L1 inhibitor may comprise a miramolecule having the following chemical formula.

[0108]

Chemical formula

[0109] Here, R1 to R13 are amino acid side chains, R a ~R n is hydrogen, methyl, or forms a ring with adjacent R groups, R14 is -C(O)NHR15, R15 is hydrogen or a glycine residue, and the glycine residue may optionally be substituted at the terminal with additional glycine residues and / or this may improve the pharmacokinetic properties. In some aspects, the PD-L1 inhibitor comprises the compounds disclosed in Patent Document 35, the entirety of which is incorporated herein by reference. In some aspects, the PD-L1 inhibitor comprises the compounds disclosed in Patent Documents 44 to 55, the entirety of each of which is incorporated herein by reference.

[0110] The PD-L1 inhibitor includes small molecule PD-L1 inhibitors disclosed in Patent Documents 56 to 64, and the entireties of each of these are incorporated herein by reference. In some embodiments, the PD-L1 inhibitor includes a combination of an anti-PD-L1 antibody disclosed herein and a small molecule PD-L1 inhibitor disclosed herein.

[0111] A software application (i.e., a software resource) may refer to computer software that causes a computing device to execute a task. In some embodiments, a software application may be referred to as an "application", an "app", or a "program". Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0112] A non-transitory memory may be a physical device used to store, temporarily or persistently, a program (e.g., a sequence of instructions) or data (e.g., program state information) for use by a computing device. The non-transitory memory may be a volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and ROM (read-only memory) / PROM (programmable read-only memory) / EPROM (erasable programmable read-only memory) / EEPROM (electronically erasable programmable read-only memory) (e.g., typically used for firmware such as a boot program). Examples of volatile memory include, but are not limited to, RAM (random access memory), DRAM (dynamic random access memory), SRAM (static random access memory), PCM (phase change memory), and disks or tapes.

[0113] FIG. 22 is a schematic diagram of an exemplary computing device 2200 that may be used to implement the systems and methods described in this application. Computing device 2200 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown in this application, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the embodiments of the invention described in this application and / or claimed.

[0114] The computing device 2200 includes a processor 2210, a memory 2220, a storage device 2230, a high-speed interface / controller 2240 connected to the memory 2220 and a high-speed expansion port 2250, and a low-speed interface / controller 2260 connected to a low-speed bus 2270 and the storage device 2230. Each of the components 2210, 2220, 2230, 2240, 2250, and 2260 is interconnected using various buses and may be implemented, as appropriate, on a common motherboard or in other ways. The processor 2210 may process instructions for execution within the computing device 2200, including instructions stored in the memory 2220 or the storage device 2230 for displaying graphical information for a GUI (graphical user interface) on an external input / output device such as a display 2280 connected to the high-speed interface 2240. In other embodiments, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and multiple types of memories. Also, multiple computing devices 2200 may be connected so that each device provides a portion of the required operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0115] Memory 2220 stores information non-temporarily inside the computing device 2200. Memory 2220 may be a computer-readable medium, one or more volatile memory devices, or one or more non-volatile memory devices. The non-temporary memory 2220 may be a physical device used to store, temporarily or persistently, a program (e.g., a sequence of instructions) or data (e.g., program state information) for use by the computing device 2200. Examples of non-volatile memory include, but are not limited to, flash memory and ROM (read-only memory) / PROM (programmable read-only memory) / EPROM (erasable programmable read-only memory) / EEPROM (electrically erasable programmable read-only memory) (e.g., typically used for firmware such as a boot program). Examples of volatile memory include, but are not limited to, RAM (random access memory), DRAM (dynamic random access memory), SRAM (static random access memory), PCM (phase change memory), and disks or tapes.

[0116] The memory device 2230 may have the ability to function as the mass storage of the computing device 2200. In some embodiments, the memory device 2230 is a computer-readable medium. In various different embodiments, the memory device 2230 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices including a storage area network or other configured devices. In additional embodiments, the computer program product is tangibly embodied as an information carrier. The computer program product includes instructions that, when executed, perform one or more methods such as those described above. The information carrier is a computer or machine-readable medium such as the memory 2220, the memory device 2230, or the memory on the processor 2210.

[0117] The high-speed controller 2240 manages bandwidth-intensive operations for the computing device 2200, while the low-speed controller 2260 manages less bandwidth-intensive operations. Such an assignment of duties is merely illustrative. In some embodiments, the high-speed controller 2240 is connected to the memory 2220, the display 2280, and the high-speed expansion port 2250 (e.g., via a graphics processor or accelerator). The high-speed expansion port 2250 may receive various expansion cards (not shown). In some embodiments, the low-speed controller 2260 is connected to the memory device 2230 and the low-speed expansion port 2290. The low-speed expansion port 2290 may include various communication ports (e.g., USB, Bluetooth®, Ethernet®, wireless Ethernet), and may also be connected to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router (e.g., via a network adapter).

[0118] As shown in the drawings, the computing device 2200 may be implemented in many different forms. For example, it may be implemented as a standard server 2200a or multiple multiples of such a group of servers 2200a, as a laptop computer 2200b, or as part of a rack server system 2200c.

[0119] The various implementations of the systems and techniques described in this application may be realized in digital electronics and / or optical circuits, integrated circuits, ASICs (Application Specific Integrated Circuits) specifically designed therefor, computer hardware, firmware, software, and / or combinations thereof. These various implementations may be executable and / or interpretable on a programmable system including at least one programmable processor, which may be special-purpose or general-purpose, connected to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device, and may include implementations of one or more computer programs.

[0120] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or in assembly / machine language. According to the usage in this application, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, device, and / or apparatus (e.g., including magnetic disks, optical disks, memory, Programmable Logic Device (PLD)) used to provide machine instructions and / or data to a programmable processor, and include a machine-readable medium that receives a machine-readable signal representing the machine instructions. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0121] The processes and logic flows described herein may be executed by one or more programmable processors, also referred to as data processing hardware, that execute one or more computer programs to function by operating on input data to generate output. The processes and logic flows may also be executed by special purpose logic circuits, such as, for example, an FPGA (field programmable gate array), an ASIC (application specific integrated circuit). Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read only memory or a random access memory or both. Essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as, for example, magnetic, magneto-optical disks, or optical disks, or is functionally connected to communicate with or both. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including by way of example semiconductor memory devices, such as, for example, EPROM, EEPROM, and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuits.

[0122] To provide interactive scanning with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, and optionally a keyboard and pointing device, such as a mouse or trackball, that can be used when the user provides input to the computer. Similarly, other types of devices may be used to provide interactive scanning with a user. For example, the feedback provided to the user may be any form of perceptual feedback, such as visual feedback, auditory feedback, or tactile feedback, and the input received from the user may be in any form, including voice, speech, or tactile input. Further, the computer may interact with the user by sending and receiving documents to and from the devices used by the user, such as by sending a web page to a web browser in response to a request received from a web browser in the user's client device.

[0123] Numerous embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other embodiments may be included within the scope of the appended claims.

Claims

1. A computer-implemented method (400) that causes a data processing hardware (142) to perform a predetermined operation when executed on the data processing hardware (142), comprising: The operation includes: Receiving an input tissue structure image (110) related to a patient diagnosed with cancer, the input tissue structure image (110) including a plurality of image pixels; Processing the input tissue structure image (110) using a cell classification model (550) to generate one or more lymphocyte density maps (125) inside the input tissue structure image (110); Performing morphological image processing (130) on the one or more lymphocyte density maps (125) to identify one or more TLS regions (135) inside the input tissue structure image (110), the TLS regions (135) being represented by respective clusters of lymphocyte cells; For each corresponding TLS region (135) of the one or more TLS regions (135) identified in the input tissue structure image (110), Extracting each set of TLS features (140) from each cluster of lymphocyte cells representing the corresponding TLS region (135); Processing each set of the TLS features (140) using a TLS classification model (350) to classify the corresponding TLS region (135) as one of a first TLS maturation state (312), a second TLS maturation state (312), and a third TLS maturation state (312). A computer-implemented method (400).

2. The operation further includes identifying a tumor region (115) inside the input tissue structure image (110) by processing the input tissue structure image (110) using a tumor detection model (450), Generating the one or more lymphocyte density maps (125) by processing the input tissue structure image (110) includes generating the one or more lymphocyte density maps (125) by performing single-cell imaging analysis on the tumor region (115) identified inside the input tissue structure image (110) and then processing the input tissue structure image (110) using the cell classification model (550). The computer-implemented method (400) according to Claim 1.

3. The tumor detection model (450) is A plurality of image tiles (370) rasterized from a set of histopathological images of an entire slide, the plurality of image tiles (370) being manually annotated respectively as including a tumor or non-tumor, Training the tumor detection model (450) using a neural network (374) to train the tumor detection model (450) with the plurality of image tiles (370) so as to teach the tumor detection model (450) a method of identifying a tumor region (115) within a tissue structure image, The computer-implemented method (400) according to claim 2.

4. The cell classification model (550) obtains a plurality of image patches (382) each having a corresponding plurality of human cells and a manual annotation that labels each human cell as a tumor cell, a lymphocyte cell, or a non-malignant cell, Training the cell classification model (550) using a neural network (386) to train the cell classification model (550) with the plurality of image patches (382) so as to teach the cell classification model (550) a method of classifying individual cells in a tissue structure image as a tumor cell, a lymphocyte cell, or a non-malignant cell, The computer-implemented method (400) according to any one of claims 1 to 3.

5. The TLS classification model (350) obtains a training dataset (305) including a plurality of training tissue structure images (310) each including a tumor microenvironment and each having a manual annotation (312), the manual annotation (312) identifies one or more TLS regions (135) in the training tissue structure image (310), the one or more TLS regions (135) being represented by respective clusters of lymphocyte cells, and for each corresponding TLS region (135), a ground truth TLS maturation state (312) indicating that the corresponding TLS region (135) has a first TLS maturation state (312), a second TLS maturation state (312), or a third TLS maturation state (312), obtaining the training dataset (305), extracting each set of TLS features (140) from each cluster of lymphocyte cells representing the TLS region (135) for each TLS region (135), Teaching the TLS classification model (350) to learn a method for predicting the ground truth TLS grade for each corresponding TLS region (135), by causing the TLS classification model (350) to learn each set of learning TLS features (140) extracted for each TLS region (135). A computer-implemented method (400) according to any one of claims 1 to 4. **Claim 6** Causing the TLS classification model (350) to learn includes causing the TLS classification model (350) to learn using a CART (classification and regression trees) algorithm. A computer-implemented method (400) according to claim 5. **Claim 7** The first TLS maturation state (312) comprises a dense aggregate of at least a threshold number of lymphocytes, excluding high endothelial venules or germinal centers (313). The second TLS maturation state (312) comprises an immature TLS that includes high endothelial venules and at least a threshold number of dense aggregates of lymphocytes, excluding germinal centers (313). The third TLS maturation state (312) comprises a mature TLS that includes high endothelial venules and germinal centers (313) and at least a threshold number of dense aggregates of lymphocytes. A computer-implemented method (400) according to any one of claims 1 to 6. **Claim 8** The operations are For each corresponding TLS region (135) of the one or more TLS regions (135) identified in the input tissue structure image (110), generating each pixel mask (112) that emphasizes at least the periphery of the corresponding TLS region (135); Generating an output image (110A) that enhances the input tissue structure image (110) by superimposing each pixel mask (112) generated for each TLS region (135) over the input tissue structure image (110); Further comprising providing the output image (110A) for display on a screen in communication with the data processing hardware (142). A computer-implemented method (400) according to any one of claims 1 to 7. **Claim 9** Each pixel mask (112) generated for each corresponding TLS region (135) classified as the first maturation state comprises a first pixel mask (112). Each of the pixel masks (112) generated for each corresponding TLS region (135) classified as the second maturation state includes a second pixel mask (112) that is visually distinguishable from the second pixel mask (112). Each of the pixel masks (112) generated for each corresponding TLS region (135) classified as the third maturation state includes a third pixel mask (112) that is visually distinguishable from the first pixel mask (112) and the second pixel mask (112). The computer-implemented method (400) according to claim 8.

10. Each set of the TLS features (140) extracted from each cluster of the lymphocytes includes the area of the corresponding TLS region (135), the roundness of the corresponding TLS region (135), and the distortion of the corresponding TLS region (135). The computer-implemented method (400) according to any one of claims 1 to 9.

11. The operation further includes determining an overall TLS score (152) for the input tissue structure image (110) based on the TLS maturation state (312) for the one or more TLS regions (135) identified in the tissue structure image and the TLS features (140) extracted from the one or more TLS regions (135) identified in the tissue structure image. The computer-implemented method (400) according to any one of claims 1 to 10.

12. The operation further includes determining a treatment recommendation (192) for treating the patient using immunotherapy based on the overall TLS score (152). The computer-implemented method (400) according to claim 11.

13. The immunotherapy includes at least one of a PD-1 inhibitor and a PD-L1 inhibitor. The computer-implemented method (400) according to claim 12.

14. The operation further includes determining a prediction score for the patient's response to immunotherapy based on the TLS maturation state (312) for the one or more TLS regions (135) identified in the tissue structure image and the TLS features (140) extracted from the one or more TLS regions (135) identified in the tissue structure image. The computer-implemented method (400) according to any one of claims 11 to 13.

15. Data processing hardware (142) and memory hardware (144) that communicates with the data processing hardware (142), a system (100) comprising: the memory hardware (144) stores instructions that cause the data processing hardware (142) to perform a predetermined operation when executed on the data processing hardware (142), the operation is: receiving an input tissue structure image (110) related to a patient diagnosed with cancer, the input tissue structure image (110) including a plurality of image pixels; processing the input tissue structure image (110) using a cell classification model (550) to generate one or more lymphocyte density maps (125) inside the input tissue structure image (110); performing morphological image processing (130) on the one or more lymphocyte density maps (125) to identify one or more TLS regions (135) inside the input tissue structure image (110), the TLS regions (135) being represented by each cluster of lymphocyte cells; for each corresponding TLS region (135) of the one or more TLS regions (135) identified in the input tissue structure image (110), extracting each set of TLS features (140) from each cluster of lymphocyte cells representing the corresponding TLS region (135); processing each set of the TLS features (140) using a TLS classification model (350) to classify the corresponding TLS region (135) as one of a first TLS maturation state (312), a second TLS maturation state (312), and a third TLS maturation state (312), a system (100).

16. the operation further includes identifying a tumor region (115) inside the input tissue structure image (110) by processing the input tissue structure image (110) using a tumor detection model (450). Generating the one or more lymphocyte density maps (125) by processing the input tissue structure image (110) includes performing single-cell imaging analysis on the tumor region (115) identified within the input tissue structure image (110), and processing the input tissue structure image (110) using the cell classification model (550) to generate the one or more lymphocyte density maps (125). The system (100) according to claim 15.

17. The tumor detection model (450) obtains a plurality of image tiles (370) rasterized from a set of histopathological images of an entire slide, each of which is manually annotated as including a tumor or non-tumor. The tumor detection model (450) is learned by using a neural network (374) to cause the tumor detection model (450) to learn the plurality of image tiles (370) so as to teach the tumor detection model (450) a method of identifying a tumor region (115) within a tissue structure image. The system (100) according to claim 16.

18. The cell classification model (550) obtains a plurality of image patches (382), each of which includes a corresponding plurality of human cells and a manual annotation that labels each human cell as a tumor cell, a lymphocyte cell, or a non-malignant cell. The cell classification model (550) is learned by using a neural network (386) to cause the cell classification model (550) to learn the plurality of image patches (382) so as to teach the cell classification model (550) a method of classifying individual cells in a tissue structure image as tumor cells, lymphocyte cells, or non-malignant cells. The system (100) according to any one of claims 15 to 17.

19. The TLS classification model (350) obtains a training dataset (305) including a plurality of training tissue structure images (310), each of which includes a tumor microenvironment and is provided with a manual annotation (312), and the manual annotation (312) is one or more TLS regions (135) in the training tissue structure image (310), and the one or more TLS regions (135) are each represented by a respective cluster of lymphocyte cells For each corresponding TLS region (135), a ground truth TLS maturity state (312) indicating that the corresponding TLS region (135) has a first TLS maturity state (312), a second TLS maturity state (312), or a third TLS maturity state (312) and obtaining the learning dataset (305) that identifies the same; For each TLS region (135), extracting each set of TLS features (140) from each cluster of the lymphocytes representing the TLS region (135); learning a method for predicting a ground truth TLS grade for each corresponding TLS region (135) by causing the TLS classification model (350) to learn each set of learning TLS features (140) extracted for each TLS region (135) so as to teach the TLS classification model (350); The system (100) according to any one of claims 15 to 18.

20. Causing the TLS classification model (350) to learn includes causing the TLS classification model (350) to learn using a CART (classification and regression trees) algorithm. The system (100) according to claim 19.

21. The first TLS maturity state (312) comprises a dense aggregate of at least a threshold number of lymphocytes that does not include high endothelial venules or germinal centers (313). The second TLS maturity state (312) comprises an immature TLS that includes high endothelial venules and does not include germinal centers (313) and has a dense aggregate of at least the threshold number of lymphocytes. The third TLS maturity state (312) comprises a mature TLS that includes high endothelial venules and germinal centers (313) and has a dense aggregate of at least the threshold number of lymphocytes. The system (100) according to any one of claims 15 to 20.

22. The operation is generating each pixel mask (112) that emphasizes at least the periphery of the corresponding TLS region (135) for each corresponding TLS region (135) of the one or more TLS regions (135) identified in the input tissue structure image (110); By superimposing each of the pixel masks (112) generated for each of the TLS regions (135) on the input tissue structure image (110), an output image (110A) that enhances the input tissue structure image (110) is generated. Further including providing the output image (110A) for display on a screen that communicates with the data processing hardware (142). The system (100) according to any one of claims 15 to 21.

23. Each of the pixel masks (112) generated for each corresponding TLS region (135) classified as the first maturation state includes a first pixel mask (112). Each of the pixel masks (112) generated for each corresponding TLS region (135) classified as the second maturation state includes a second pixel mask (112) that is visually distinguishable from the second pixel mask (112). Each of the pixel masks (112) generated for each corresponding TLS region (135) classified as the third maturation state includes a third pixel mask (112) that is visually distinguishable from the first pixel mask (112) and the second pixel mask (112). The system (100) according to claim 22.

24. Each set of the TLS features (140) extracted from each cluster of the lymphocytes includes the area of the corresponding TLS region (135), the roundness of the corresponding TLS region (135), and the distortion of the corresponding TLS region (135). The system (100) according to any one of claims 15 to 23.

25. The operation further includes determining an overall TLS score (152) for the input tissue structure image (110) based on the TLS maturation state (312) for the one or more TLS regions (135) identified in the tissue structure image and the TLS features (140) extracted from the one or more TLS regions (135) identified in the tissue structure image. The system (100) according to any one of claims 15 to 24.

26. The operation further includes determining a treatment recommendation (192) for treating the patient using immunotherapy based on the overall TLS score (152). The system (100) according to claim 25.

27. The immunotherapy includes at least one of a PD-1 inhibitor and a PD-L1 inhibitor. The system (100) according to claim 26. **Claim 28** The operation further includes determining a prediction score of the patient's response to immunotherapy based on the TLS maturation state (312) related to the one or more TLS regions (135) identified in the tissue structure image and the TLS features (140) extracted from the one or more TLS regions (135) identified in the tissue structure image. The system (100) according to any one of claims 25 to 27.

Images