Machine learning-based identification, classification, and quantification of tertiary lymphoid tissue-like structures.

Abstract

The method (400) includes receiving an input tissue structure image (110), generating one or more lymphocyte density maps (125) inside the input tissue structure image by processing the input tissue structure image using a cell classification model (550), and identifying one or more TLS regions (135) inside the input tissue structure image by performing morphological image processing (130) on the one or more lymphocyte density maps. Each TLS region is represented by each cluster of lymphocyte cells. For each corresponding TLS region of the one or more TLS regions identified in the input tissue structure image, the method includes extracting each set of TLS features (140) from each cluster of lymphocyte cells, and classifying the corresponding TLS region as one of a first TLS maturation state, a second TLS maturation state, and a third TLS maturation state by processing each set of TLS features using a TLS classification model (350).

Description

[Technical Field]

[0001] This disclosure relates to machine learning-based identification, classification, and quantification of tertiary lymphoid tissue-like structures, for example, in tumor biopsy specimens. [Background technology]

[0002] Tertiary lymphoid structures (TLSs) (e.g., tertiary lymphoid organs or ectopic lymphoid follicles) are ectopic lymphoid tissues consisting of B cells, T cells, and supporting cells that originate in non-lymphoid organs and are often found in tumors. TLSs support the differentiation of naive T cells into effector and memory T cells and frequently occur in areas of chronic inflammation. While TLSs are observed in the clinicopathological setting, they are not currently evaluated for diagnostic pathology or to guide treatment. Several studies have shown associations between TLSs and immuno-oncology (IO) treatment outcomes across multiple signs (see, e.g., Non-Patent Literature 1). The presence of TLSs in various tumors has shown associations with outcomes in the non-IO setting, and recently, TLSs have been found to predict the response to IO treatment in melanoma, osteosarcoma, and RCC (reduced-compulsive cytology). See, e.g., Non-Patent Literature 2. In the research environment, TLS were evaluated by manual visual methods based on hematoxylin and eosin staining (H&E) and immunohistochemistry (IHC) staining. For quantification, image analysis of IHC or immunofluorescence (IF) staining was used. These correlations depend on the maturation and localization of TLS within the tumor microenvironment (TME). [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] U.S. Patent No. 8008449 [Patent Document 2] U.S. Patent No. 6808710 [Patent Document 3] U.S. Patent No. 7488802 [Patent Document 4] U.S. Patent No. 8168757 [Patent Document 5] U.S. Patent No. 8354509 [Patent Document 6] U.S. Patent Application Publication No. 2016 / 0272708, [Patent Document 7] International Publication No. 2012 / 145493 [Patent Document 8] International Publication No. 2008 / 156712 [Patent Document 9] International Publication No. 2015 / 112900 [Patent Document 10] International Publication No. 2015 / 112800 [Patent Document 11] International Publication No. 2014 / 206107 [Patent Document 12] International Publication No. 2015 / 035606 [Patent Document 13] International Publication No. 2015 / 085847 [Patent Document 14] International Publication No. 2014 / 179664 [Patent Document 15] International Publication No. 2017 / 020291 [Patent Document 16] International Publication No. 2017 / 020858 [Patent Document 17] International Publication No. 2016 / 197367 [Patent Document 18] International Publication No. 2017 / 024515 [Patent Document 19] International Publication No. 2017 / 025051 [Patent Document 20] International Publication No. 2017 / 123557 [Patent Document 21] International Publication No. 2016 / 106159 [Patent Document 22] International Publication No. 2014 / 194302 [Patent Document 23] International Publication No. 2017 / 040790 [Patent Document 24] International Publication No. 2017 / 133540 [Patent Document 25] International Publication No. 2017 / 132827 [Patent Document 26] International Publication No. 2017 / 024465 [Patent Document 27] International Publication No. 2017 / 025016 [Patent Document 28] International Publication No. 2017 / 106061 [Patent Document 29] International Publication No. 2017 / 019846 [Patent Document 30] International Publication No. 2017 / 132825 [Patent Document 31] U.S. Patent Application Publication No. 2015 / 0079109 [Patent Document 32] U.S. Patent No. 8900587 [Patent Document 33] U.S. Patent No. 8779105 [Patent Document 34] International Publication No. 2013 / 173223 [Patent Document 35] International Publication No. 2014 / 151634 [Patent Document 36] U.S. Patent No. 9580507 [Patent Document 37] U.S. Patent No. 7943743 [Patent Document 38] U.S. Patent No. 8217149 [Patent Document 39] International Publication No. 2011 / 066389 [Patent Document 40] International Publication No. 2013 / 079174 [Patent Document 41] International Publication No. 2013 / 181634 [Patent Document 42] International Publication No. 2016 / 149201 [Patent Document 43] International Publication No. 2017 / 034916 [Patent Document 44] International Publication No. 2016 / 039749 [Patent Document 45] International Publication No. 2016 / 149351 [Patent Document 46] International Publication No. 2016 / 077518 [Patent Document 47] International Publication No. 2016 / 100285 [Patent Document 48] International Publication No. 2016 / 100608 [Patent Document 49] International Publication No. 2016 / 126646 [Patent Document 50] International Publication No. 2016 / 057624 [Patent Document 51] International Publication No. 2017 / 151830 [Patent Document 52] International Publication No. 2017 / 176608 [Patent Document 53] International Publication No. 2018 / 085750 [Patent Document 54] International Publication No. 2018 / 237153 [Patent Document 55] International Publication No. 2019 / 070643 [Patent Document 56] International Publication No. 2015 / 034820 [Patent Document 57] International Publication No. 2015 / 160641 [Patent Document 58] International Publication No. 2018 / 044963 [Patent Document 59] International Publication No. 2017 / 066227 [Patent Document 60] International Publication No. 2018 / 009505

Patent document 61

Patent document 62

Patent document 63

Patent document 64

Non-licensed literature

[0004] [Non-licensed document 1] Sautes-Fridman, et al, 2019, Nat Rev Cancer 19:307 and Vanhersecke, et al, "Mature tertiary lymphoid structures predict immune checkpoint inhibitor efficacy in solid tumors independently of PD-L1 expression," Nat Cancer, 2021 [Non-licensed document 2] Cabrita, et al, 2020, Nature 577:561, Petitprez, et al, 2020, Nature 577:556, Helmink, et al, 2020, Nature 577:549, Bruno, N&V, 2020, Nature 577:474, Sautes-Fridman, et al, 2019, Nat Rev Cancer 19:307 [Non-licensed document 3] Buisseret L, Desmedt C, Garaud S, et al. Reliability of tumor-infiltrating lymphocyte and tertiary lymphoid structure assessment in human breast cancer. Mod Pathol. 2017;30(9):1204-1212. doi:10.1038 / modpathaol.2017.43.

Non-Patent Document 4

Non-Patent Document 5

Non-Patent Document 6

Non-Patent Document 7

Non-Patent Document 8

Non-Patent Document 9

Non-Patent Document 10

Summary of the Invention

[0005] One aspect of the present disclosure provides a computer-implemented method for causing data processing hardware to perform an operation that, when executed on data processing hardware, includes receiving an input tissue structure image relating to a patient diagnosed with cancer. The input tissue structure image comprises a plurality of image pixels. The operation also includes processing the input tissue structure image using a cell classification model to generate one or more lymphocyte density maps within the input tissue structure image, and performing morphological image processing on one or more lymphocyte density maps to identify one or more TLS regions within the input tissue structure image. Each TLS region is represented by a cluster of lymphocyte cells. With respect to each corresponding TLS region of one or more TLS regions identified in the input tissue structure image, the operation also includes extracting each set of TLS features from each cluster of lymphocyte cells representing the corresponding TLS region, and processing each set of TLS features using a TLS classification model to classify the corresponding TLS region as one of a first TLS maturity state, a second TLS maturity state, and a third TLS maturity state.

[0006] Embodiments of this disclosure may include one or more of the following optional features. In some embodiments, the operation also includes identifying tumor regions within an input tissue structure image by processing the input tissue structure image using a tumor detection model. Here, generating one or more lymphocyte density maps by processing the input tissue structure image may include generating one or more lymphocyte density maps by processing the input tissue structure image using a cell classification model by performing single-cell imaging analysis on the tumor regions identified within the input tissue structure image. In these embodiments, the tumor detection model may be trained by training the tumor detection model with a neural network to learn how to identify tumor regions within a tissue structure image, which are rasterized image tiles from a set of histopathological images of an entire slide, each manually annotated as containing tumor or non-tumor, and how to learn how to identify tumor regions within a tissue structure image.

[0007] In some embodiments, the cell classification model is trained by using a neural network to train the cell classification model on multiple image patches, thereby teaching the cell classification model how to acquire multiple image patches and how to classify individual cells in tissue structure images as tumor cells, lymphocytes, or non-malignant cells. Each image patch includes a corresponding set of human cells and manual annotations labeling each human cell as a tumor cell, lymphocyte, or non-malignant cell.

[0008] In some embodiments, the TLS classification model is trained by acquiring a training dataset containing multiple training tissue structure images, each containing a tumor microenvironment and each having manual annotations. The manual annotations identify one or more TLS regions in the training tissue structure images and, for each corresponding TLS region, a ground truth TLS maturity state indicating that the corresponding TLS region contains a first TLS maturity state, a second TLS maturity state, or a third TLS maturity state. Each TLS region is represented by each cluster of lymphocyte cells. In these embodiments, the TLS classification model is further trained by training the TLS classification model with each set of training TLS features extracted for each TLS region, instructing the TLS classification model to learn how to extract each set of TLS features from each cluster of lymphocyte cells representing the TLS region and how to predict the ground truth TLS grade for each corresponding TLS region. Training the TLS classification model may include training the TLS classification model using the CART (classification and regression trees) algorithm.

[0009] The first TLS maturation state may include dense aggregates of at least a threshold number of lymphocytes, which do not contain high endothelial venules or germinal centers. The second TLS maturation state may include immature TLS, which include high endothelial venules but do not contain germinal centers, and include at least the above threshold number of dense aggregates of lymphocytes. The third TLS maturation state may include mature TLS, which include high endothelial venules and germinal centers, and include at least a threshold number of dense aggregates of lymphocytes. Each set of TLS features extracted from each cluster of lymphocyte cells may include the area of ​​the corresponding TLS region, the roundness of the corresponding TLS region, and the distortion of the corresponding TLS region.

[0010] In some embodiments, the operation further includes generating a pixel mask that enhances at least the surrounding area of ​​each corresponding TLS region for each of the one or more TLS regions identified in the input tissue structure image, generating an output image that enhances the input tissue structure image by superimposing each pixel mask generated for each TLS region onto the input tissue structure image, and providing the output image for display on a screen that communicates with data processing hardware. In these embodiments, each pixel mask generated for each corresponding TLS region classified as a first mature state includes a first pixel mask, and each pixel mask generated for each corresponding TLS region classified as a second mature state includes, 1 Each pixel mask generated for each corresponding TLS region classified as a third mature state includes a second pixel mask that is visually identifiable from the first and second pixel masks, and each pixel mask generated for each corresponding TLS region includes a third pixel mask that is visually identifiable from the first and second pixel masks.

[0011] In some embodiments, the operation also includes determining an overall TLS score for an input tissue image based on the TLS maturity status of one or more TLS regions identified in the tissue image and TLS features extracted from one or more TLS regions identified in the tissue image. In these embodiments, the operation may also include determining a treatment recommendation to treat the patient with immunotherapy based on the overall TLS score, where the immunotherapy may include at least one of a PD-1 inhibitor or a PD-L1 inhibitor. The operation may also include determining a predictive score for the patient's response to immunotherapy based on the TLS maturity status of one or more TLS regions identified in the tissue image and TLS features extracted from one or more TLS regions identified in the tissue image.

[0012] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware that communicates with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform an operation which includes receiving an input tissue structure image relating to a patient diagnosed with cancer. The input tissue structure image includes a plurality of image pixels. The operation also includes processing the input tissue structure image using a cell classification model to generate one or more lymphocyte density maps within the input tissue structure image, and performing morphological image processing on one or more lymphocyte density maps to identify one or more TLS regions within the input tissue structure image. Each TLS region (135) is represented by a cluster of lymphocyte cells. For each corresponding TLS region of one or more TLS regions identified in the input tissue structure image, the operation also includes extracting each set of TLS features from each cluster of lymphocyte cells representing the corresponding TLS region, and processing each set of TLS features using a TLS classification model to classify the corresponding TLS region as one of a first TLS maturity state, a second TLS maturity state, and a third TLS maturity state.

[0013] This embodiment may include one or more of the following optional features. In some embodiments, the operation also includes identifying tumor regions within the input tissue structure image by processing the input tissue structure image using a tumor detection model. Here, generating one or more lymphocyte density maps by processing the input tissue structure image may include generating one or more lymphocyte density maps by processing the input tissue structure image using a cell classification model by performing single-cell imaging analysis on the tumor regions identified within the input tissue structure image. In these embodiments, the tumor detection model may be trained by training the tumor detection model with a neural network to learn how to identify tumor regions within the tissue structure image, which are rasterized image tiles from a set of histopathology images of an entire slide, each manually annotated as containing tumor or non-tumor, and how to learn how to identify tumor regions within the tissue structure image.

[0014] In some embodiments, the cell classification model is trained by using a neural network to train the cell classification model on multiple image patches, thereby teaching the cell classification model how to acquire multiple image patches and how to classify individual cells in tissue structure images as tumor cells, lymphocytes, or non-malignant cells. Each image patch includes a corresponding set of human cells and manual annotations labeling each human cell as a tumor cell, lymphocyte, or non-malignant cell.

[0015] In some embodiments, the TLS classification model is trained by acquiring a training dataset containing multiple training tissue structure images, each containing a tumor microenvironment and each having manual annotations. The manual annotations identify one or more TLS regions in the training tissue structure images and, for each corresponding TLS region, a ground truth TLS maturity state indicating that the corresponding TLS region contains a first TLS maturity state, a second TLS maturity state, or a third TLS maturity state. Each TLS region is represented by each cluster of lymphocyte cells. In these embodiments, the TLS classification model is further trained by training the TLS classification model with each set of training TLS features extracted for each TLS region, instructing the TLS classification model to learn how to extract each set of TLS features from each cluster of lymphocyte cells representing the TLS region and how to predict the ground truth TLS grade for each corresponding TLS region. Training the TLS classification model may include training the TLS classification model using the CART (classification and regression trees) algorithm.

[0016] The first TLS maturation state may include dense aggregates of at least a threshold number of lymphocytes, which do not contain high endothelial venules or germinal centers. The second TLS maturation state may include immature TLS, which include high endothelial venules but do not contain germinal centers, and include at least the above threshold number of dense aggregates of lymphocytes. The third TLS maturation state may include mature TLS, which include high endothelial venules and germinal centers, and include at least a threshold number of dense aggregates of lymphocytes. Each set of TLS features extracted from each cluster of lymphocyte cells may include the area of ​​the corresponding TLS region, the roundness of the corresponding TLS region, and the distortion of the corresponding TLS region.

[0017] In some embodiments, the operation further includes generating a pixel mask that enhances at least the surrounding area of ​​each corresponding TLS region for each of the one or more TLS regions identified in the input tissue structure image, generating an output image that enhances the input tissue structure image by superimposing each pixel mask generated for each TLS region onto the input tissue structure image, and providing the output image for display on a screen that communicates with data processing hardware. In these embodiments, each pixel mask generated for each corresponding TLS region classified as a first mature state includes a first pixel mask, and each pixel mask generated for each corresponding TLS region classified as a second mature state includes, 1 Each pixel mask generated for each corresponding TLS region classified as a third mature state includes a second pixel mask that is visually identifiable from the first and second pixel masks, and each pixel mask generated for each corresponding TLS region includes a third pixel mask that is visually identifiable from the first and second pixel masks.

[0018] In some embodiments, the operation also includes determining an overall TLS score for an input tissue image based on the TLS maturity status of one or more TLS regions identified in the tissue image and TLS features extracted from one or more TLS regions identified in the tissue image. In these embodiments, the operation may also include determining a treatment recommendation to treat the patient with immunotherapy based on the overall TLS score, where the immunotherapy may include at least one of a PD-1 inhibitor or a PD-L1 inhibitor. The operation may also include determining a predictive score for the patient's response to immunotherapy based on the TLS maturity status of one or more TLS regions identified in the tissue image and TLS features extracted from one or more TLS regions identified in the tissue image.

[0019] Details of one or more embodiments of this disclosure are described in the accompanying drawings and the following description. Other embodiments, features, and advantages will be apparent from the specification and drawings and from the claims. [Brief explanation of the drawing]

[0020] [Figure 1] This is a schematic diagram of an exemplary system for identifying, classifying, and quantifying tertiary lymphoid tissue-like structures (TLSs) in histological images of the tumor microenvironment. [Figure 2A] A table listing exemplary TLS features is shown below. [Figure 2B] A table listing exemplary TLS features is shown below. [Figure 2C] A table listing exemplary TLS features is shown below. [Figure 2D] A table listing exemplary TLS features is shown below. [Figure 2E] A table listing exemplary TLS features is shown below. [Figure 2F] A table listing exemplary TLS features is shown below. [Figure 2G] A table listing exemplary TLS features is shown below. [Figure 2H] A table listing exemplary TLS features is shown below. [Figure 2I] A table listing exemplary TLS features is shown below. [Figure 2J] A table listing exemplary TLS features is shown below. [Figure 2K] A table listing exemplary TLS features is shown below. [Figure 3A] This is a schematic diagram illustrating an example of the training process used to train a TLS classification model. [Figure 3B] This is a schematic diagram illustrating an example of the training process used to train a tumor detection model. [Figure 3C] This is a schematic diagram illustrating an example of the learning process used to train a cell classification model. [Figure 4] This is a flowchart illustrating an exemplary configuration of a method for identifying, classifying, and quantifying TLS in tissue structure images of the tumor microenvironment. [Figure 5A] This is an exemplary confusion matrix comparing accuracy between a TLS classification model and a pathologist. [Figure 5B] This is an exemplary confusion matrix comparing accuracy between a TLS classification model and a pathologist. [Figure 6A] This is an exemplary plot comparing the performance of a TLS classification model with that of a pathologist. [Figure 6B] This is an exemplary plot comparing the performance of a TLS classification model with that of a pathologist. [Figure 6C] This is an exemplary plot comparing the performance of a TLS classification model with that of a pathologist. [Figure 7] This shows an example input tissue structure image and the corresponding classified TLS maturity state. [Figure 8] This shows an exemplary input tissue structure image representing a mature TLS state, and a corresponding output image including each pixel mask. [Figure 9] This shows an exemplary input tissue structure image representing an immature TLS maturation state, and a corresponding output image including each pixel mask. [Figure 10] This shows an exemplary input tissue structure image representing the TLS maturation state of lymphocyte aggregates, and a corresponding output image including each pixel mask. [Figure 11] The following are exemplary output images containing TLS regions corresponding to the mature TLS state, the immature TLS state, and the lymphocyte aggregate TLS state, respectively. [Figure 12] An example of an unconverted output image and a converted output image are shown. [Figure 13] An example of an unconverted output image and a converted output image are shown. [Figure 14] An example of an unconverted output image and a converted output image are shown. [Figure 15] An example of an unconverted output image and a converted output image are shown. [Figure 16] An example of an unconverted output image and a converted output image are shown. [Figure 17] This is a schematic diagram of the processing flow for verifying TLS features extracted using transcriptome analysis correlation. [Figure 18]An illustrative table of 12 chemokine gene signatures is shown. [Figure 19] An example feature table is shown. [Figure 20A] An illustrative graphical representation of correlation data is shown. [Figure 20B] An illustrative graphical representation of correlation data is shown. [Figure 20C] An illustrative graphical representation of correlation data is shown. [Figure 21A] This shows an illustrative graphical representation of a correlation diagram that verifies the extracted TLS features (140). [Figure 21B] This shows an illustrative graphical representation of a correlation diagram that verifies the extracted TLS features (140). [Figure 21C] This shows an illustrative graphical representation of a correlation diagram that verifies the extracted TLS features (140). [Figure 21D] This shows an illustrative graphical representation of a correlation diagram that verifies the extracted TLS features (140). [Figure 22] This is a schematic diagram of an exemplary computing device that may be used to implement the system and method described herein. [Modes for carrying out the invention]

[0021] Similar reference symbols in various drawings indicate similar components.

[0022] Tertiary lymphoid tissue-like structures (TLSs) are ectopic lymphoid organs that arise in non-lymphoid tissues, such as in sites of chronic inflammation and tumors. TLSs are vascularized lymphoid structures that arise in benign and tumorous tissues with chronic inflammation. TLSs are highly organized structures that resemble secondary lymphoid structures (e.g., lymph nodes). A TLS may consist of a B-cell zone containing active germinal centers, surrounded by a B-cell zone containing various types of dendritic cells (DCs), T cells, high endothelial venules (HEVs), and / or other supporting cells within the structural matrix. Unlike lymph nodes, TLSs lack a fibrous capsule and are directly exposed to the tumor microenvironment (TME). TLSs are more abundant in the invasive margins / stroma compared to the tumor core. The presence of TLSs is associated with favorable outcomes in the treatment of multiple signs (e.g., melanoma with nivolumab, or with nivolumab and ipilimumab). TLS structures can be classified as lymphoid aggregates (LA) (i.e., the first stage of maturation), immature TLSs (imTLS) (e.g., grade 1) (i.e., the second stage of maturation), or mature TLSs (mTLS) (e.g., grade 2) (i.e., the third stage of maturation) in which germinal centers (GCs) are present. In some cases, TLSs are absent (e.g., grade 0). While the biological mechanisms behind their formation are not fully understood, TLSs are known to play an important role in antitumor immune responses. For example, the presence of TLSs has been associated with favorable prognosis and improved response to immunotherapy across numerous cancer types.

[0023] The conventional approach to detecting TLS in patients involves using tissue staining techniques for immunocytoplasmic cell lineage markers through multiplexed immunohistochemistry or immunofluorescence techniques. However, multiplexed imaging is not routinely applicable given its cost, high complexity, small field of view, and scaling difficulties, which limits its use in research settings. On the other hand, hematoxylin-eosin (H&E) staining is widely available and remains the clinical standard in histopathology. Evaluating H&E-stained slides based on pathologist assessment is time-intensive and labor-intensive, and manual, qualitative assessments performed manually by pathologists are often inaccurate and subject to inter-observer variability.

[0024] Embodiments of the present invention relate to leveraging machine learning techniques to train a model using deep learning to detect the presence of TLS regions in H&E-stained tissue structure images and to classify each TLS region into one of three TLS maturation states. The first TLS maturation state includes dense aggregates of at least a threshold number of lymphocytes that do not include high endothelial venules or germinal centers. In some embodiments, the threshold number is equal to 100. The second TLS maturation state includes immature TLS associated with the above threshold number of dense aggregates of lymphocytes that include high endothelial venules but do not include germinal centers. The third TLS maturation state includes mature TLS associated with dense aggregates of at least a threshold number of lymphocytes that include high endothelial venules and germinal centers. More specifically, the embodiment includes generating one or more lymphocyte density maps by processing an input tissue structure image (e.g., an H&E-stained tissue structure image) using a cell classification model, identifying one or more TLS regions within the input tissue structure image, each represented by a cluster of lymphocyte cells, by performing morphological image processing on one or more lymphocyte density maps, and for each corresponding TLS region, extracting each set of TLS features from each cluster of lymphocyte cells representing the corresponding TLS region. Subsequently, the learned TLS classification model receives each set of TLS features extracted for each corresponding TLS region and classifies the corresponding TLS region as one of a first TLS maturity state, a second TLS maturity state, and a third TLS maturity state.

[0025] Embodiments of the present invention further relate to calculating a TLS score for an input tissue structure image based on TLS maturity states output from a TLS classification model and TLS features relating to TLS regions identified in the input tissue structure image. The TLS score determinationr may determine a TLS score for each of the three TLS maturity states, based on the total tumor area and, further, the total TLS area of ​​each of the TLS regions classified for each of the three TLS maturity states. The TLS score determinationr may then calculate an overall TLS score for the patient associated with the input tissue structure image based on a linearly weighted sum of each total TLS area divided by the tumor area. As will be described in detail later, the overall TLS score may be used to predict various prognostic values ​​for the patient, such as predicting survival outcomes like overall survival and progression-free survival. That is, a higher overall TLS score indicates significantly improved overall survival and progression-free survival compared to a lower overall TLS score. Thus, the overall TLS score may be used to predict prognostic outcomes instead of using tumor stage prediction, and / or prognostic outcomes predicted using tumor stage / grade may be further refined by the overall TLS score.

[0026] For each corresponding TLS region among the one or more TLS regions identified in the input tissue structure image, the image auger may generate a pixel mask that enhances at least the periphery of the corresponding TLS region, and then generate an output image that enhances the input tissue structure image by superimposing each pixel mask generated for each TLS region onto the input tissue structure image. The output image generated by the image auger may be provided for display on a screen for viewing by a healthcare professional (HCP). Here, the image auger receives classification output from a TLS classification model and generates a visually distinct pixel mask for each of three different TLS maturation states. For example, the pixel mask generated for a TLS region classified as a first maturation state may include a first color, the pixel mask generated for a TLS region classified as a second maturation state may include a different second color, and the pixel mask generated for a TLS region classified as a third maturation state may include a third color different from the first and second colors. In some embodiments, the pixels generated for a TLS region classified as a third maturation state enhance at least the periphery of the corresponding TLS region and further enhance the area of ​​pixels enclosed by germinal centers.

[0027] Embodiments of the present invention further relate to a training process for training a TLS classification model. The training process involves acquiring a training dataset containing multiple training tissue structure images. Each of the multiple training tissue structure images contains a tumor microenvironment and includes manual annotations from a pathologist. The manual annotations identify the presence of TLS regions in each training tissue structure image, each represented by a cluster of lymphocyte cells. The manual annotations further identify ground truth TLS maturity states for each corresponding TLS region, indicating that the corresponding TLS region contains a first TLS maturity state, a second TLS maturity state, or a third maturity state. Next, the training process extracts each set of training TLS features from each cluster of lymphocyte cells representing each TLS region. Each set of training TLS features may include the area of ​​the TLS region, its roundness (i.e., the ratio of the area of ​​the TLS region multiplied by 4π to the square of the perimeter of the TLS region), and the density distortion of each cluster of lymphocyte cells representing each TLS region. Based on each set of training TLS features extracted for each TLS region, the training process uses the CART (classification and regression trees) algorithm to train the TLS model to learn how to predict the ground truth TLS grade for each corresponding TLS region.

[0028] In particular, the cell classification model is trained to learn how to classify individual cells in a tissue structure image as tumor cells, lymphocytes, or non-malignant cells. As used in this application, lymphocytes may include T cells and B cells. The cell classification model may be trained using a Mask R-CNN deep learning model to learn how to divide and classify individual nuclei into tumor cells, lymphocytes, and other non-malignant cells.

[0029] In some embodiments, image preprocessing is performed on the input tissue structure image by processing the input tissue structure image using a tumor detection model to identify tumor regions within the input tissue structure image, thereby generating one or more lymphocyte density maps by performing single-cell image analysis on the identified tumor regions within the input tissue structure using a cell classification model. The tumor detection model may be trained on multiple rasterized image tiles from a set of histopathology images of an entire slide, each manually annotated as containing tumors or non-tumors. More specifically, the deep learning neural network is trained on multiple image tiles so as to teach the tumor detection model how to learn to identify tumor regions within the tissue structure image. The deep learning neural network may include a ResNet18 deep learning model.

[0030] A key advantage is that the deep learning-based single-cell analysis technique disclosed herein provides the ability to accurately identify, classify, and quantify the presence of TLS regions from an image of an entire H&E-stained slide, without suffering any of the shortcomings of other techniques that employ patch or tile-based approaches for image analysis. Since TLS can vary considerably in size, density, and morphology, there are significant challenges in using conventional patch-based approaches to identify and interpret TLS regions. As will become apparent, the technique disclosed herein includes quantifying the spatial distribution of lymphocytes, thereby providing an accurate and interpretable model for classifying TLS according to their maturity state.

[0031] Similarly, manual and qualitative assessments of TLS performed by pathologists lack automated enumeration and quantitative characterization of TLS. By the same concept, such manual and qualitative assessments of TLS performed by pathologists are inaccurate and subject to inter-observer variability when assessed against H&E-stained slides. See Non-Patent Literature 3.

[0032] Referring to Figure 1, in some embodiments, the system 100 includes a client device 111 that inputs a tissue image 110 of a patient diagnosed with cancer into a TLS classification model 350 to identify, classify, and quantify the presence of TLS regions within the tissue image for use as a predictive biomarker for the efficacy and prognosis of immune checkpoint inhibitors (ICIs). The input tissue image 110 may optionally include metadata 11 containing information such as the type of cancer the patient is diagnosed with, the tumor stage / grade, and / or the patient's demographic information. The input tissue image 110 may include a whole slide image (WSI) stained with hematoxylin and eosin (H&E). The input tissue image 110 contains multiple image pixels. The input tissue image 110 characterizes a human tumor biopsy specimen. The input tissue structure image 110 may include the tumor microenvironment relating to any number of cancers, including but not limited to bladder cancer (BLCA), breast cancer (BRCA), stomach adenocarcinoma (STAD), lung adenocarcinoma (LLUAD) (e.g., non-small cell lung cancer adenocarcinoma (NSCLC-AD)), and / or lung squamous cell carcinoma (LUSC) (e.g., non-small cell lung cancer squamous (NSCLC-SQ)).

[0033] A client device 111 may be associated with a user 10, such as a healthcare professional (HCP), who may communicate with a remote system 141 via a network 132. The remote system 141 may be a distributed system (e.g., a cloud environment) having scalable / elastic resources 142. Resources 142 include computing resources 144 (e.g., data processing hardware) and / or storage resources 146 (e.g., memory hardware). In some embodiments, the remote system 141 runs a TLS identification and quantification application 160 (also simply referred to as "Application 160"), which is configured to run a TLS classification model 450 and other components such as a tumor detection model 450, a cell classification model 550, a lymphocyte aggregator 120, a morphological image processing module 130, a TLS extractor 145, a TLS score determinationr 150, and an image augmenter 360. Here, the client device 111 may access an application 160 running on a remote system 141, or it may input the organizational structure input image 110 to the TLS classification model 350 via a GUI (graphical user interface) running on the client device 111. The GUI may be displayed to the user 10 via the screen 114 of the client device 111. Additionally or alternatively, the client device 111 may run an application 160 that implements the ability to perform any combination of the TLS classification model 350 and / or other components on the client device 111 to identify, classify, and quantify the presence of TLS regions 135 within the organizational structure image 110.

[0034] The TLS identification and quantification application 160 may verify the TLS details 190 and / or treatment recommendations 192 based on the identified TLS regions 135, which have been classified and quantified using the TLS classification model 350. The application 160 may return the TLS details 190 and / or treatment recommendations 192 to the client device 111, causing the client device to display the TLS details 190 and / or treatment recommendations 192 on the screen 114 of the client device 111. The TLS details 190 may include an overall TLS score 152 for the input tissue structure image 110, and may also include, but are not limited to, other details such as the number of TLS regions associated with a first maturity state (e.g., TLS1) classified by the TLS classification model 350, the number of TLS regions associated with a second maturity state (e.g., TLS2) classified by the TLS classification model 350, and the number of TLS regions associated with a third maturity state (e.g., TLS3) classified by the TLS classification model 350. Here, the first TLS maturation state includes dense aggregates of at least a threshold number of lymphocytes, which do not include high endothelial venules or germinal centers. In some embodiments, the threshold number is equal to 100. The second TLS maturation state includes immature TLS associated with the above threshold number of dense aggregates of lymphocytes, which include high endothelial venules but do not include germinal centers. The third TLS maturation state includes mature TLS associated with dense aggregates of lymphocytes, which include high endothelial venules and germinal centers. The TLS details 190 provided for display on screen 114 may further include an output image 110A that enhances the input tissue structure image 110 by superimposing each pixel mask 112 generated for each of the TLS regions onto the input tissue structure image 110. The treatment recommendation 192 may indicate whether to apply (or not apply) immunotherapy to the patient in order to treat the patient. For example, the immunotherapy may include a PD-1 inhibitor (e.g., an anti-PD-1 antibody) or a PD-L1 inhibitor (e.g., an anti-PID-L1 antibody). In one embodiment, the immunotherapy includes nivolumab, which is an immune checkpoint inhibitor.

[0035] Treatment recommendation 192 may further include predicted prognostic outcomes for the patient based on TLS detail 190, such as overall survival (OS) (i.e., in months) and progression-free survival (PFS) (in months). Treatment recommendation 192 may also show predicted OS and / or PFS with immunotherapy, contrasted with predicted OS and / or PFS without immunotherapy. Prognostic outcomes predicted by application 160 may be communicated to patients, healthcare providers, and / or relatives of patients in order to make better testing and treatment decisions for the specific health condition being diagnosed, or to perform risk stratification of the clinical trial.

[0036] In some embodiments, the input tissue structure image 110 undergoes initial image preprocessing to ensure sufficient image quality. The input tissue structure image may include a 40x magnification. However, WSI slides scanned at lower magnifications (e.g., 20x) may be used. To minimize the impact of image artifacts, the image preprocessing may involve downsampling the entire slide image with a 32 factor and applying appropriate color factors to remove areas with write, folding, and blur artifacts.

[0037] In the illustrated embodiment, the tumor detection model 450 identifies one or more tumor regions 115 within the input tissue structure image 110 by processing the input tissue structure image 110. Each tumor region 115 may be represented by a group of corresponding pixels in the input tissue structure image 110 in which the tumor region 115 is located. In particular, since only TLS inside or around the tumor region 115 is relevant, the tumor detection model 450 may separate cancerous tissue from normal tissue, allowing post-processing for TLS identification and quantification to focus on the tumor regions 115 in the input tissue structure image 110. The tumor detection model 450 may include a pre-trained symptom-specific tissue segmentation model configured to distinguish cancer, associated stroma, and necrosis from normal tissue by processing the input tissue structure image 110. Figure 3B shows an exemplary tumor detection model training process 300b that may be used to train the tumor detection model 450. The training process 300b obtains a number of rasterized image tiles 370 from a set of histopathological images across an entire slide. The histopathology images may include publicly available and already annotated H&E-stained WSIs from patients with colorectal cancer and gastric cancer. Each image tile 370 may include manual annotations 372 indicating the location of tumor and non-tumor regions (including adipose tissue, mucus, stroma, or muscle) within the histopathology image of the entire slide. The image tiles may consist of 512 × 512 image tiles, each 0.5 micrometers in size. The learning process 300b includes training the tumor detection model 450 with a plurality of image tiles 370 so as to teach the tumor detection model 450 how to learn to identify tumor regions within the tissue structure images using a neural network 374. In some embodiments, the neural network 374 includes a ResNet 18 deep learning network, and the loss module 378 calculates a learning loss 380 based on predictions 376 output by the ResNet 18 network for ground truth annotations 372.The training process 300b may update the parameters of ResNet 18 based on the training loss 380 until the parameters of ResNet 18 converge and a trained tumor detection model 450 is obtained. The loss module 378 may use a cross-entropy loss function and counteract overfitting by applying L2 regularization. The training process 300b may expand tumor segmentation by expanding images in 0.5 mm increments to include the invasion margin. The training process may further augment the image tiles 370 used for training by applying horizontal / vertical flipping and translation.

[0038] Referring again to Figure 1, after the tumor detection model 450 identifies the tumor region 115, the cell classification model 550 processes the input tissue structure image 110 by performing a single-cell imaging analysis on the identified tumor region 115 within the input tissue structure image 110 (i.e., on the image pixels corresponding to the tumor region 115) to generate a classified tumor region 115C. That is, the single-cell imaging analysis performed by the cell classification model 550 classifies individual cells / nuclei as tumor cells, lymphocytes (i.e., B cells and T cells, dendritic cells (DCs), high endothelial venules (HEVs)), and non-malignant cells. As used in this application, the learned cell classification model 550 functions as a lymphocyte mask for classifying which cells in the tumor region 115 contain lymphocytes. Thus, the classified tumor region 115C may correspond to a lymphocyte mask that identifies all lymphocytes classified and segmented by the cell classification model 550 within the tumor region 115 within the input tissue structure image 110. Subsequently, application 160 runs the lymphocyte aggregator 120, which processes the classified tumor regions 115C output by the cell classification model 550 into a predefined grid (e.g., 16 × 16 μm). 2 The number of lymphocytes per unit square in the grid is counted. This generates one or more lymphocyte density maps 125 within the input tissue structure image 110.

[0039] Figure 3C shows an exemplary cell classification model training process 300c that may be used to train a cell classification model 550 on multiple image patches 382. Each image patch (i.e., image tile) 382 is manually annotated to characterize a corresponding group of human cells, labeling each human cell as a tumor cell, lymphocyte, or non-malignant cell. The multiple image patches may include 1,358 image patches from 66 patients in a published dataset, with manual annotations 384 including 17,582 tumor cells, 22,550 lymphocytes, and 10,675 other non-malignant cells. The training process 300c includes training the cell classification model 550 on the multiple image patches 382 using a neural network 386 to teach the cell classification model 550 how to classify individual cells in tissue structure images as tumor cells, lymphocytes, or non-malignant cells. In some embodiments, the neural network 386 includes a Mask R-CNN deep learning network, and the loss module 392 calculates a learning loss 390 based on the predictions 388 output by the Mask R-CNN network 386 for ground truth annotations 384. The learning process 300b may update the parameters of the Mask R-CNN based on the learning loss 392 until the parameters of the Mask R-CNN converge and a learned cell classification model 550 is obtained. As used in this application, the learned cell classification model 550 functions as a lymphocyte mask for classifying which cells in the tumor region 115 contain lymphocytes. The loss module 392 may update the Mask R-CNN via the learning loss 392 using stochastic gradient descent techniques. The learning process may further augment the image patches 382 used for learning by applying horizontal / vertical flipping and translation.

[0040] Referring again to Figure 1, in some embodiments, the TLS identification and quantification application 160 identifies one or more TLS regions 135 within the input tissue structure image 110 by performing morphological imaging 130 on one or more lymphocyte density maps 125. In particular, each TLS region represents each cluster of lymphocyte cells. The morphological imaging 130 may indicate the pixel location corresponding to each identified TLS region 135 within the input tissue structure image 110. Each TLS region 135 may correspond to a TLS mask. In some embodiments, the morphological imaging 130 performed on the lymphocyte density map 125 applies thresholding to exclude lymphocyte clusters having an area smaller than a predefined threshold area from being identified as TLS regions. The predefined threshold area is 0.0384 mm². 2 It may be equal to.

[0041] For each identified TLS region 135, application 160 runs a TLS feature extractor 145 configured to extract each set of TLS features 140 from each cluster of lymphocyte cells representing the corresponding TLS region 135. The set of TLS features 140 may include human interpretable features (HIFs) associated with the TLS region 135. In some embodiments, some of the TLS features include sample-level features that include at least one of the outline count, area, shape, or location of the corresponding TLS region 135. The TLS features 140 extracted from each cluster of lymphocyte cells representing the corresponding TLS region 135 may include the area of ​​the TLS region 135, the roundness of the TLS region 135 (i.e., the ratio of the area of ​​the TLS region 135 multiplied by 4π to the square of the circumference of the TLS region 135), and the density distortion of each cluster of lymphocyte cells representing the TLS region 135. The TLS feature 140 may additionally or alternatively include at least one of the following: the area of ​​germinal centers within the object in the tissue, the area of ​​the object in the tissue, the centroid x of the object in the tissue, the centroid y of the object in the tissue, the longest distance of the object from the tumor, the circumference of the object in the tissue, the shortest distance of the object from the tumor, the total germinal centers within the object in the tissue, or the ratio of the area of ​​germinal centers within the object to the area of ​​the object in the tissue. Some of the TLS features 140 may include sample-level features that include one or more of the following: area of ​​TLS region 135, total count of lymphocytes, area ratio, count ratio, maximum area, maximum longest distance from tumor, maximum circumference, maximum shortest distance from tumor, maximum total area, maximum total count, average area, average longest distance from tumor, average circumference, average shortest distance from tumor, average total area, average total count, median area, median longest distance from tumor, median circumference, median shortest distance from tumor, median total area, median total count, minimum area, minimum longest distance from tumor, minimum circumference, minimum shortest distance from tumor, minimum total area, or minimum total count.

[0042] Figures 2A to 2K show multiple tables listing TLS features 140 that can be extracted by the TLS extractor 145. Each table contains multiple columns listing (1) the feature name, (2) the feature type which identifies whether the feature is an identifier, metadata, or a feature, (3) the feature description which describes the extracted feature, and the human-interpretable feature (HIF) type which indicates whether the feature is an identifier, metadata, raw feature, minimum feature, maximum feature, intermediate feature, mean feature, ratio feature, or sum feature.

[0043] Referring again to Figure 1, the TLS classification model 350 may classify the corresponding TLS region 135 as one of the first TLS maturation state (TLS1), the second TLS maturation state (TLS2), or the third TLS maturation state (TLS3) by processing each set of TLS features 140. The first maturation state may be associated with lymphocyte aggregates, the second maturation state may be associated with each cluster of lymphocyte cells having primary follicles without germinal centers, and the third maturation state may be associated with each cluster of lymphocyte cells having primary follicles and secondary cells having germinal centers. Assuming that TLS2 and TLS3 tend to have a round shape and are usually larger than TLS1, and that TLS3 has a distinctive germinal center with a lower lymphocyte density, the aforementioned area, roundness, and distortion TLS features 140 can be interpreted by the TLS classification model 350, which has been trained to accurately classify each TLS region 135. Each TLS region 135 classified by the TLS classification model may correspond to a prognostic biomarker. The TLS classification model 350 may output a TLS state 312 indicating the maturity state of each TLS region 135 classified by the TLS classification model 350.

[0044] In some embodiments, application 160 runs an image augmenter 360 configured to augment the input tissue structure image 110 based on TLS states 312 output from a TLS classification model 350 for one or more TLS regions 135 identified in the input tissue structure image 110. Here, the image augmenter 360 may generate each pixel mask 112 that enhances at least the surrounding area of ​​each corresponding TLS region 135 based on the maturity state of the corresponding TLS region 135 (e.g., TLS1, TLS2, or TLS3). The image augmenter 360 may generate a first pixel mask 112 for the TLS region 135 classified as TLS1, a second pixel mask 112 different from the first pixel mask 112 for the TLS region 135 classified as TLS2, and a third pixel mask 112 different from the first and second pixel masks 112 for the TLS region 135 classified as TLS3. That is, the different pixel masks 112 may be visually distinguishable from one another. In some embodiments, different pixel masks 112 are associated with different colors. The image auger 360 generates an output image 110A that augments the input tissue structure image 110 by superimposing each pixel mask 112 generated for each of the TLS regions 135 onto the input tissue structure image 110. The pixel masks 112 are superimposed as graphical features that highlight at least the periphery of each corresponding TLS region 135, thereby acting as visual cues indicating the location and corresponding classification (e.g., TLS1, TLS2, or TLS3) of each TLS region 135 identified in the output image 110A. As will become apparent, the image auger 360 may apply one or more post-processing rules to generate the output image 110A. As described in the previous paragraph, the application 160 may provide the output image 110 to the client device 111 for display on the screen 114 as TLS details 190.

[0045] In addition to the maturity state, the TLS classification model 350 and / or the TLS feature extractor 145 may be further configured to output / extract topological information associated with the TLS region 135, such as the coordinates of the TLS region 135, their proximity to the tumor bed, and their position relative to the tumor and / or stromal compartments. Thus, the image enhancer 360 or the image generator may process the topological information and any combination of the input tissue structure image, the TLS state 312, the TLS region 135, and the TLS features to generate a topology or heat map as an output image 110A visually indicating the topological information associated with the TLS region 135 of interest.

[0046] Continuing to refer to FIG. 1, the application 160 may further execute a TLS score determiner 150 for calculating an overall TLS score 152 for a patient based on the TLS features 140 and the corresponding TLS maturity states 312 for all TLS regions 135 identified in the input tissue structure image 110. The TLS score determiner 150 may determine the total area of the tumor region 115 (denoted as "area" tumor ). The TLS score determiner 150 may further determine each individual TLS area for each of the three TLS maturity states. For example, the TLS score determiner 150 may determine a first TLS area (denoted as "area" TLS1 ) based on the total area of the TLS regions classified as the first maturity state, a second TLS area (denoted as "area" TLS2 ) based on the total area of the TLS regions classified as the second maturity state, and a third TLS area (denoted as "area" TLS2 ) based on the total area of the TLS regions classified as the second maturity state. In some embodiments, the TLS score determiner 150 calculates the overall TLS score 152 as a linearly weighted sum of the individual TLS areas divided by the tumor area, as follows:

[0047] TLS score = (w1 × area TLS1 + w2 × area TLS2 + w3 × area TLS3) (1)

[0048] Here, w1, w2, and w3 are the corresponding weights. The optimal corresponding weights may be selected by performing a Cox regression analysis of overall survival for each of the individual TLS areas. In one embodiment, w1 is equal to 0.81, w2 is equal to 0.84, and w3 is equal to 1.0. This suggests that the TLS regions classified as a third mature state (e.g., mature TLS) play the most important role in the antitumor immune response.

[0049] In particular, statistical analyses applied to the overall TLS score of 152 and the individual TLS scores indicated by the first, second, and third TLS areas may be used to predict various prognostic values ​​for patients, such as predicted survival outcomes, including but not limited to overall survival and progression-free survival. Overall survival may be defined as the time from diagnosis to death or last follow-up. Progression-free survival may be defined as the time from diagnosis to disease progression, death, or last follow-up. Univariate and multivariate analyses may be performed with the Cox proportional hazards model. Multivariate analyses may include clinical and pathological variables such as tumor stage and grade. Kaplan-Meier analysis and log-rank tests may be used to assess the stratification of patients by risk group. TLS scores may be evaluated in relation to tumor status or grade. Higher overall TLS scores indicate significantly improved overall survival and progression-free survival compared to lower overall TLS scores. For patients with low overall TLS scores, overall survival and progression-free survival are even better than in the absence of identified TLS areas. Thus, the overall TLS score may be used to predict prognosis instead of using tumor stage prediction, and / or prognosis predicted using tumor stage / grade may be further refined by the overall TLS score.

[0050] In some scenarios, application 160 performs post-processing to adjust the output image 110A based on any combination of TLS features 140, one or more TLS scores 152, and TLS states 312. Specifically, application 160 may modify the pixel mask 112 by applying one or more post-processing rules 362, such as fixing small exposed germinal centers, fixing TLS regions 135 that do not have germinal centers classified as a third mature state (mature TLS), fixing mosaics to address multiple-class predictions in the same structure due to confusion by the TLS classification model, applying object-level masking to eliminate false-positive predictions of TLS within cancerous and necrotic tissue regions, and / or applying cutoffs.

[0051] Referring to Figure 3A, in some embodiments, an exemplary TLS classification model training process 300a trains a TLS classification model 350 to learn how to predict TLS states for TLS regions identified in tissue structure images. The training process 300a acquires a training dataset 305 containing a plurality of training tissue structure images 310, 310a-n. Each training tissue structure image 310 may include a tumor microenvironment and may include manual annotations 312 from a qualified pathologist. The manual annotations 312 identify one or more TLS regions 312a in the training tissue structure image 310 and, for each identified TLS region 312a, a ground truth TLS maturity state 312b indicating that the corresponding TLS region 312a contains a first TLS maturity state, a second TLS maturity state, or a third TLS maturity state. Each annotated TLS region 312a in the training tissue structure image 310 is represented by a cluster of lymphocyte cells.

[0052] The learning process 300a executes the TLS feature extraction module 320, which receives each training tissue structure image 310 and extracts each set of training TLS features 140 for each TLS region 312a. That is, for each TLS region 312a annotated in the training tissue structure image 310, the TLS feature extraction module 320 may extract each set of training TLS features 140 from each cluster of lymphocyte cells representing the TLS region 312a. The TLS feature extraction module 320 may also include a pre-trained tumor extraction model 450 and a pre-trained cell classification model 550 for generating a lymphocyte density map. The feature extraction module 320 may also include any other components, or combinations of components, executed by the application 160.

[0053] The training TLS features may include, but are not limited to, the area 140a of the TLS region, the roundness 140b (i.e., the ratio of the area of ​​the TLS region 312a multiplied by 4π to the square of the perimeter of the TLS region), and the density distortion 140c of each cluster of lymphocyte cells representing the TLS region 312a. Based on each set of training TLS features 140 extracted for each TLS region 312a, the training process 300a trains the TLS classification model 350 using the CART (classification and regression trees) algorithm 340 to learn how to predict the ground truth TLS state 312b for each corresponding TLS region 312a. In some embodiments, the training process 300a trains the CART algorithm 340 using the scikit-learn package from Python programming language version 3.6.11 (Python Software Foundation) with default parameter settings (criterion=gini;splitter=best;min_samples_split=2). The maximum tree depth was determined to be 4 using 5-fold cross-validation on the training dataset 305. Assuming the relative importance of TLS3, the class weights for TLS1, TLS2, and TLS3 may be empirically set to 1, 2, and 3, respectively, during training.

[0054] Figure 4 is a flowchart illustrating an exemplary configuration of operation for a method 400 for identifying, classifying, and quantifying TLS regions 135 within an input tissue structure image 110. Method 400 may be performed in data processing hardware 142 of a remote system 141 and / or on a client device 111. In operation 402, method 400 includes receiving an input tissue structure image 110 relating to a patient diagnosed with cancer. The input tissue structure image contains multiple image pixels. The input tissue structure image 110 may include an H&E-stained image of a tumor sample from the patient.

[0055] In operation 404, method 400 includes generating one or more lymphocyte density maps 125 within the input tissue structure image 110 by processing the input tissue structure image 110 using a cell classification model 550. In operation 406, method 400 includes identifying one or more TLS regions 135 within the input tissue structure image 110 by performing morphological image processing on one or more lymphocyte density maps 125, where each TLS region 135 is represented by a cluster of lymphocyte cells.

[0056] In operation 408, method 400 includes extracting each set of TLS features 140 from each cluster of lymphocyte cells representing each corresponding TLS region 135. In operation 410, method 400 includes classifying each corresponding TLS region 135 as one of a first TLS maturation state, a second TLS maturation state, and a third TLS maturation state by processing each set of TLS features using a TLS classification model 350. The first TLS maturation state includes lymphocyte aggregates of at least a threshold number of lymphocytes that do not include high endothelial venules or germinal centers. The second TLS maturation state includes dense aggregates of at least the above threshold number of lymphocytes that include high endothelial venules but do not include germinal centers. The third TLS maturation state includes dense aggregates of at least a threshold number of lymphocytes that include high endothelial venules and germinal centers.

[0057] As an advantage, after training the TLS classification model 350, the accuracy of the TLS classification model 350 in identifying and classifying TLS regions 135 within the input tissue structure image 110 is comparable to (or, in some scenarios, even better than) the accuracy of a pathologist manually classifying TLSs. For example, the confusion matrices 500 shown in Figures 5A and 5B show confusion matrices 500 comparing the accuracy 510 between a pathologist and the trained TLS classification model 350. In particular, the first confusion matrix 500,500a (Figure 5A) shows the normalized accuracy 510 of the TLS classification model 350, and the second confusion matrix 500,500b (Figure 5B) shows the normalized accuracy of the pathologist annotator. Here, the confusion matrix 500 shows the accuracy 510 for each TLS maturation state 312 (e.g., mature TLS, immature TLS, germinal centers, lymphocyte aggregates, and others). The ground truth maturity status of these input tissue structure images 110 was generated by majority agreement of five experienced pathologists. Furthermore, Figures 6A–6C show plots 600 comparing the TLS identification and classification performance between the TLS classification model 350 and pathologist annotators. Specifically, the first plot 600,600a (Figure 6A) shows a comparison of accuracy scores 610, the second plot 600,600b (Figure 6B) shows a comparison of F1 scores 620, and the third plot 600,600c (Figure 6C) shows a comparison of recall scores 630. Here, each plot 600 graphically represents the score for each of the different TLS maturity statuses 312.

[0058] Figure 7 shows the input tissue structure images 700 corresponding to each of the TLS maturity states 312 classified by the TLS classification model 350. Therefore, the input tissue structure image 110 (Figure 1) may also be interchangeably referred to as the input tissue structure image 700 with respect to Figure 7. In particular, the input tissue structure image 700 contains the classified TLS maturity states 312 but is not annotated as the output image 110A. In some embodiments, the input tissue structure image 700 corresponds to the entire area of ​​the input tissue structure image 700. In other embodiments, the input tissue structure image corresponds only to the tumor region 115 detected by the tumor detection model 450, or only to the TLS region 135 identified by the morphological image processor 130 (Figure 1), within the input tissue structure image 700.

[0059] In the illustrated embodiment, the first input tissue structure images 700, 700a correspond to the first TLS maturation state 312, 312a, which shows the lymphocyte aggregate maturation state. In particular, the input tissue structure image 700 corresponding to the first TLS maturation state 312a may include dense aggregates of at least a threshold number of lymphocytes (e.g., 100 lymphocytes) that do not include high endothelial venules and germinal centers. The second input tissue structure images 700, 700b correspond to the second TLS maturation state 312, 312b, which shows the immature TLS maturation state. The input tissue structure image 700 corresponding to the second TLS maturation state 312b may include dense aggregates of at least a threshold number of lymphocytes (e.g., 100 lymphocytes) that include high endothelial venules (in contrast to the first TLS maturation state 312a) but do not include germinal centers. The third input tissue structure images 700, 700c correspond to the third TLS maturation state 312, 312c, which show a mature TLS maturation state. The input tissue structure image 700 corresponding to the third TLS maturation state 312c may include dense aggregates of at least a threshold number of lymphocytes (e.g., 100 lymphocytes), including high endothelial venules and germinal centers 313 (in contrast to the first and second TLS maturation states 312a, 312b).

[0060] Continuing to refer to Figure 7, the fourth input tissue structure images 700, 700d show germinal center 313. In some embodiments, germinal center 313 is not a distinct TLS maturation state 312, but rather a feature of mature TLS maturation state 312c. In other embodiments, the TLS classification model 350 classifies germinal center 313 as a distinct TLS maturation state 312, independent of other TLS maturation states 312. The input tissue structure image 700, including germinal center 313, contains a lighter-colored, less dense region at the center of a mature TLS (e.g., a third TLS maturation state 312c) surrounded by a dense lymphocyte region. Although not shown in Figure 7, the TLS classification model 350 may also classify a fourth TLS maturation state (not shown) that represents a non-TLS region (e.g., a zero TLS region present in the input tissue structure image 700) or other region. As used in this application, the first TLS maturation state 312a, the second TLS maturation state 312b, and the third TLS maturation state 312c may be interchangeably referred to as lymphocyte aggregate TLS maturation state 312a, immature TLS maturation state 312b, and mature TLS maturation state 312c, respectively.

[0061] Figures 8 to 10 show exemplary input tissue structure images 110 and the corresponding output images (e.g., TLS-enhanced tissue structure images) 110A generated by the image augmenter 360 (Figure 1). In other words, application 160 receives the exemplary input tissue structure image 110 (right side) shown in Figures 8 to 10 as input and generates the output image 110A (left side) as output. In some embodiments, the image augmenter 360 generates each pixel mask 112 that enhances at least the periphery of the corresponding TLS region 135. In other embodiments, each pixel mask 112 enhances the entire area of ​​the corresponding TLS region 135. Thus, the image augmenter 360 may generate an output image 110A that enhances the input tissue structure image 110 by superimposing each pixel mask 112 generated for each of the TLS regions 135 onto the input tissue structure image 110.

[0062] Furthermore, the image augmenter 360 generates first pixel masks 112,112a for each corresponding TLS region 135 classified as a first TLS maturity state 312a, second pixel masks 112,112b for each corresponding TLS region 135 classified as a second maturity state 312b, and third pixel masks 112,112c for each corresponding TLS region 135 classified as a third maturity state 312c. In particular, each pixel mask 112 is visually distinguishable from other pixel masks 112 so that the output image 110A visually shows different maturity states 312 using visually distinct pixel masks 112. In this way, the output image 110A is displayed on the screen 114 of the user device 111 so that the user 10 (Figure 1) can easily visualize the different TLS maturity states 312 contained in the output image 110A. Optionally, the image augmenter 360 may generate a fourth pixel mask 112,112d for each corresponding TLS region 135 classified as a non-TLS region.

[0063] For example, Figure 8 shows a graphical representation 800 of an input tissue structure image 110 (right) representing the tissue of a mature TLS mature state 312c, and a corresponding output image 110A (left) that includes a third pixel mask 112c highlighting the area of ​​the TLS region 135 classified as a mature TLS mature state 312c. Furthermore, the mature TLS mature state 312c includes a germinal center 313 encompassed by the TLS region 135 corresponding to the mature TLS mature state 312c. For this purpose, the third pixel mask 112c includes an inner third pixel mask 112c1 highlighting the area of ​​the germinal center 313 and an outer third pixel mask 112c2 highlighting the area of ​​the mature TLS mature state 312c.

[0064] Figure 9 shows a graphical representation 900 of an input tissue structure image 110 (right) representing tissue in an immature TLS mature state 312b, and a corresponding output image 110A (left) that includes a second pixel mask 112b highlighting the area of ​​TLS region 135 classified as immature TLS mature state 312b. In yet another embodiment, Figure 10 shows a graphical representation 1000 of an input tissue structure image 110 (right) representing tissue in a lymphocyte aggregate TLS mature state 312a, and a corresponding output image 110A (left) that includes a first pixel mask 112a highlighting the area of ​​TLS region 135 classified as lymphocyte aggregate TLS mature state 312a. Furthermore, the output image 110A shown in graphical representations 800, 900, and 1000 each further includes a fourth pixel mask 112d highlighting the area of ​​output image 110A corresponding to a non-TLS mature TLS region (e.g., a non-TLS mature state).

[0065] Referring here to Figure 11, in some embodiments, the input tissue structure image 110 includes several classified TLS maturation states 312. For example, the graphical representation 1100 shows an output image 110A that includes three TLS regions 135 corresponding to the first, second, and third TLS maturation states 312a-c, respectively. Here, each pixel mask 112 superimposed on the input tissue structure image easily indicates to the user different identified TLS regions 135 and their corresponding classified TLS maturation states 312. As shown in Figure 11, the output image 110A includes a first TLS region 135,135a classified as lymphocyte aggregate TLS maturation state 312a, a second TLS region 135,135b classified as immature TLS maturation state 312b, and a third TLS region 135,135c classified as mature TLS maturation state 312c including germinal centers 313. Furthermore, below the output image 110A, an enlarged view of the identified TLS region 135 is shown adjacent to the corresponding input tissue structure image 110. For example, the first TLS region 135a includes a first pixel mask 112a that highlights the area of ​​the first TLS region 135a as a lymphocyte aggregate TLS mature state 312a, the second TLS region 135b includes a second pixel mask 112b that highlights the area of ​​the second TLS region 135b as an immature TLS mature state 312b, and the third TLS region 135c includes a third pixel mask 112c that highlights the area of ​​the third TLS region 135c as a mature TLS mature state 312c. Adjacent to each enlarged TLS region 135 is the corresponding portion of the input tissue structure image 110 input to the application 160 that corresponds to the TLS region 135.

[0066] Referring again to Figure 1, in some embodiments, the image augmenter 360 applies one or more post-processing rules 362 before generating the output image 110A. That is, in some scenarios, the TLS classification model 350 classifies the TLS region 135 as a specific TLS maturity state 312 that does not meet a threshold (e.g., a post-processing threshold). Therefore, applying the post-processing rules 362 filters out and removes the classified TLS maturity states 312 that do not meet one or more post-processing rules. In this way, the image augmenter 360 corrects any false positives, or otherwise incorrect classifications, generated by the TLS classification model 350 by applying the post-processing rules 362. For example, the post-processing rules 362 may include, but are not limited to, fixing small exposed germinal centers, fixing mature TLSs without germinal centers, fixing mosaics to address multiple class predictions in the same structure due to model confusion, performing object-level masking to remove false positive predictions of TLSs within cancerous and necrotic tissue regions, and / or applying cutoffs.

[0067] Figures 12 to 15 show the output images 110A generated by the image augmenter 360, both with and without the application of post-processing rule 362. When the image augmenter 360 does not apply post-processing rule 362, the output image 110A may be referred to as the untransformed output image 110A,110A1. On the other hand, when the image augmenter 360 applies post-processing rule 362, the output image 110A may be referred to as the transformed output image 110A,110A2. For example, Figure 12 shows a graphical representation 1200 of the output image 110A when post-processing rule 362 is applied to fix (i.e., filter) small exposed germinal centers. As shown in Figure 12, the untransformed output image 110A1 includes germinal centers 313, which are classified as mature TLS mature state 312c and non-TLS mature state, and are partially surrounded by TLS regions 135 indicated by their respective pixel masks 112. Here, the post-processing rule 362 defines that, for germinal centers 313 that do not meet the threshold amount (e.g., 70% of germinal centers 313 surrounded by mature TLS) of the TLS region 135 classified as mature TLS state 312c surrounding the germinal center 313, the image auger 360 reclassifies the germinal center 313 as TLS state 312 surrounding the majority of the germinal center 313.

[0068] For example, as shown in Figure 12, the unconverted output image 110A1 includes the outer third pixel mask 212c2 (e.g., representing the mature TLS mature state 312c), which only partially surrounds the inner third pixel mask 212c1 (e.g., representing the germinal center 313), thereby failing to meet the threshold amount. Therefore, since most of the area around the germinal center 313 is surrounded by the non-TLS region, the image auger 360 reclassifies the germinal center 313 as a non-TLS mature state. As a result, the converted output image 110A2 removes (i.e., filters out) the germinal center 313 so that the converted output image 110A2 includes only the fourth pixel mask 112d. Alternatively, the post-processing rule 362 may define that, with respect to germinal centers 313 having an area that does not meet a threshold area (e.g., 4480 μm2), the image enhancer 360 reclassifies the germinal center 313 as a TLS mature state 312 that surrounds most of the area around the germinal center 313.

[0069] Referring here to Figure 13, in some embodiments, the post-processing rule 362 is configured to fix the classified mature TLS mature state 312c that does not have a germinal center 313. Here, the image augmenter 360 reclassifies the TLS region classified as mature TLS mature state 312c that is not connected to a germinal center 313 as immature TLS mature state 312b. For example, the region of mature TLS mature state 312c may need to completely encompass a germinal center 313 or partially encompass a germinal center 313 that meets a threshold. As shown in Figure 13, the graphical representation 1300 includes an untransformed output image 110A1 which includes a second pixel mask 112b (e.g., showing immature TLS mature state 312b), an inner third pixel mask 112c1 (e.g., showing a germinal center 313), and an outer third pixel mask 112c2 (e.g., showing mature TLS mature state 312c). In this embodiment, the outer third pixel mask 112c2 does not encompass the germinal center 313 by threshold. That is, the outer third pixel mask 112c2 only partially encompasses the germinal center 313, but not enough to satisfy the threshold. Therefore, in this scenario, the image augmenter 360 reclassifies the mature TLS mature state 312c and the germinal center 313 as immature TLS mature state 312b, as shown in the transformed output image 110A2 having a second pixel mask 112b. The output image 110A also includes a fourth pixel mask 112d corresponding to the non-TLS region of the output image 110.

[0070] Referring here to Figure 14, in some embodiments, the post-processing rule 362 is configured to fix a mosaic 1402 contained in the output image 110A. Here, the mosaic 1402 represents a single TLS region 135 containing multiple classified TLS maturity states 312. In some configurations, if the mosaic 1402 contains at least immature TLS maturity states 312b and lymphocyte aggregate TLS maturity states 312a, the image augmenter 360 reclassifies the entire mosaic 1402 as immature TLS maturity state 312b, based on the determination that the mosaic 1402 contains an immature TLS maturity state 312b with a threshold ratio (e.g., 70 percent). Otherwise, the image augmenter 360 reclassifies the entire mosaic 1402 as lymphocyte aggregate TLS maturity state 312a. In other configurations, if mosaic 1402 contains at least immature TLS mature state 312b and mature TLS mature state 312c, the image augmenter 360 reclassifies the entire mosaic 1402 as mature TLS mature state 312c based on its determination that the mosaic contains a threshold ratio of mature TLS mature state 312c (e.g., 70 percent). Otherwise, the image augmenter 360 reclassifies the entire mosaic 1402 as immature TLS mature state 312b. In yet another configuration, if mosaic 1402 contains at least lymphocyte aggregate TLS mature state 312a and mature TLS mature state 312c, the image augmenter 360 reclassifies the entire mosaic 1402 as mature TLS mature state 312c based on its determination that the mosaic 1402 contains a threshold ratio of mature TLS mature state 312c (e.g., 70 percent). Otherwise, the image enhancer 360 reclassifies the entire mosaic 1402 as lymphocyte aggregate TLS mature state 312a.

[0071] As shown in Figure 14, the graphical representation 1400 includes an unconverted output image 110A1 showing a mosaic 1402 containing a first pixel mask 112a (e.g., representing lymphocyte aggregate TLS mature state 312a) and a second pixel mask 112b (e.g., representing immature TLS mature state 312b). Here, the mosaic 1402 does not satisfy the threshold ratio for immature TLS mature state 312b. Thus, the image augmenter 360 reclassifies the entire area of ​​the mosaic 1402 as lymphocyte aggregate mature state 312a, as shown in the converted output 110A2 which includes the first pixel mask 112a. In particular, the converted output 110A2 removes the mosaic 1402 because the TLS region contains only a single TLS mature state 312. Output image 110A also includes a fourth pixel mask 112d corresponding to the non-TLS region of output image 110.

[0072] Referring here to Figure 15, in some embodiments, the post-processing rule 362 is configured to eliminate false positive predictions of TLS maturity state 312 within cancerous and necrotic tissue areas. In particular, the image augmenter 360 determines whether the ratio of cancer and necrosis in the object or TLS area classified as a first, second, or third TLS maturity state 312a, 312b, 312c satisfies a threshold ratio (e.g., 20 percent) for the object or TLS area. In response to determining that the ratio of cancer and necrosis satisfies the threshold ratio, the image augmenter 360 reclassifies the TLS maturity state 312 as a non-TLS maturity state. For example, as shown in Figure 15, the graphical representation 1500 includes an untransformed output image 110A1 containing a cancer pixel mask 1502 and a necrotic pixel mask 1504. In this embodiment, the cancer pixel mask 1502 and the necrotic pixel mask 1504 satisfy the tissue threshold ratio, and therefore the image auger 360 reclassifies the cancer pixel mask 1502 and the necrotic pixel mask 1504 as non-TLS mature state 312d. Thus, the converted output image 110A2 contains only the fourth pixel mask 112d corresponding to the non-TLS region of the converted output image 110A.

[0073] Referring here to Figure 16, in some embodiments, the post-processing rule 362 is configured to apply a cutoff that filters out classified TLS maturation states 312 that do not meet any of the minimum threshold area, maximum threshold area, and / or maximum number of germinal centers 313. TLS maturation states 312 that do not meet the thresholds are reclassified as non-TLS regions. In particular, lymphocyte aggregate TLS maturation state 312a may have a minimum threshold area (e.g., 0.0008 mm²) and no maximum threshold area. On the other hand, immature TLS maturation state 312b may include both a minimum threshold area (e.g., 0.018 mm²) and a maximum threshold area (e.g., 2.0 mm²). Mature TLS maturation state 312c may have the maximum threshold number of germinal centers 313 (e.g., 8 germinal centers 313). For example, if the mature TLS mature state 312c contains a large number of germinal centers 313 that exceed the maximum threshold number, the image auger 360 reclassifies the mature TLS mature state 312c as a non-TLS region. As shown in Figure 16, the graphical representation 1600 includes the untransformed output image 110A1, which includes the first pixel mask 112a, the second pixel mask 112b, the inner third pixel mask 112c1, the outer third pixel mask 112c2, and the fourth pixel mask 112d. However, none of the pixel masks 112 meet the cutoff threshold, and therefore the image auger 360 reclassifies each of the first, second, and third TLS mature states 312a-c as the fourth TLS classification state 312d, as shown in the transformed output image 110A2. That is, the transformed output image 110A2 contains only the fourth TLS classification state 312d.

[0074] Figure 17 shows a processing flow chart 1700 for verifying extracted TLS features 140 using ribonucleic acid (RNA) sequencing analysis or transcriptome analysis correlation. Specifically, various gene signatures of TLS related to either chemokines or cell populations were studied. For example, Figure 18 shows a table 1800 of 12 chemokine gene signatures derived by correlating metagenes related to combustion and associated with improved patient survival in colorectal cancer, melanoma, and breast cancer. The 12 chemokine gene signatures in table 1800 include CCL2, CCL3, CCL4, CCL5, CCL8, CCL18, CCL19, CCL21, CXCL9, CXCL10, CXCL11, and CXCL13. In another example, eight gene signatures representing follicular helper T cells (TFH cells), particularly CXCL13, characterize breast cancer. In yet another example, 19 gene signatures associated with T helper type 1 (Th1) cells and B cells indicate the presence of TLS. Despite various gene signatures correlating with the presence of TLS, only a limited number of recent studies have investigated the most accurate TLS gene signatures.

[0075] Referring again to Figure 17, while immunohistochemistry-based TLS detection in tissue sections is a robust and specific approach, the processing flow diagram 1700 aims to compare several gene signatures extracted from TLS-positive cancer tissue. As will become clear, the diversity of gene expression between different cancer types leads to a better understanding of gene signatures correlated with the presence of TLS. In particular, the processing flow diagram 1700 includes a TLS feature extraction module 320, a transcriptome module 1710, a feature selector 1720, and a clustering module 1730. The TLS feature extraction module 320 is configured to receive input tissue structure images 110 as input and extract TLS features 140 corresponding to each input tissue structure image 110. For example, the TLS feature extraction module 320 may extract TLS features 140 using a TLS feature extractor 145 (Figure 1).

[0076] The transcriptome module 1710 is configured to receive input tissue structure images 110 as input and to generate gene expression signatures (GES) 1712 for each input tissue structure image 110 as output. Here, the transcriptome module 1710 may generate GES 1712 by extracting RNA sequences from each input tissue structure image 110. Using the TLS features 140 and GES 1712 generated for each input tissue structure image 110, the feature selector 1720 generates a feature table 1722. That is, for each input tissue structure image 110, the feature extractor 1720 pairs the TLS features 140 and GES 172 derived from each input tissue structure image 110 in the feature table 1722. The feature table 1722 includes pairings for all received input tissue structure images 110. Thus, the feature table 1722 structures the TLS features 140 and GES 1712 so that the clustering module 1730 can determine the correlation between the TLS features and GES 1712. In some embodiments, the feature table 1722 includes the number of other TLS features 140 and corresponding annotations for each TLS feature 140 in the set of input tissue structure images, as shown in table 1900 (Figure 19). In some embodiments, the feature selector 1720 may filter the feature table 1722 to include only specific TLS features 140. For example, the feature selector 1720 may generate the feature table 1722 by applying a linear regression lasso penalty.

[0077] Continuing to refer to Figure 17, the clustering module 1730 is configured to receive the clustering table 1722 as input and generate correlation data 1732 as output. In particular, using gene signature data indicating the presence and classification of TLS, the clustering module 1730 may verify that the extracted TLS features 140 correlate with the presence and classification of TLS in the input tissue structure image 110. Furthermore, the clustering module 1730 may further determine gene signatures indicating the presence and classification of TLS in tissue by modifying the extracted TLS features 140. That is, the clustering module 1730 may further determine gene signatures that can identify TLS that are not yet known.

[0078] For example, Figures 20A to 20C show a graphical representation 200 of exemplary correlation data 1732 (Figure 17) that verifies that TLS feature 140 strongly correlates with GES in exemplary breast cancer gene (BRCA) analysis. In particular, graphical representation 2000a (Figure 20A) shows a correlation diagram 2002 showing that TLS maturation states 312 and TLS feature 140 correspond to TLS-inducing genes shown in Table 2004 in BRCA analysis. Correlation diagram 2002 also shows the TLS-inducing genes that occur in each of the first, second, and third TLS maturation states 312a to c, and the TLS-inducing genes that correlate with each individual TLS maturation state 312. Further processing of correlation diagram 2002 by the clustering module 1730 (Figure 17) can generate signatures for a given cancer. In short, correlation diagram 2002 highlights that TLS feature 140 strongly correlates with GES related to the input tissue structure image 110 of BRCA.

[0079] Figure 20B shows a graphical representation of the hierarchical clustering plot 2000b. Here, the plot includes cluster 1 corresponding to low-expression breast cancer samples, cluster 2 corresponding to medium-expression breast cancer samples, and cluster 3 corresponding to high-expression breast cancer samples. Along the Y-axis, the first, second, and third TLS maturation states 312a-c and TLS-inducing genes are plotted for each breast sample cluster. Figure 20C shows a graphical representation of the plot 2000c, with the X-axis showing a timeline in months and the Y-axis showing the overall survival rate of patients from the breast cancer samples. Thus, the graphical representation 2000c shows that breast cancer samples in clusters containing upregulated chemokines have higher long-term overall survival rates.

[0080] Figures 21A to 21D show exemplary graphical representations 2100 of correlation diagrams verifying TLS features 140 extracted by gene signatures. The graphical representations 2100 correlate TLS features 140 and gene signatures across different cancer types (X axis), including BRCA, bladder cancer (BLCA), lung adenocarcinoma (LUAD), lung squamous cell carcinoma (LUSC), and gastric adenocarcinoma (STAD). Each graphical representation 2100 also plots TLS-inducing genes along the Y axis. For example, graphical representation 2100a (Figure 21A) includes TLS features 140 in the proportional area of ​​mature TLS maturation state 312c, graphical representation 2100b (Figure 21B) includes TLS features 140 in the proportional area of ​​immature TLS maturation state 312b, and graphical representation 2100c (Figure 21C) includes TLS features 140 in the proportional area of ​​lymphocyte aggregate TLS maturation state 312c. Figure 2DC shows a graphical representation of plot 2000d, where the X-axis represents a timeline of months and the Y-axis represents the overall survival rate of patients from LUAD cancer samples and BRCA cancer samples. Thus, graphical representation 2100d shows that the proportional areas for different TLS maturation states 312 correlate with a subset of TLS-inducing genes, and in particular, the proportional area for mature TLS maturation state 312c demonstrates prognostic values ​​in LUAD and BRCA samples. Thus, graphical representation 2000c shows that breast cancer samples in clusters containing upregulated chemokines have higher long-term overall survival rates.

[0081] The anti-PD-1 antibodies known in this technology can be used in the configuration and method described herein. Various human monoclonal antibodies that have high affinity and specifically bind to PD-1 are disclosed in Patent Document 1. The anti-PD-1 human antibodies disclosed in Patent Document 1 have been demonstrated to exhibit one or more of the following characteristics: (a) 1 × 10⁻¹⁶ as determined by surface plasmon resonance using a Biacore biosensor system -7 K below M D (b) binding to human PD-1; (c) substantially not binding to human CD28, CTLA-4, or ICOS; (d) increasing T cell proliferation in mixed lymphocyte reaction (MLR) analysis; (e) increasing interferon-γ production in MLR analysis; (f) binding to human PD-1 and cynomolgus monkey PD-1; (g) inhibiting the binding of PD-L1 and / or PD-L2 to PD-1; (h) stimulating antigen-specific memory responses; (i) stimulating antibody responses; and (j) inhibiting tumor cell proliferation in vivo. Anti-PD-1 antibodies available in this disclosure include monoclonal antibodies that specifically bind to human PD-1 and exhibit at least one, and in some embodiments at least five, of the above-described properties.

[0082] Other anti-PD-1 monoclonal antibodies are described, for example, in Patent Documents 2-30, each of which is incorporated by reference.

[0083] In some embodiments, the anti-PD-1 antibody is selected from the following group: Nivolumab (also known as OPDIVO®, 5C4, BMS-936558, MDX-1106, and ONO-4538), Pembrolizumab (Merck; also known as KEYTRUDA®, lambrolizumab, and MK-3475; see Patent Document 8), PDR001 (Novartis; see Patent Document 9), MEDI-0680 (AstraZeneca; also known as AMP-514; see Patent Document 7), Semiprimab (Regeneron; also known as REGN-2810; see Patent Document 10), JS001 (TAIZHOU JUNSHI PHARMA; also known as toripalimab; see Non-Patent Document 4), BGB-A317 (Beigene; also known as Tislelizumab; see Patent Documents 12 and 31), INCSHR1210 (Jiangsu Hengrui Medicine (also known as SHR-1210; see Patent Document 13; Non-Patent Document 4), TSR-042 (Tesaro Biopharmaceutical; also known as ANB011; see Patent Document 14), GLS-010 (Wuxi / Harbin Gloria Pharmaceuticals; also known as WBP3055; see Non-Patent Document 4), AM-0001 (Armo), STI-1110 (Sorrento Therapeutics; see Patent Document 22), AGEN2034 (Agenus; see Patent Document 23), MGA012 (Macrogenics; see Patent Document 29), BCD-100 (Biocad; Non-Patent Document 5), and IBI308 (Innovent; see Patent Documents 26, 27, 30, and 24).

[0084] Nivolumab is a fully human IgG4 (S228P) PD-1 immune checkpoint inhibitor antibody that selectively prevents interaction with PD-1 ligands (PD-L1 and PD-L2), thereby inhibiting the downregulation of antitumor T cell function (Patent Document 1, Non-Patent Document 6). Pembrolizumab is a humanized monoclonal IgG4 (S228P) antibody that targets the human cell surface receptor PD-1 (programmed death-1 or programmed cell death-1). Pembrolizumab is described, for example, in Patent Documents 5 and 32.

[0085] The anti-PD-1 antibodies usable in the disclosed configurations and methods also include isolated antibodies that specifically bind to human PD-1 and that cross-compete with any of the anti-PD-1 antibodies disclosed herein, such as nivolumab, upon binding to human PD-1 (see, for example, Patent Documents 1, 33, and 34). In some embodiments, the anti-PD-1 antibody binds to the same epitope as any of the anti-PD-1 antibodies described herein, such as nivolumab. The ability of antibodies to cross-compete upon binding to an antigen indicates that these monoclonal antibodies bind to the same epitope region of the antigen and sterically interfere with the binding of other cross-competing antibodies to that particular epitope region. These cross-competing antibodies are expected to have functional properties very similar to those of a reference antibody, such as nivolumab, due to their binding to the same epitope region of PD-1. Cross-competing antibodies can be easily identified based on their ability to cross-compete with nivolumab in standard PD-1 binding analyses such as Biacore analysis, ELISA analysis, or flow cytometry (see, for example, Patent Document 34).

[0086] In some embodiments, antibodies that cross-compete with human PD-1 or bind to the same epitope region of human PD-1 antibodies, such as nivolumab, are monoclonal antibodies. For administration to human subjects, these cross-competing antibodies are chimeric antibodies, modified antibodies, or humanized or human antibodies. Such chimeric, modified, humanized, or human monoclonal antibodies can be prepared and isolated by methods well known in this art.

[0087] The anti-PD-1 antibodies usable in the configuration and methods of this disclosure also include the antigen-binding portion of the antibody described above. The antigen-binding function of the antibody has been well demonstrated to be performable by a full-length antibody fragment.

[0088] An anti-PD-1 antibody suitable for use in the configurations and methods disclosed is an antibody that binds to PD-1 with high specificity and affinity, inhibits the binding of PD-L1 or PD-L2, and prohibits the immunosuppressive effect of the PD-1 signaling pathway. In any of the configurations or methods disclosed herein, the anti-PD-1 "antibody" includes an antigen-binding moiety or fragment that binds to the PD-1 receptor and exhibits functional properties similar to those of the whole antibody with respect to inhibiting ligand binding and upregulating the immune system. In certain embodiments, the anti-PD-1 antibody or its antigen-binding moiety cross-competes with nivolumab when binding to human PD-1.

[0089] In some embodiments, the anti-PD-1 antibody is administered once every 2, 3, 4, 5, 6, 7, or 8 weeks in a dose range of 0.1 mg / kg to 20.0 mg / kg per body weight, for example, once every 2, 3, or 4 weeks in a dose range of 0.1 mg / kg to 10.0 mg / kg per body weight. In other embodiments, the anti-PD-1 antibody is administered once every 2 weeks in a dose range of approximately 2 mg / kg, approximately 3 mg / kg, approximately 4 mg / kg, approximately 5 mg / kg, approximately 6 mg / kg, approximately 7 mg / kg, approximately 8 mg / kg, approximately 9 mg / kg, or 10 mg / kg per body weight. In other embodiments, the anti-PD-1 antibody is administered once every three weeks at a dose of approximately 2 mg / kg, 3 mg / kg, 4 mg / kg, 5 mg / kg, 6 mg / kg, 7 mg / kg, 8 mg / kg, 9 mg / kg, or 10 mg / kg per body weight. In one embodiment, the anti-PD-1 antibody is administered once every three weeks at a dose of approximately 5 mg / kg per body weight. In another embodiment, the anti-PD-1 antibody, for example, nivolumab, is administered once every two weeks at a dose of approximately 3 mg / kg per body weight. In yet another embodiment, the anti-PD-1 antibody, for example, pembrolizumab, is administered once every three weeks at a dose of approximately 2 mg / kg per body weight.

[0090] Anti-PD-1 antibodies useful for this disclosure may be administered in uniform doses. In some embodiments, anti-PD-1 antibodies are administered in uniform doses ranging from about 100 to about 1000 mg, about 100 mg to about 900 mg, about 100 mg to about 800 mg, about 100 mg to about 700 mg, about 100 mg to about 600 mg, about 100 mg to about 500 mg, about 200 mg to about 1000 mg, about 200 mg to about 900 mg, about 200 mg to about 800 mg, about 200 mg to about 700 mg, about 200 mg to about 600 mg, about 200 mg to about 500 mg, about 200 mg to about 480 mg, or about 240 mg to about 480 mg. In one embodiment, the anti-PD-1 antibody is administered in a uniform dose of at least about 200 mg, at least about 220 mg, at least about 240 mg, at least about 260 mg, at least about 280 mg, at least about 300 mg, at least about 320 mg, at least about 340 mg, at least about 360 mg, at least about 380 mg, at least about 400 mg, at least about 420 mg, at least about 440 mg, at least about 460 mg, at least about 480 mg, at least about 500 mg, at least about 520 mg, at least about 540 mg, at least about 550 mg, at least about 560 mg, at least about 580 mg, at least about 600 mg, at least about 620 mg, at least about 640 mg, at least about 660 mg, at least about 680 mg, at least about 700 mg, or at least about 720 mg at dosing intervals of about 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10 weeks. In another embodiment, the anti-PD-1 antibody is administered in uniform doses ranging from approximately 200 mg to approximately 800 mg, from approximately 200 mg to approximately 700 mg, from approximately 200 mg to approximately 600 mg, and from approximately 200 mg to approximately 500 mg, with dosing intervals of approximately 1, 2, 3, or 4 weeks.

[0091] In some embodiments, the anti-PD-1 antibody is administered once every three weeks in a uniform dose of approximately 200 mg. In other embodiments, the anti-PD-1 antibody is administered once every two weeks in a uniform dose of approximately 200 mg. In yet another embodiment, the anti-PD-1 antibody is administered once every two weeks in a uniform dose of approximately 240 mg. In a given embodiment, the anti-PD-1 antibody is administered once every four weeks in a uniform dose of approximately 480 mg.

[0092] In some additional embodiments, nivolumab is administered in a uniform dose of approximately 240 mg once every two weeks. In some embodiments, nivolumab is administered in a uniform dose of approximately 240 mg once every three weeks. In some embodiments, nivolumab is administered in a uniform dose of approximately 360 mg once every three weeks. In some embodiments, nivolumab is administered in a uniform dose of approximately 480 mg once every four weeks.

[0093] Alternatively, pembrolizumab is administered in a uniform dose of approximately 200 mg once every two weeks. In some embodiments, pembrolizumab is administered in a uniform dose of approximately 200 mg once every three weeks. In some embodiments, pembrolizumab is administered in a uniform dose of approximately 400 mg once every four weeks.

[0094] In some embodiments, the PD-1 inhibitor is a small molecule. In some embodiments, the PD-1 inhibitor comprises millamolecule. In some embodiments, the PD-1 inhibitor comprises a macrocyclic peptide. The PD-1 inhibitor may also comprise BMS-986189. In some additional embodiments, the PD-1 inhibitor comprises the inhibitor disclosed in Patent Document 35, which is incorporated herein by reference in its entirety. In some embodiments, the PD-1 inhibitor comprises INCMGA00012 (Incyte Corporation). In some embodiments, the PD-1 inhibitor comprises a combination of the anti-PD-1 antibody disclosed herein and a PD-1 small molecule inhibitor.

[0095] In some embodiments, in any of the methods disclosed herein, the anti-PD-1 antibody is replaced with an anti-PD-L1 antibody. Anti-PD-L1 antibodies known in the art are usable in the configurations and methods of the disclosure. Examples of anti-PD-L1 antibodies useful in the configurations and methods of the disclosure include the antibody disclosed in Patent Document 36. The anti-PD-L1 human monoclonal antibody disclosed in Patent Document 36 has been demonstrated to exhibit one or more of the following characteristics: (a) 1 × 10⁻¹⁶ as determined by surface plasmon resonance using a Biacore biosensor system -7 K below M D (b) having the ability to bind to human PD-L1; (c) increasing T cell proliferation in mixed lymphocyte reaction (MLR) analysis; (d) increasing interferon-γ production in MLR analysis; (e) stimulating antibody reactions; and (f) reversing the effect of regulatory T cells on T cell effector cells and / or dendritic cells. Anti-PD-L1 antibodies available in this disclosure include monoclonal antibodies that specifically bind to human PD-L1 and exhibit at least one, and in some embodiments at least five, of the above-described properties.

[0096] The anti-PD-L1 antibody may be selected from the following group: BMS-936559 (also known as 12A4 and MDX-1105; see, for example, Patent Documents 37 and 34), atezolizumab (Roche; also known as TECENTRIQ®; MPDL3280A, RG7446; see Patent Document 38; also see Non-Patent Document 7), durvalumab (AstraZeneca; also known as IMFINZI® and MEDI-4736; see Patent Document 39), avelumab (Pfizer; also known as BAVENCIO® and MSB-0010718C; see Patent Document 40), STI-1014 (Sorrento; see Patent Document 41), CX-072 (Cytomx; see Patent Document 42), KN035 (3D Med / Alphamab; see Non-Patent Document 8), LY3300054 (Eli Lilly Co. (see, for example, Patent Document 43), BGB-A333 (BeiGene; see Non-Patent Document 9), and CK-301 (Checkpoint Therapeutics; see Non-Patent Document 10).

[0097] Atezolizumab is a fully humanized IgG1 monoclonal anti-PD-L1 antibody. Durvalumab is a human IgG1 kappa monoclonal anti-PD-L1 antibody. Avelumab is a human IgG1 lambda monoclonal anti-PD-L1 antibody. The anti-PD-L1 antibodies available in the disclosed configurations and methods also include isolated antibodies that specifically bind to human PD-L1 and cross-compete with any of the anti-PD-L1 antibodies disclosed herein, such as atezolizumab, durvalumab, and / or avelumab, upon binding to human PD-L1. In some embodiments, the anti-PD-L1 antibody binds to the same epitope as any of the anti-PD-L1 antibodies described herein, such as atezolizumab, durvalumab, and / or avelumab. The ability of antibodies to cross-compete upon binding to an antigen indicates that these antibodies bind to the same epitope region of the antigen and sterically interfere with the binding of other cross-competing antibodies to that particular epitope region. These cross-competing antibodies are expected to have very similar functional properties to those of a reference antibody, such as atezolizumab and / or avelumab, due to their binding to the same epitope region of PD-L1. Cross-competing antibodies can be readily identified based on their ability to cross-compete with atezolizumab and / or avelumab in standard PD-L1 binding analyses such as Biacore analysis, ELISA analysis, or flow cytometry (see, for example, Patent Document 34).

[0098] Antibodies that cross-compete with human PD-L1 or bind to the same epitope region of human PD-L1 antibodies, such as atezolizumab, durvalumab, and / or avelumab, are monoclonal antibodies. For administration to human subjects, these cross-competing antibodies are chimeric antibodies, modified antibodies, or humanized or human antibodies. Such chimeric, modified, humanized, or human monoclonal antibodies can be prepared and isolated by methods well known in this technology.

[0099] The anti-PD-L1 antibodies usable in the configuration and methods of this disclosure also include the antigen-binding moiety of the antibody described above. The antigen-binding function of the antibody has been well demonstrated to be performable by a full-length antibody fragment.

[0100] An anti-PD-L1 antibody suitable for use in the configurations and methods disclosed is an antibody that binds to PD-L1 with high specificity and affinity, inhibits PD-1 binding, and prohibits the immunosuppressive effect of the PD-1 signaling pathway. In any of the configurations or methods disclosed herein, the anti-PD-L1 "antibody" includes an antigen-binding moiety or fragment that binds to PD-L1 and exhibits functional properties similar to those of the whole antibody with respect to inhibiting receptor binding and upregulating the immune system. In certain embodiments, the anti-PD-L1 antibody or its antigen-binding moiety cross-competes with atezolizumab, durvalumab, and / or avelumab when binding to human PD-L1.

[0101] The anti-PD-L1 antibody useful for this disclosure may be any PD-L1 antibody that specifically binds to PD-L1, for example, an antibody that cross-competes with durvalumab, avelumab, or atezolizumab when binding to human PD-1, for example, an antibody that binds to the same epitope as durvalumab, avelumab, or atezolizumab. In certain embodiments, the anti-PD-L1 antibody is durvalumab. In other embodiments, the anti-PD-L1 antibody is avelumab. In some embodiments, the anti-PD-L1 antibody is atezolizumab.

[0102] In some examples, anti-PD-L1 antibodies are administered once every 2, 3, 4, 5, 6, 7, or 8 weeks in doses ranging from approximately 0.1 mg / kg to approximately 20.0 mg / kg, specifically approximately 2 mg / kg, 3 mg / kg, 4 mg / kg, 5 mg / kg, 6 mg / kg, 7 mg / kg, 8 mg / kg, 9 mg / kg, 10 mg / kg, 11 mg / kg, 12 mg / kg, 13 mg / kg, 14 mg / kg, 15 mg / kg, 16 mg / kg, 17 mg / kg, 18 mg / kg, 19 mg / kg, or 20 mg / kg per body weight.

[0103] The anti-PD-L1 antibody may be administered once every three weeks at a dose of approximately 15 mg / kg of body weight. In another embodiment, the anti-PD-L1 antibody is administered once every two weeks at a dose of approximately 10 mg / kg of body weight.

[0104] In some scenarios, the anti-PD-L1 antibody useful for this disclosure is a uniform dose. In some embodiments, the anti-PD-L1 antibody is administered in a uniform dose ranging from about 200 mg to about 1600 mg, about 200 mg to about 1500 mg, about 200 mg to about 1400 mg, about 200 mg to about 1300 mg, about 200 mg to about 1200 mg, about 200 mg to about 1100 mg, about 200 mg to about 1000 mg, about 200 mg to about 900 mg, about 200 mg to about 800 mg, about 200 mg to about 700 mg, about 200 mg to about 600 mg, about 700 mg to about 1300 mg, about 800 mg to about 1200 mg, about 700 mg to about 900 mg, or about 1100 mg to about 1300 mg. In some embodiments, the anti-PD-L1 antibody is administered in a uniform dose of at least about 240 mg, at least about 300 mg, at least about 320 mg, at least about 400 mg, at least about 480 mg, at least about 500 mg, at least about 560 mg, at least about 600 mg, at least about 640 mg, at least about 700 mg, at least about 720 mg, at least about 800 mg, at least about 840 mg, at least about 880 mg, at least about 900 mg, at least 960 mg, at least about 1000 mg, at least about 1040 mg, at least about 1100 mg, at least about 1120 mg, at least about 1200 mg, at least about 1280 mg, at least about 1300 mg, at least about 1360 mg, or at least about 1400 mg, with dosing intervals of about 1, 2, 3, or 4 weeks. In some embodiments, the anti-PD-L1 antibody is administered once every three weeks in a uniform dose of approximately 1200 mg. In other embodiments, the anti-PD-L1 antibody is administered once every two weeks in a uniform dose of approximately 800 mg. In yet another embodiment, the anti-PD-L1 antibody is administered once every two weeks in a uniform dose of approximately 840 mg.

[0105] Atezolizumab is administered once every three weeks in a uniform dose of approximately 1200 mg. In some cases, atezolizumab is administered once every two weeks in a uniform dose of approximately 800 mg. In other cases, atezolizumab is administered once every two weeks in a uniform dose of approximately 840 mg. Optionally, avelumab may be administered once every two weeks in a uniform dose of approximately 800 mg.

[0106] In some embodiments, durvalumab is administered at a dose of approximately 10 mg / kg once every two weeks. In other embodiments, durvalumab is administered at a uniform dose of approximately 800 mg / kg once every two weeks. Durvalumab may optionally be administered at a uniform dose of approximately 1200 mg / kg once every three weeks.

[0107] The PD-L1 inhibitor may contain small molecules or miramolecules. The PD-L1 inhibitor may contain macrocyclic peptides. In some examples, the PD-L1 inhibitor contains BMS-986189. The PD-L1 inhibitor may contain miramolecules having the following chemical formula.

[0108] [ka]

[0109] Here, R1~R13 are amino acid side chains, and R a ~R n R14 is a hydrogen, methyl, or a ring with an adjacent R group, where R14 is -C(O)NHR15 and R15 is a hydrogen or glycine residue, the glycine residue may optionally be substituted with an additional glycine residue and / or terminally, which may improve pharmacokinetic properties. In some embodiments, the PD-L1 inhibitor includes the compounds disclosed in Patent Document 35, the whole of which is incorporated herein by reference. In some embodiments, the PD-L1 inhibitor includes the compounds disclosed in Patent Documents 44-55, the whole of each of which is incorporated herein by reference.

[0110] PD-L1 inhibitors include small molecule PD-L1 inhibitors disclosed in Patent Documents 56-64, each of which is incorporated herein by reference. In some embodiments, the PD-L1 inhibitor includes a combination of an anti-PD-L1 antibody disclosed herein and a small molecule PD-L1 inhibitor disclosed herein.

[0111] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some embodiments, a software application may be referred to as an “application,” “app,” or “program.” Exemplary applications include, but are not limited to, system diagnostic applications, system administration applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0112] Non-temporary memory may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by a computing device. Non-temporary memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and ROM (read-only memory) / PROM (programmable read-only memory) / EPROM (erasable programmable read-only memory) / EEPROM (electronically erasable programmable read-only memory) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, RAM (random access memory), DRAM (dynamic random access memory), SRAM (static random access memory), PCM (phase change memory), and disk or tape.

[0113] Figure 22 is a schematic diagram of an exemplary computing device 2200 that may be used to carry out the system and method described herein. The computing device 2200 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are intended to be illustrative only and are not intended to limit the embodiments of the invention described herein and / or claimed herein.

[0114] The computing device 2200 includes a processor 2210, memory 2220, storage device 2230, a high-speed interface / controller 2240 connected to memory 2220 and high-speed expansion port 2250, and a low-speed bus 2270 and a low-speed interface / controller 2260 connected to storage device 2230. Each of the components 2210, 2220, 2230, 2240, 2250, and 2260 is interconnected using various buses and may be implemented on a common motherboard or in other ways as appropriate. The processor 2210 may process instructions to be executed within the computing device 2200, including instructions stored in memory 2220 or storage device 2230 for displaying graphical information for a GUI (graphical user interface) on an external input / output device such as a display 2280 connected to the high-speed interface 2240. In other embodiments, multiple processors and / or multiple buses may be used together with multiple memories and multiple types of memories as appropriate. Furthermore, multiple computing devices 2200 may be connected (for example, as a server bank, a group of blade servers, or a multiprocessor system) so that each device provides the necessary part of the operation.

[0115] Memory 2220 stores information non-temporarily within the computing device 2200. Memory 2220 may be a computer-readable medium, one or more volatile memory devices, or one or more non-volatile memory devices. Non-temporarily stored memory 2220 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 2200. Examples of non-volatile memory include, but are not limited to, flash memory and ROM (read-only memory) / PROM (programmable read-only memory) / EPROM (erasable programmable read-only memory) / EEPROM (electronically erasable programmable read-only memory) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, RAM (random access memory), DRAM (dynamic random access memory), SRAM (static random access memory), PCM (phase change memory), and disk or tape.

[0116] The storage device 2230 may have the capability to function as mass storage for the computing device 2200. In some embodiments, the storage device 2230 is a computer-readable medium. In various different embodiments, the storage device 2230 may be an array of devices including a floppy disk drive, a hard disk drive, an optical disk drive, or a tape drive, flash memory or other similar solid-state memory devices, or a storage area network or other configuration of devices. In additional embodiments, a computer program product is tangibly embodied as an information carrier. The computer program product includes instructions that, when executed, perform one or more of the methods described above. The information carrier is a computer or machine-readable medium, such as memory 2220, storage device 2230, or memory on the processor 2210.

[0117] The high-speed controller 2240 manages bandwidth-intensive operations for the computing unit 2200, while the low-speed controller 2260 manages less bandwidth-intensive operations. Such duty cycle assignments are merely illustrative. In some embodiments, the high-speed controller 2240 is connected to memory 2220, display 2280, and (e.g., via a graphics processor or accelerator) to a high-speed expansion port 2250. The high-speed expansion port 2250 may accept various expansion cards (not shown). In some embodiments, the low-speed controller 2260 is connected to storage device 2230 and a low-speed expansion port 2290. The low-speed expansion port 2290 may include various communication ports (e.g., USB, Bluetooth®, Ethernet®, wireless Ethernet) and may also be connected to one or more input / output devices such as a keyboard, pointing device, scanner, or networking device such as a switch or router via a network adapter, for example.

[0118] The computing device 2200 may be implemented in a number of different forms, as shown in the drawings. For example, it may be implemented as a standard server 2200a or multiples of a group of such servers 2200a, as part of a laptop computer 2200b, or as part of a rack server system 2200c.

[0119] Various implementations of the systems and technologies described herein may be realized by digital electronic and / or optical circuits, integrated circuits, particularly designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations of one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be application-specific or general-purpose, connected to a storage system, at least one input device, and at least one output device to receive and transmit data and instructions to them.

[0120] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” mean any computer program product, non-temporary computer-readable medium, device, and / or equipment (including, for example, magnetic disks, optical disks, memory, and PLDs (Programmable Logic Devices)) used to provide machine instructions and / or data to a programmable processor, and include machine-readable medium that receives machine instructions as machine-readable signals. The term “machine-readable signal” means any signal used to provide machine instructions and / or data to a programmable processor.

[0121] The processing and logic flows described herein may be executed by one or more programmable processors, also called data processing hardware, which execute one or more computer programs to perform functions by acting on input data and producing outputs. Processing and logic flows may also be executed by purpose-specific logic circuits, e.g., FPGAs (field programmable gate arrays), ASICs (application-specific integrated circuits). Processors suitable for executing computer programs include, as examples, both general-purpose and purpose-specific microprocessors, and any one or more processors of any type of digital computer. Generally, processors receive instructions and data from read-only memory or random-access memory or both. Essential components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks, or are functionally connected to receive or transfer data to each other, or both. However, a computer may not have such devices. Computer-readable media suitable for storing computer program instructions and data include, as examples, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices such as EPROMs, EEPROMs, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROMs and DVD-ROM disks. Processors and memory may be supplemented with or incorporated into logic circuits for specific purposes.

[0122] To provide interactive scanning with a user, one or more aspects of the present disclosure may be implemented in a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen, and optionally a keyboard and pointing device, such as a mouse or trackball, which the user can use to provide input to the computer. Similarly, other types of devices may be used to provide interactive scanning with a user, for example, the feedback provided to the user may be any form of perceptual feedback, such as visual feedback, auditory feedback, or tactile feedback, and the input from the user may be received in any form, including voice, speech, or tactile input. Furthermore, the computer may interact with the user by sending and receiving documents to and from devices used by the user, for example, by sending a web page to a web browser in response to a request received from a web browser on the user's client device.

[0123] Numerous embodiments have been described. Nevertheless, it will be understood that various modifications may be made without deviating from the spirit and scope of this disclosure. Accordingly, other embodiments may be included in the appended claims.

Claims

1. A computer-implemented method (400) for causing data processing hardware (142) to perform a predetermined operation when executed on the data processing hardware (142), The above operation is, The system receives an input tissue structure image (110) relating to a patient diagnosed with cancer, which includes a plurality of image pixels. By processing the input tissue structure image (110) using a cell classification model (550), one or more lymphocyte density maps (125) are generated within the input tissue structure image (110). By performing morphological image processing (130) on one or more lymphocyte density maps (125) as described above, one or more TLS regions (135) within the input tissue structure image (110) are identified, and each TLS region (135) is represented by a cluster of lymphocyte cells. With respect to each corresponding TLS region (135) of the one or more TLS regions (135) identified in the above input tissue structure image (110), From each cluster of lymphocyte cells representing the corresponding TLS region (135) described above, each set of TLS features (140) is extracted, By processing each set of the above TLS features (140) using the TLS classification model (350), the corresponding TLS region (135) is classified as one of the first TLS maturity state (312), the second TLS maturity state (312), and the third TLS maturity state (312). This includes determining an overall TLS score (152) for the input tissue structure image (110) based on the TLS maturation state (312) for one or more TLS regions (135) identified in the input tissue structure image and the TLS features (140) extracted from the one or more TLS regions (135) identified in the input tissue structure image. Computer-implemented methods (400).

2. The above operation further includes identifying tumor regions (115) within the input tissue structure image (110) by processing the input tissue structure image (110) using a tumor detection model (450), Generating one or more lymphocyte density maps (125) by processing the above input tissue structure image (110) includes generating one or more lymphocyte density maps (125) by processing the above input tissue structure image (110) using the above cell classification model (550) by performing single-cell imaging analysis on the tumor region (115) identified within the above input tissue structure image (110), The computer-implemented method (400) according to claim 1.

3. The above tumor detection model (450) is, Obtaining multiple rasterized image tiles (370) from a collection of histopathological images of the entire slide, each of which is manually annotated as containing tumor or non-tumor, This is learned by training the tumor detection model (450) with the multiple image tiles (370) using a neural network (374) so ​​that the tumor detection model (450) learns how to identify tumor regions (115) within tissue structure images. The computer-implemented method (400) according to claim 2.

4. The above cell classification model (550) is, The process involves obtaining multiple image patches (382), each comprising a corresponding set of human cells and manual annotations labeling each human cell as a tumor cell, lymphocyte, or non-malignant cell. This is learned by training the cell classification model (550) with the multiple image patches (382) using a neural network (386) so that the cell classification model (550) learns how to classify individual cells in tissue structure images as tumor cells, lymphocytes, or non-malignant cells. The computer-implemented method (400) according to claim 1.

5. The above TLS classification model (350) is, The method involves obtaining a training dataset (305) that includes multiple training tissue structure images (310), each containing a tumor microenvironment and each having manual annotations (312), wherein the manual annotations (312) are One or more TLS regions (135) in the above-mentioned learning tissue structure image (310), wherein one or more TLS regions (135) are represented by each cluster of lymphocyte cells, With respect to each corresponding TLS region (135), a ground truth TLS maturity state (312) is shown to indicate that the corresponding TLS region (135) has a first TLS maturity state (312), a second TLS maturity state (312), or a third TLS maturity state (312). To identify the above training dataset (305), With respect to each TLS region (135), each set of TLS features (140) is extracted from each cluster of lymphocyte cells representing the TLS region (135), The TLS classification model (350) is trained by having it learn each set of training TLS features (140) extracted for each TLS region (135), instructing the TLS classification model (350) to learn how to predict the ground truth TLS grade for each corresponding TLS region (135). The computer-implemented method (400) according to claim 1.

6. Training the above TLS classification model (350) includes training the above TLS classification model (350) using the CART (classification and regression trees) algorithm. The computer-implemented method (400) according to claim 5.

7. The first TLS maturation state (312) described above comprises a dense aggregate of at least a threshold number of lymphocytes, which does not include high endothelial venules or germinal centers (313). The second TLS maturation state (312) described above comprises an immature TLS having dense aggregates of at least the threshold number of lymphocytes, which include high endothelial venules and do not include germinal centers (313). The third TLS maturation state (312) described above comprises a mature TLS having dense aggregates of at least the threshold number of lymphocytes, including high endothelial venules and germinal centers (313). The computer-implemented method (400) according to claim 1.

8. The above operation is, With respect to each corresponding TLS region (135) of the one or more TLS regions (135) identified in the above input tissue structure image (110), a pixel mask (112) is generated that emphasizes at least the area surrounding the corresponding TLS region (135). By superimposing the respective pixel masks (112) generated for each of the above TLS regions (135) onto the input tissue structure image (110), an output image (110A) that enhances the input tissue structure image (110) is generated. The further includes providing the output image (110A) for display on a screen that communicates with the data processing hardware (142), The computer-implemented method (400) according to claim 1.

9. Each pixel mask (112) generated with respect to each corresponding TLS region (135) classified as the first TLS maturity state comprises the first pixel mask (112), Each pixel mask (112) generated for each corresponding TLS region (135) classified as the second TLS maturity state comprises a second pixel mask (112) that is visually distinguishable from the first pixel mask (112). Each pixel mask (112) generated for each corresponding TLS region (135) classified as the third TLS maturity state comprises a third pixel mask (112) that is visually distinguishable from the first pixel mask (112) and the second pixel mask (112). The computer-implemented method (400) according to claim 8.

10. Each set of TLS features (140) extracted from each cluster of lymphocyte cells comprises the area of ​​the corresponding TLS region (135), the roundness of the corresponding TLS region (135), and the distortion of the corresponding TLS region (135). The computer-implemented method (400) according to claim 1.

11. The above operation further includes determining a treatment recommendation (192) to treat the patient using immunotherapy, based on the overall TLS score (152). The computer-implemented method (400) according to claim 1.

12. The above immunotherapy comprises at least one of a PD-1 inhibitor and a PD-L1 inhibitor. The computer-implemented method (400) according to claim 11.

13. The above operation further includes determining a predictive score for the patient's response to immunotherapy based on the TLS maturation state (312) of one or more TLS regions (135) identified in the input tissue structure image and the TLS features (140) extracted from the one or more TLS regions (135) identified in the input tissue structure image. The computer-implemented method (400) according to claim 1.

14. Data processing hardware (142) and The memory hardware (144) communicates with the above data processing hardware (142) and A system (100) comprising, The memory hardware (144) stores instructions that cause the data processing hardware (142) to perform a predetermined operation when executed on the data processing hardware (142). The above operation is, The system receives an input tissue structure image (110) relating to a patient diagnosed with cancer, which includes a plurality of image pixels. By processing the input tissue structure image (110) using a cell classification model (550), one or more lymphocyte density maps (125) are generated within the input tissue structure image (110). By performing morphological image processing (130) on one or more lymphocyte density maps (125) as described above, one or more TLS regions (135) within the input tissue structure image (110) are identified, and each TLS region (135) is represented by a cluster of lymphocyte cells. With respect to each corresponding TLS region (135) of the one or more TLS regions (135) identified in the above input tissue structure image (110), From each cluster of lymphocyte cells representing the corresponding TLS region (135) described above, each set of TLS features (140) is extracted, By processing each set of the above TLS features (140) using the TLS classification model (350), the corresponding TLS region (135) is classified as one of the first TLS maturity state (312), the second TLS maturity state (312), and the third TLS maturity state (312). This includes determining an overall TLS score (152) for the input tissue structure image (110) based on the TLS maturation state (312) for one or more TLS regions (135) identified in the input tissue structure image and the TLS features (140) extracted from the one or more TLS regions (135) identified in the input tissue structure image. System (100).

15. The above operation further includes identifying tumor regions (115) within the input tissue structure image (110) by processing the input tissue structure image (110) using a tumor detection model (450), Generating one or more lymphocyte density maps (125) by processing the above input tissue structure image (110) includes generating one or more lymphocyte density maps (125) by processing the above input tissue structure image (110) using the above cell classification model (550) by performing single-cell imaging analysis on the tumor region (115) identified within the above input tissue structure image (110), The system (100) according to claim 14.

16. The above tumor detection model (450) is, Obtaining multiple rasterized image tiles (370) from a collection of histopathological images of the entire slide, each of which is manually annotated as containing tumor or non-tumor, This is learned by training the tumor detection model (450) with the multiple image tiles (370) using a neural network (374) so ​​that the tumor detection model (450) learns how to identify tumor regions (115) within tissue structure images. The system (100) according to claim 15.

17. The above cell classification model (550) is, The process involves obtaining multiple image patches (382), each comprising a corresponding set of human cells and manual annotations labeling each human cell as a tumor cell, lymphocyte, or non-malignant cell. This is learned by training the cell classification model (550) with the multiple image patches (382) using a neural network (386) so that the cell classification model (550) learns how to classify individual cells in tissue structure images as tumor cells, lymphocytes, or non-malignant cells. The system (100) according to claim 14.

18. The above TLS classification model (350) is, The method involves obtaining a training dataset (305) that includes multiple training tissue structure images (310), each containing a tumor microenvironment and each having manual annotations (312), wherein the manual annotations (312) are One or more TLS regions (135) in the above-mentioned learning tissue structure image (310), wherein one or more TLS regions (135) are represented by each cluster of lymphocyte cells, With respect to each corresponding TLS region (135), a ground truth TLS maturity state (312) is shown to indicate that the corresponding TLS region (135) has a first TLS maturity state (312), a second TLS maturity state (312), or a third TLS maturity state (312). To identify the above training dataset (305), With respect to each TLS region (135), each set of TLS features (140) is extracted from each cluster of lymphocyte cells representing the TLS region (135), The TLS classification model (350) is trained by having it learn each set of training TLS features (140) extracted for each TLS region (135), instructing the TLS classification model (350) to learn how to predict the ground truth TLS grade for each corresponding TLS region (135). The system (100) according to claim 14.

19. Training the above TLS classification model (350) includes training the above TLS classification model (350) using the CART (classification and regression trees) algorithm. The system (100) according to claim 18.

20. The first TLS maturation state (312) described above comprises a dense aggregate of at least a threshold number of lymphocytes, which does not include high endothelial venules or germinal centers (313). The second TLS maturation state (312) described above comprises an immature TLS having dense aggregates of at least the threshold number of lymphocytes, which include high endothelial venules and do not include germinal centers (313). The third TLS maturation state (312) described above comprises a mature TLS having dense aggregates of at least the threshold number of lymphocytes, including high endothelial venules and germinal centers (313). The system (100) according to claim 14.

21. The above operation is, With respect to each corresponding TLS region (135) of the one or more TLS regions (135) identified in the above input tissue structure image (110), a pixel mask (112) is generated that emphasizes at least the area surrounding the corresponding TLS region (135). By superimposing the respective pixel masks (112) generated for each of the above TLS regions (135) onto the input tissue structure image (110), an output image (110A) that enhances the input tissue structure image (110) is generated. The further includes providing the output image (110A) for display on a screen that communicates with the data processing hardware (142), The system (100) according to claim 14.

22. Each pixel mask (112) generated with respect to each corresponding TLS region (135) classified as the first TLS maturity state comprises the first pixel mask (112), Each pixel mask (112) generated for each corresponding TLS region (135) classified as the second TLS maturity state comprises a second pixel mask (112) that is visually distinguishable from the first pixel mask (112). Each pixel mask (112) generated for each corresponding TLS region (135) classified as the third TLS maturity state comprises a third pixel mask (112) that is visually distinguishable from the first pixel mask (112) and the second pixel mask (112). The system (100) according to claim 21.

23. Each set of TLS features (140) extracted from each cluster of lymphocyte cells comprises the area of ​​the corresponding TLS region (135), the roundness of the corresponding TLS region (135), and the distortion of the corresponding TLS region (135). The system (100) according to claim 14.

24. The above operation further includes determining a treatment recommendation (192) to treat the patient using immunotherapy, based on the overall TLS score (152). The system (100) according to claim 14.

25. The above immunotherapy comprises at least one of a PD-1 inhibitor and a PD-L1 inhibitor. The system (100) according to claim 24.

26. The above operation further includes determining a predictive score for the patient's response to immunotherapy based on the TLS maturation state (312) of one or more TLS regions (135) identified in the input tissue structure image and the TLS features (140) extracted from the one or more TLS regions (135) identified in the input tissue structure image. The system (100) according to claim 14.

Images