Systems and methods for detecting tertiary lymphoid structures - Patents.com

Abstract

Computer-implemented methods, machine learning models, and machine-readable media are provided for detecting tertiary lymphoid structures (TLS) and / or the possible presence or absence of TLS in a tissue image and / or a subject.

Description

[Technical Field]

[0001] The present invention relates generally to machine learning and computer vision, and more particularly to image preprocessing and classification. [Background technology]

[0002] Histopathological image analysis (HIA) is an important component of diagnosis in many medical fields, including oncology. Tertiary lymphoid structures (TLS) are ectopic lymphoid structures that develop at sites of chronic inflammation, including tumors. TLS exist in tumors at various stages of maturation, ultimately leading to germinal center formation. The presence of mature TLS in tumor compartments, characterized by mature follicles containing germinal centers, is associated with improved survival in patients with solid tumors treated with cancer immunotherapy. Therefore, TLS has the potential to be used as a predictor of patient susceptibility to immune checkpoint inhibitors. However, pathological assessment of TLS status remains time-consuming and typically requires additional analysis, including immunohistochemical staining. Therefore, there is a need for methods and systems for accurate and efficient detection of TLS in tissue images and subjects. Summary of the Invention

[0003] A method and apparatus for a device for classifying images is described.

[0004] In one aspect, disclosed herein is a computer-implemented method for detecting the presence or absence of tertiary lymphoid structures (TLS) in a subject, comprising: receiving a digitized tissue image of a sample obtained from the subject; tiling the tissue image into a set of tiles; extracting a plurality of feature vectors from each of the tiles, each feature of the one or more feature vectors representing a local descriptor of the tile; classifying the tissue image for TLS status using at least the plurality of feature vectors and a classification model trained on a training set of tissue images with known TLS annotations, wherein the TLS status indicates the presence or absence of TLS in the subject.

[0005] In some embodiments, the classifying step comprises: applying a first neural network to the one or more feature vectors of the plurality of feature vectors, the first neural network assigning a tile score to each tile of the set of tiles based on the one or more feature vectors of the plurality of feature vectors, the tile score representing the likelihood that the tile contains a TLS; applying a second neural network to each tile, the second neural network aggregating a subset of the tile scores for the set of tiles to determine a TLS state in the tissue image; Including, the first neural network is trained using a training set of tissue images containing known local annotations of the presence or absence of TLS at the tile level; The first and second neural networks are trained using a training set of tissue images that contain known global annotations regarding the presence or absence of TLS at the tissue image level.

[0006] In some embodiments, the first neural network comprises a 1D convolutional layer. In some embodiments, the first neural network uses MoCo features as input and outputs a likelihood that the tile contains a TLS. In some embodiments, the second neural network comprises a multi-layer perceptual model.

[0007] In some embodiments, the computer-implemented methods provided herein include detecting one or more locations in the tissue image where TLS is present.

[0008] In some embodiments of the computer-implemented method provided herein, each tile comprises a plurality of pixels. In some embodiments, the method further comprises: receiving a digitized tissue image of a sample obtained from the subject; tiling the tissue image into a set of tiles; extracting a segmentation mask from each tile; applying a third neural network to each tile, the third neural network using the extracted segmentation mask to find pixel scores; Detecting pixel-based TLS segmentation within a tile or image, wherein the third neural network is trained using a training set of tiles with known pixel-based TLS segmentation masks within the tiles.

[0009] In some embodiments, the third neural network assigns a pixel score to each pixel of each tile in the tile set to determine the pixel-based segmentation of TLS within the tile, the pixel score representing the likelihood that the pixel contains TLS. In some embodiments, the third neural network is a U-NET semantic segmentation neural network.

[0010] In some embodiments, the extracting is performed by a ResNet50 neural network. In some embodiments, the extracting of the plurality of feature vectors comprises a Momentum Contrast (MoCo) or Momentum Contrast v2 (MoCo v2) algorithm.

[0011] In some embodiments of the computer-implemented methods provided herein, the sample is a cancer, in some embodiments, the cancer is selected from the group consisting of lung cancer, sarcoma, bladder cancer, colorectal cancer, ovarian cancer, pancreatic cancer, and melanoma, kidney cancer, head and neck cancer, liver cancer, breast cancer, gastrointestinal stromal tumor cancer, cervical cancer, endometrial cancer, gastric cancer, thyroid cancer, bile duct cancer, prostate cancer, anal cancer, vulvar cancer, skin cancer, parotid gland cancer, gastrointestinal cancer, penile cancer, esophageal cancer, and cancer of unknown primary origin.

[0012] In some embodiments of the computer-implemented methods provided herein, the training set of tiles and / or the training set of tissue images are digitized images of tissue sections of heterogeneous cancers (e.g., lung cancer, sarcoma, bladder cancer, colon cancer, ovarian cancer, pancreatic cancer, melanoma).

[0013] In some embodiments, the tissue image is a digitized whole slide image (WSI). In some embodiments, the digitized tissue image is a digitized image of a tissue section stained with a dye. In some embodiments, the dye is hemotoxylin and eosin (H&E).

[0014] In some embodiments, the classifying further comprises: picking the tile with the highest TLS tile score; and picking the tile with the lowest TLS tile score; sorting the set of tiles by

[0015] In some embodiments, the computer-implemented method provided herein comprises: receiving a digitized tissue image of a sample obtained from the subject; tiling the tissue image into a set of tiles; extracting a plurality of feature vectors from each of the tiles, each feature of the one or more feature vectors representing a local descriptor of the tile; and classifying the tissue images for TLS status using at least the plurality of feature vectors and a classification model trained on an image training set having known TLS annotations for each tissue image in the plurality of tissue images, the TLS status indicating the presence or absence of TLS in the subject; Repeating all steps of detecting the presence or absence of TLS in the subject by processing the TLS status of the plurality of tissue images; Includes.

[0016] In some embodiments, the tissue image lacks local annotation of histopathological features. In some embodiments, each tile in the set of tiles comprises approximately 224x224 pixels.

[0017] In one aspect, disclosed herein is a machine-readable medium having executable instructions that cause one or more processing units to perform a method of detecting the presence or absence of tertiary lymphoid structures (TLS) in a subject, the method comprising: receiving a digitized tissue image of a sample obtained from the subject; tiling the tissue image into a set of tiles; extracting a plurality of feature vectors from each of the tiles, each feature of the one or more feature vectors representing a local descriptor of the tile; and classifying the tissue image for TLS status using a classification model trained on at least the plurality of feature vectors and a training set of tissue images with known TLS annotations, the TLS status indicating the presence or absence of TLS in the subject.

[0018] In some embodiments, the classifying step comprises: applying a first neural network to the one or more feature vectors of the plurality of feature vectors, the first neural network assigning a tile score to each tile of the set of tiles based on the one or more feature vectors of the plurality of feature vectors, the tile score representing a likelihood that the tile comprises a TLS; applying a second neural network to each tile, the second neural network aggregating a subset of the tile scores for the set of tiles to determine a TLS state in the tissue image; Including, the first neural network is trained using a training set of tissue images with known local annotations of the presence or absence of TLS at the tile level; The first and second neural networks are trained using a training set of tissue images with known global annotations of the presence or absence of TLS at the tissue image level.

[0019] In some embodiments, the first neural network comprises a 1D convolutional layer. In some embodiments, the first neural network uses MoCo features as input and outputs a likelihood that the tile comprises a TLS. In some embodiments, the second neural network comprises a multi-layer perceptual model.

[0020] In some embodiments, a method performed by a machine-readable medium provided herein includes detecting one or more locations in the tissue image where TLS is present.

[0021] In some embodiments of the method performed by the machine-readable medium provided herein, each tile comprises a plurality of pixels. In some embodiments, the method further comprises: receiving a digitized tissue image of a sample obtained from the subject; tiling the tissue image into a set of tiles; extracting a segmentation mask from each tile; applying a third neural network to each tile, the third neural network using the extracted segmentation mask to find pixel scores; Detecting pixel-based TLS segmentation within a tile or image, wherein the third neural network is trained using a training set of tiles with known pixel-based TLS segmentation masks within the tiles; Includes.

[0022] In some embodiments, the third neural network assigns a pixel score to each pixel of each tile in the tile set to determine the pixel-based segmentation of TLS within the tile, the pixel score representing the likelihood that the pixel contains TLS. In some embodiments, the third neural network is a U-NET semantic segmentation neural network.

[0023] In some embodiments, the extracting step is performed by a ResNet50 neural network. In some embodiments, the extracting of the plurality of feature vectors comprises a Momentum Contrast (MoCo) or Momentum Contrast v2 (MoCo v2) algorithm.

[0024] In some embodiments, the extraction of the plurality of feature vectors comprises a Momentum Contrast (MoCo) or Momentum Contrast v2 (MoCo v2) algorithm.

[0025] In some embodiments of the methods performed by the machine-readable media provided herein, the sample is cancer, hi some embodiments, the cancer is selected from the group consisting of lung cancer, sarcoma, bladder cancer, colorectal cancer, ovarian cancer, pancreatic cancer, melanoma, kidney cancer, head and neck cancer, liver cancer, breast cancer, gastrointestinal stromal tumor cancer, cervical cancer, endometrial cancer, stomach cancer, thyroid cancer, bile duct cancer, prostate cancer, anal cancer, vulvar cancer, skin cancer, parotid gland cancer, gastrointestinal cancer, penile cancer, esophageal cancer, and cancer of unknown primary origin.

[0026] In some embodiments of the methods performed by the machine-readable media provided herein, the training set of tiles and / or the training set of tissue images are digitized images of tissue sections of heterogeneous cancers (e.g., lung cancer, sarcoma, bladder cancer, colon cancer, ovarian cancer, pancreatic cancer, melanoma).

[0027] In some embodiments, the tissue image is a digitized whole slide image (WSI). In some embodiments, the digitized tissue image is a digitized image of a tissue section stained with a dye. In some embodiments, the dye is hemotoxylin and eosin (H&E).

[0028] In some embodiments, the classifying further comprises: picking the tile with the highest TLS tile score; and picking the tile with the lowest TLS tile score; sorting the set of tiles by

[0029] In some embodiments, the method performed by the machine-readable medium provided herein comprises: receiving a digitized tissue image of a sample obtained from the subject; tiling the tissue image into a set of tiles; extracting a plurality of feature vectors from each of the tiles, each feature of the one or more feature vectors representing a local descriptor of the tile; and classifying the tissue images for TLS status using at least the plurality of feature vectors and a classification model trained on an image training set having known TLS annotations for each tissue image in the plurality of tissue images, the TLS status indicating the presence or absence of TLS in the subject; Repeating all steps of detecting the presence or absence of TLS in the subject by processing the TLS status of the plurality of tissue images; Includes.

[0030] In some embodiments, the tissue images lack local annotation of histopathological features. In some embodiments, each of the set of tiles comprises approximately 224x224 pixels.

[0031] The present invention is illustrated by way of example, and not limitation, in the figures of the accompanying drawings in which like reference numerals indicate like elements and in which: [Brief explanation of the drawings]

[0032] [Figure 1] FIG. 10 is an example flow diagram of a process for detecting TLS status in an image and / or the presence or absence of TLS in a subject using a machine learning model, according to an embodiment of the present disclosure.

[0033] [Figure 2] FIG. 10 is an exemplary flow diagram of a process for classifying an image for TLS status, optionally predicting TLS location within an image, and / or optionally predicting pixel-based TLS segmentation, according to an embodiment of the present disclosure.

[0034] [Figure 3] FIG. 1 shows an exemplary flow diagram for a process of training and validating a machine learning model for detecting TLS status in an image and / or the presence or absence of TLS in a subject, according to an embodiment of the present disclosure.

[0035] [Figure 4] 1 shows a receiver operating characteristic (ROC) curve for predicting the presence or absence of TLS in a subject in a cross-validation of a machine learning model, according to an embodiment of the present disclosure.

[0036] [Figure 5] 1 depicts a ROC curve valid for predicting the presence or absence of TLS in a validation cohort of subjects using a machine learning model according to an embodiment of the present disclosure.

[0037] [Figure 6A] FIG. 10 depicts an H&E stained tissue image with manual annotations by a pathologist marking the location of the TLS with a blue line, showing the location of the TLS in the tissue image, according to an embodiment of the present disclosure. [Figure 6B] FIG. 1 illustrates the location of TLS in a tissue image analyzed by a machine learning model, with brighter areas having higher tile scores indicating a higher likelihood of TLS, according to an embodiment of the present disclosure.

[0038] [Figure 7] FIG. 7 is an example flow diagram of a process 700 for detecting pixel-based TLS segmentation in an image using a machine learning model, according to an embodiment of the present disclosure.

[0039] [Figure 8] FIG. 1 illustrates an exemplary flow diagram for a training and validation process of a machine learning model for detecting pixel-based TLS segmentation in an image, according to an embodiment of the present disclosure.

[0040] [Figure 9A] FIG. 1 shows pixel-wise TLS segmentation in a tissue image, illustrating extraction of tiles and their segmentation masks from an H&E stained tissue image with manual annotation by a pathologist of TLS locations marked with blue lines. [Figure 9B] 1A-1C illustrate pixel-by-pixel TLS segmentation in tissue images, illustrating the process of training a machine learning model to predict TLS segmentation within a tile, according to an embodiment of the present disclosure.

[0041] [Figure 10] FIG. 1 illustrates an example of a computing system that may be used in conjunction with embodiments described herein. DETAILED DESCRIPTION OF THE INVENTION

[0042] Computer-implemented methods, associated systems, devices, and computer-readable media for detecting the presence or absence of tertiary lymphoid structures (TLS) are described. In some embodiments, provided herein are diagnostic tools that apply machine learning to digital images of tissue sections, e.g., whole slide images (WSI), to identify subjects with or without tumor-associated TLS and / or to aid in therapeutic decisions.

[0043] In the following description, numerous specific details are set forth to provide a thorough explanation of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention can be practiced without these specific details. In some cases, well-known components, structures, and techniques are not shown in detail in order to avoid obscuring the understanding of this specification.

[0044] Histology is the study of the microscopic characteristics of biological samples. Histopathology refers to the microscopic examination of a sample, such as a tissue, obtained or otherwise derived from a subject, such as a patient, to assess a disease state. A histopathological sample is generally obtained by processing the sample, such as a tissue, in such a way that the sample, or a portion thereof, is attached to a microscope slide. For example, a microtome or other suitable device can be used to obtain thin sections of the tissue sample, which can then be attached to a slide. Optionally, the sample can be further processed, such as by applying stains, to aid in visualization of the sample. Many stains have been developed for visualizing cells and tissues. These include, but are not limited to, hemotoxylin and eosin (H&E), methylene blue, Masson's trichrome, Congo red, Oil Red O, and safranin. H&E is routinely used by pathologists to aid in the visualization of cells within tissue samples. Hematoxylin stains cell nuclei blue, while eosin stains cytoplasm and extracellular matrix pink. Pathologists visually inspecting H&E-stained slides can use this information to evaluate the morphological characteristics of tissue. However, H&E-stained slides typically contain insufficient information to visually assess the presence or absence of specific biomarkers. Visualization of specific biomarkers (e.g., protein or RNA biomarkers) can be achieved through additional staining techniques, such as immunofluorescence, immunohistochemistry, and in situ hybridization, which rely on the use of labeled detection reagents that specifically bind to the marker of interest. While these techniques are useful for determining the expression of individual genes or proteins, they are impractical for assessing complex expression patterns involving numerous biomarkers. Global expression profiling can be achieved through genomic and proteomic approaches using separate samples derived from the same tissue source as the sample used for histopathological analysis. However, these methods are costly and time-consuming, require specialized equipment and imaging, and do not provide information correlating biomarker expression with specific regions within the tissue sample, such as within an H&E-stained image.

[0045] As used herein, "tertiary lymphoid structures" (TLS) refer to ectopic lymphoid structures that develop in non-lymphoid tissues at sites of chronic inflammation, including autoimmune disease, transplant rejection, and cancer. TLS resemble lymph nodes in structure and development, and their organization and integrity are supported by stromal cells. TLS comprise a heterogeneous population of cells, including B cells (e.g., germinal center B cells), T cells (e.g., type 1 T helper (Th1) cells, T follicular helper (Tfh) cells, and regulatory T (Treg) cells), dendritic cells (e.g., follicular dendritic cells (FDCs), mature dendritic cells), plasma cells, neutrophils, and macrophages. TLS are also characterized by the presence of high endothelial venules (HEVs), which refer to blood vessels adapted for lymphocyte transport. Fully developed TLS contain B cell follicles with activated and replicating B cell germinal centers surrounded by T cell sites. As used herein, "germinal center" refers to a microstructure that temporarily forms within the B cell zone (follicle) of secondary lymphoid organs, where mature B cells activate, proliferate, differentiate, and mutate antibody genes. TLS are interspersed with high endothelial veins, dendritic cell-lysosome-associated membrane proteins (DC-LAMPs), and dendritic cells. TLS are not encapsulated and reside within various non-lymphoid tissues, such as epithelial tissue and interstitium. While secondary lymphoid organs (SLOs) define specific structures, such as lymph nodes, TLS have diverse structures. TLS can be simple lymphocyte aggregates or organized structures present within non-lymphoid structures (Munoz-Erazo, L. et al. 2020 Cell. Mol. Immunol. 17:570-575).

[0046] The use of TLS as a prognostic indicator for cancer has been proposed (Colbeck, E. et al. Immunol. 8, 1830; Trajkovski, G. et al. 2018 Open Access Maced. J. Med. Sci. 6, 1824-1828). The presence or induction of TLS after cancer treatment can predict therapeutic efficacy and typically correlates with favorable therapeutic efficacy and / or clinical outcome. While the presence or induction of TLS can also be a prognostic factor and typically indicates favorable prognosis in various cancers, such correlation between the presence or induction of TLS and favorable therapeutic efficacy, clinical outcome, or prognosis does not exist in hepatocellular carcinoma (HCC). For example, the presence of mature TLS, characterized by mature follicles containing germinal centers, in tumor compartments has been associated with improved survival following cancer immunotherapy in patients with solid tumors. Therefore, TLS can be used to identify subjects likely to benefit from immune checkpoint inhibitors. The number, density, and location of TLS may affect a subject's prognosis or treatment outcome (Munoz-Erazo, L. et al. 2020 Cell. Mol. Immunol. 17:570-575).

[0047] Furthermore, exogenous induction of TLS can have therapeutic benefits. TLS formation through various pharmacological approaches can promote lymphocyte infiltration, tumor antigen activation, and differentiation, augment antitumor immune responses, and / or, when combined with immune checkpoint blockade, vaccines, viruses, local intratumoral agents, or interventional therapies, potentially enhance the susceptibility of immune-compromised tumors to immunotherapy. In immunologically active tumors with a perturbed tumor microenvironment and intense chronic inflammation, angiogenesis, and fibrotic stroma, the use of antiangiogenic and antiimmunosuppressive agents can normalize the immune situation, promoting TLS formation and therapeutic responses to immune checkpoint inhibitors. Several approaches have been developed to induce TLS formation using chemokines, cytokines, antibodies, antigen-presenting cells, or synthetic scaffolds. Strategies aimed at inducing TLS formation in immune-compromised and immune-compromised tumors, in this case in combination with therapeutic agents that attenuate the inflammatory environment and / or immune checkpoint inhibitors, represent promising avenues for cancer treatment.

[0048] Detection of TLS is based on morphological assessment of TLS, including their location and quantity, and relies on local annotation of critical areas within images by expert pathologists. For example, H&E staining allows for the detection of TLS in formalin-fixed, paraffin-embedded tumor sections. Mature TLS correspond to lymphoid follicles containing dense cellular clusters resembling germinal centers found in secondary lymphoid structures (SLOs). Less differentiated structures, such as lymphoid aggregates and lymphoid follicles lacking germinal centers, can also be detected by pathological examination of H&E-stained tissue slides. Currently, pathological detection and evaluation of such TLS is time-consuming and labor-intensive.

[0049] Additional analyses, including immunohistochemistry, are useful for detecting TLS. Immunohistochemistry (IHC) of serial tumor sections or dual or multiple labeling techniques using markers present in cells or tissues that comprise TLS can also be used to detect TLS, followed by evaluation of the density, size, and cellular content of TLS on scanned images using, for example, quantitative digital pathology software. Common cell types present in TLS and exemplary markers that can be used to detect TLS are listed in Table 1 below.

[0050] Table 1. Cell types present in TLS and their markers TIFF2025541702000002.tif130169

[0051] Various gene signatures of TLS can also be used to aid in the pathological detection of TLS in samples, such as cancer histological samples. Examples of gene signatures of TLS are shown in Table 2 below. The characteristics, functions, roles and meanings of TLS are discussed in the following documents, which are incorporated herein by reference in their entireties: Sautes-Fridman et al., 2019 Nat.Rev.Cancer 19(6);Sautes-Fridman et al.Cancer 19(6);Sautes-Fridman et al., 2016 Front.Immunol.7:407;Vanherseche et al., 2021 Nat.Cancer 2(8):794-802.

[0052] Table 2. Gene signatures for TLS detection TIFF2025541702000003.tif192169 TIFF2025541702000004.tif172169 TIFF2025541702000005.tif177169 TIFF2025541702000006.tif203169

[0053] Even with histological analysis using IHC or the gene signature analysis provided herein, detecting TLS remains time-consuming, laborious, and expensive, making it unsuitable for high-throughput applications. To overcome this problem, the present disclosure provides an imaging pipeline for analyzing pathological tissue images without using local annotations. This pipeline is initially based on segmenting a large image (e.g., WSI) into smaller images (e.g., 224x224 pixel images) and detecting regions of interest within the images to perform classification using Otsu's method. In this way, the classification works on smaller images, which is much less computationally expensive than a single large image. These smaller images are fed into a ResNet-type convolutional neural network, which can extract feature vectors from each smaller image. The feature vectors comprise local descriptors for that smaller image. From the extracted feature vectors, a score is calculated for each subimage as a local tile-level (instance) descriptor. The top and bottom instances are used as inputs to a multilayer perceptron (MLP) to perform classification on them.

[0054] In one embodiment, the device classifies histology images (e.g., cancer histology images) using one or more neural network models to determine labeling for the images. In this embodiment, the tissue images may be large images for which processing the entire image using only a neural network model is computationally impractical. In particular, the device reduces the amount of computing resources (e.g., time and / or memory requirements) required to perform image classification tasks on these large images. This reduction in resources further improves the device's performance in performing image classification tasks. Furthermore, the device can classify whole slide images (WSIs), even when these types of images are too large to fit into the memory of graphics processing units typically used to train machine learning models. In a further embodiment, the device reduces the dimensionality of the data, thereby providing better generalization error and being more efficient in terms of model accuracy.

[0055] References herein to "one embodiment" or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the present invention. The appearance of the phrase "in one embodiment" in various places herein does not necessarily all refer to the same embodiment. The term "exemplary" is used herein to mean "exemplary" rather than "ideal." It should be understood from this disclosure that the present invention is not limited to the examples set forth herein.

[0056] For any method described herein, the order of steps presented, whether in the text or in an accompanying flow diagram, should not be construed to imply that the steps must be performed in the order presented, unless the context otherwise specifies or requires. Rather, the order of steps is presented in one embodiment of the provided method, and in general, such steps can alternatively be performed in a different order or simultaneously. The processes depicted in the following figures can be performed by processing logic comprising hardware (e.g., circuitry, dedicated logic, etc.), software (such as might be executed on a general-purpose computer system or dedicated machine), or a combination of both. Although the processes are described below in terms of several sequential operations, it should be understood that some of the described operations may be performed in a different order. Furthermore, some operations may be performed in parallel rather than sequentially.

[0057] Computing methods used to implement the methods provided herein may include, for example, machine learning, artificial intelligence (AI), deep learning (DL), neural networks, classification and / or clustering algorithms, and regression algorithms.

[0058] The terms "server," "client," and "device" are intended to refer generally to data processing systems, rather than specifically to particular form factors of servers, clients, and / or devices.

[0059] References herein to "local annotations" refer to metadata (e.g., text, markings, numbers, and / or other types of metadata) that apply to a portion of an image, rather than to the entire image. For example, in one embodiment, a local annotation can be a marking of a region of interest within an image, such as a histology image. Exemplary local annotations include markings that outline or otherwise identify a portion of an image, e.g., a tumor region in an image, a stromal region in an image, identification of a cell type within an image, or identification of a biological structure comprised of multiple cells within an image, e.g., TLS. In contrast, a "global annotation" herein refers to metadata that applies to the entire image. Exemplary global annotations include labels that identify the entire image, data regarding how the image was acquired, labels that identify characteristics of the subject from whom the image was derived, e.g., labels indicating the age, sex, diagnosis, etc., of the subject from whom the image was derived, and / or any other data that applies to the entire image. In some embodiments, a global annotation can indicate the presence, amount, or location of TLS that are known or understood to be present in the subject from whom the image was derived. In other embodiments, the global annotations may indicate known characteristics of the subject from which the image is derived, such as survival time (e.g., survival time following collection of the specimen represented in the image) or response to a given treatment. In some embodiments described herein, images with global annotations may be used in the absence of local annotations.

[0060] "Patient" refers to a subject who exhibits symptoms and / or complications of a disease or condition (e.g., malignancy, cancer), is receiving treatment from a clinician (e.g., an oncologist), has been diagnosed with the disease or condition, and / or is at risk for developing the disease or condition. The term "patient" includes human and animal subjects. All references to a subject in this disclosure should be understood to include the possibility that the subject is a "patient," unless the context clearly dictates otherwise.

[0061] As used herein, a "subject" is an animal, such as a mammal, including a primate (such as a human, monkey, and chimpanzee) or a non-primate (such as a cow, pig, and horse) that would benefit from a method according to the present disclosure. In some embodiments of the present invention, the subject is a human, such as a human diagnosed with cancer. The subject may be a female human. The subject may be a male human. In some embodiments, the subject is an adult subject.

[0062] As used herein, "predict" or "predicting" in the context of this disclosure refers to determining the likelihood of the presence or absence of a state (e.g., TLS) in the past, present, or future. In some embodiments, a model (e.g., a 1D convolutional layer with MoCo features followed by multi-layer perception) can predict the likelihood of a TLS state by one or more of the following measures of test accuracy: an odds ratio of 1 or greater, preferably about 2 or greater but less than about 0.5, about 3 or greater but less than about 0.33, about 4 or greater but less than about 0.25, about 5 or greater but less than about 0.2, or about 10 or greater but less than about 0.1; a specificity of greater than 0.5, preferably at least about 0.6, at least about 0.7, at least about 0.8, at least about 0.9, or at least about 0.95, with a corresponding sensitivity of greater than 0.2, preferably at least about 0.3, at least about 0.4, at least about 0.5, at least about 0.6, at least about 0.7, at least about 0.8, at least about 0.9, or at least about 0.95; a sensitivity of at least 0.5, preferably at least about 0.6, at least about 0.7, at least about 0.8, at least about 0.9, or at least about 0.95, with a corresponding sensitivity of at least 0.2, preferably at least about 0.3, at least about 0.4, at least about 0.5, at least about 0.6, at least about 0.7, at least about 0.8, at least about 0.9, or at least about 0.95; A sensitivity of at least about 75% combined with a specificity of at least about 75%; a positive likelihood ratio (calculated as sensitivity / (1-specificity)] of 1 or greater, preferably at least about 2, at least about 3, at least about 4, at least about 5, at least about 10; or A negative likelihood ratio [calculated as (1-sensitivity) / specificity] of less than 1, preferably about 0.5 or less, about 0.33 or less, about 0.25 or less, or about 0.1 or less.

[0063] As used herein, "tumor" refers to the abnormal growth of cells or tissues and / or the resulting mass. In some embodiments, the tumor tissue or tumor cells are malignant (e.g., cancerous). As used herein, "cancer" or "malignant tumor" refers to a plurality of cells in which the abnormal cells divide uncontrolled and have the ability to invade or metastasize to adjacent or distant tissues or organs. Exemplary malignant tumors or cancers include lung cancer, sarcoma, bladder cancer, colon cancer, ovarian cancer, pancreatic cancer, and melanoma.

[0064] I. Methods for detecting TLS in subjects In some embodiments, provided herein are computer-implemented methods for detecting the presence or absence of TLS in a subject. In one embodiment, the computer-implemented method comprises: receiving a digitized tissue image of a sample obtained from the subject; tiling the tissue image into a set of tiles; extracting a plurality of feature vectors from each of the tiles, each feature of the one or more feature vectors representing a local descriptor of the tile; classifying the tissue images for TLS status using a classification model trained using at least the plurality of feature vectors and an imaging training set having known TLS annotations, the TLS status indicating the presence or absence of TLS in the subject; and In some embodiments, the tissue image lacks local annotation of histopathological features and / or global annotation of TLS status.

[0065] As used herein, a "tile" refers to a subsection of an image. As used herein, "tiling" refers to dividing an image or region of interest into tiles.

[0066] As used herein, the term "digitized image" or "digital image" refers to an electronic image represented by a collection of pixels that can be displayed, processed, and / or analyzed by a computer. In some aspects of the present disclosure, digital images of histology slides, e.g., H&E-stained slides, enable computational evaluation of tissue samples in addition to or instead of visual inspection by a pathologist. In some embodiments, the digital image can be acquired by a digital camera or other optical device capable of capturing a digital image from a slide or portion thereof. In other embodiments, the digital image can be acquired by scanning a non-electronic image of a slide or portion thereof. In some embodiments, the digital image used in the applications provided herein is a whole slide image. As used herein, the term "whole slide image (WSI)" refers to an image that includes all or nearly all of a tissue section, e.g., a tissue section present on a tissue slide. In some embodiments, a WSI includes an image of the entire slide. In other embodiments, the digital image used in the applications provided herein is a tissue section, e.g., a selected portion of a tissue section present on a tissue slide. In some embodiments, the digital image is acquired after the tissue section has been treated with a stain, e.g., H&E.

[0067] As used herein, "TLS" status refers to the presence, location, or quantity of TLS in a subject, a tissue image, a tile within a tissue image, or a pixel within a tile. TLS status can be expressed in a binary format (e.g., presence or absence), a classification, a continuous range (e.g., a numerical quantity), a description (e.g., a description of the location or quantity of TLS), or a combination thereof. TLS status can be based on a probability score assigned to a subject, an imager, a tile, or a pixel, or components thereof. For example, at the subject level, TLS status refers to the presence or absence of TLS in a subject, e.g., a cancer site, and can be determined based on a probability score (e.g., an imager score) assigned to a tissue image of a specimen (e.g., a tumor specimen) obtained from the subject. At the tissue image level, TLS status can refer to the presence or absence of TLS in an image, an imager, or a location (e.g., segmentation within an image) and can be determined based on an image score assigned to the image or a tile score assigned to a tile within the image. At the tile level, the TLS state may refer to the presence or absence, amount, or location of TLS within the tile (e.g., segmentation within the tile) and may be determined based on a tile score assigned to the tile or a pixel score assigned to a pixel within the tile. At the pixel level, the TLS state may refer to the presence or absence or amount of TLS within the pixel and may be determined based on a pixel score assigned to the pixel.

[0068] As used herein, a "score," "probability score," or "risk score" refers to the likelihood that a particular condition, such as TLS, is present or is present in a therapeutically relevant manner (e.g., present in at least some clinically relevant amount, present in at least some clinically relevant location). In some embodiments, the score is expressed as a category. In other embodiments, the score is expressed as a continuous range. In one embodiment, the score represents the likelihood of TLS being present in a pixel, tile, image, or subject.

[0069] According to one embodiment, a device classifies at least one histological input image by applying a first convolutional neural network to segment the image between at least one region of interest containing information useful for classification and at least one background region containing little or no information useful for classification. The device further tiles the region of interest of the image into a set of tiles. The device further applies a second convolutional neural network to extract a feature vector for each tile, where the features are local descriptors of the tile. The device further processes the extracted feature vectors of the tiles to classify the image. In one embodiment, by segmenting the input image, the device processes a reduced number of tiles and avoids processing the entire image.

[0070] In one embodiment, the first convolutional neural network is a semantic segmentation neural network that classifies pixels of the input image as one of two categories: (a) regions of interest, and (b) background regions. Furthermore, the tiling step (b) can be performed by applying a fixed tiling grid to the image, the tiles having a predetermined size. Furthermore, at least one level of zoom can be applied to the tiles. For example, multiple levels of zoom can be applied to the tiles, and tiles from different levels of zoom can be combined. Furthermore, the device can optionally randomly sample tiles and / or pad the set of tiles with blank tiles so that the set of tiles comprises a predetermined number of tiles.

[0071] In a further embodiment, the second convolutional neural network can be a residual neural network, such as a ResNet50 or ResNet101 residual neural network with the last layer removed using the previous layer as the output, or a VGG neural network. This second convolutional neural network can be a pre-trained neural network, allowing for the use of state-of-the-art advanced neural networks without having a large image database and the computing resources to train this neural network.

[0072] In one embodiment, the device can calculate at least one score for a tile from the extracted feature vector, the tile score representing the tile's contribution to the classification of the image. Using the tile scores, the device can sort the set of tile scores, select a subset of the tile scores based on their value and / or their rank in the sorted set, and apply a classifier to the retained tile scores to classify the image. The device can further apply this classification to multiple input images, and the device can aggregate groups of corresponding tiles from different input images.

[0073] In an alternative embodiment, the device may also aggregate clusters of adjacent tiles. In this embodiment, aggregating the clusters of tiles may include concatenating tiles of the cluster, picking a single tile from the cluster according to predetermined criteria, using the cluster as a multidimensional object, or aggregating values ​​via, for example, an average or max-pooling operation. Additionally, the device may apply an autoencoder to the extracted feature vector to reduce the dimensionality of the features. In an embodiment, the image is a histopathology slide, the region of interest is a tissue region, and the classification of the image is a diagnostic classification.

[0074] In an alternative embodiment, if local annotations, such as the presence of a tumor at a site on a slide, are available, a hybrid technique can be used to take these annotations into account. To do so, the device can train a machine learning model for two simultaneous tasks: (1) local prediction of the presence of a tumor and / or other macroscopic features on each tile, and (2) prediction of a set of global labels. A more complex architecture can be used on the device (or multiple devices), which includes, on the one hand, the classification system described above to process the set of 2,048 features. On the other hand, the device applies a convolutional neural network to convert the features of N tiles into an N*2,048 feature vector. Based on this vector, the device trains a convolutional neural network to predict the presence or absence of a tumor (or other macroscopic feature) for each tile. The device can take both the prediction output and the N*2,048 feature vector and apply a weight pooling operation to the concatenation of these two vectors to obtain a 2,048 feature vector for the input image. The imaging device concatenates the output of the classification model with the resulting 2,048 feature vector and attempts to predict a set of global labels for the image (e.g., survival, tumor size, necrosis, and / or other types of predictions) based on this vector. The model's loss includes both global and local predictions. In this embodiment, adding information derived from local annotations to the computational flow can improve the overall model performance.

[0075] 1 is an exemplary flow diagram of a process 100 for detecting TLS status in an image and / or the presence or absence of TLS in a subject using a machine learning model, according to an embodiment of the present disclosure. At block 102, process 100 receives an image, one or more machine learning (ML) models, and optionally other inputs. In some embodiments, the image is a digitized histology image of a sample obtained from the subject, such as a digitized whole slide image (WSI).

[0076] At block 104, process 100 detects regions of interest (ROIs) within the image. As used herein, a "region of interest" (ROI) in an image can be any region semantically relevant to the task being performed, and particularly in the context of histopathology, can be a region corresponding to a tissue, organ, bone, cell, bodily fluid, etc. In a further embodiment, process 100 segments the image into regions of interest and background regions. In this embodiment, extracting regions of interest from an input image can reduce the amount of computing required to classify the input image. For example, in one embodiment, histopathology slides (or other types of images) can contain empty areas of the image with little or no tissue, so it is useful to employ methods known as "tissue detection" or "material detection" to assess whether regions of the slide contain tissue. More generally, when the goal is to classify large images, it is important to identify regions of interest in the image and distinguish them from background regions. These regions of interest are regions of the image that contain valuable information for the classification process. Background regions are regions of the image that contain little or no valuable information; background regions can be considered noise for the task at hand. To accomplish this task, various different types of image segmentation schemes can be used. For example, in an embodiment, Otsu's method can be used to segment the image. Otsu's method is a simple thresholding method based on the intensity histogram of the image. In this embodiment, image segmentation using Otsu's method has shown fairly good results when the image contains two classes of pixels that follow a bimodal distribution, such as foreground and background pixels, or more specifically, tissue and non-tissue. However, this method is known to perform poorly on complex images when the histogram of intensity levels cannot be assumed to have a bimodal distribution. Therefore, a more robust method is needed to improve the overall efficiency of the method.

[0077] In another embodiment, and to improve the robustness of image segmentation and enable it to tackle complex images (such as histopathology images), a semantic segmentation neural network, such as the U-NET semantic segmentation neural network, SegNet, DeepLab, or another type of semantic segmentation neural network, can be used for image segmentation. In this embodiment, a semantic segmentation neural network can be used that does not rely on a particular distribution of intensity histograms. Furthermore, the use of such a neural network enables image segmentation that takes into account multi-channel images, such as red-green-blue (RGB) images. In this way, segmentation can take advantage of image semantics rather than just relying on the histogram of pixel intensities. In one embodiment, the semantic segmentation neural network is trained to segment tissue in a histopathology image from the background of the image, distinguishing stained or unstained tissue from the background.

[0078] In another embodiment, another advantage of using the U-NET segmentation neural network is that this network type was developed for biomedical image segmentation and fits the typical constraints of biomedical data, which is to have small datasets with very high dimensions. In fact, because the U-NET segmentation neural network is a model with few parameters to train, it is possible to train this network with few training examples. In yet another embodiment, by using data augmentation techniques on the training data, more training examples can be obtained from the same training set, and this architecture can achieve very good results.

[0079] In further embodiments, the original image can be downsampled to make the image segmentation step less computationally expensive. As described below, and in some embodiments, because part of the image analysis is performed at the tile level (which is a subsection of the image), using semantic segmentation on a downsampled version of the image does not degrade the quality of the segmentation. This allows the downsampled image to be used without degrading the quality of the segmentation. In one embodiment, to obtain the segmentation mask of the original full-resolution image, process 100 simply needs to upscale the segmentation mask generated by the neural network.

[0080] At block 106, process 100 tiles the ROI into a set of tiles. Tiling an image can involve dividing the original image into smaller, more manageable images called tiles. In one embodiment, the tiling operation is performed by applying a fixed grid to the entire slide image using a segmentation mask generated by the segmentation method and picking tiles that contain tissue or other regions of interest. In one embodiment, additional or alternative selection methods, such as random subsampling, that leave only a predetermined number of slides, can be used to further reduce the number of tiles to process.

[0081] Tiling can be performed to enhance image preprocessing capabilities. For example, and in embodiments, the use of tiling techniques is useful in histopathology analysis due to the large size of whole slide images. More broadly, when dealing with specialized images such as histopathology slides, satellite images, or other types of large images, the resolution of imaging devices used in these fields can grow as quickly as the capacity of the random access memory associated with the sensor. This increase in imaging device size makes it difficult to store batches of images, or even single images, within a computer's random access memory. This difficulty is further exacerbated when attempting to store such large images in the dedicated memory of a graphics processing unit (GPU). This situation makes it computationally challenging to process slide images, or other similarly sized images, as a whole.

[0082] In one embodiment, tiling the image (or the image minus the background) addresses this challenge by dividing the original image (or the image minus the background) into smaller, more manageable images (i.e., tiles). In one embodiment, the tiling operation is performed by applying a fixed grid to the entire slide image using a segmentation mask generated by the segmentation method and selecting tiles that contain tissue or other types of regions of interest for subsequent classification processing. To further reduce the number of tiles to be processed, additional or alternative selection methods can be used, such as random subsampling, which leaves only a predetermined number of slides. For example, in one embodiment, the image (or the image minus the background) is divided into fixed-size tiles (e.g., each tile has a size of 224 x 224 pixels). The tiles can be any uniform size within the slide. The tiles can also be square. For example, the tiles can have a width and / or depth of approximately 10 to 500 μm. The tiles can have approximately 20 to 1,000 pixels per side. The number of tiles generated depends on the size of the substance to be detected and can vary from several hundred tiles to 50,000 tiles or more than two tiles. In one embodiment, the number of tiles is limited to a configurable fixed number (e.g., 10,000 tiles) based at least on computing time and memory requirements. In specific embodiments, tiles have a size of approximately 224 x 224 pixels and approximately 112 μm x 112 μm. In these embodiments, a digitized full slide image can have approximately 10,000 tiles per image.

[0083] In one embodiment, augmentation can be applied to each set of tiles. In some embodiments, a first set of features is extracted from a first batch of augmented tiles. A second set of features is extracted from a second batch of augmented tiles. In some embodiments, the augmented tiles include magnified or rotated views or color-enhanced views. For example, orientation is not important for histological slides, so slides can be rotated at various angles. Slides can also be enlarged or zoomed in. Contrast loss between pairs of the first and second sets of extracted features can be used to bring matching pairs of tiles closer together and dissimilar pairs of tiles further apart. Contrast loss can be applied to focus on positive feature pairs from the first and second sets, but not on negative pairs.

[0084] At block 108, process 100 extracts a feature vector from each tile. In one embodiment, feature extraction is performed by a ResNet50 neural network. In one embodiment, each feature is extracted by applying a trained feature extractor trained with a machine learning algorithm using a training set of images. For example and in one embodiment, the machine learning algorithm is MoCo (Momentum Contrast) or MoCo v2 (Momentum Contrast v2). In one embodiment, the trained machine learning model is the machine learning model trained in FIG. 3 described herein.

[0085] In some embodiments, a machine learning algorithm, such as MoCo or MoCo v2, extracts multiple feature vectors from each tile. The extraction of the multiple feature vectors can be performed using a convolutional neural network, such as a ResNet50 neural network. The number of feature vectors can be any number, such as about 1,000, about 1,500, about 2,000, about 3,000, about 4,000, or more.

[0086] In some embodiments, a machine learning algorithm, such as MoCo or MoCo v2, extracts multiple feature vectors from the digital image, and the extraction of the multiple feature vectors is performed using a first convolutional neural network. In one embodiment, process 100 can use any feature extraction neural network to extract features, such as a ResNet-based architecture (e.g., ResNet-50, ResNet-101, ResNetX), a Visual Geometry Group (VGG) neural network, an Inception neural network, or a custom neural network designed specifically for the task. In some embodiments, a non-neural network feature extractor, such as SIFT or CellProfiler, can be used for feature extraction. Furthermore, the feature extraction neural network used can be pre-trained, as these are trained on very large datasets and therefore have optimal generalization accuracy. In one embodiment, the first neural network includes a 1D convolutional layer. In one embodiment, the 1D convolutional layer uses MoCo features as input and outputs the likelihood that the tile includes a TLS.

[0087] In one embodiment, process 100 uses a ResNet-50 neural network, which can provide features well suited for image analysis without requiring excessive computing resources. For example, in one embodiment, ResNet-50 can be used for histopathology image analysis. In this example, the ResNet-50 neural network relies on residual blocks, which allow the neural network to be deeper and still improve its accuracy, because simple convolutional neural network architectures can suffer from poor accuracy when the number of layers becomes too large. In one embodiment, the weights of the ResNet-50 neural network used for feature extraction can be pre-trained on the ImageNet dataset. In one embodiment, using a neural network pre-trained on a large, independent image dataset allows for good features to be obtained regardless of the type of image, even when the input image is unusual, such as a histopathology image (or other type of image). In this embodiment, process 100 uses a ResNet-50 convolutional neural network to extract 2,048 features per tile. If process 100 extracts, for example, 10,000 tiles, process 200 generates a matrix of 2,048 x 10,000. Furthermore, if process 200 is run with a large number of images as input, process 100 generates a tensor with dimensions of number of images x number of features / tiles x number of tiles.

[0088] In one embodiment, process 100 processes and extracts features for a given slide by passing each selected tile through a ResNet-50 neural network, which outputs a feature vector for that tile. In this embodiment, the feature vector can be a vector of dimension 2048 or another size. Additionally, process 100 can apply an autoencoder to the feature vector to provide further dimensionality reduction (e.g., reducing the dimension of the feature vector to 256 or another dimension). In one embodiment, an autoencoder can be used when the machine learning model is susceptible to overfitting. For example, in one embodiment, process 100 can reduce the length of a 2048 feature vector to a 512-length feature vector. In this example, process 100 uses an autoencoder with a single hidden layer architecture (of 512 neurons). This prevents overfitting of the model by finding a few peculiar features in the training dataset and reduces computing time and memory requirements. In one embodiment, the classification model is trained on a small subset of image tiles, for example, 200 randomly selected tiles from each slide (out of a total of 411,400 tiles).

[0089] Process 100 can optionally perform a zero-padding operation on the feature vectors to derive a minimum number of features. In this embodiment, process 100 can perform zero-padding to add feature vectors to the set of feature vectors for the image if the number of feature vectors is below the minimum number of feature vectors. In this embodiment, each zero-padded feature vector has a null value.

[0090] At block 110, process 100 classifies the image for TLS status using the extracted feature vector and the machine learning model. The computer-implemented method may process all tiles within the ROI for feature vector extraction and / or classification. In some embodiments, the computer-implemented method further comprises selecting a subset of tiles to apply the machine learning model to. In some embodiments, the subset of tiles is selected by random sampling. In this embodiment, process 100 may output that the digital image is TLS-positive or TLS-negative, with this designation being a global classification for the digital image. Alternatively, process 100 may determine which tiles are TLS-positive based on the tile scores for those tiles. In one embodiment, TLS image classification is further described in FIG. 2 below.

[0091] In some embodiments, the classification step, such as block 110 of process 100, applying a first neural network to the one or more feature vectors of the plurality of feature vectors, the first neural network assigning a tile score to each tile of the set of tiles based on the one or more feature vectors of the plurality of feature vectors, the tile score representing the likelihood that the tile contains a TLS; applying a second neural network to each tile, the second neural network aggregating a subset of the tile scores for the set of tiles to determine a TLS state in the tissue image; Including, the first neural network is trained using a training set of tissue images containing known local annotations of the presence or absence of TLS at the tile level; The first and second neural networks are trained using a training set of tissue images that contain known global annotations regarding the presence or absence of TLS at the tissue image level.

[0092] 2 is an example flow diagram of a process 200 for classifying an image for TLS status and optionally predicting TLS locations within the image, according to an embodiment of the present disclosure. In one embodiment, the process 100 executes FIG. 2 to classify an image for TLS status.

[0093] At block 202, process 200 receives extracted features from each tile, a machine learning model, and optionally other inputs. In one embodiment, the extracted features are the extracted features determined at block 108 above. At block 204, process 200 determines a tile score for each tile based on the extracted feature vector using a first neural network trained using a training image set with local annotations. In some embodiments, the first neural network is a convolutional 1D layer. In some embodiments, the tile score represents the likelihood that the tile has a TLS. In one embodiment, the tile score is expressed as a continuous range. For example, in one embodiment, process 200 determines a tile score for each tile that is a number between 0 and 1, where a score of 0 indicates that the tile is least likely to have a TLS and a score of 1 indicates that the tile is most likely to have a TLS. In one embodiment, process 200 uses a concatenated neural network to reduce each of the feature vectors to one or more scores. In one embodiment, process 200 can reduce the feature vector to a single score using a fully connected neural network, or to multiple scores representing different characteristics of the tile using one fully connected neural network that outputs different scores, or multiple fully connected neural networks that each output a different score. These scores associated with a tile are sorted, and a subset of tiles is selected for image classification.

[0094] For example, in one embodiment, process 200 can use a convolutional 1D layer to create a score for each tile. In this example, as described above with a 2,048-length feature vector, the convolutional layer performs a weighted sum between the tile's 2,048 features to derive this score, and the weights in this sum are learned by the model. Furthermore, because the convolutional 1D layer is unbiased, zero-padded tiles have a score of 0 and thus serve as a completely uninformative reference for the tile. Process 200 then selects the N highest scores and the M lowest scores and uses these as inputs for classification, as described below. This architecture guarantees which tiles will be used for prediction.

[0095] In one embodiment, process 200 uses the tile score vector as input to a dense multilayer neural network to provide the desired classification (e.g., the TLS status of the image). This classification can be any task that associates a label with data provided as input to the classifier. In one embodiment, by using a classifier trained on the digital image input, the classifier can therefore label the histopathology image provided as input without having to process the entire image, which can be computationally prohibitive, since the input data is derived through an entire pipeline. For example, and in one embodiment, the label can be any type of label, such as a binary value representing the prognosis of a given pathology, a score, a probability, or a numerical label representing a prediction of a physical quantity, such as a survival prediction or a response to treatment prediction, and / or a scalar label as described above, or a vector, matrix, or tensor of such labels representing structured information. For example, and in one embodiment, process 200 can output a TLS-positive or TLS-negative status, indicating whether the digital image is predicted to contain or not contain TLS. In one embodiment, process 200 uses a multilayer perceptron (MLP) with two fully connected layers of 200 neurons and 100 neurons with sigmoidal operation. In this embodiment, the MLP is used as the core of a prediction algorithm that converts tile scores into labeling. In one embodiment, process 200 predicts a single label (e.g., a probability score) for an image, but in an alternative embodiment, process 200 can predict which tiles may contain TLS. In this embodiment, process 200 can determine that a particular score is a threshold for determining the presence of TLS in a tile.

[0096] The tissue image can be classified based on at least one set of tile scores, which are derived from the tissue image tile feature vectors generated from the neural network. In block 204, process 200 calculates a tile score for each tile using the associated feature vector for that tile. For example, in one embodiment, process 200 can use a convolutional 1D layer to create a score for each tile. In the above example with a feature vector of length 2,048, the convolutional layer performs a weighted sum between all 2,048 features of the tile to obtain this score, and the weights of this sum are learned by the model. Furthermore, because the convolutional 1D layer is unbiased, zero-padded tiles have a score of 0, and therefore serve as a completely uninformative reference for the tile.

[0097] In block 206, process 200 sorts the tiles by tile score. In one embodiment, process 200 sorts the tile set to determine the top N and / or bottom M scores for block 206.

[0098] At block 208, process 200 selects the highest tile score (N) and the lowest tile score (M). In one embodiment, process 200 selects a subset of tiles to be used in a subsequent classification step. In one embodiment, this subset of tiles can be tiles with the top N highest scores and the bottom M lowest scores, the top N highest scores, the bottom M lowest scores, and / or any weighted combination of scores. In one embodiment, the ranges of values ​​for N and / or M can be the same or different. Additionally, the ranges for N and / or M can conform to a static numeric range (e.g., 10, 20, 100, or some other numeric value), a range, a percentage, a labeling (e.g., small, large, or some other labeling), be set via a user interface component (a slider, a user input, and / or another type of user interface component), and / or be some other value. In one embodiment, process 200 further concatenates these scores into an imaging score vector, which can be taken as input for image classification.

[0099] In some embodiments, the classifying further comprises: picking the tile with the highest TLS tile score; and picking the tile with the lowest TLS tile score; sorting the set of tiles by

[0100] In one embodiment, when studying histological whole slide images (or slides), a single patient may be associated with multiple slides taken with different stains, at different locations on the same sample, from multiple organs, or at different time points. In this embodiment, slides from a single patient may be aggregated in multiple ways. In one embodiment, process 200 may concatenate slides to form larger slides that are processed in the same or similar ways (segmentation, tiling, feature extraction, and classification) as normal.

[0101] In another embodiment, a slide may not contain enough useful tissue to extract as tiles to apply the feature extraction step and provide features to the classifier. In this case, the input to the classifier is zero-padded, i.e., for each missing tile, a zero-padded feature is added to the actual feature calculated by the feature extractor.

[0102] At block 210, process 200 applies a second neural network to the tiles using the tile scores. In one embodiment, the second neural network is a multi-layer perceptual model. In one embodiment, process 200 uses a subset of the tiles, i.e., tile scores N and M. At block 212, process 200 uses the results of the second neural network applied to the tile scores to determine a TLS global score for the tissue image. In one embodiment, the TLS global score can be a positive occurrence of TLS or a negative occurrence of TLS. In this embodiment, the number of tiles for the image can be on the order of 10,000 tiles. In further embodiments, the number of tiles for the image can be one or more than one. In one embodiment, and to reduce computational complexity, the classification system samples the tiles to reduce the number of tiles used in computing the neural network. In some embodiments, the classification system samples the tiles randomly or with other types of sampling mechanisms. For example, in one embodiment, the classification system randomly samples the tiles to reduce the number of tiles from on the order of 10,000 tiles to on the order of several thousand tiles (e.g., 3,000 tiles).

[0103] In some embodiments, the computer-implemented methods provided herein further include detecting one or more locations in the tissue image where TLS is present.

[0104] For example, process 200 can optionally proceed to block 214, where process 200 outputs predicted TLS locations within the tissue image. In some embodiments, in block 214, process 200 overlays the tile scores assigned to each tile onto the tissue image so that regions assigned high and low tile scores are visualized. For example, as shown in FIG. 6B, process 200 can detect TLS locations within the tissue image. Regions are color-coded according to their assigned tile scores, with brighter regions corresponding to higher tile scores and indicating a higher likelihood of TLS. The detection of TLS locations corresponds to the pathologist's manual annotation of TLS locations in FIG. 6A, where regions marked with blue lines indicate TLS locations.

[0105] In some embodiments, the computer-implemented method provided herein comprises: receiving a digitized tissue image of a sample obtained from the subject; tiling the tissue image into a set of tiles; extracting a plurality of feature vectors from each of the tiles, each feature of the one or more feature vectors representing a local descriptor of the tile; and classifying the tissue images for TLS status using at least the plurality of feature vectors and a classification model trained on an image training set having known TLS annotations for each tissue image in the plurality of tissue images, the TLS status indicating the presence or absence of TLS in the subject; Repeating all steps of detecting the presence or absence of TLS in the subject by processing the TLS status of the plurality of tissue images; Includes.

[0106] The machine learning model of the computer-implemented method provided herein can be trained and validated. In a specific embodiment, the first neural network is trained using a training set of tissue images with known local annotations of the presence or absence of TLS at the tile level, and the first neural network and the second neural network are trained using a training set of tissue images with known global annotations of the presence or absence of TLS at the tissue image level.

[0107] In one embodiment, the model used in FIGS. 1 and 2 can be trained and validated using a training set of images. In this embodiment, process 300 uses input labels from the training set of images to train a one-dimensional convolutional neural network and MLP classification model to generate scores. In this embodiment, the process iteratively trains the model by calculating a score set for the training images, predicting labels, determining the difference between the predicted labels and the input labels, and optimizing the model (e.g., calculating new weights for the model) based on the difference until the difference is within a threshold. In one embodiment, process 300 trains the model to predict a single labeling (e.g., a tile score) for an image. In an alternative embodiment, the process can be trained to predict multiple global labels for an image. In one embodiment, the process can be trained to perform a multi-task learning environment to predict multiple global labels. For example, in one embodiment, a machine learning model (e.g., an MLP and / or other model described elsewhere) can be trained to predict multiple labels at once in a multi-task learning environment using the resulting feature vectors.

[0108] FIG. 3 shows a flow diagram of one embodiment of a process 300 for training and validating first and second neural networks for detecting TLS status in tissue images or subjects. In one embodiment, the neural networks include one or more separate models used in the classification processes described in FIGS. 1 and 2. In FIG. 3, process 300 begins at block 302 by receiving a training set of tissue images, local and global annotations, a machine learning model, and optionally other inputs. In one embodiment, process 300 receives the images, local and global annotations, the first and second neural networks used in FIGS. 1 and 2 above, and optionally other inputs as described in FIG. 1. In one embodiment, the first neural network is a 1D convolutional layer. In one embodiment, the 1D convolutional layer uses MoCo features as input and outputs the likelihood that a tile has TLS. In one embodiment, the second neural network is a multi-layer perceptual model. In one embodiment, the first neural network is trained using a training set of tissue images with known local annotations of the presence or absence of TLS at the tile level, and the first neural network and the second neural network are trained using training sets of tissue images with known global annotations of the presence or absence of TLS at the tissue image level. Process 300 performs a processing loop (blocks 304-314) for each training image to generate a set of feature vectors from the tiles and predict the TLS status of the image. At block 306, process 300 detects regions of interest (ROIs) within the training images. In one embodiment, process 300 detects ROIs as described above in block 104 of FIG. 1. At block 308, process 300 tiles the ROIs into a set of tiles. In one embodiment, process 300 tiles the ROIs as described above in block 106 of FIG. 1. At block 310, process 300 extracts feature vectors from the tiles.In one embodiment, process 300 extracts feature vectors as described in block 108 of FIG. 1 above. For example, in one embodiment, process 300 uses a ResNet-50 convolutional neural network to determine a feature vector for each tile of the tiled segmentation image, as described in FIG. 1 above. In one embodiment, process 300 generates a set of feature vectors for training images. Additionally, process 300 can perform data augmentation during training of the method to improve generalization error. This data augmentation can be performed by applying various transformations to the tiles, such as rotating, translating, flipping, cropping, blurring, adding noise to the image, changing the intensity of specific colors, or changing the contrast. In one embodiment, process 300 determines tile scores using a first neural network as described in block 204 of FIG. 2 above. In one embodiment, process 300 continues to sort the tiles by tile score as described in block 206 of FIG. 2 above. In one embodiment, process 300 proceeds to select N (the highest tile score) and M (the lowest tile score), as described in block 208 of FIG. 2 above. In one embodiment, process 300 proceeds to apply a second neural network to N, M, and the tiles, as described in block 210 of FIG. 2 above. At block 312, process 300 uses the extracted feature vectors and a machine learning model to predict the TLS status (e.g., the presence, amount, or location of TLS) in each tissue image. In one embodiment, process 300 uses the extracted feature vectors and a machine learning model, as described in block 110 of FIG. 1 above, to predict the TLS status (e.g., the presence, amount, or location of TLS) in each tissue image, and outputs a TLS global score, as described in block 212 of FIG. 2 above, and / or outputs the TLS location in the tissue image, as described in block 214 of FIG. 2 above. The process loop ends at 314.

[0109] To determine the validity of the training, process 300 evaluates whether the predictions of TLS states by the machine learning model and the global and local annotations of TLS states in the training dataset have converged in block 316. If they have converged, process 300 validates the machine learning model in block 318. If they have not converged, process 300 adjusts the machine learning model in block 320 and re-executes the processing loop (blocks 304-314).

[0110] In FIG. 3 , process 300 trained a machine learning model used to classify images. The accuracy of the machine learning model can be evaluated by validating the machine learning model using a training set of images as input and calculating one or more labels. The validation process can be performed by receiving a validation image set and the trained machine learning model; and processing the validation image set. In one embodiment, the validation image set is the same as the training set. In another embodiment, the validation set can be different from the training image set. For example, in an embodiment, a labeled image set for a particular type of image (e.g., tissue images) is used, with some images selected for use in training the model, and other images from this set can be used to validate the trained model. In one embodiment, the model is a machine learning model, such as the MLP model described above.

[0111] In some embodiments of the computer-implemented method provided herein, each tile includes a plurality of pixels. In some embodiments, the method further comprises detecting a pixel-based segmentation of TLS within the tile. Such determining step may include applying a machine learning model to each tile of the tile set, the model being trained using a training set of tiles comprising known pixel-based TLS segmentation masks within the tile. As used herein, "TLS segmentation" or "segmentation of TLS" refers to the localization of TLS in a given region. The segmentation of TLS in a given region (e.g., a tile, a tissue image) may be represented by the presence or absence of TLS in unit areas (e.g., pixels) tiled within the given region. In some embodiments, the model assigns a pixel score to each pixel of each tile of the set of tiles to determine the pixel-based segmentation of TLS within the tile, the pixel score representing the likelihood that the pixel contains a TLS.

[0112] For example, as shown in Figure 9A, a training set of images can have local manual annotations of tiles and their segmentation masks from H&E-stained tissue images based on a pathologist's manual annotation of TLS locations marked with blue lines. As shown in Figure 9B, a machine learning model can be trained to predict TLS segmentation by processing the training set of images with known pixel-based segmentation masks and comparing the output with the annotations.

[0113] FIG. 7 is an exemplary flow diagram of a process 700 for detecting pixel-based TLS segmentation in an image using a machine learning model, according to an embodiment of the present disclosure. At block 702, process 700 receives an image, one or more machine learning (ML) models, and optionally other inputs. In some embodiments, the image is a digitized tissue image of a sample obtained from a subject, such as a digitized WSI. At block 704, process 700 detects a region of interest (ROI) in the image, as described in block 104 of FIG. 1. At block 706, process 700 tiles the ROI into a set of tiles, as described in block 106 of FIG. 1. At block 708, process 700 extracts a pixel-based segmentation mask from each tile. At block 710, process 700 determines a pixel score for each pixel in the tile. At block 712, process 700 detects pixel-based segmentation in the tile or image using a third neural network. In some embodiments, determining pixel scores and / or detecting pixel-based segmentations within a tile or image is performed using a third neural network trained using a training set of images with local annotations of pixel-based segmentation masks, as described herein and, for example, in FIG. 8. In one embodiment, the third neural network is a semantic segmentation neural network, such as the U-NET semantic segmentation neural network, or another type of semantic segmentation neural network. In one embodiment, a semantic segmentation neural network that does not rely on a particular distribution of intensity histograms can be used. Furthermore, the use of such neural networks enables image segmentation that takes into account multi-channel images, such as red-green-blue (RGB) images. Thus, segmentation can utilize image semantics rather than just relying on the histogram of pixel intensities.At block 714, the process 700 outputs a pixel-based TLS segmentation of the tissue image.

[0114] FIG. 8 shows a flow diagram of one embodiment of a process 800 for training and validating a machine learning model for detecting TLS segmentations in tissue images or subjects. In one embodiment, the machine learning model trained includes one or more separate models used in the segmentation detection process described in FIG. 7. In one embodiment, the machine learning model trained is the third neural network described in FIG. 7. In FIG. 8, process 800 begins at block 802 by receiving a training set of tissue images, local and global annotations, a machine learning model, and optionally other inputs. In one embodiment, process 800 receives images, local manual annotations, the third neural network used in FIG. 7, and optionally other inputs described in FIG. 7. In one embodiment, the third neural network is a semantic segmentation neural network, such as the U-NET semantic segmentation neural network. In one embodiment, the third neural network is trained using a training set of tissue images with known local manual annotations of TLS locations (i.e., TLS segmentations) in the images, as shown in FIG. 9A. Process 800 performs a processing loop (blocks 804-816) for each training image to generate a set of segmentation masks from the tiles and predict a pixel-based TLS segmentation of the image. At block 806, process 800 detects regions of interest (ROIs) in the training images. In one embodiment, process 800 detects the ROIs as described in block 704 of FIG. 7 above. At block 808, process 800 tiles the ROIs into a set of tiles. In one embodiment, process 800 tiles the ROIs as described in block 706 of FIG. 7 above. At block 810, process 800 extracts segmentation masks from the tiles. In one embodiment, process 800 extracts the segmentation masks as described in block 708 of FIG. 7 above.In one embodiment, process 800 generates a set of segmentation masks for the training images. At block 812, process 800 detects pixel scores using the extracted segmentation masks and a third neural network. In one embodiment, process 800 detects pixel scores using the extracted segmentation masks and a third neural network, as described in block 712 of FIG. 7. At block 814, process 800 predicts a pixel-based TLS segmentation within the image. In one embodiment, process 800 predicts a pixel-based TLS segmentation within the image, as described in block 714 of FIG. 7. The process loop ends at 816.

[0115] To determine the validity of the training, at block 818, process 800 evaluates whether the machine learning model's predictions of TLS segmentation and the local manual annotations of TLS states in the training dataset have converged. If so, at block 820, process 800 validates the machine learning model. If not, at block 822, process 800 adjusts the machine learning model and re-runs the processing loop (blocks 804-816). The validation process can be performed by receiving a validation image set and the trained machine learning model; and processing the validation image set as described above.

[0116] In some embodiments, a trained feature extractor can be realized after a certain number of epochs of training. In some embodiments, training is performed until the accuracy is at or near 1 (or 100%), or the AUC is at or near 1 (or 100%), or the loss is near zero-padded. In some embodiments, during training of a feature extractor with a contrastive loss, useful metrics may not be abundantly available. Therefore, one of the available metrics of a downstream task, such as AUC, can be monitored to see how the feature extractor is performing. As an example, a feature extractor trained for a particular epoch can be used to train a downstream weakly supervised task and evaluate performance. If such additional training can improve downstream performance, such additional training may be justified.

[0117] In some embodiments of the computer-implemented method provided herein, the sample is cancer. The method provided herein can be used for cancer samples of heterogeneous primary sites, disease stages, pathological types, subject profiles, or clinical / treatment courses. The method provided herein can also be used for cancer samples of specific primary sites, conditions, pathological types, subject profiles, or clinical / treatment courses.

[0118] For example, the methods provided herein can be used for any cancer occurring in an organ or tissue within the body. In some embodiments, the cancer is selected from the group consisting of lung cancer, sarcoma, bladder cancer, colon cancer, ovarian cancer, pancreatic cancer, and melanoma. The cancer sample can be derived from the subject at any time relevant to the diagnosis of cancer. For example, the sample can be obtained from the subject before or after a pathological diagnosis of cancer. The sample can be obtained from the subject before or after treatment for cancer (e.g., immunotherapy, chemotherapy, surgery, radiation).

[0119] In some embodiments of the computer-implemented methods provided herein, the training set of tiles and / or the training set of tissue images are digitized images of tissue sections of heterogeneous cancers (e.g., lung cancer, sarcoma, bladder cancer, colon cancer, ovarian cancer, pancreatic cancer, melanoma).

[0120] In some embodiments, the tissue image is a digitized whole slide image (WSI). In some embodiments, the digitized tissue image is an image of a tissue section of a sample stained with a dye to visualize the underlying tissue structure. The dye can be hematoxylin and eosin (H&E). Other common stains that can be used to visualize tissue structures in the input image include, for example, Masson's Trichome stain, Periodic Acid-Schiff stain, Prussian Blue stain, Gomori's Trichome stain, Alcian Blue stain, or Ziehl-Neelsen stain.

[0121] In some embodiments, the machine learning model is a Deep Multiple Instance Learning model. In some embodiments, the machine learning model is a Weldon model. In some embodiments, the machine learning model is applied to an entire group of tiles. In some embodiments, the machine learning model is applied to a subset of tiles. The training images can include digital images of tissue sections from multiple control subjects. In some cases, the training images lack local annotations. The training images can include images associated with one or more global labels that indicate one or more TLS features of the patient from whom the sample was derived.

[0122] The machine learning model described herein can identify TLS locations in tissue images. TLS locations can be identified, for example, by selecting a cohort of tiles with the highest M score and the lowest N score. The highest and lowest cohorts of tiles identified by the model as having the best correlation with the presence or absence of TLS can be analyzed by a pathologist to determine TLS features within the tiles. For example, in some embodiments, TLS features can be determined by analyzing a cohort of tiles with M scores in the top 20%, e.g., top 15%, top 10%, top 5%, top 2%, top 1%, etc., and / or N scores in the bottom 20%, e.g., bottom 15%, bottom 10%, bottom 5%, bottom 2%, bottom 1%, etc., of all tiles evaluated by the model.

[0123] TLS features include one or more of the following features and combinations thereof: structural features found in secondary lymphoid organs (SLOs) (e.g., lymph nodes, tonsils, spleen, Peyer's patches, or mucosa-associated lymphoid tissue); non-inclusion, non-lymphoid tissue, such as Peyer's patches and pre-existing lymphoid follicles; lymphocyte aggregates; B-cell follicles with activated and replicating B-cell germinal centers surrounded by T-cell sites; Mature dendritic cells in the T cell zone and / or follicular dendritic cells (i.e., follicles) in the B cell zone, and A heterogeneous population of cells and structures, including discrete B cell zones, T cell zones, a marginal zone with activated macrophages and dendritic cells, a reticular fibroblast (RFC) network (or RFC-like stromal network), vasculature permissive for immune cell extravasation (e.g., high endothelial veins are vessels that express peripheral nodal addressin (PNAd) and are specialized for the extravasation of circulating immune cells), dendritic cell-lysosome-associated membrane protein (DC-LAMP), and at least one of dendritic cells.

[0124] In some embodiments, the tissue image for quantifying the presence of TLS includes one or more, two or more, three or more, four or more, five or more, or six of the aforementioned TLS features. In some embodiments, the tissue image is a whole slide image. In other embodiments, the tissue image is a section of the whole slide image, e.g., a tile derived from the whole slide image.

[0125] The presence or absence of TLS features described herein can be determined in an image obtained from a subject's tissue. The image can be, for example, a whole slide image (WSI), or a portion thereof, e.g., a tile derived from the WSI. In an exemplary embodiment, the tissue is derived from a biopsy obtained from the subject, e.g., a cancer biopsy. Tissue sources suitable for biopsy are known in the art and include, but are not limited to, tissue samples obtained from needle biopsy, endoscopic biopsy, or surgical biopsy. In an exemplary embodiment, the imaging device is derived from a thoracocentesis biopsy, thoracoscopic biopsy, open thoracostomy biopsy, parathoracic needle biopsy, laparoscopic biopsy, or open thoracostomy biopsy.

[0126] Tissue sections can be processed for image analysis using any suitable method and stain for histopathological analysis, for example, hematoxylin and eosin, alkaline phosphatase, methylene blue, Hoechst stain, and / or 4',6-diamino-2-phenylindole (DAPI).

[0127] The classification algorithm can calculate a TLS score for the subject, which indicates the TLS status. The TLS score can be categorical, e.g., the presence or absence of TLS. The TLS score can also be a continuous probability score. The continuous TLS score for a subject, e.g., a cancer subject, can be plotted against scores obtained from multiple subjects with known TLS status to determine the subject's TLS status.

[0128] II. Computing Systems and Machine-Readable Media As shown in FIG. 10, computer system 1000, which is one form of data processing system, includes a microprocessor 1005 and a bus 1003 coupled to a ROM (Read Only Memory) 1007, as well as volatile RAM 1009 and non-volatile memory 1013. Microprocessor 1005 may include one or more CPUs, GPUs, special purpose processors, and / or combinations thereof. Microprocessor 1005 may communicate with a cache 1004 and may retrieve and execute instructions from memories 1007, 1009, and 1013 to perform the operations described above. Bus 1003 interconnects these various components and also interconnects these components 1005, 1007, 1009, and 1013 to peripheral devices such as a display controller and display device 1015 and input / output (I / O) devices 1011, which may be a mouse, keyboard, modem, network interface, printer, and other devices known in the art. Input / output devices 1011 are typically coupled to the system through an input / output controller 1017. Volatile RAM (Random Access Memory) 1009 is typically implemented as dynamic RAM (DRAM) and requires continuous power to refresh or maintain the data in the memory.

[0129] Non-volatile memory 1013 may be, for example, a magnetic hard drive or a magnetic optical drive or an optical drive or a DVD RAM or flash memory or other type of memory system that retains data (e.g., large amounts of data) even after power is removed from the system. Typically, non-volatile memory 1013 is also random access memory, although this is not required. While FIG. 10 illustrates non-volatile memory 1013 as a local device directly coupled to the rest of the components in the data processing system, it will be understood that the present invention may utilize non-volatile memory that is remote from the system, such as a network storage device coupled to the data processing system via a network interface, such as a modem, an Ethernet interface, or a wireless network. Bus 1003 may include one or more buses connected to each other through various bridges, controllers, and / or adapters, as is well known in the art.

[0130] Portions of the foregoing may be implemented using logic circuitry, such as dedicated logic circuitry, or in a microcontroller or other form of processing core that executes program code instructions. Accordingly, the processes taught by the above discussion may be performed in program code, such as machine-executable instructions, that cause a machine that executes those instructions to perform a particular function. In this context, a “machine” may be a machine that translates intermediate-form (or “abstract”) instructions into processor-specific instructions (e.g., an abstract execution environment, such as a “virtual machine” (e.g., a Java Virtual Machine), an interpreter, a common language runtime, or a high-level language virtual machine), and / or an electronic circuit (e.g., a “logic circuit” implemented with transistors) located on a semiconductor chip that is designed to execute instructions, such as a general-purpose processor and / or a special-purpose processor. The processes taught by the above discussion may also be performed by (in place of or in combination with) electronic circuitry designed to perform processes (or portions thereof) without the execution of program code.

[0131] The present invention also relates to apparatus for performing the operations described herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such computer program may be stored on a computer-readable storage medium such as, but not limited to, any type of disk, including floppy disks, optical disks, CD-ROMs, and magneto-optical disks, read-only memory (ROM), RAM, EPROM, EEPROM, magnetic or optical cards, or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0132] A machine-readable medium includes any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer). For example, machine-readable media include read-only memory ("ROM"); random-access memory ("RAM"); magnetic disk storage media; optical storage media; flash memory devices; and the like.

[0133] The article of manufacture can be used to store program code. The article of manufacture storing the program code can be embodied in, but is not limited to, one or more memories (e.g., one or more flash memories, random access memories (static, dynamic, or other)), optical disks, CD-ROMs, DVD-ROMs, EPROMs, EEPROMs, magnetic or optical cards, or other types of machine-readable media suitable for storing electronic instructions. The program code can also be downloaded from a remote computer (e.g., a server) to a requesting computer (e.g., a client) by a data signal embodied in a propagation medium (e.g., via a communications link (e.g., network connection)).

[0134] The foregoing detailed descriptions are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the tools used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. The operations require physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0135] It should be noted, however, that all of these and similar terms should be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. As is clear from the above discussion, unless specifically stated otherwise, throughout this specification discussions utilizing terms such as "segmentation," "tiling," "receiving," "computing," "extracting," "processing," "applying," "augmenting," "normalizing," "pre-training," "sorting," "selecting," "aggregating," "sorting," and the like will be understood to refer to operations and processes of a computer system, or similar electronic computing device, that manipulate and transform data represented as physical (electronic) quantities in the registers and memory of the computer system into other data similarly represented as physical quantities in the memory or registers of the computer system, or other such information storage, transmission, or display device.

[0136] The processes and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the described operations. The required structure for a variety of these systems will appear from the description herein. Further, the present invention is not described with reference to any particular programming language. It will be understood that a variety of programming languages ​​can be used to implement the teachings of the invention as described herein.

[0137] III.Product In some embodiments, provided herein are products capable of detecting TLS status (e.g., the presence, quantity, or location of TLS) in a subject, a tissue image, a tile within a tissue image, or a pixel within a tile. In some embodiments, the products are attached to a scanner. In some embodiments, the scanner is capable of scanning pathology slides, such as H&E slides. In some embodiments, the products are particularly useful for medical facilities, clinics, or providers, including those without expertise in cancer pathology, including those diagnosing or prognosing cancer (e.g., lung cancer, sarcoma, bladder cancer, colon cancer, ovarian cancer, pancreatic cancer, and melanoma) or diagnosing TLS. In some embodiments, the products are useful for identifying personalized or targeted therapy options for a subject's cancer.

[0138] IV. General considerations The articles "a" and "an" are used herein to refer to one or to more than one (i.e., to at least one) of the grammatical object of the article. By way of example, "an element" means one element or more than one element, e.g., a plurality of elements.

[0139] As used herein, the term "comprising" is used to mean, and is used interchangeably with, the phrase "including but not limited to." The term "comprising" does not necessarily imply that there must be additional elements beyond the listed elements.

[0140] The term "about" or "approximately," when referring to a numerical value or numerical range, means that the referenced numerical value or numerical range is an approximation within experimental variability (or within statistical experimental error); thus, the numerical value or numerical range may vary, for example, by 1% to 20% of the stated numerical value or numerical range. In some embodiments, "about" refers to a value within 20% of the stated value. In more preferred embodiments, "about" refers to a value within 10% of the stated value. In even more preferred embodiments, "about" refers to a value within 1% of the stated value.

[0141] Unless otherwise indicated, all numerical values ​​expressing quantities of ingredients, properties such as molecular weights, reaction conditions, and the like used in the specification and claims are understood to be modified in all instances by the term "about." Accordingly, unless otherwise indicated, the numerical properties set forth in the specification and appended claims are approximations that may vary depending upon the desired properties sought to be obtained in embodiments of the invention. Notwithstanding that the numerical ranges and parameters setting forth the broad scope of the invention are approximations, the numerical values ​​set forth in the specific examples are reported as precisely as possible. However, any numerical value inherently contains certain errors necessarily resulting from error found in its respective measurement.

[0142] The term "at least" before a number or series of numbers is understood to include the number immediately adjacent to the term "at least," and all subsequent numbers or integers that can be logically included, as is clear from the context. When "at least" is before a series of numbers or a range, it is understood that "at least" can modify each number in the series or range.

[0143] As used herein, "not greater than" or "less than" is to be understood from the context as the value adjacent to the phrase and the logical lower value or integer, up to zero (where negative values ​​are not possible). When "not greater than" precedes a series of numbers or a range, it is understood that "not greater than" can modify each number in the series or range.

[0144] As used herein, "up to," as in "up to 10," is understood in the context of non-negative integers to include up to and including 10, i.e., 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10.

[0145] When a range of values ​​is provided, it is understood that each intervening value (e.g., to one-tenth of the unit of the lower limit unless the context clearly dictates otherwise) between the upper and lower limit of that range, and any other stated or intervening value within that stated range, is included in the invention. The upper and lower limits of these smaller ranges may independently be included in the smaller ranges and are also included in the invention, subject to any specifically excluded limit in the stated range. When the stated range includes one or both of the limits, ranges excluding either or both of those included limits are also included in the invention. [Example]

[0146] Example 1: Detection of TLS in a cohort of cancer patients The machine learning model for predicting TLS status was cross-validated using a pan-cancer cohort of 289 H&E-stained whole slide images (WSIs) from 289 patients (one slide from each patient) from the Bergonier Institute, in which TLS status was manually inspected and annotated by an expert pathologist. The cohort included 113 patients with non-small cell lung cancer (NSCLC) (39.1%), 45 patients with sarcoma (15.6%), 30 patients with bladder cancer (10.4%), 26 patients with colorectal cancer (9.0%), 10 patients with kidney cancer (3.7%), 10 patients with head and neck cancer (3.7%), 9 patients with ovarian cancer (3.1%), 7 patients with liver cancer, 5 patients with breast cancer, 5 patients with gastrointestinal stromal tumor cancer, 4 patients with cervical cancer, 4 patients with endometrial cancer, 4 patients with gastric cancer, 3 patients with thyroid cancer, 2 patients with bile duct cancer, 2 patients with prostate cancer, 2 patients with anal cancer, 2 patients with vulvar cancer, 1 patient with skin cancer, 1 patient with parotid gland cancer, 1 patient with gastrointestinal cancer, 1 patient with penile cancer, 1 patient with cancer of unknown primary origin, and 1 patient with esophageal cancer. A deep learning (DL) model was trained on WSI to predict TLS status (presence or absence of TLS) at the patient level.

[0147] The models were evaluated using five-fold cross-validation. The best-performing DL model provided two key components—a predictive score for the presence of TLS in a small region of the WSI measuring 112 μm × 112 μm (i.e., one tile) (tile score) followed by aggregation at the patient level—with a ROC AUC score of 0.917 (standard deviation 0.036) (Figure 4). The trained DL model provided a subject's TLS status with a sensitivity of 90%, a specificity of 68%, a sensitivity of 85%, a specificity of 85%, a sensitivity of 80%, and a specificity of 87%.

[0148] The transferability of the DL model was evaluated using a validation cohort (PEMBROSARC) consisting of 236 sarcoma WSIs (subjects). In this cohort, 47 WSIs (subjects) (19.9% ​​of the total cohort) were TLS-positive (i.e., TLS was present). The PEMBROSARC trial was the first clinical trial to use TLS status as an inclusion criterion (Italiano A. et al. 2022 Nat. Med. 28:1199-1206). The DL model detected TLS status in subjects with an ROC AUC score of 0.89, demonstrating a sensitivity of 90% and a specificity of 64%, a sensitivity of 85% and a specificity of 88%, and a sensitivity of 80% and a specificity of 86%. In summary, this study demonstrated the predictive power of the DL model for detecting TLS status in subjects based on images of H&E-stained tissue slides. The DL model presented in this paper can be implemented in pathology laboratories and medical facilities as an efficient pre-screening tool for TLS status in subjects.

[0149] The foregoing discussion describes only some exemplary embodiments of the present invention, and those skilled in the art will readily recognize from this discussion, the accompanying drawings, and the following claims that various modifications are possible without departing from the spirit and scope of the present invention.

Claims

1. 1. A computer-implemented method for detecting the presence or absence of tertiary lymphoid structures (TLS) in a subject, comprising: receiving a digitized histology image of a sample obtained from the subject; tiling the tissue image into a set of tiles; extracting a plurality of feature vectors from each of the tiles, each feature of one or more of the feature vectors representing a local descriptor of the tile; classifying the tissue image for a TLS status using at least the plurality of feature vectors and a classification model trained on a training set of tissue images with known TLS annotations, the TLS status indicating the presence or absence of TLS in the subject; and 11. A computer-implemented method comprising:

2. The classification is applying a first neural network to the one or more feature vectors of the plurality of feature vectors, the first neural network assigning a tile score to each tile of the set of tiles based on the one or more feature vectors of the plurality of feature vectors, the tile score representing a likelihood that the tile contains a TLS; applying a second neural network to each tile, the second neural network aggregating a subset of the tile scores for the set of tiles to determine the TLS state in the tissue image; Including, the first neural network is trained using a training set of tissue images containing known local annotations of the presence or absence of TLS at the tile level; the first neural network and the second neural network are trained using a training set of tissue images containing known global annotations regarding the presence or absence of TLS at the tissue image level; The computer-implemented method of claim 1 .

3. the first neural network includes a 1D convolutional layer; The computer-implemented method of claim 2 .

4. the second neural network includes a multi-layer perceptual model; 4. A computer-implemented method according to claim 2 or 3.

5. detecting one or more locations in the tissue image where TLS is present; A computer-implemented method according to any one of claims 1 to 4.

6. each said tile comprising a plurality of pixels; The method further comprises: receiving a digitized histology image of a sample obtained from the subject; tiling the tissue image into a set of tiles; extracting a segmentation mask from each of the tiles; applying a third neural network to each tile, the third neural network using the extracted segmentation mask to find pixel scores; Detecting pixel-based TLS segmentation within the tiles or the image, wherein the third neural network is trained using a training set of tiles with known pixel-based TLS segmentation masks within the tiles; The computer-implemented method of any one of claims 1 to 5, comprising:

7. the third neural network assigns a pixel score to each pixel of each tile in the set of tiles to determine the pixel-based segmentation of TLS within the tile, the pixel score representing the likelihood that the pixel comprises a TLS; The computer-implemented method of claim 6.

8. The third neural network is a U-NET semantic segmentation neural network.

8. A computer-implemented method according to claim 6 or 7.

9. The extracting is performed by a ResNet50 neural network and / or a Momentum Contrast (MoCo) or Momentum Contrast v2 (MoCo v2) algorithm. The method according to any one of claims 1 to 8.

10. the sample is cancer; A computer-implemented method according to any one of claims 1 to 9.

11. the cancer is selected from the group consisting of lung cancer, sarcoma, bladder cancer, colorectal cancer, ovarian cancer, pancreatic cancer, melanoma, kidney cancer, head and neck cancer, liver cancer, breast cancer, gastrointestinal stromal tumor cancer, cervical cancer, endometrial cancer, stomach cancer, thyroid cancer, bile duct cancer, prostate cancer, anal cancer, vulvar cancer, skin cancer, parotid gland cancer, gastrointestinal cancer, penile cancer, esophageal cancer, and cancer of unknown primary origin; The computer-implemented method of claim 10.

12. The training set of tiles and / or the training set of tissue images are digitized images of tissue sections of heterogeneous cancers; A computer-implemented method according to any one of claims 1 to 11.

13. The tissue image is a digitized whole slide image (WSI). A computer-implemented method according to any one of claims 1 to 12.

14. the digitized tissue image is a digitized image of a tissue section stained with a dye; A computer-implemented method according to any one of claims 1 to 13.

15. the dye is hemotoxylin and eosin (H&E); 15. The computer-implemented method of claim 14.

16. The classification is picking the tile with the highest TLS tile score; and picking the tile with the lowest TLS tile score; further comprising sorting the set of tiles by A computer-implemented method according to any one of claims 2 to 15.

17. repeating all steps of claim 1 for a plurality of tissue images; detecting the presence or absence of TLS in the subject by processing TLS status of a plurality of tissue images; 17. The computer-implemented method of any one of claims 1 to 16, comprising:

18. the tissue image lacks local annotation of histopathological features; A computer-implemented method according to any one of claims 1 to 17.

19. each tile of said set of tiles comprises approximately 224 x 224 pixels; A computer-implemented method according to any one of claims 1 to 18.

20. 1. A machine-readable medium having executable instructions for causing one or more processing units to perform a method for detecting the presence or absence of tertiary lymphoid structures (TLS) in a subject, the method comprising: The method comprises: receiving a digitized histology image of a sample obtained from the subject; tiling the tissue image into a set of tiles; extracting a plurality of feature vectors from each of the tiles, each feature of one or more of the feature vectors representing a local descriptor of the tile; classifying the tissue image for TLS status using at least the plurality of feature vectors and a classification model trained on a training set of images with known TLS annotations, the TLS status indicating the presence or absence of TLS in the subject; and 1. A machine-readable medium comprising:

21. The classification is applying a first neural network to the one or more feature vectors of the plurality of feature vectors, the first neural network assigning a tile score to each tile of the set of tiles based on the one or more feature vectors of the plurality of feature vectors, the tile score representing a likelihood that the tile contains a TLS; applying a second neural network to each tile, the second neural network aggregating a subset of the tile scores for the set of tiles to determine the TLS state in the tissue image; Including, the first neural network is trained using a training set of tissue images containing known local annotations of the presence or absence of TLS at the tile level; the first neural network and the second neural network are trained using a training set of tissue images containing known global annotations regarding the presence or absence of TLS at the tissue image level; 21. The machine-readable medium of claim 20.

22. the first neural network includes a 1D convolutional layer; 22. The machine-readable medium of claim 21.

23. the second neural network includes a multi-layer perceptual model; 23. The machine-readable medium of claim 21 or 22.

24. detecting one or more locations in the tissue image where TLS is present; The machine-readable medium of any one of claims 20 to 23.

25. each said tile comprising a plurality of pixels; The method further comprises: receiving a digitized histology image of a sample obtained from the subject; tiling the tissue image into a set of tiles; extracting a segmentation mask from each of the tiles; applying a third neural network to each tile, the third neural network using the extracted segmentation mask to find pixel scores; Detecting pixel-based TLS segmentation within the tiles or the image, wherein the third neural network is trained using a training set of tiles with known pixel-based TLS segmentation masks within the tiles; 25. The machine-readable medium of any one of claims 20 to 24, comprising:

26. the third neural network assigns a pixel score to each pixel of each tile in the set of tiles to determine the pixel-based segmentation of TLS within the tile, the pixel score representing the likelihood that the pixel comprises a TLS; 26. The machine-readable medium of claim 25.

27. The third neural network is a U-NET semantic segmentation neural network.

27. The machine-readable medium of claim 25 or 26.

28. The extracting is performed by a ResNet50 neural network and / or a Momentum Contrast (MoCo) or Momentum Contrast v2 (MoCo v2) algorithm. The method according to any one of claims 1 to 8.

29. the sample is cancer; The machine-readable medium of any one of claims 20 to 28.

30. the cancer is selected from the group consisting of lung cancer, sarcoma, bladder cancer, colorectal cancer, ovarian cancer, pancreatic cancer, melanoma, kidney cancer, head and neck cancer, liver cancer, breast cancer, gastrointestinal stromal tumor cancer, cervical cancer, endometrial cancer, stomach cancer, thyroid cancer, bile duct cancer, prostate cancer, anal cancer, vulvar cancer, skin cancer, parotid gland cancer, gastrointestinal cancer, penile cancer, esophageal cancer, and cancer of unknown primary origin; 30. The machine-readable medium of claim 29.

31. The training set of tiles and / or the training set of tissue images are digitized images of tissue sections of heterogeneous cancers; The machine-readable medium of any one of claims 20 to 30.

32. The tissue image is a digitized whole slide image (WSI). The machine-readable medium of any one of claims 20 to 31.

33. the digitized tissue image is a digitized image of a tissue section stained with a dye; The machine-readable medium of any one of claims 1 to 32.

34. the dye is hemotoxylin and eosin (H&E); 34. The machine-readable medium of claim 33.

35. The classification is picking the tile with the highest TLS tile score; and picking the tile with the lowest TLS tile score; further comprising sorting the set of tiles by The machine-readable medium of any one of claims 21 to 34.

36. repeating all steps of claim 17 for a plurality of tissue images; detecting the presence or absence of TLS in the subject by processing TLS status of a plurality of tissue images; 36. The machine-readable medium of any one of claims 19 to 35, comprising:

37. the tissue image lacks local annotation of histopathological features; The machine-readable medium of any one of claims 19 to 36.

38. each tile of said set of tiles comprises approximately 224 x 224 pixels; 38. The method according to any one of claims 19 to 37.

Images