Systems and methods for biomarker detection

Abstract

A method for detecting a biomarker with a machine learning-based detection system includes: identifying, with the detection system, a plurality of tiles corresponding to whole-slide image data of a tissue sample; generating, with the detection system, tile-level embedding data based on the plurality of tiles; generating, with the detection system, cell-level embedding data based on the plurality of tiles; and generating, with the detection system, a slide-level prediction based on the tile-level embedding data and the cell-level embedding data, wherein the slide-level prediction indicates the presence or absence of the biomarker in the tissue sample.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. Provisional Patent Application No. 63 / 488,253, filed March 3, 2023 ("EXPLAINABLE PLUG AND PLAY FOR FEATURE REPRESENTATION IN HISTOPATHOLOGY"), U.S. Provisional Patent Application No. 63 / 506,866, filed June 8, 2023 ("CELL OF ORIGIN PREDICTION FOR DIFFUSE LARGE B CELL LYMPHOMAS"), and U.S. Provisional Patent Application No. 63 / 507,704, filed June 12, 2023 ("CELL OF ORIGIN PREDICTION FOR DIFFUSE LARGE B CELL LYMPHOMAS"). LYMPHOMAS"), and U.S. Provisional Application No. 63 / 515,655, filed July 26, 2023 ("DEEP LEARNING BASED WHOLE SLIDE IMAGE ANALYSIS FOR IDENTIFICATION OF MYC-DRIVEN HIGH-GRADE B-CELL LYMPHOMA"), the entire contents of which are incorporated herein by reference.

[0002] Field Aspects of some embodiments of the present disclosure relate to systems and methods for biomarker detection. [Background technology]

[0003] background Various forms of cancer are one of the leading causes of death worldwide. Early diagnosis plays a key role in achieving optimal treatment outcomes for people with cancer. Identification of cancer biomarkers allows for finer tumor classification, resulting in better diagnosis and prognosis and enabling more informed treatment decisions. For many cancers, clinically viable and reliable biomarkers have yet to be identified, and biomarker identification techniques have limitations that may limit their clinical use. Meanwhile, histological analysis of hematoxylin and eosin (H&E)-stained pathology slides is widely used in cancer diagnosis and prognosis. However, visual inspection of H&E-stained slides is insufficient for classifying some tumors because the morphological differences that can distinguish between subtypes exceed the limits of human detection.

[0004] The above information disclosed in this Background section is intended to enhance understanding of the background art only, and therefore, the information described in this Background section does not necessarily constitute prior art. Summary of the Invention

[0005] summary Aspects of some embodiments of the present disclosure are directed to a biomarker detection system that extracts both tile-level and cell-level embedding data from a WSI and combines the embeddings to simultaneously capture histological and cytological features, improving model performance and explainability, such that the detection system can make more accurate predictions regarding the presence of specific biomarkers in samples represented by the WSI.

[0006] According to some embodiments of the present disclosure, there is provided a method for detecting a biomarker by a machine learning-based detection system, the method including: identifying, by the detection system, a plurality of tiles corresponding to whole-slide image data of a tissue sample; generating, by the detection system, tile-level embedding data based on the plurality of tiles; generating, by the detection system, cell-level embedding data based on the plurality of tiles; and generating, by the detection system, a slide-level prediction based on the tile-level embedding data and the cell-level embedding data, wherein the slide-level prediction indicates the presence or absence of a biomarker in the tissue sample.

[0007] In some embodiments, identifying the plurality of tiles includes receiving, by a detection system, whole-slide image data corresponding to the tissue sample, and extracting, by the detection system, the plurality of tiles from the whole-slide image data.

[0008] In some embodiments, the whole slide image data comprises at least one of a digitized image or a region of interest (ROI) map of the patient's tissue sample stained with hematoxylin and eosin (H&E) dye.

[0009] In some embodiments, the method further includes performing, by the detection system, stain normalization based on the plurality of tiles to generate a plurality of normalized tiles, and generating the tile-level embedding data includes generating, by the detection system, the tile-level embedding data from the plurality of normalized tiles.

[0010] In some embodiments, performing stain normalization includes generating a plurality of normalized tiles based on the plurality of tiles according to a first model of the detection system.

[0011] In some embodiments, the first model comprises a fully convolutional neural network.

[0012] In some embodiments, generating the tile-level embedding data includes generating, by a second model of the detection system, a plurality of tile-level feature vectors based on the plurality of tiles, the number of tile-level feature vectors corresponding to the number of tiles.

[0013] In some embodiments, the second model includes at least one of a residual network (ResNet) or a transformer network, and the number of tile-level feature vectors is the same as the number of tiles.

[0014] In some embodiments, generating the cell-level embedding data includes extracting, by a detection system, a plurality of cell patches based on the plurality of tiles; and generating, by the detection system, the cell-level embedding data based on the plurality of cell patches.

[0015] In some embodiments, extracting the plurality of cell patches includes detecting a plurality of cells in each of the plurality of tiles by a segmentation model of the detection system, and generating a plurality of cell patches by the detection system based on the plurality of tiles and the plurality of cells in each of the plurality of tiles, wherein one cell patch of the plurality of cell patches includes a portion of one of the plurality of tiles that encompasses a single cell of the plurality of cells.

[0016] In some embodiments, extracting the plurality of cell patches includes extracting, by the detection system, the plurality of cell patches from a plurality of normalized tiles corresponding to the plurality of tiles.

[0017] In some embodiments, generating the cell-level embedding data includes generating a plurality of cell-level feature vectors based on a plurality of cell patches by a third model of the detection system, wherein the number of cell-level feature vectors corresponds to the number of tiles and the number of cell patches.

[0018] In some embodiments, the third model includes at least one of a residual network (ResNet) or a transformer network, and the number of cell-level feature vectors is the number of tiles multiplied by the number of cell patches.

[0019] In some embodiments, generating the cell-level embedding data further includes combining, by the detection system, a plurality of cell-level feature vectors to generate the cell-level embedding data, wherein the cell-level embedding data includes a plurality of embedding vectors, the number of embedding vectors corresponding to the number of tiles.

[0020] In some embodiments, one embedding vector of the plurality of embedding vectors comprises the mean of a number of cell-level feature vectors of the plurality of cell-level feature vectors and the standard deviation of the number of cell-level feature vectors of the plurality of cell-level feature vectors.

[0021] In some embodiments, the method further includes aggregating, by the detection system, the tile-level embedding data and the cell-level embedding data to generate aggregated embedding data, and generating the slide-level prediction is performed by a fourth model of the detection system and is based on the aggregated embedding data.

[0022] In some embodiments, aggregating the tile level embedding data and the cell level embedding data includes, by the detection system, concatenating the tile level embedding data and the cell level embedding data to generate aggregated embedding data, wherein the vector length of the aggregated embedding data is equal to the sum of the vector lengths of the tile level embedding data and the cell level embedding data.

[0023] In some embodiments, the fourth model includes at least one of a multi-instance learning (MIL) network, an attention-based MIL (AMIL) network, or a Transformer.

[0024] In some embodiments, the slide-level predictions include a MYC-driven high-grade B-cell lymphoma (HGBL) signature.

[0025] In some embodiments, the method further includes transmitting the slide-level prediction to a display device for display to a user.

[0026] According to some embodiments of the present disclosure, there is provided a detection system for detecting a biomarker, comprising: a processor; and a memory storing instructions that, when executed by the processor, cause the processor to: identify a plurality of tiles corresponding to whole-slide image data of a tissue sample; generate tile-level embedding data based on the plurality of tiles; generate cell-level embedding data based on the plurality of tiles; and generate a slide-level prediction based on the tile-level embedding data and the cell-level embedding data, wherein the slide-level prediction indicates the presence or absence of the biomarker in the tissue sample.

[0027] In some embodiments, identifying the plurality of tiles includes receiving whole-slide image data corresponding to a tissue sample and extracting the plurality of tiles from the whole-slide image data, wherein the whole-slide image data includes at least one of a digitized image or a region-of-interest (ROI) map of the patient's tissue sample stained with hematoxylin and eosin (H&E) dye.

[0028] In some embodiments, generating the tile-level embedding data includes generating a plurality of tile-level feature vectors based on a plurality of tiles, the number of tile-level feature vectors corresponding to the number of tiles.

[0029] In some embodiments, the detection system further includes performing stain normalization based on the plurality of tiles to generate a plurality of normalized tiles, and generating the tile-level embedding data includes generating the tile-level embedding data from the plurality of normalized tiles.

[0030] In some embodiments, generating the cellular-level embedding data includes extracting a plurality of cell patches based on the plurality of tiles, and generating the cellular-level embedding data based on the plurality of cell patches.

[0031] In some embodiments, extracting the plurality of cell patches includes detecting a plurality of cells in each of the plurality of tiles and generating a plurality of cell patches based on the plurality of tiles and the plurality of cells in each of the plurality of tiles, wherein one cell patch of the plurality of cell patches includes a portion of one of the plurality of tiles that includes a single cell of the plurality of cells.

[0032] In some embodiments, generating the cell-level embedding data includes generating a plurality of cell-level feature vectors based on a plurality of cell patches, the number of cell-level feature vectors corresponding to the number of tiles and the number of cell patches.

[0033] In some embodiments, generating the cell-level embedding data further includes combining a plurality of cell-level feature vectors to generate the cell-level embedding data, wherein the cell-level embedding data includes a plurality of embedding vectors, the number of the embedding vectors corresponding to the number of tiles, and one embedding vector of the plurality of embedding vectors includes an average of a certain number of cell-level feature vectors among the plurality of cell-level feature vectors and a standard deviation of the number of cell-level feature vectors among the plurality of cell-level feature vectors.

[0034] In some embodiments, the detection system further includes aggregating the tile-level embedding data and the cell-level embedding data to generate aggregated embedding data, and generating the slide-level prediction is performed by a fourth model of the detection system and is based on the aggregated embedding data.

[0035] In some embodiments, aggregating the tile level embedding data and the cell level embedding data comprises concatenating the tile level embedding data and the cell level embedding data to generate aggregated embedding data, wherein the vector length of the aggregated embedding data is equal to the sum of the vector lengths of the tile level embedding data and the cell level embedding data.

[0036] Non-limiting and non-exhaustive embodiments according to the present disclosure are described with reference to the following figures, in which like reference numerals refer to like parts throughout the various views unless otherwise specified. [Brief explanation of the drawings]

[0037] [Figure 1] FIG. 1 is a block diagram illustrating a biomarker detection system according to some embodiments of the present disclosure.

[0038] [Figure 2] FIG. 1 is a block diagram illustrating a whole slide image (WSI) preprocessor and its operation, according to some embodiments of the present disclosure.

[0039] [Figure 3A] FIG. 1 is a block diagram illustrating the internal structure of a tile-level analyzer and a cell-level analyzer of a biomarker detection system, according to some embodiments of the present disclosure.

[0040] [Figure 3B] FIG. 1 is a block diagram illustrating a cell patch extractor of a cell level analyzer according to some embodiments of the present disclosure.

[0041] [Figure 4] 1 is a flow chart illustrating a process for detecting biomarkers by a biomarker detection system according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0042] Detailed Description Aspects of some exemplary embodiments are described in more detail below with reference to the accompanying drawings, in which like reference numerals refer to like elements throughout. However, the present invention may be embodied in a variety of different forms and should not be construed as limited to only the embodiments illustrated herein. Rather, these embodiments are provided as examples so that this disclosure will be thorough and complete, and will fully convey the aspects and features of the present invention to those skilled in the art. Accordingly, processes, elements, and techniques that are not necessary for those skilled in the art to fully understand the aspects and features of the present invention may not be described. Unless otherwise noted, like reference numerals indicate like elements throughout the accompanying drawings and specification, and therefore, descriptions thereof will not be repeated. In the drawings, relative sizes of elements, layers, and regions may be exaggerated for clarity.

[0043] In general, testing for cancer biomarkers can improve the accuracy of tumor classification, which leads to better diagnostic, prognostic, and treatment decisions. However, biomarker testing can be time-consuming and unavailable in some situations. Visual inspection of hematoxylin and eosin (H&E)-stained pathology slides is widely used in cancer diagnosis and prognosis. However, this may be insufficient to classify some tumors because morphological differences between molecularly defined subtypes can exceed the limits of human detection.

[0044] The introduction of digital pathology (DP) has enabled the application of machine learning (ML) methods to extract otherwise inaccessible diagnostic and prognostic information from H&E-stained whole slide images (WSIs). Current ML approaches use embeddings derived from slide-level aggregations of data extracted across multiple tiles of WSIs, each containing many cells, and these often fail to capture useful information from individual cells within each tile.

[0045] Aspects of some embodiments of the present disclosure are directed to a biomarker detection system that extracts both tile-level and cell-level embeddings from a WSI and combines the embeddings to simultaneously capture histological and cytological features, improving model performance and explainability, such that the detection system can make more accurate predictions regarding the presence of specific biomarkers in samples represented by the WSI.

[0046] As an example, the detection system can be utilized to identify MYC-driven high-grade B-cell lymphoma (HGBL) based on morphology from H&E-stained WSI. HGBL is an aggressive lymphoma that often harbors MYC rearrangements (MYC-R) and possesses a molecular signature resulting from aberrant MYC activation. Identifying and classifying HGBL is challenging, and current classification systems recognize diffuse large B-cell lymphoma (DLBCL), or HGBL with MYC and BCL2 rearrangements (MYC-R / BCL2-R; double-hit; molecularly defined), as well as HGBL not otherwise specified (morphologically defined). MYC-R occurs in up to 45% of cases, and the double-hit signature (DHITsig) occurs in 54% of cases. Existing methods for molecular classification, such as fluorescence in situ hybridization (FISH), are expensive, time-consuming, and not widely available. Morphological classification is subjective and associated with high inter-reader variability.

[0047] In such examples, the detection system can be applied to WSI to extract cytological features from single cells, extract histomorphological features from larger tissue regions, and quantify high-grade morphology, characterized by monomorphic sheets of dense cells with round, intermediate-sized nuclei and finely dispersed chromatin, thus making accurate predictions regarding the presence of molecular alterations associated with HGBL, such as MYC-R biomarkers. In some examples, the detection system can be used to predict MYC gene rearrangements in Burkitt lymphoma, similarly avoiding FISH testing, or to predict gene expression signatures, such as double-hit signatures (DHITsig) or molecular high-grade (MGH) signatures in DLBCL / HGBL, avoiding expression profiling. In further examples, the detection system utilizes HPS as a biomarker to characterize / identify specific subpopulations of DLBCL / HGBL patients with common biology / pathophysiology, enabling other applications such as patient selection or stratification in clinical trials.

[0048] FIG. 1 is a block diagram illustrating a biomarker detection system 100 according to some embodiments of the present disclosure.

[0049] According to some embodiments, a biomarker detection system (also referred to as a detection system) 100 is configured to analyze both tile-level and cell-level features of given whole slide image (WSI) data 10 and generate corresponding predictions 20 regarding the presence or absence of specific biomarkers (e.g., the MYC-R biomarker). In some examples, the detection system 100 utilizes machine learning-based models to identify MYC-driven HGBL based on morphology from H&E-stained WSI, although embodiments of the present disclosure are not limited thereto. The detection system 100 may be utilized to detect or predict the presence of any suitable biomarker, such as mutations in individual genes (e.g., loss-of-function single nucleotide mutations in the TP53 gene), gene mutation signatures (e.g., an MCD signature based on the co-occurrence of MYD88 and CD79b mutations), expression levels of individual genes or proteins (e.g., MYC), gene expression profiles or signatures (e.g., cell-of-origin signatures), infiltration of immune cells (e.g., lymphocytes) in the microenvironment, etc.

[0050] The WSI data 102 provided to the biomarker detection system 100 may include one or more digitized images of a patient tissue sample (e.g., a tumor tissue sample) stained with hematoxylin and eosin dye. The H&E dye stains cell nuclei, extracellular matrix and cytoplasm, and other cellular structures with different colors, thus allowing the pathologist and the detection system 100 to distinguish between different cellular structures. The overall pattern of coloration from the stain also indicates the general arrangement and distribution of cells, providing a view of the structure of the tissue sample. In some examples, the whole-slide image data 102 may include one or more image tiles extracted (e.g., randomly selected and extracted) from viable tumor regions of the stained tissue sample.

[0051] The prediction 20 output by the biomarker detection system 100 may be a binary output (e.g., "0" or "1," or "+" or "-") indicating the presence or absence of the biomarker for which the detection system 100 is trained. In some examples, the prediction 20 may be a confidence level or probability that the biomarker is present in the tissue sample associated with the WSI data. However, these are merely examples, and embodiments of the present disclosure are not limited thereto.

[0052] According to some embodiments, the biomarker detection system 100 includes a tile-level analyzer 120 , a cell-level analyzer 130 , and an aggregator 150 , as well as a biomarker predictor 160 .

[0053] In some embodiments, the tile-level analyzer 120 is configured to receive multiple tiles corresponding to the WSI data 10 of the tissue sample, analyze the tiles at the tile level, and generate (e.g., extract) tile-level embedding data based on the multiple tiles. The cell-level analyzer 130 is also configured to receive multiple tiles, analyze the tiles at the cellular level, and generate (e.g., extract) cellular-level embedding data based on the multiple tiles. The aggregator 150 is configured to aggregate (e.g., combine) the tile-level embedding data and the cell-level embedding data to generate aggregated embedding data. The biomarker predictor 160 then generates slide-level predictions 20 based on the aggregated embedding data.

[0054] Once the biomarker detection system 100 generates a prediction 20, the prediction may be transmitted to a server (e.g., a remote server or a cloud server) 30 for further processing and / or to a display device 40 for display to a user.

[0055] By analyzing WSI data 10 at both the tile and cell levels, the accuracy of slide-level predictions 20 can be significantly improved. This is due, at least in part, to the fact that tiles extracted from WSI data 10 may contain different types of cells, as well as non-cellular tissues such as stroma and blood vessels, and non-biological features (e.g., glass). When using tile-level embedding data for prediction, cell density and the proportion of non-cellular tissue per tile can be dominant predictors. While cell-level embeddings can be valuable for downstream classification tasks, it may be possible to extract useful information based on the morphological appearance of individual cells that would otherwise be masked by more dominant features in the tile-level embedding.

[0056] In some embodiments, the biomarker detection system 100 also includes a WSI processor 110 configured to preprocess the WSI data 100 to ensure uniformity within tiles that are fed to the tile-level analyzer 120 and the cell-level analyzer 130. Given that different laboratories generating whole-slide images based on tissue samples may use different stains and / or settings, the resulting WSI generated by such laboratories may have different staining (e.g., different coloring). Thus, in some embodiments, the WSI processor 110 performs stain normalization, i.e., standardizes staining across all tiles, and then generates multiple normalized tiles, which are then passed to the tile-level analyzer 120 and the cell-level analyzer 130 for further analysis and processing. The WSI processor 110 may also perform the function of extracting tiles from the original WSI.

[0057] However, embodiments of the present disclosure are not limited in this respect. For example, one or more functions of WSI processor 110 may be omitted from this component, integrated into other component blocks, or omitted entirely from biomarker detection system 100. In some examples, stain normalization may be omitted from WSI processor 110, and this function may be integrated into the input stage of the other of tile-level analyzer 120 and cell-level analyzer 130. Furthermore, stain normalization functionality may be omitted from biomarker detection system 100, and each of tile-level analyzer 120 and cell-level analyzer 130 may operate on raw tiles with potentially different staining profiles.

[0058] FIG. 2 is a block diagram illustrating the WSI preprocessor 110 and its operation, according to some embodiments of the present disclosure.

[0059] In some examples, WSI data 10 includes WSI 11 and a region of interest (ROI) map 12 that identifies regions of WSI 11 that are relevant for analysis by biomarker detection system 100. ROI map 12 may be generated by applying a series of filters to WSI 11.

[0060] In some examples, the filters may include at least one of a background filter, an out-of-focus filter, a crush filter, a pen mark filter, a hemorrhage filter, a necrosis filter, an adipose tissue filter, or a non-lymphatic filter. The background filter can remove portions of the WSI 11 that do not contain any tissue by detecting portions containing tissue and discarding all others. The out-of-focus filter can remove portions of the WSI 11 that contain tissue that is out of focus, i.e., blurred due to either suboptimal image acquisition or slide preparation. The crush filter can remove portions of the WSI 11 that contain tissue with crush artifacts, i.e., clusters of cells that are deformed or damaged due to suboptimal tissue processing. The pen mark filter can remove portions of the WSI 11 that contain pen marks made on a physical glass slide, as may be common in an anatomical pathology laboratory. The hemorrhage filter can remove portions of the WSI 11 that contain tissue with signs of hemorrhage, i.e., excessive extravascular accumulation of red blood cells that hide tumor tissue. The necrosis filter can remove portions of the WSI 11 that contain necrotic tissue. This can demonstrate a variety of characteristics, ranging from eosinophilic tissue debris without intact tumor cells to cells with nuclear alterations including pyknosis, karyorrhexis, karyorrhexis, and cytoplasmic vacuolization and / or eosinophilia. The adipose tissue filter can remove portions of the WSI 11 that encompass adipose tissue, including adipocytes and associated connective tissue. The non-lymphoid tissue filter can remove portions of the WSI 11 that encompass lymphoid tissue. The filters can identify lymphoid tissue that may be encountered in anatomical sites where lymphoma may arise, such as lymphoma tissue, lymph node parenchyma, lymphocyte-rich stroma, and lymphoid aggregates in lymph nodes and extranodal anatomical sites. Positively classified WSI portions that encompass lymphoid tissue can be retained, while negatively classified areas that do not encompass lymphoid tissue can be removed.

[0061] Each of the above filters may represent a function parameterized by a convolutional neural network (CNN) that takes a WSI or a portion thereof as input and returns a single Boolean value as output. The CNN model underlying each filter may be trained to identify a specific histological concept in the WSI and classify the portion according to the presence or absence of that concept. An output of 0 may mean that the component did not identify the concept within the given WSI portion, and an output of 1 may mean that the filter identified the concept within the WSI portion.

[0062] The application of the above filters produces an analysis region of interest as output. This region may be continuous or may span multiple parts, i.e., not be continuously connected. In some examples, the region identified by ROI map 12 includes areas rich in lymphoid elements (e.g., lymphoma tissue, lymph node parenchyma, lymphocyte-rich stroma, lymphoid aggregates) that may be encountered in lymph nodes and extranodal anatomical sites. The ROI may be free of artifacts and non-lymphoid tissue (e.g., background areas, out-of-focus areas, pulverized tissue, pen-marked areas, hemorrhagic tissue, necrotic tissue, and adipose tissue). In some examples, ROI map 12 may be further inspected and modified by a human user (e.g., a pathologist) as desired.

[0063] In some embodiments, the WSI preprocessor 110 includes a tile extractor 112 and a stain normalizer 114 .

[0064] The tile extractor 112 may apply the ROI map 12 to the WSI 11 (e.g., overlay the ROI map 12 on the WSI 11) to identify a region of interest within the WSI 11 and then extract multiple, equally sized, non-overlapping tiles 113 from the region of interest within the WSI 11. In some examples, the tile extractor 112 may also extract tiles from the WSI 11 and discard tiles that do not fall within the ROI (e.g., tiles with more than 10% overlap with non-ROI regions). In some examples, the tile extractor 112 may extract multiple (e.g., more than 30,000) non-overlapping tiles of 256×256 pixels from the WSI 11, which may have been digitized at 40x magnification.

[0065] To accommodate the different staining that tiles 113 may exhibit (e.g., as represented by tiles 113a, 113b, 113c, and 113d), stain normalizer 114 normalizes the staining across multiple tiles to generate multiple normalized tiles 115 that have uniform staining regardless of the staining used in WSI 11.

[0066] In some embodiments, the stain normalizer 114 includes a first model that may utilize a U-Net architecture having a neural network (e.g., a convolutional neural network) that represents the input image in short form as a vector and then upscales the image with the desired (e.g., normalized) stain. However, embodiments of the present disclosure are not limited thereto, and the first model of the stain normalizer 114 may use any suitable architecture.

[0067] The stain normalizer 114 provides the normalized tiles 115 to a tile-level analyzer 120 and a cell-level analyzer 130 for tile-level and cell-level embedding extraction, respectively.

[0068] 3A is a block diagram showing in more detail the internal structure of the tile-level analyzer 120 and the cell-level analyzer 130, according to some embodiments of the present disclosure. FIG. 3B is a block diagram showing the cell patch extractor 132 of the cell-level analyzer 130, according to some embodiments of the present disclosure.

[0069] 3A , in some embodiments, the tile-level analyzer 120 includes a second model that receives multiple tiles (e.g., non-overlapping normalized tiles) 115 and generates multiple tile-level feature vectors 121 as tile-level embedding data based on the received input tiles 115. Each tile-level feature vector 121 may represent a measurement of a tissue structure pattern associated with a histopathological diagnosis or biomarker evaluation. Here, the number of tile-level feature vectors 121 (N, an integer greater than 1) corresponds to (e.g., is the same as) the number of tiles 115 (N). Furthermore, each tile-level feature vector may have a length of L1, where L1 may be 1024, in some examples.

[0070] In some embodiments, the second model is a convolutional neural network (CNN) architecture, such as a residual neural network (ResNet) architecture (e.g., ResNet50) or a modified ResNet architecture (e.g., modified ResNet50). In some examples, the modified ResNet architecture may exclude the last blocks, i.e., the average pooling, flattening, and fully connected (FC) layers of the ResNet architecture, to improve alignment with cellular content and increase model interpretability. However, embodiments of the present disclosure are not limited to CNNs, and any suitable neural network, such as a Transformer, may be used as the second model.

[0071] To enable the use of large-scale unlabeled clinical imaging datasets, the second model can be trained on public and / or proprietary datasets. For example, a modified ResNet50 model can be trained using several publicly available datasets from The Cancer Genome Atlas (TCGA), namely, the TGCA-Breast Invasive Carcinoma (TGCA-BRCA), TGCA-Lung Adenocarcinoma (TGCA-LUAD), TGCA-Thyroid Carcinoma (TGCA-THCA), and TGCA-Diffuse Large B-Cell Lymphoma (TGCA-DLBCL) datasets, via the Autonomous Latent Representation Learning (BYOL) method. In some examples, the second model can also be trained using one or more datasets from commercial sources that include different tissues, tumor types, and diseases, including breast, lung, thyroid, lymph node, and tonsil tissues, as well as cancers, including follicular lymphoma and DLBCL.

[0072] 3A , in some embodiments, the cell-level analyzer 120 is configured to extract a plurality of cell patches based on the plurality of tiles 115 and generate cell-level embedding data based on the plurality of cell patches. According to some embodiments, the cell-level analyzer 120 includes a cell patch extractor 132, a cell feature generator 134, and a cell feature combiner 136.

[0073] In some embodiments, the cell patch extractor 132 is configured to receive multiple tiles (e.g., N normalized tiles) 115 and extract multiple cell patches (e.g., M patches) 133 from each tile 115. The cell feature generator 134 is configured to generate multiple cell-level feature vectors 135 based on the extracted cell patches 133. The cell feature combiner 136 then combines the cell-level feature vectors 135 to generate cell-level embedding data.

[0074] Referring to FIG. 3B, according to some embodiments, the cell patch extractor 132 includes a segmentation model 140 and a patch generator 142.

[0075] In some embodiments, segmentation model 140 is configured to detect cells (e.g., cell nuclei) within tile 115 and generate segmentation mask 141, which defines the outline of each cell and effectively separates each cell (e.g., nuclei) from the background. Segmentation model 140 may be a deep learning-based neural network trained for object detection, such as a StarDist network trained to distinguish between cells (e.g., nuclei) and the background.

[0076] The patch generator 142 applies the segmentation mask 141 to the corresponding tile 115 and generates (extracts) multiple patches (e.g., non-overlapping, equal-sized patches) 133 from the corresponding tile 115, such that each patch contains a single cell near or at its center. In some examples, each patch 133 may be a 32x32 pixel cell image crop centered around a segmented nucleus. The patch generator 142 may also remove the background surrounding the cells in each patch 133, i.e., set their pixel values ​​to black (RGB 0,0,0). The patch generator 142 may generate M (an integer greater than 1) patches 133 based on each tile that contains some or all of the cells detected by the segmentation model 140.

[0077] 3B illustrates some examples of the cell patch extractor 132, although embodiments of the present disclosure are not limited thereto. For example, the cell patch extractor 132 may simply apply a fixed grid to the tile 115 and subdivide it into multiple equally sized patches, some of which may contain no cells or only partial cells.

[0078] 3, the cell feature generator 134 may include a third model that is identical or substantially similar to the second model of the tile level analyzer 120 and extract cell-level features in a manner similar to the extraction of tile-level features outlined above. For example, the second and third models may be trained with the same data, or the second model of the tile level analyzer 120 may be trained on tile data, while the third model of the cell feature generator 134 is trained on cell data.

[0079] In some examples, the third model receives NxM cell patches 133 and generates a corresponding number of cell-level feature vectors 135 (NxM vectors) of length L2 (e.g., 256). That is, the cell feature generator 13 may generate one feature vector 135 for each cell patch 133.

[0080] In some examples, the cell patch images 133 may be downscaled by a factor of 32 by the backbone ResNet50 model so that the 32x32 pixel images have a spatial resolution of 1:1 at the output from the tensor. To ensure that the cell-level embedding encompasses cell-related features, before average pooling in ResNet50, the spatial image resolution may be increased to 16x16 pixels at the output from the ResNet50 CNN by upscaling the 32x32 pixel cell patch images to 128x128 pixels and skipping the last four blocks of the Resnet50 network.

[0081] Due to the heterogeneity in the size of detected cells, each 32x32 pixel cell patch image may contain different proportions of cellular and non-cellular features. A high proportion of non-cellular features in an image can cause the resulting embedding to be dominated by non-cellular tissue features or other background features. Therefore, to limit the information used to create the cell-level embedding to only cellular features, the cell patch extractor 132 may remove portions of the cell patch image 133 outside the segmented nuclei by setting their pixel values ​​to black (RGB 0,0,0). Finally, in some embodiments, to prevent the size of individual nuclei or the amount of background in each cell patch image from dominating over the cellular features, the global average pooling layer of the cell feature generator 134 (e.g., a modified ResNet50) averages only features within the boundary of the segmented nuclei, rather than averaging across the entire output tensor from the CNN layer. This allows the cell feature generator 134 to focus on the shape and coloration of the nuclei themselves, rather than the surrounding background (information that can be extracted from the surrounding background may already be captured by tile-level analysis).

[0082] According to some embodiments, the cell feature combiner 136 is configured to combine multiple cell-level feature vectors 135 to generate cell-level embedding data including multiple embedding vectors 137 of length L3 (e.g., 512). The number of embedding vectors 137 may correspond to (e.g., be equal to) the number of tiles 115 (N).

[0083] The cell feature combiner 136 may apply one or more statistical measures to cell-level feature vectors 135 associated with the same tile 115 to characterize a population. In some embodiments, the cell feature combiner 136 may calculate the arithmetic mean / median / average and standard deviation across the cell-level feature vectors 135 of one tile 115 to generate one mean / median / average vector (e.g., of length 256) and one standard deviation vector (e.g., of length 256) per tile 115. The cell feature combiner 136 may concatenate the two resulting vectors to form an embedding vector 137. Thus, each embedding vector 137 may include the mean, median, or arithmetic mean vector of multiple (e.g., M) cell-level feature vectors 135 corresponding to the same tile 115 and the standard deviation of the same set of cell-level feature vectors 135.

[0084] However, embodiments of the present disclosure are not limited thereto. For example, instead of performing the above statistical calculations, the cell feature combiner 136 may include an attention block that operates at the tile level, determines how important each tile 115 is, assigns corresponding weights to the cell-level feature vectors 135 of the same tile 115, and combines them by adding the weighted cell-level feature vectors 135 across the tile.

[0085] In some embodiments, aggregator 150 concatenates the tile-level embedding data and the cell-level embedding data to generate aggregated embedding data 151, where the vector length L4 of aggregated embedding data 151 may be equal to the sum of the vector lengths of the tile-level embedding data and the cell-level embedding data. In some examples, the arithmetic mean and standard deviation of the vectors of the cell-level embedding for each tile concatenated with each corresponding tile-level embedding may result in a combined embedded representation with a total size of 1536 pixels (1024 + 256 + 256).

[0086] According to some embodiments, the biomarker predictor 160 includes a fourth model having a multi-instance learning framework that generates WSI level predictions 20 based on the aggregated embedding data 151. In some examples, the fourth model may include a multi-instance learning (MIL) network based on softmax or transformer attention mechanisms, a weakly supervised classifier with max, min, or average pooling, a graph convolutional network, etc.

[0087] In some examples, the prediction 20 may be a continuous probability score representing the probability of the presence of a biomarker (e.g., MYC gene rearrangement) in tumor cells encompassed in the WSI data 10. The prediction score may be further thresholded at a predetermined threshold to classify the WSI into one of two possible outcomes: biomarker negative, where the score is below the threshold, or biomarker positive, where the score is equal to or above the threshold.

[0088] In some examples, biomarker detection system 100 may be used to aid in ruling out MYC rearrangement in scans containing aggressive B-cell lymphomas with morphology and phenotypes consistent with diffuse large B-cell lymphoma (DLBCL) or high-grade B-cell lymphoma (HGBL). In such cases, a negative HPS label may indicate that MYC gene rearrangement has been ruled out, and a positive HPS label may indicate that MYC gene rearrangement has not been ruled out.

[0089] This allows biomarker detection system 100 to identify cases that can safely be omitted from laborious and expensive molecular cytogenetic testing, allowing pathologists to focus on molecular characterization of the remaining cases, if necessary, using more advanced and comprehensive molecular tests (e.g., FISH testing using multiple probes for IG heavy and light chain loci, cancer genome profiling using next-generation sequencing). Given the cost and limitations of current FISH testing, a combined sequential testing approach beginning with a highly sensitive digital screening test (via biomarker detection system 100) to rule out MYC-R, followed by a highly specific molecular confirmatory test (e.g., FISH testing), may reduce resource consumption and improve overall test performance.

[0090] In some examples, biomarker detection system 100 may be used to predict MYC gene rearrangements in Burkitt's lymphoma to avoid FISH testing, or to predict gene expression signatures, such as double-hit signatures (DHITsig) or molecular high-grade (MGH) signatures in DLBCL / HGBL to avoid expression profiling. In further examples, biomarker detection system 100 utilizes HPS as a biomarker to characterize / distinguish specific subpopulations of DLBCL / HGBL patients with common biology / pathophysiology, further enabling applications such as patient selection or stratification in clinical trials.

[0091] FIG. 4 is a flow diagram illustrating a process 400 for detecting biomarkers by biomarker detection system 100 according to some embodiments of the present disclosure.

[0092] In some embodiments, the biomarker detection system 100 receives whole slide image data 10 corresponding to a tissue sample and identifies (S402) a plurality of tiles 115 corresponding to the whole slide image data 10 of the tissue sample by extracting the plurality of tiles 115 from the whole slide image data 10. The whole slide image data 10 may include at least one of a digitized WSI 11 or a region of interest (ROI) map 12 of a patient's tissue sample stained with hematoxylin and eosin (H&E) dye.

[0093] In some embodiments, the biomarker detection system 100 generates (S404) tile-level embedding data 121 based on the plurality of tiles (e.g., normalized tiles) 115. A second model of the tile-level analyzer 120 may then generate the plurality of tile-level feature vectors 121 based on the plurality of tiles 115. The second model may include at least one of a residual network (ResNet) or a transformer network.

[0094] The biomarker detection system 100 also generates (e.g., simultaneously generates) cell-level embedding data 137 based on the multiple tiles (S406). In this case, the detection system 100 may extract multiple cell patches 133 based on the multiple tiles 115 and generate cell-level embedding data 137 based on the multiple cell patches 133.

[0095] In some examples, extracting the cell-level embedding data 137 includes detecting a plurality of cells in each of the plurality of tiles 115 using the segmentation model 140, and generating a plurality of cell patches 133 based on the plurality of tiles 115 and the plurality of cells in each of the plurality of tiles. Each cell patch 133 may include a portion of one of the plurality of tiles 115 that encompasses a single cell.

[0096] In some examples, generating the cell-level embedding data 137 includes generating, by a third model of the cell feature generator 134, a plurality of cell-level feature vectors 135 based on the plurality of cell patches 133, and combining the plurality of cell-level feature vectors 135 to generate the cell-level embedding data 137. The cell-level embedding data 137 may include a plurality of embedding vectors.

[0097] In some embodiments, the biomarker detection system 100 aggregates (S408) the tile-level embedded data 121 and the cell-level embedded data 137 to generate aggregated embedded data 151. The detection system then concatenates the tile-level embedded data and the cell-level embedded data to generate the aggregated embedded data.

[0098] According to some embodiments, the biomarker detection system 100 (e.g., biomarker predictor 160) generates (S410) slide-level predictions 20 based on the aggregate embedding data (i.e., based on the tile-level embedding data 121 and the cell-level embedding data 137). The slide-level predictions 20 may indicate the presence or absence of a biomarker (e.g., MYC-R) in the tissue sample.

[0099] In some examples, the output of the biomarker detection system 100 may be utilized to exclude MYC rearrangements in scans containing aggressive B-cell lymphomas with morphology and phenotypes consistent with diffuse large B-cell lymphoma (DLBCL) or high-grade B-cell lymphoma (HGBL). In some examples, the biomarker detection system 100 may be used to predict gene expression signatures, such as the double-hit signature (DHIT signature) or molecular high-grade (MGH) signature, in DLBCL / HGBL. In a further example, the biomarker detection system 100 identifies HPS as a biomarker to characterize / identify specific subpopulations of DLBCL / HGBL patients with common biology / pathophysiology. The predictive capabilities of the detection system 100 may also enable applications such as patient selection or stratification in clinical trials.

[0100] According to various embodiments of the present disclosure, biomarker detection system 100 is implemented using one or more processing or electronic circuits configured to perform various operations as described above. Types of electronic circuits may include central processing units (CPUs), graphics processing units (GPUs), artificial intelligence (AI) accelerators (e.g., vector processors that may include vector arithmetic logic units configured to efficiently perform operations common to neural networks, such as dot products and softmaxes), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), digital signal processors (DSPs), etc. For example, in some circumstances, aspects of embodiments of the present disclosure are implemented in program instructions stored in non-volatile computer-readable memory that, when executed by an electronic circuit (e.g., a CPU, a GPU, an AI accelerator, or a combination thereof), perform the described operations. The operations performed by biomarker detection system 100 may be performed by a single electronic circuit (e.g., a single CPU, a single GPU, etc.) or may be allocated among multiple electronic circuits (e.g., multiple GPUs, or a CPU in conjunction with a GPU). The multiple electronic circuits may be local to each other (e.g., located on the same die, in the same package, or in the same embedded device or computer system) and / or remote from each other (e.g., in communication over a network such as a local personal area network such as Bluetooth®, in communication over a local area network such as a local wired and / or wireless network, and / or in communication over a wide area network such as the Internet, where some operations are performed locally and other operations are performed on a server hosted by a cloud computing service). One or more electronic circuits that operate to implement biomarker detection system 100 may be referred to herein as a computer or computer system, which may include a memory that stores instructions that, when executed by the one or more electronic circuits, implement the systems and methods described herein.

[0101] Terms such as "first," "second," and "third" may be used herein to describe various elements, components, regions, layers, and / or sections, but it is understood that these elements, components, regions, layers, and / or sections should not be limited by these terms. These terms are used to distinguish one element, component, region, layer, or section from another element, component, region, layer, or section. Thus, a first element, component, region, layer, or section discussed below may be referred to as a second element, component, region, layer, or section without departing from the spirit and scope of the inventive concept.

[0102] The terms used herein are for the purpose of describing particular embodiments and are not intended to limit the concept of the present invention. As used herein, the singular forms "a" and "an" are intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that the terms "includes," "including," "comprises," "comprising," "has," "have," and "having," when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0103] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. For example, the phrase "A and / or B" refers to A, B, or A and B. Phrases such as "one or more" and "at least one," when preceding a list of elements, modify the entire list of elements and not each individual element of the list. For example, the phrases "one or more of A, B, and C," "at least one of A, B, or C," "at least one of A, B, and C," and "at least one selected from the group consisting of A, B, and C" refer to A only, B only, C only, both A and B, both A and C, both B and C, or all of A, B, and C.

[0104] Furthermore, the use of "may" when describing embodiments of the inventive concepts refers to "one or more embodiments of the inventive concepts." Also, the term "exemplary" is intended to refer to an example or illustration.

[0105] When an element or layer is referred to as "on," "connected to," "coupled to," or "adjacent to" another element or layer, it will be understood that the element or layer can be directly on, directly connected to, directly coupled to, or directly adjacent to the other element or layer, or there can be one or more intervening elements or layers. When an element or layer is referred to as "directly on," "directly connected to," "directly coupled to," "contacting," "directly contacting," or "directly adjacent to" another element or layer, there are no intervening elements or layers present.

[0106] As used herein, the terms "substantially," "about," and similar terms are used as terms of approximation, not degree, and are intended to account for inherent variations in measurements or calculations that will be appreciated by those of ordinary skill in the art.

[0107] As used herein, the terms "use," "using," and "used" may be considered synonymous with the terms "utilize," "utilizing," and "utilized," respectively.

[0108] As one or more embodiments may be implemented differently, the particular process order may be implemented differently from the order described, for example, (i) the disclosed operations of the process are merely examples and may involve various additional operations not explicitly covered, and (ii) the temporal order of operations may be varied.

[0109] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by those skilled in the art to which the concept of the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant art and / or this specification, and should not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0110] While aspects of several exemplary embodiments of systems and methods for biomarker detection have been described and illustrated herein, various modifications and variations may be made, as will be understood by those skilled in the art, without departing from the spirit and scope of the embodiments according to the present disclosure. It should therefore be understood that pathology slide production systems and methods according to the principles of the present disclosure may be embodied in other ways than those specifically described herein. The present disclosure is also defined in the following claims and their equivalents.

Claims

1. A method for detecting biomarkers using a machine learning-based detection system, The detection system identifies multiple tiles corresponding to the entire slide image data of the tissue sample, The detection system generates tile-level embedding data based on the plurality of tiles, The detection system generates cell-level embedding data based on the plurality of tiles, The detection system generates a slide level prediction based on the tile-level embedding data and the cell-level embedding data, wherein the slide level prediction indicates the presence or absence of the biomarker in the tissue sample. Methods that include...

2. Identifying the plurality of tiles is The detection system receives the complete slide image data corresponding to the tissue sample, The detection system extracts the multiple tiles from the total slide image data. Includes, The method according to claim 1, wherein the total slide image data includes at least one of a digitized image of the patient's tissue sample stained with hematoxylin and eosin (H&E) dyes or a region of interest (ROI) map.

3. The detection system further includes performing stain normalization based on the plurality of tiles in order to generate a plurality of normalized tiles, The generation of the aforementioned tile-level embedded data is The method according to claim 1, comprising generating tile-level embedding data from the plurality of normalized tiles using the detection system.

4. Performing the aforementioned staining normalization is A first model of the detection system includes generating the plurality of normalized tiles based on the plurality of tiles, The method according to claim 3, wherein the first model includes a fully convolutional neural network.

5. The generation of the aforementioned tile-level embedded data is A second model of the detection system includes generating a plurality of tile-level feature vectors based on the plurality of tiles, The second model includes at least one of a residual network (ResNet) or a transformer network, The method according to claim 1, wherein the number of tile-level feature vectors is the same as the number of tiles.

6. The generation of the aforementioned cell-level embedded data is The detection system extracts multiple cell patches based on the multiple tiles, The detection system generates the cell-level embedding data based on the plurality of cell patches. The method according to claim 1, including the method described in claim 1.

7. Extracting the aforementioned multiple cell patches is The segmentation model of the detection system detects multiple cells within each of the multiple tiles, The detection system generates a plurality of cell patches based on the plurality of tiles and the plurality of cells in each of the plurality of tiles, wherein one of the plurality of cell patches includes a portion of the plurality of tiles that contains a single cell among the plurality of cells. The method according to claim 6, including the method described in claim 6.

8. The generation of the aforementioned multiple cell patches is The method according to claim 7, comprising generating the plurality of cell patches from a plurality of normalized tiles corresponding to the plurality of tiles using the detection system.

9. The generation of the aforementioned cell-level embedded data is A third model of the detection system includes generating multiple cell-level feature vectors based on the multiple cell patches, The method according to claim 7, wherein the number of cell-level feature vectors corresponds to the number of the plurality of tiles and the number of cell patches.

10. The third model includes at least one of a residual network (ResNet) or a transformer network. The method according to claim 9, wherein the number of the cell-level feature vectors is the value obtained by multiplying the number of the plurality of tiles by the number of the cell patches.

11. The generation of the aforementioned cell-level embedded data is The detection system further includes combining the plurality of cell-level feature vectors to generate the cell-level embedding data, wherein the cell-level embedding data includes the plurality of embedding vectors. The method according to claim 9, wherein the number of embedding vectors corresponds to the number of the plurality of tiles.

12. The method according to claim 11, wherein one of the plurality of embedding vectors includes the mean of a certain number of cell-level feature vectors and the standard deviation of the aforementioned number of cell-level feature vectors.

13. The detection system further includes aggregating the tile-level embedding data and the cell-level embedding data in order to generate aggregated embedding data. The generation of the slide level prediction is performed by the fourth model of the detection system, based on the aggregated embedding data, The method according to claim 1, wherein the fourth model includes at least one of a multi-instance learning (MIL) network, an attention-based MIL (AMIL) network, or a transformer.

14. The aggregation of the tile-level embedded data and the cell-level embedded data is The method according to claim 13, wherein the detection system includes concatenating the tile-level embedding data and the cell-level embedding data in order to generate the aggregated embedding data, the vector length of the aggregated embedding data being equal to the sum of the vector lengths of the tile-level embedding data and the cell-level embedding data.

15. The further includes transmitting the slide level prediction to a display device for display to the user, The method according to claim 1, wherein the slide level prediction includes a MYC-driven high-grade B-cell lymphoma (HGBL) signature.

Images