Image tiling aware of the architecture for processing of pathological slides

Abstract

Techniques for image tiling aware of an architecture for processing pathological slides are disclosed herein. In certain aspects, accessing an image and generating tiling elements for the image based on (i) the number of downsampling layers implemented in a machine learning model, (ii) the size of a kernel applied during a convolutional operation in the machine learning model, (iii) the number of convolutional layers implemented by the machine learning model at each level or each resolution, or any combination thereof, extracting tiles from the image using the tiling elements, inputting each tile into the machine learning model, generating a convolutional portion of the image for each tile using at least a convolutional layer, a kernel, and a downsampling layer, generating a convolutional version of the image using the convolutional portions of the image, and outputting the convolutional version of the image are provided in a computer-implemented method.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit of U.S. Provisional Patent Application No. 63 / 392,346, filed Jul. 26, 2022, which is hereby incorporated by reference in its entirety for all purposes.

[0002] Field The present disclosure relates to digital pathology, and more particularly to techniques for image tiling with an architecture - aware for processing pathology slides.

Background Art

[0003] Background Digital pathology involves scanning slides (e.g., histopathology or cytopathology glass slides) into digital images interpretable on a computer screen. Tissues and / or cells within the digital image can then be examined by digital pathology image analysis and / or interpreted by a pathologist for various reasons such as disease diagnosis, evaluation of response to treatment, and development of drugs to fight disease. To examine tissues and / or cells (substantially transparent) within the digital image, pathology slides may be prepared using various staining assays (e.g., immunohistochemistry) that selectively bind to tissue and / or cell components. Immunofluorescence (IF) is a technique for analyzing assays that bind a fluorescent dye to an antigen. Multiple assays responsive to different wavelengths may be utilized on the same slide. These multiplexed IF slides enable an understanding of the complexity and heterogeneity of the immune landscape of the tumor microenvironment and the potential impact on tumor response to immunotherapy. In some assays, the target antigen for a stain in a tissue may be referred to as a biomarker. Digital pathology image analysis can then be performed on the digital image of the stained tissue and / or cells to identify and quantify staining related to antigens (e.g., biomarkers representing various cells such as tumor cells) in the biological tissue.

[0004] Artificial intelligence and machine learning-based methods and / or techniques have shown great promise in digital pathology image analysis, such as cell detection, counting, localization, classification, and patient prognosis prediction. Many computing systems equipped with machine learning techniques, including convolutional neural networks (CNNs), have been proposed for image classification and digital pathology image analysis, such as cell detection and classification. For example, a CNN can have a series of convolutional layers as hidden layers, and this network structure enables the extraction of representative features for object / image classification and digital pathology image analysis. In addition to object / image classification, machine learning techniques have also been implemented for image segmentation. Image segmentation is the process of dividing a digital image into multiple segments (sets of pixels, also known as image objects). The purpose of segmentation is to simplify and / or transform the representation of the image to be more meaningful and easier to analyze. For example, image segmentation is typically used to find objects such as cells and boundaries (lines, curves, etc.) within an image. To perform image segmentation on large data (e.g., whole slide pathology images), the image is first divided into a number of small patches. A computing system equipped with machine learning techniques is trained to classify each pixel of these patches, and all pixels of the same class are combined into one segmented region of each patch, and then all segmented patches are combined into one segmented image (e.g., a segmented whole slide pathology image). Subsequently, based on the representative features associated with the segmented regions, machine learning techniques may be further implemented to predict or further classify the segmented regions (e.g., positive cells for a given biomarker, negative cells for a given biomarker, or cells without staining expression). Summary of the Invention

[0005] Summary Artificial intelligence and machine learning-based methods have achieved excellent performance in digital pathology. However, limitations in the memory size within the system memory of computer hardware, particularly the central processing unit (CPU), graphics processing unit (GPU), or accelerator cards, prevent processing relatively large images such as those from digital pathology imaging as a whole at the original resolution. Fully convolutional topologies such as U-Net are typically trained on downsampled images and inferred for images of the original size and resolution by simply dividing larger images into smaller (typically overlapping) tiles, making predictions for these tiles, and stitching them back together as the prediction of the whole image. Nevertheless, tiling can introduce several issues, such as artifacts at the tile boundaries and the need for additional processing to blend the results appropriately. Therefore, selecting an appropriate tiling method, including tile size and overlap, is essential to ensure good performance and avoid artifacts in the final output. A framework is disclosed herein for implementing a tiling method that adapts itself to the architecture of the deep neural network processing the extracted tiles and is aware of architectures that also show flexibility with respect to performance-related size constraints.

[0006] In various embodiments, accessing an image and generating tiling elements for the image, wherein generating the tiling elements includes determining the number of downsampling layers implemented in a machine learning model used for processing the image, determining the size of a kernel applied during a convolutional operation in the machine learning model, and generating a non-overlapping core region and an overlapping border region of the tiling elements, the overlapping border region surrounding the non-overlapping core region, the size of the non-overlapping core region being determined based on a dimensionality reduction attribute N calculated from the number of iterations of downsampling and the size of the kernel; generating tiling elements for the image; extracting tiles from the image using the tiling elements, each tile having the same non-overlapping core region and the same overlapping border region as the tiling elements; inputting each tile into a machine learning model; for each tile, generating a convolutional portion of the image using at least a convolutional layer, a kernel, and a downsampling layer; generating a convolutional version of the image using the convolutional portion of the image; and outputting the convolutional version of the image.

[0007] In some embodiments, the machine learning model is configured to include convolutional layers for each of one or more resolutions, each iteration of downsampling resulting in an evaluation of a different resolution version of the image, and generating the tiling elements further includes determining the number of convolutional layers implemented by the machine learning model for each level or each resolution, the dimensionality reduction attribute N being calculated from the number of iterations of downsampling, the size of the kernel, and the number of convolutional layers.

[0008] In some embodiments, convolutional layers without padding or border processing reduce each tile such that only the non-overlapping core region is used to generate each convolutional portion of the image.

[0009] In some embodiments, extracting tiles from an image using a tiling element involves sliding the tiling element through the image until the tiling element crosses the entire image, and for each portion of the image where the tiling elements are aligned as the tiling element slides through the image, a corresponding tile is extracted from that portion of the image, and each tile will have the same dimensions as the tiling element including an overlapping border region and a non-overlapping core region.

[0010] In some embodiments, when the tiling element slides through the image, the non-overlapping core regions of the tiles extracted from adjacent portions of the image are adjacent to each other but do not overlap each other, and the overlapping border regions of the tiles extracted from adjacent portions of the image overlap each other.

[0011] In some embodiments, for each tile, generating a convolutional portion of the image includes: (i) sliding a kernel over the tile one step at a time; (ii) for each step, positioning the center of the kernel at a particular location on the tile; (iii) in the sliding process, obtaining a result by multiplying the values of the kernel element-by-element with the corresponding values within the input region on the tile covered by the kernel; (iv) summing the results of the element-by-element multiplications to obtain a single value representing the output of the kernel at the particular location; and (v) repeating (i)-(iv) for all possible locations on the tile where the kernel can fit, and each convolutional layer outputs a new feature map, and each element of the new feature map represents the output of the application of the kernel to a particular location on the tile.

[0012] In some embodiments, neither padding nor border processing is applied to each tile.

[0013] In various embodiments, identifying tiling criteria for a machine learning model, the tiling criteria being set such that it is necessary for the size attribute of the input image to exceed a predetermined spatial metric threshold to meet the tiling criteria; accessing an image; determining a plurality of dimensions of the image; defining a particular size attribute of the image using at least one of the plurality of dimensions; determining whether the tiling criteria are met based on the particular size attribute of the image; determining the number of iterations of downsampling to be performed in the machine learning model in response to a determination that the tiling criteria are met; determining the kernel size of a filter to be applied in the machine learning model; determining a tiling specification for aligning the filter to various portions of the entire image based on the number of iterations of downsampling and the kernel size; for each portion of the various portions of the image defined by the corresponding tiling specification, generating a convolutional portion of the image using the filter, that portion, and the machine learning model; generating a convolutional version of the image using the convolutional portions of the image; and outputting the convolutional version of the image. A computer-implemented method is provided that includes these steps.

[0014] In some embodiments, the machine learning model is configured to include one or more convolutional layers for each of one or more resolutions, each iteration of downsampling results in an evaluation of an image version at a different resolution, and the method further includes determining, for each resolution, the number of convolutional layers to be implemented by the machine learning model in response to a determination that the tiling criteria are met, and for each portion of the various portions of the image defined by the corresponding specification, the convolutional portion of the image is further generated by using the number of convolutional layers to be implemented by the machine learning model.

[0015] In some embodiments, determining a tiling specification for aligning a filter to various parts of an entire image includes determining a border distance of the filter corresponding to a contaminated convolution space corresponding to an incomplete representation of the corresponding tile of the image where the filter alignment occurs.

[0016] In some embodiments, determining a specification for aligning a filter to various parts includes determining one or more shifts of a horizontal movement of the filter based on the width of the filter and determining one or more shifts of a vertical movement of the filter based on the height of the filter.

[0017] In some embodiments, generating a convolved version of an image includes, for each part of various parts of the image defined by a corresponding tiling specification, identifying a portion of a convolved part of the image predicted to be uncontaminated by edge effects based on a number of iterations of downsampling and a kernel size, and using those portions of the convolved parts of the image to generate the convolved version.

[0018] In some embodiments, the image includes a digital pathology image.

[0019] In some embodiments, the machine learning model is a deep convolutional neural network having at least three convolutional layers.

[0020] In some embodiments, the computer-implemented method further includes a user determining a diagnosis of an object related to the image, the diagnosis being determined based on inferences output by a machine learning model or another machine learning model using the convolved version of the image.

[0021] In some embodiments, the computer-implemented method further includes a user performing a treatment on the object based on (i) inferences output by a machine learning model or another machine learning model using the convolved version of the image, and / or (ii) a diagnosis of the object.

[0022] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium that houses instructions that, when executed by the one or more data processors, cause the one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0023] In some embodiments, a computer program product is provided that is tangibly embodied on a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0024] The terms and expressions used are used as terms of description and not of limitation, and there is no intention, in using such terms and expressions, of excluding any equivalents of the features shown and described or portions thereof, it being recognized that various modifications are possible within the scope of the invention as claimed. Accordingly, while the invention is specifically disclosed by way of embodiments and any features thereof, alterations and modifications of the concepts herein disclosed may be resorted to by those skilled in the art, and such alterations and modifications are to be regarded as within the scope of the invention as defined by the appended claims.

Brief Description of the Drawings

[0025] Aspects and features of various embodiments will become more apparent by way of example with reference to the accompanying drawings.

[0026]

Figure 1

[0027]

Figure 2

[0028]

Figure 3

[0029]

Figure 4

[0030]

Figure 5

[0031]

Figure 6

[0032]

Figure 7

[0033]

Figure 8

[0034]

Figure 9

Mode for Carrying Out the Invention

[0035] Detailed Description Specific embodiments are described, but these embodiments are presented by way of example only and are not intended to limit the scope of protection. The apparatuses, methods, and systems described herein can be embodied in various other forms. Further, various omissions, substitutions, and changes in the form of the exemplary methods and systems described herein may be made without departing from the scope of protection.

[0036] I. Overview When pathological slides are digitized, they become large images (e.g., gigapixel images), mainly because the slides are scanned at high magnifications (e.g., 20x or 40x). As a result, the entire whole-slide image (WSI) typically cannot be accommodated in the system memory (e.g., the memory of a CPU, GPU, or accelerator card) used to execute a machine learning model, especially when dealing with convolutional neural networks, and the execution of downstream image processing algorithms on the whole-slide image may be hindered. To address this reality, an image tiling approach can be applied in tiling techniques to slice a large image into smaller overlapping patches or tiles. These tiles are then processed individually by a machine learning model. In these cases, the integrity of the image processing results generated for the WSI depends on how the image processing algorithm is defined with respect to the tile size, stride, and overlap, as well as what kind of continuous processes are used in the algorithms related to the tiling technique and / or the image tiling approach (e.g., how to handle the boundaries). Therefore, selecting appropriate parameters and processes for the image processing algorithm is essential to ensure good performance, avoid artifacts in the final output, and ultimately improve the accuracy of the results generated by an image processing algorithm that processes images using tiling techniques.

[0037] One type of downstream image processing algorithm that can be performed in the context of a pathology slide is an algorithm that uses one or more convolutional neural networks (CNNs) in its pipeline. When applying a CNN to process an image, each tile is passed through the CNN in the same way as a normal image, and a core operation known as convolution is applied to each tile. The convolution operation of a CNN is applied to each tile by sliding a kernel over the tile to extract features and generating a corresponding feature map for each tile. A kernel is a small matrix of learnable weights that is multiplied by the input to extract relevant features. As the kernel slides over the image and aligns with various parts of a given input image, a convolution value is calculated for each alignment position. The convolution value is then assigned to a specific pixel (e.g., the pixel aligned with the center of the kernel) of each part of the given input image that is aligned with the kernel. Figure 1 shows an example of a 3×3 kernel 105 applied to a 5×5 image 110. Since the kernel 105 can fit at three different horizontal positions and three different vertical positions within the image 110 with a stride of 1, the output convolution features or feature map 115 generated by the convolution operation is 3×3 pixels. Once all tiles are processed, their corresponding feature maps are combined to create a final feature map for the entire image. This combination can be achieved in various ways, such as averaging overlapping regions, adopting the maximum value, or using more sophisticated techniques such as blending. In particular, according to this specific approach, the output feature map or image will be smaller than the input image.

[0038] To control the spatial dimension of the output feature map (e.g., to control the size of the output feature map, maintain the spatial dimension, and prevent information loss at the boundaries of the input image), padding techniques can be used. Padding techniques involve adding additional pixels around the edges of the input image, such as tiles, before applying the convolution, which allows the kernel to cover more space in the image. Generally, there are three types of padding: valid, same, and causal. In valid padding (shown in Figure 1), no padding is applied. The convolution filter is restricted to move only on pixels that completely overlap with the input image. If the filter exceeds the image boundary, it is simply ignored. As a result, the spatial dimension of the output feature map is smaller compared to the input image.

[0039] Same padding is a type of padding commonly used to maintain the spatial dimension of the input image in the output feature map. For an odd-sized kernel (e.g., 3×3), an equal number of padding pixels (zeros) are added to all sides of the input image. For an even-sized kernel (e.g., 2×2), one more row and column of padding pixels are added to the bottom and right ends of the input image. The main purpose of same padding is to ensure that the central pixel of the kernel remains aligned with the center of the input image. Figure 2 shows the same padding technique where one-pixel padding with a value of 0 is added to the input image 205. This technique generates an output feature map or image 210 of the same size as the input image 205 (without padding). However, the convolution of the padded part of the input image 205 with the kernel 215 may be contaminated by the padding.

[0040] In a standard convolution operation with same padding or valid padding, the kernel is centered at each position of the input sequence, and the convolution operation is applied to all elements within the receptive field of the kernel. This means that when processing a sequence, the output at a particular time step can depend on elements both in the past and future of the sequence. However, in some cases, it is essential to enforce a causal relationship between the input and output in order to accurately model real-world situations. Causal padding addresses this problem by adding padding only to the left side of the input sequence. This ensures that during convolution, the kernel cannot access future elements of the sequence, thus maintaining the causal relationship between the input and output. By preventing the kernel from peeking into the future, causal padding enables the CNN to model sequential data in a way that is consistent with the temporal order of the input.

[0041] Some CNNs use consecutive kernel applications. In these cases, the size reduction in the output or the contamination in the output worsens through the kernel application. Some CNNs use downsampling techniques such that the intensity (or other attributes) of multiple adjacent pixels are aggregated into a single value. In these cases, if a single pixel has a contaminated value, its influence may "spread" to multiple adjacent pixels. In addition to or instead of this, when kernel application occurs in the downsampled space, the reduced output size can be magnified with respect to its overall impact on the original size of the input image. For example, in U-Net, at each resolution, multiple convolutions are performed. Thus, if the valid padding technique shown in Figure 1 is performed for multiple convolutions, the size of the output image decreases dramatically with respect to the size of the input image due to convolution. On the other hand, if the same padding technique shown in Figure 2 is performed, contamination spreads dramatically from the boundaries due to multiple convolutions and downsamplings.

[0042] To address these and other issues, a tiling process (e.g., architecture-aware image tiling) for generating a transformed image using a strategic tiling approach has been developed and disclosed herein. The strategic tiling approach determines how to configure the tiles and how to aggregate the outputs from the output of the machine learning model to produce an output that reduces or eliminates the impact of kernel convolution on the prediction of image characteristics, using the architecture specifications of the machine learning model (e.g., kernel size, number of convolutional layers, number of downsampling layers, etc.). More specifically, the amount of contamination or size reduction with respect to the machine learning model is calculated based on the kernel size, number of convolutional layers, number of downsampling layers, or any combination thereof in the architecture of the machine learning model. The tiling approach according to the present disclosure for constructing the tiles has tiling elements (TEs) that slide through the whole WSI until the whole WSI is covered. Each TE has an overlapping border region of size N / 2 (where N is the amount of contamination or size reduction) that encloses a non-overlapping core region. As the TE moves across the WSI, the TE is used to extract tiles from each part of the WSI, which can then be input into the machine learning model, and the convolutional layer (without border processing) trims the tiles to the non-overlapping core region, and the non-overlapping core region is used to generate a feature map as the output of the machine learning model. As a result, the aggregation of the outputs of the machine learning model for each tile provides the result of the machine learning model in the WSI.

[0043] In an exemplary embodiment, accessing an image and generating tiling elements for the image, wherein generating the tiling elements includes determining the number of downsampling layers implemented in a machine learning model used for processing the image, determining the size of a kernel applied during a convolution operation in the machine learning model, and generating a non-overlapping core region and an overlapping border region of the tiling elements, the overlapping border region surrounding the non-overlapping core region, the size of the non-overlapping core region being determined based on a dimensionality reduction attribute N calculated from the number of iterations of downsampling and the size of the kernel; generating tiling elements for the image; extracting tiles from the image using the tiling elements, each tile having the same non-overlapping core region and the same overlapping border region as the tiling elements; inputting each tile into a machine learning model; for each tile, generating a convolutional portion of the image using at least a convolutional layer, a kernel, and a downsampling layer; generating a convolutional version of the image using the convolutional portion of the image; and outputting the convolutional version of the image. A computer-implemented method is provided that includes the foregoing steps.

[0044] In another exemplary embodiment, identifying tiling criteria for a machine learning model, the tiling criteria being set such that it is necessary for the size attribute of the input image to exceed a predetermined spatial metric threshold in order to meet the tiling criteria; accessing the image; determining a plurality of dimensions of the image; defining a particular size attribute of the image using at least one of the plurality of dimensions; determining whether the tiling criteria are met based on the particular size attribute of the image; in response to a determination that the tiling criteria are met, determining the number of iterations of downsampling to be performed in the machine learning model; determining the kernel size of the kernel to be applied in the machine learning model; based on the number of iterations of downsampling and the kernel size, determining a tiling specification for aligning the kernel to various parts of the entire image; for each part of the various parts of the image defined by the corresponding tiling specification, using the kernel, that part, and the machine learning model to generate a convolutional part of the image; using the convolutional parts of the image to generate a convolutional version of the image; and outputting the convolutional version of the image.

[0045] Conveniently, architecture-aware image tiling enables machine learning models, such as deep neural networks, to process large images without requiring excessive memory. Since the network operates on smaller tiles, the memory requirements are reduced. Further, by dividing the image into tiles, it becomes possible to process them in parallel, significantly accelerating the computation, especially on hardware that supports parallel processing, such as GPUs. Additionally, architecture-aware image tiling enables the machine learning model to handle images of different sizes. The network can process each tile regardless of its size, making it adaptable to real-world situations where images can have various dimensions. Further, conventional image processing algorithms calculate the convolution operation at the boundaries and discard the invalid regions in the output. In contrast, the architecture-aware image tiling of the present disclosure has a computational advantage (and thus a computational resource advantage) of not calculating the convolution near the boundaries. Since the results of the convolution near the boundaries are always invalid, the architecture-aware image tiling does not waste computational resources on what will ultimately be discarded. In summary, the establishment of appropriate optimized tiling is important for the successful analysis of large pathological images, especially in view of the data load generated by each WSI. The architecture-aware tiling method of the present disclosure adapts itself to the architecture of the machine learning model that processes the tiles after extraction and further demonstrates flexibility with respect to performance-related size constraints.

[0046] II. Definitions As used herein, when an act is "based on" something, this means that the act is at least partially based on at least a part of that something.

[0047] As used herein, the terms "substantially", "approximately", and "about" are defined as being generally as specified (including where exactly as specified), but not necessarily exactly as specified, as would be understood by one of ordinary skill in the art. In any disclosed embodiment, the terms "substantially", "approximately", or "about" may be replaced with "within [a percentage] of" the specified content, where the percentage includes 0.1, 1, 5, and 10%.

[0048] As used herein, the terms "sample", "biological sample", "tissue", or "tissue sample" refer to any sample containing biomolecules (such as proteins, peptides, nucleic acids, lipids, carbohydrates, or combinations thereof) obtained from any organism, including viruses. Other examples of organisms include mammals (such as veterinary animals like humans, cats, dogs, horses, cows, and pigs, as well as laboratory animals like mice, rats, and primates), insects, annelids, spiders, marsupials, reptiles, amphibians, bacteria, and fungi. Biological samples include tissue samples (such as tissue sections and tissue needle biopsies), cell samples (such as cytological smear samples like Pap smears or blood smears, or samples of cells obtained by microdissection), or cell fractions, fragments, or organelles (such as those obtained by lysing cells and separating their components by centrifugation, etc.). Other examples of biological samples include blood, serum, urine, semen, feces, cerebrospinal fluid, interstitial fluid, mucus, tears, sweat, pus, biopsy tissue (such as obtained by surgical biopsy or needle biopsy), nipple aspirate, earwax, milk, vaginal fluid, saliva, swabs (such as oral swabs), or any material containing biomolecules derived from an initial biological sample. In certain embodiments, the term "biological sample" as used herein refers to a sample (such as a homogenized or liquefied sample) prepared from a tumor or a part thereof obtained from a subject.

[0049] As used herein, the terms "biological material", "biological structure", or "cellular structure" refer to natural materials or structures that include all or part of a biological structure (e.g., cell nucleus, cell membrane, cytoplasm, chromosome, DNA, cell, cell mass, etc.).

[0050] As used herein, the term "digital pathology image" refers to a digital image of a stained sample.

[0051] As used herein, the term "cell detection" refers to the detection of the location and characteristics of cell or cellular structure (e.g., cell nucleus, cell membrane, cytoplasm, chromosome, DNA, cell, cell mass, etc.) pixels.

[0052] As used herein, the term "target region" refers to a region of an image that includes the image data intended to be evaluated in an image analysis process. The target region includes any region, such as a tissue region of an image intended to be analyzed in an image analysis process (e.g., tumor cells or staining expression).

[0053] As used herein, the term "tile" or "tile image" refers to a single image corresponding to a part of the entire image or the entire slide. In some embodiments, a "tile" or "tile image" refers to a region of a whole slide scan or a region of interest having (x,y) pixel dimensions (e.g., 1000 pixels × 1000 pixels). For example, considering an entire image divided into M columns of tiles and N rows of tiles, each tile within the M×N mosaic includes a part of the entire image. That is, the tile at position M1, N1 includes the first part of the image, the tile at position M1, N2 includes the second part of the image, and the first part and the second part are different. In some embodiments, the tiles can each have the same dimensions (pixel size × pixel size). In some examples, the tiles can overlap partially and can represent the overlapping regions of the entire slide scan or the region of interest.

[0054] As used herein, the terms "patch", "image patch", or "mask patch" refer to a container of pixels corresponding to a portion of an entire image, an entire slide, or an entire mask. In some embodiments, a "patch", "image patch", or "mask patch" refers to a region of an image or mask, or a region of interest having (x,y) pixel dimensions (e.g., 256 pixels × 256 pixels). For example, a 1000 pixel × 1000 pixel image divided into 100 pixel × 100 pixel patches will contain 10 patches (each patch containing 1000 pixels). In other embodiments, a patch has (x,y) pixel dimensions and each "patch", "image patch", or "mask patch" overlaps with another "patch", "image patch", or "mask patch" sharing one or more pixels.

[0055] III. Generation of Digital Pathology Images Digital pathology involves the interpretation of digitized images to accurately diagnose a subject and guide treatment decisions. In a digital pathology solution, an image analysis workflow can be established to automatically detect or classify biological objects of interest, such as positive and negative tumor cells. An exemplary workflow for a digital pathology solution includes obtaining a tissue slide and scanning a preselected region or the entire tissue slide with a digital image scanner (e.g., a whole slide image (WSI) scanner) to obtain a digital image, performing image analysis on the digital image using one or more image analysis algorithms, and potentially detecting and quantifying each object of interest based on the image analysis (e.g., quantitative or semi - quantitative scoring such as positive, negative, moderate, weak, etc.) (e.g., counting or identifying the object - specific area or cumulative area for each object of interest).

[0056] Figure 3 shows an exemplary network 300 for generating and analyzing digital pathology images. A fixation / embedding system 305 uses a fixative (e.g., a liquid fixative such as a formaldehyde solution) and / or an embedding material (e.g., a histological wax such as paraffin wax, and / or one or more resins such as styrene or polyethylene) to fix and / or embed a tissue sample (e.g., a sample containing at least a portion of at least one tumor). Each sample can be fixed by exposing the sample to the fixative for a predetermined period (e.g., at least 3 hours) and then dehydrating the sample (e.g., via exposure to an ethanol solution and / or a clearing intermediate agent). The embedding material can infiltrate when the sample is in a liquid state (e.g., when heated).

[0057] Fixation and / or embedding of the sample is used to preserve the sample and delay its degradation. In histology, fixation generally refers to an irreversible process that uses chemicals to retain the chemical composition, preserve the natural sample structure, and maintain the cell structure from degradation. Fixation may also harden the cells or tissue for sectioning. The fixative may enhance the preservation of the sample and cells using cross-linked proteins. The fixative may bind to and cross-link some proteins and denature other proteins by dehydration, which can harden the tissue and inactivate enzymes that would otherwise degrade the sample. The fixative may also kill bacteria.

[0058] Fixatives can be administered, for example, by perfusion and immersion of the prepared sample. A variety of fixatives can be used, including methanol, Bouin's fixative and / or formaldehyde fixatives, such as neutral buffered formalin (NBF) or paraffin-formaldehyde (paraformaldehyde-PFA). If the sample is a liquid sample (e.g., a blood sample), the sample may be smeared onto a slide and dried prior to fixation. The fixation process can help preserve the structure of the sample and cells for histological examination purposes, but fixation can conceal tissue antigens, thereby potentially reducing antigen detection. Thus, formalin is generally considered a limiting factor in immunohistochemistry because it can crosslink antigens and mask epitopes. In some instances, additional processes are performed to reverse the effects of crosslinking, including treating the fixed sample with anhydrous citric acid (a reversible protein crosslinker) and heating.

[0059] Embedding can involve infiltrating the sample (e.g., a fixed tissue sample) with a suitable histological wax, such as paraffin wax. Histological waxes can be insoluble in water or alcohol but may be soluble in paraffin solvents such as xylene. Thus, it may be necessary to replace the water in the tissue with xylene. To do so, the sample can first be dehydrated by gradually replacing the water in the sample with alcohol, which can be accomplished by passing the tissue through ethyl alcohol of increasing concentration (e.g., 0 to about 100%). After replacing the water with alcohol, the alcohol can be replaced with xylene, which is miscible with alcohol. Since histological wax is soluble in xylene, the melted wax can fill the space previously filled with water that has been filled with xylene. The sample filled with wax can be cooled to form a hardened block, which can be clamped to a microtome, vibratome, or compressotome to cut sections. In some cases, deviating from the exemplary procedures above can result in infiltration of paraffin wax, which can inhibit the penetration of antibodies, chemicals, or other fixatives.

[0060] Next, a tissue slicer 310 can be used to slice the fixed and / or embedded tissue sample (e.g., a tumor sample). Slicing is a process of cutting thin slice pieces (e.g., 4-5 μm thick) of the tissue sample from the tissue block for the purpose of attaching the tissue block onto a microscope slide for examination. Slicing may be performed using a microtome, a vibratome or a compressotome. In some cases, the tissue can be rapidly frozen in dry ice or isopentane and then cut with a cold knife in a refrigerated cabinet (e.g., a cryostat). Other types of coolants such as liquid nitrogen can be used to freeze the tissue. Sections for use with brightfield and fluorescence microscopes are generally about 4-10 μm thick. In some cases, the sections can be embedded in epoxy resin or acrylic resin, which may enable thinner sections (e.g., <2 μm) to be cut. These sections may then be attached to one or more glass slides. A coverslip may be placed on top to protect the sample sections.

[0061] Since tissue sections and the cells therein are substantially transparent, slide preparation typically further includes staining the tissue sections (e.g., automated staining) to make the relevant structures more visible. In some cases, the staining is performed manually. In some cases, the staining is performed semi-automatically or automatically using a staining system 320. The staining process includes exposing the tissue sample or sections of the fixed liquid sample to one or more different stains (e.g., sequentially or simultaneously) to develop different characteristics of the tissue.

[0062] For example, staining can be used to mark specific types of cells and / or to flag specific types of nucleic acids and / or proteins to assist microscopy. The staining process generally involves adding a dye or stain to a sample to confirm or quantify the presence of a specific compound, structure, molecule, or feature (e.g., an intracellular feature). For example, staining can help identify or highlight specific biomarkers from tissue sections. In other examples, staining can be used to identify or highlight cell organelles within biological tissues (e.g., muscle fibers or connective tissue), cell populations (e.g., different blood cells), or individual cells.

[0063] One exemplary type of tissue staining is histochemical staining, which uses one or more chemical dyes (e.g., acidic dyes, basic dyes, chromogenic substrates) to stain tissue structures. Histochemical staining may be used to show general aspects of tissue morphology and / or cell microanatomy (e.g., distinguish cell nuclei from cytoplasm, show lipid droplets, etc.). An example of histochemical staining is H&E. Other examples of histochemical staining solutions include trichrome staining solutions (e.g., Masson's trichrome), periodic acid Schiff (PAS), silver staining solutions, and iron staining solutions. The molecular weight of histochemical staining reagents (e.g., dyes) is generally 500 kilodaltons (kD) or less, although some histochemical staining reagents (e.g., alcian blue, phosphomolybdic acid (PMA)) may have molecular weights up to 2000 or 3000 kD. An example of a high molecular weight histochemical staining reagent is α-amylase (˜55 kD), which may be used to show glycogen.

[0064] Another type of tissue staining is IHC, also called "immunostaining", which uses a primary antibody that specifically binds to a target antigen of interest (also called a biomarker). IHC can be either direct or indirect. In direct IHC, the primary antibody is directly conjugated to a label (e.g., a chromophore or a fluorophore). In indirect IHC, first the primary antibody is bound to the target antigen, and then a secondary antibody conjugated to a label (e.g., a chromophore or a fluorophore) is bound to the primary antibody. Since antibodies have a molecular weight of about 150 kD or more, the molecular weight of IHC reagents is much larger than that of histochemical staining reagents.

[0065] For performing staining, various types of staining protocols may be used. For example, an exemplary IHC staining protocol involves using a hydrophobic barrier line around the sample (e.g., tissue section) to prevent leakage of reagents from the slide during incubation, treating the tissue section with reagents to block endogenous sources of non-specific staining (e.g., enzymes, free aldehyde groups, immunoglobulins, other irrelevant molecules that can mimic specific staining), incubating the sample with a permeabilization buffer to facilitate penetration of antibodies and other staining reagents into the tissue, incubating the tissue section with a primary antibody at a specific temperature (e.g., room temperature, 6 - 8 °C) for a certain period (e.g., 1 - 24 hours), rinsing the sample using a wash buffer, then incubating the sample (tissue section) with a secondary antibody at another specific temperature (e.g., room temperature) for another period, rinsing the sample again with a water buffer, incubating the rinsed sample with a chromogen (e.g., DAB: 3,3’-diaminobenzidine), and washing away the chromogen to stop the reaction. In some examples, subsequently, counterstaining is used to distinguish the overall "landscape" of the sample and serves as a reference for the main color used for detection of the tissue target. Counterstaining agents may include, for example, hematoxylin (a blue to purple stain), methylene blue (a blue stain), toluidine blue (a stain that stains nuclei dark blue and polysaccharides pink to red), nuclear fast red (also called Kernechtrot dye, a red stain), methyl green (a green stain), non-nuclear chromogenic stains, such as eosin (a pink stain), etc. Those skilled in the art will recognize that other immunohistochemical staining techniques can be implemented to perform the staining.

[0066] In another example, an H&E staining protocol can be implemented for tissue section staining. The H&E staining protocol includes applying a hematoxylin stain mixed with a metal salt or mordant to the sample. The sample can then be rinsed with a weak acid solution to remove excess staining (differentiation), followed by bluing in a weak alkaline water. After application of hematoxylin, the sample can be counterstained with eosin. It will be understood that other H&E staining techniques can be implemented.

[0067] In some embodiments, depending on what the feature of interest is, different types of staining agents can be used to perform staining. For example, DAB can be used for various tissue sections for IHC staining, and DAB results in the brown color that displays the feature of interest in the stained image. In another example, since the DAB color may be masked by melanin pigment, alkaline phosphatase (AP) can be used for skin tissue sections for IHC staining. Regarding primary staining techniques, applicable staining agents can include, for example, basic and acidic staining, hematin and hematoxylin, silver nitrate, trichrome staining agents, etc. Acidic dyes can react with cationic or basic components in tissues or cells, such as proteins and other components in the cytoplasm. Basic dyes can react with anionic or acidic components in tissues or cells, such as nucleic acids. As described above, an example of a staining system is H&E. Eosin may be a negatively charged pink acidic dye, and hematoxylin may be a purple or blue basic dye containing hematin and aluminum ions. Other examples of staining can include periodic acid-Schiff reaction (PAS) staining, Masson's trichrome, alcian blue, Fontana-Masson, reticulin staining, etc. In some embodiments, different types of staining agents can be used in combination.

[0068] Next, the sections may be attached to corresponding slides and scanned or imaged by the imaging system 325 to generate raw digital pathology images 330a - n. To magnify the stained samples, a microscope (e.g., an electron microscope or an optical microscope) can be used. For example, an optical microscope can have a resolution of less than 1 μm, such as about several hundred nanometers. An electron microscope can be used to observe finer details in the nanometer or sub - nanometer range. The imaging device (combined with or separated from the microscope) images the magnified biological sample to acquire image data such as a multi - channel image (e.g., multi - channel fluorescence) having several (e.g., 10 - 16, etc.) channels. The imaging device can include, but is not limited to, a camera (e.g., an analog camera, a digital camera, etc.), optical elements (e.g., one or more lenses, a sensor focus lens group, a microscope objective lens, etc.), an imaging sensor (e.g., a charge - coupled device (CCD), a complementary metal - oxide - semiconductor (CMOS) image sensor, etc.), photographic film, etc. In a digital embodiment, the imaging device can include a plurality of lenses that cooperate to demonstrate on - the - fly focusing. An image sensor, such as a CCD sensor, can image a digital image of the biological sample. In some embodiments, the imaging device is a bright - field imaging system, a multi - spectral imaging (MSI) system, or a fluorescence microscope system. The imaging device can utilize invisible electromagnetic radiation (e.g., UV light) or other imaging techniques to capture the image. For example, the imaging device may include a microscope and a camera configured to capture the image magnified by the microscope. The image data received by the analysis system may be the same as the raw image data captured by the imaging device and / or may be derived from the raw image data.

[0069] Next, an image of the stained section can be stored in a remote system 335 such as a storage device or a server. The image can be stored locally, remotely, and / or in a cloud server. Each image can be stored in association with an identifier of the subject and a date (e.g., the date the sample was collected and / or the date the image was captured). The image can further be sent to another system (e.g., a system associated with a pathologist, an automated or semi-automated image analysis system, a machine learning training and deployment system, or any combination thereof, as described in more detail herein).

[0070] In some cases, the network 300 can include an image analysis system 340 for training and executing a machine learning model. Examples of machine learning models can be a deep convolutional neural network, U-Net, V-Net, residual neural network, or recurrent neural network. The machine learning model can be trained and / or used to (e.g.) predict whether a biomedical image includes a depiction of a set of tumor cells or other structural and / or functional biological entities related to a disease, whether the biomedical image is related to the diagnosis of a disease, whether the biomedical image is related to the classification of a disease (e.g., stage, subtype, etc.), and / or whether the biomedical image is related to the prognosis prediction of a disease. The prediction can characterize the presence, amount, and / or size of a set of tumor cells or other structural and / or functional biological entities, the diagnosis of a disease, the classification of a disease, and / or the prognosis prediction of a disease.

[0071] The training controller 345 can execute code for training a machine learning model and / or other machine learning models using one or more training data sets 350. Each training data set 350 can include a set of training biomedical images from images 330a-n (e.g., obtained from the remote system 335). Each of the biomedical images can include a digital pathology image, a CT image, an MRI image, an ultrasound image, etc., that depicts one or more biological objects (e.g., a set of one or more types of cells). Each of the biomedical images can depict a portion of a sample such as a tissue sample (e.g., colorectal tissue, bladder tissue, breast tissue, pancreatic tissue, lung tissue, or stomach tissue), a blood sample, or a urine sample. In some cases, each of the one or more biomedical images depicts a plurality of tumor cells or a plurality of other structural and / or functional biological entities. The training data set 350 may be collected (e.g.) from the imaging system 325.

[0072] The training controller 345 can identify an indicator of a representative distribution of disease characteristics. The analysis system 340 can access one or more databases of clinical data related to the disease, perform an analysis, determine a representative distribution of the characteristics, and communicate the indicator of the representative distribution to the training controller 345. Alternatively, the training controller 345 can receive the indicator via user input from a remote system 335 that may be associated with, for example, a physician, nurse, hospital, pharmacist, etc. The representative distribution can correspond to the realistic ratios of various characteristics to each other. For example, a particular variant of a disease may be present in a certain percentage of subjects having the disease. Thus, the representative distribution can indicate a particular percentage of a particular variant. Other characteristics can include the recurrence rate of the disease, international prognostic index risk factors, and / or demographic factors of the disease. In addition to clinical characteristics, the representative distribution can also include technical characteristics that can identify device and process-related characteristics. For example, the technical characteristics can include the type of staining protocol used by the staining system 320 (e.g., the type of dye and / or the number of dyes), the type of scanner of the imaging system 325, one or more types of data acquisition methods of the imaging system 325, etc. The technical characteristics may be determined based on the disease, as different diseases may be better visualized using a particular scanner or staining protocol.

[0073] The training controller 345 can generate a training dataset 350 to have a distribution of characteristics corresponding to a representative distribution of the characteristics of the disease. The distribution can correspond to the representative distribution in such a way that one or more measured traits of the distribution of the training dataset 345 are close enough to the representative distribution (e.g., within a predetermined absolute value range from the representative distribution, within a predetermined percentage range from the representative distribution, etc.), the integral (or, a normalized integral) of the overlap between the distribution and the representative distribution exceeds a threshold value, and so on. The generation of the training dataset 350 can include the training controller 345 defining or selecting a set of images 330 having a representative distribution. For example, breast cancer tumors can have a representative distribution of ductal subtypes that occur in 80 - 85% of cases, as well as lobular and other rare subtypes that occur in 15 - 20% of cases. Thus, the images included in the training dataset 350 can have a distribution of characteristics equal to, or close enough to (e.g., within 1%), the distribution of characteristics shown in the representative distribution.

[0074] In some cases, the training controller 345 can determine that the image 330 lacks representative data for one or more characteristics of the disease. For example, the disease is breast cancer, and the representative distribution can indicate that 1% of breast cancer cases are associated with biopsies that stain positive for the lobular subtype. However, the training controller 345 can determine that the image 330 lacks the combination of characteristics of biopsies that stain positive for the lobular subtype that can constitute 1% of the training dataset 350. When making this determination, the training controller 345 can output a notification indicating that the image 330 does not contain representative data for a particular combination of characteristics to the remote system 335. Then, the training controller 345 can receive an adjustment of the characteristic values from the remote system 335. For example, the adjustment can indicate that biopsies that stain positive for the lobular subtype include 0.5% instead of 1% of the training dataset 350. Based on the adjustment, the training controller 345 can generate the training dataset 350.

[0075] Analysis system 340 can include a label mapper 355 that maps images 330 from imaging system 325 that include tumor cells or other structural and / or functional biological entities associated with a disease to a "tumor" label and maps images 330 that do not include tumor cells or other structural and / or functional biological entities associated with a disease to a "non-tumor" label. The mapping data may be stored in a mapping data store (not shown). The mapping data can identify each image mapped to either a tumor label or a non-tumor label.

[0076] In some cases, the labels associated with training data set 350 may be received or may be derived from data received from remote system 350. The received data can include, for example, one or more medical records corresponding to a particular subject to which one or more of images 330 correspond. The medical records can indicate, for example, whether the subject had a tumor with respect to a period of time during which one or more input image elements associated with the subject were collected or a defined subsequent period, and / or the stage of progression of the subject's tumor (e.g., by specifying an expert diagnosis or characterization along standard measures and / or such metrics as total metabolic tumor volume (TMTV)). The received data can further include pixels of the location of tumors or tumor cells within one or more of the images associated with the subject. Thus, the medical records can include, or be used to identify, one or more labels for each training image. In some cases, the images or scans input to one or more classifier subsystems are received from remote system 335. For example, remote system 335 may receive image 330 from image generation system 305 and then transmit image 330 or the scan to analysis system 340 (e.g., along with an identifier of the subject and one or more labels).

[0077] The training controller 345 can use the mapping of the training dataset 350 to train the machine learning model. More specifically, the training controller 345 can access the architecture of the model, define the (fixed) hyperparameters of the model (which are parameters that affect the learning process such as the learning rate, the size / complexity of the model, etc.), and train the model such that a set of parameters is learned. More specifically, the set of parameters can be learned by identifying the parameter values associated with a low or minimum loss, cost, or error generated by comparing the predicted output (obtained using a given parameter value) with the actual output. In some cases, the machine learning model can be configured to iteratively fit a new model to improve the estimation accuracy of the output (e.g., a metric or identifier corresponding to an estimate or likelihood for a portion of an image including a depiction of tumor cells or other structural and / or functional biological entities). Training the machine learning model using a training dataset having a distribution corresponding to a representative distribution can result in a trained machine learning model that can more accurately detect depictions of tumor cells or other structural and / or functional biological entities associated with a disease compared to a machine learning model trained on a training dataset that is different from the representative distribution.

[0078] The machine learning (ML) execution handler 360 can process unlabeled data using the architecture of the machine learning model and the learned parameters to generate results. For example, the ML execution handler 360 can access biomedical images not represented in the training data set 350. For example, the biomedical image may be a histopathological image of a specimen slice not represented in the training data set 350. In some embodiments, the generated biomedical image is stored in a memory device. The image may be generated using the imaging system 325. In some embodiments, the image is generated or obtained from a microscope or other device capable of imaging the image data of a microscope slide holding a specimen, as described herein. In some embodiments, the biomedical image is generated or obtained using a 2D scanner such as one capable of scanning image tiles. Alternatively, the image may have been previously generated (e.g., scanned) and stored in a memory device (or, in that case, obtained from a server via a communication network).

[0079] In some examples, the biomedical image can be supplied by the ML execution handler 360 to a trained machine learning model having an architecture (e.g., U-Net) configured with the parameters learned and used in training. In other cases, the biomedical image is first decomposed by the ML execution handler 360 into smaller parts (e.g., tiles) as described in more detail herein, and then each tile can be supplied by the ML execution handler 360 to a trained machine learning model having an architecture (e.g., U-Net) configured with the parameters learned and used in training. The trained machine learning model may or may not have been trained on the training data set 350 having a distribution of characteristics corresponding to a representative distribution. The trained machine learning model can output predictions such as a feature map or image, classification, or a prediction as to whether the image depicts tumor cells or other structural and / or functional biological entities related to a disease.

[0080] The validation controller 365 can supply one or more biomedical images to the trained machine learning model to evaluate the performance metric of the prediction output by the trained machine learning model. For example, one or more validation data sets 370 of biomedical images can be generated to test the accuracy, precision, sensitivity, and / or F-score of the trained machine learning model in predicting the depiction of tumor cells of a disease. Each validation data set 370 can include a set of validation biomedical images from images 330a-n. In some cases, each of the one or more biomedical images depicts multiple tumor cells or multiple other structural and / or functional biological entities. The validation data set 370 may be collected (for example) from the image generation system 305.

[0081] The verification controller 365 can generate a verification dataset 370 to have a distribution of characteristics corresponding to a representative distribution of disease characteristics. The distribution can correspond to the representative distribution such that one or more measured traits of the distribution of the verification dataset 370 are close enough to the representative distribution. Since the representative distribution can change over time, the verification dataset 370 can be evaluated and modified as needed. In some cases, the verification controller 365 can perform a gap analysis between the representative distribution and the verification dataset 370 to evaluate the verification dataset 370. For example, when generating the verification dataset 370, the verification controller 365 can determine the distribution of disease characteristics within the verification dataset 370. The verification controller 365 can then compare the representative distribution with the determined distribution of the verification dataset 370 and determine whether there is a difference between the representative distribution and the determined distribution. If there is a difference, the verification controller 365 can modify the verification dataset 370 to mitigate the difference. For example, in the representative distribution, 50% of the cases of diffuse large B-cell lymphoma, which is a germinal center B-cell subtype, and 50% of the cases are of the activated B-cell subtype, and if the verification controller 365 determines that 45% of the images in the verification dataset 370 are related to the germinal center B-cell subtype and 55% are related to the activated B-cell subtype, the verification controller 365 can modify the verification dataset 370 to include 50% of the images related to the germinal center B-cell subtype and 50% of the images related to the activated B-cell subtype.

[0082] Similar to the training dataset 350, if the verification controller 365 determines that the images 330a - n do not contain representative data of the characteristics, the verification controller 365 can output a notification to the remote system 335 and can receive an adjustment of the amount of characteristics in the verification dataset 370. The verification controller 365 can then generate the verification dataset 370 accordingly.

[0083] In some cases, the ML execution handler 360 can access the validation dataset 370 and process the validation dataset 370 using the trained machine learning model. For each image in the validation dataset 370, the ML execution handler 360 can generate a prediction of the depiction of tumor cells or other structural and / or functional biological entities within the image. The validation controller 365 can compare the prediction to the ground truth of the image depicting the tumor cells or other structural and / or functional biological entities (e.g., based on the labels generated by the label mapper 355). Based on the comparison, the validation controller 365 can determine the performance metric of the trained machine learning model. The performance metric can include the accuracy, precision, sensitivity, and / or F-score of the trained machine learning model in predicting the depiction of tumor cells or other structural and / or functional biological entities of the disease.

[0084] The validation controller 365 can identify a threshold criterion for the metric associated with the prediction. For example, the threshold criterion can be a lower limit of 0.8 regarding the accuracy of the prediction of the depiction of tumor cells or other structural and / or functional biological entities. If the validation controller 365 determines that the trained machine learning model meets the threshold criterion by exceeding 0.8 regarding accuracy, the validation controller 365 can utilize the trained machine learning model for subsequent processing of biomedical images. For example, the validation controller 365 can make the trained machine learning model available for other entities or systems to process biomedical images related to the disease. Once made available, the trained machine learning model can receive a biomedical image and output a prediction of the biomedical image depicting tumor cells or other structural and / or functional biological entities.

[0085] Alternatively, if the verification controller 365 determines that the metric does not meet the threshold criteria, the verification controller 365 may still utilize the trained machine learning model for subsequent processing of the biomedical image, but the subsequent processing of the biomedical image may result in predictions and reliability levels of the predictions output by the trained machine learning model. The reliability level may be quantitative (e.g., percentage or decimal) or qualitative (e.g., low, medium, or high indication). By outputting the reliability level, the user can determine whether to trust the prediction or whether to perform additional processing on the biomedical image before being able to determine the presence of tumor cells or other structural and / or functional biological entities.

[0086] In some cases, when the trained machine learning model is utilized for subsequent processing of the biomedical image and the subsequent processing of the biomedical image is performed by the ML execution handler 360, the image property evaluator 375 identifies a predicted property evaluation of the biomedical image based on the execution of the image processing. The execution itself may generate results including property evaluations, or the execution may include results that can be used by the image property evaluator 375 to determine the predicted property evaluations of the specimen. For example, the subsequent processing may include property evaluations of the presence, quantity, and / or size of a set of tumor cells or other structural and / or functional biological entities predicted to be present in the biomedical image. In addition to or instead of this, the subsequent processing may include property evaluations of other structural and / or functional biological entities predicted to be present in the biomedical image, the diagnosis of a disease predicted to be present in the biomedical image, the classification of a disease predicted to be present in the biomedical image, and / or the prognosis prediction of a disease predicted to be present in the biomedical image. The image property evaluator 375 may apply rules and / or transformations to map probabilities and / or confidence levels to the property evaluations. As an example, if the result is that the probability that the biomedical image contains a set of tumor cells exceeds 50%, a first property evaluation may be assigned, otherwise, a second property evaluation may be assigned.

[0087] The communication interface 380 can collect results and communicate the results (or a processed version thereof) to a user device (e.g., related to a medical technician or caregiver) or another system. For example, the results can be communicated to the remote system 335. In some cases, the communication interface 380 can generate an output that identifies the presence, quantity, and / or size of a set of tumor cells or other structural and / or functional biological entities, a disease diagnosis, a disease classification, and / or a disease prognosis prediction. The output can then be presented, rendered, and / or transmitted, thereby facilitating, for example, the display of output data on a display of a computing device. The results can be used to make a diagnosis, determine a treatment plan, or evaluate an ongoing treatment of tumor cells.

[0088] It will be appreciated that modifications to the processes described with respect to the network 300 are contemplated. For example, if the sample is a liquid sample, embedding and / or sectioning can be omitted from the process.

[0089] IV. Image Tiling Aware of Architectures for Analysis of Pathology Slides or Scans U-Net is a common deep neural network architecture used in computer vision and digital pathology image analysis. The following discussion regarding image tiling with architecture awareness pertains to the application of this tiling technique to the U-Net architecture, but it should be understood that the same discussion and calculations can be applied to other CNNs such as, but not limited to, V-Net, residual neural networks, or recurrent neural networks. As shown in FIG. 4, U-Net 400 may include a contracting path 405 and an expanding path 410, thereby obtaining a u-shaped architecture. The contracting path 405 is a CNN network that includes repeated application of convolutions (e.g., 3×3 convolutions), followed by a rectified linear unit (ReLU) and a max pooling operation (e.g., 2×2 max pooling) for downsampling after each convolution. The input for the convolution operation is a two-dimensional tile or three-dimensional volume (e.g., an input image of size n×n, where n is the number of input features or pixels), and a set of "k" kernels (also called feature extractors) of respective sizes (e.g., f×f, where f is any number, e.g., 3 or 5). The output of the convolution operation is also a two-dimensional tile or three-dimensional volume (also called an output image or feature map) of size (m×m×k, where m is the number of output features and k is the convolution kernel size).

[0090] Each block 415 of the reduction path 315 includes one or more convolutional layers (indicated by the gray horizontal arrows), and the number of feature channels changes, for example, from 1 to 64 as the convolutional process increases the depth of the input image. The downward gray arrows between each block 415 are max pooling processes that halve the size of the input image. In each downsampling step or pooling operation, the number of feature channels may be doubled. During reduction, the spatial information of the image data decreases, but the feature information increases. Thus, information that was present in an image of, for example, 572×572 before pooling is (approximately) the same information that is present in an image of, for example, 284×284 after pooling. Here, when the convolutional operation is applied again to a subsequent process or layer, the filters of the subsequent process or layer can see a larger context, that is, as the input image progresses deeper into the network, the size of the input image decreases, but the receptive field increases (the receptive field (context) is the area of the input image that the kernel covers at any given point in time). After block 415 is executed, two more convolutions may be executed in block 420, but max pooling need not be performed. The image after block 420 has been resized, for example, to 28×28×1024 (this size is merely exemplary, and the size at the end of process 420 can vary depending on the starting size of the input image - size n×n×channels).

[0091] After the image is resized to, for example, 28x28x1024, the image is concatenated with the corresponding image from the downsampling path (see the horizontal gray bar 425 from the downsampling path 405) to create an image of size 56x56x1024 together. The reason for concatenation is to combine information from the previous layer to obtain a more accurate prediction (i.e., the high-resolution features from the downsampling path 405 are combined with the upsampled output from the upsampling path 410). This process continues as a series of upconvolutions (upsampling operators) that halve the number of channels, concatenation with the feature map cropped correspondingly from the downsampling path 405, repeated application of convolutions (e.g., two 3×3 convolutions) each followed by a rectified linear unit (ReLU), and a final convolution (e.g., a 1×1 convolution) in block 430 to generate the feature map or output image.

[0092] As described herein, when the convolutional layers of the downsampling path 405 and the upsampling path 410 perform a convolutional operation with boundary processing, the size of the output is the same as the input. However, regions near the image boundary contain "contaminated" values because the image boundary is padded with values that are not reasonable image values (see FIG. 5). When the convolutional layers of the downsampling path 405 and the upsampling path 410 perform a convolutional operation without boundary processing, the image size is reduced (see FIG. 6). As shown in FIGS. 5 and 6, the dimension reduction attribute N corresponds to the degree of size reduction or contamination suffered by the output from a machine learning model having an architecture such as U-Net400 when applying the kernel to the input image with or without padding, respectively.

[0093] One approach for calculating the dimension reduction attribute N when the machine learning model being used has a standard U-Net architecture is to define the attribute according to the following equations (1 - 3). TIFF2025524956000002.tif30170Where F is the kernel size, C is the number of convolutional layers at each level, and D is the number of pooling layers in the U-Net architecture. However, the formula for calculating the dimensionality reduction attribute N is not limited to these, but it should be understood that it may be adjusted to consider various architectural features including kernel size, convolutional layers, pooling layers, up-convolution, concatenation, normalization linear units, etc. in other architectures.

[0094] As shown in FIG. 7, architecture-aware image tiling involves implementing a tiling method that slides a tiling element (TE) 705 through a whole slide image (WSI) 710 until it traverses the entire WSI 710. The TE 705 includes an overlapping border region 715 of size N / 2 (based on the dimensionality reduction attribute N calculated in Equations 1-3) that surrounds a non-overlapping core region 720 (the hatched region in FIG. 7). For each portion of the WSI 710 where the TE 705 aligns as it slides through the WSI 710, a corresponding tile 705' is extracted from that portion of the WSI 710. Each tile will have the same dimensions as the TE 705, including an overlapping border region 715' of size N / 2 and a non-overlapping core region 720'. As shown in FIG. 7, when the TE 705 slides through the WSI 710, the core regions 720 corresponding to the core regions 720' of the tiles 705' extracted from adjacent portions of the WSI 710 are in contact with each other (adjacent to each other) but do not overlap (thus, non-overlapping regions). In contrast, the border regions 715 corresponding to the border regions 715' of the tiles 705' extracted from adjacent portions of the WSI 710 not only contact each other but also overlap each other (thus, overlapping regions). This approach facilitates ensuring that adjacent outputs do not overlap for convolutions where the kernel is perfectly aligned with non-padding values, and that there are no gaps corresponding to pixels where the kernel is perfectly aligned with non-padding values.

[0095] When each tile 705' extracted as TE705 slides through WSI710, it is input into a machine learning model, and the convolutional layer (without padding or border handling) trims the tile 705' so that only the core region 720' is used to generate the feature map or image output of the machine learning model. As a result, the aggregation of the feature map or image output of the machine learning model provides the overall result of the machine learning model in WSI710.

[0096] However, since each WSI710 can easily generate thousands of tiles 705' (depending on the size of the tissue and the size of TE705), in addition to generating more accurate results, it may be necessary to further optimize the downstream image processing to achieve efficient data transfer, result storage, data retrieval, etc. between the CPU and the GPU. To further optimize and achieve the most time-efficient image processing pipeline, specific size constraints can be introduced to the overlapping border region 715 beyond the size constraint, i.e., the dimension reduction attribute N, introduced based on the architecture. The size constraints can be determined based on the performance efficiency P of the machine learning model and the system that executes it. The performance efficiency P can be calculated in a manner similar to that described herein with respect to the dimension reduction attribute N, but the equation can be adjusted to consider various performance characteristics such as the data transfer speed between the CPU and the GPU, the available storage or memory, the data transfer speed for retrieving data from various subsystems, the processing speed, etc., although not limited to these. As shown in FIG. 8, architecture-aware image tiling can facilitate an efficient and accurate pipeline for image processing that can still be faithful to the deep neural network architecture while accommodating any size constraints on the overlapping border region 715 related to the performance efficiency of the machine learning model and the system that executes it.

[0097] FIG. 9 shows a flowchart illustrating a process 900 for image processing using image tiling aware of architectures according to various embodiments. The process 900 shown in FIG. 9 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of each system, hardware, or combinations thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The process 900 presented in FIG. 9 and described below is intended to be exemplary and non-limiting. Although FIG. 9 shows various processing steps performed in a particular sequence or order, it is not limited thereto. In certain alternative embodiments, the steps may be executed in several different orders or some steps may be executed in parallel. In certain embodiments, such as the embodiments shown in FIGS. 3-8, the processing shown in FIG. 9 may be performed by a network including an image generation system and an analysis system (e.g., the image generation system 305 and the analysis system 340 described in connection with FIG. 3) using a machine learning model having a deep neural network architecture to generate and process those images, or may be executed as part of such a network.

[0098] Process 900 begins at block 905 that accesses an image. The image may be a digital pathology image, such as a whole slide image that includes one or more types of cells. The image may have been generated in multiple dimensions by an imaging system, and thus, part of accessing the image may include identifying or determining the multiple dimensions of the image. Additionally, certain size attributes of the image may be identified or determined based on the multiple dimensions of the image. For example, a two-dimensional whole slide image may have a size attribute of 100,000 x 100,000 pixels. In some cases, tiling criteria are identified for a machine learning model used to process the image before accessing the image. The tiling criteria are configured such that a system, including the machine learning model and the memory resources that execute it, can handle the processing of the image with a given size attribute. If the size attribute of the image exceeds a predetermined spatial metric threshold, the tiling criteria are met and the image needs to be tiled prior to processing by the machine learning model. If the size attribute of the image does not exceed the predetermined spatial metric threshold, the tiling criteria are not met and the image can be processed directly by the machine learning model as is. In other words, a determination is made as to whether a system, including the machine learning model and the memory resources that execute it, can handle the processing of the image with a given size attribute. If it cannot, the image is divided into smaller parts using an image tiling process aware of the architecture disclosed herein, and the tiles generated from this process are input into the machine learning model. Alternatively, if it can, an image tiling process aware of the architecture is not necessary and the image is simply used as a direct input to the machine learning model.

[0099] In block 910, tiling elements of the image are generated. In some cases, the generation of the tiling elements is performed in response to a determination that the size attribute of the image exceeds a predetermined spatial metric threshold. The generation includes (i) determining the number of iterations of downsampling implemented in the machine learning model, (ii) determining the size of the kernel applied in the convolutional operation in the machine learning model, (iii) determining the number of convolutional layers implemented by the machine learning model for each level or each resolution, or any combination thereof. The generation further includes generating a non-overlapping core region and an overlapping border region of the tiling element. The overlapping border region surrounds the non-overlapping core region, and the size of the non-overlapping core region is determined based on the dimension reduction attribute N calculated from the number of iterations of downsampling, the size of the kernel, the number of convolutional layers, or any combination thereof. In some cases, the machine learning model is configured to include convolutional layers for each of one or more resolutions, and each iteration of downsampling results in the evaluation of an image version at a different resolution.

[0100] In block 915, tiles are extracted from the image using the tiling elements. Each tile has the same non-overlapping core region and the same overlapping border region as the tiling element. The extraction of tiles from the image using the tiling elements includes sliding the tiling element through the image until the tiling element traverses the entire image, and for each portion of the image where the tiling element is aligned as it slides through the image, a corresponding tile is extracted from that portion of the image, and each tile will have the same dimensions as the tiling element including the overlapping border region and the non-overlapping core region.

[0101] In some cases, when the tiling element slides through the image, the non-overlapping core regions of the tiles extracted from adjacent portions of the image are adjacent to each other but do not overlap each other, and the overlapping border regions of the tiles extracted from adjacent portions of the image overlap each other.

[0102] In block 920, each tile is input into the machine learning model.

[0103] In block 925, the convolutional portion of the image is generated for each tile using at least a convolutional layer, a kernel, and a downsampling layer. The convolutional layer without padding or border processing shrinks each tile so that only the non-overlapping core region is used to generate each of the convolutional portions of the image. In some cases, for each tile, generating the convolutional portion of the image involves (i) sliding the kernel one step at a time over the tile, (ii) for each step, placing the center of the kernel at a specific position on the tile, (iii) in the sliding process, the values of the kernel are multiplied element-wise with the corresponding values in the input region on the tile covered by the kernel to obtain a result, (iv) the results of the element-wise multiplication are added together to obtain a single value representing the output of the kernel at a specific position, and (v) repeating (i)-(iv) for all possible positions on the tile where the kernel can fit. Each convolutional layer outputs a new feature map, and each element of the new feature map represents the output of the application of the kernel to a specific position on the tile. During convolution, no padding or border processing is applied to each tile.

[0104] In block 930, the convolutional version of the image is generated using the convolutional portion of the image.

[0105] In block 935, the convolutional version of the image is output.

[0106] In block 940, a machine learning model or another machine learning model uses the convolutional version of the image to detect, characterize, classify, or combine some or all regions or objects in the image, and outputs inferences based on the detection, characterization, classification, or combination thereof.

[0107] In optional block 945, a diagnosis of an object related to an image or a convolutional version of the image is determined based on inferences output by a machine learning model or another machine learning model.

[0108] In optional block 950, a treatment is performed on an object related to an image or a convolutional version of the image. In some cases, the treatment is performed based on (i) an inference output by a machine learning model or another machine learning model, and / or (ii) the diagnosis of the object determined in block 945.

[0109] V. Additional Considerations Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium including instructions that, when executed by the one or more data processors, cause the one or more data processors to perform some or all of one or more of the methods disclosed herein and / or some or all of one or more of the processes. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein and / or some or all of one or more of the processes.

[0110] The terms and expressions used are used as terms for explanation rather than limitation, and there is no intention to exclude any equivalents of the features shown and described by using such terms and expressions. It is recognized that various modifications are possible within the scope of the claimed invention. Therefore, although the present invention described in the claims is specifically disclosed by embodiments and any features, modifications and variations of the concepts disclosed herein may be reclassified by those skilled in the art, and it should be understood that such modifications and variations are considered to be within the scope of the present invention as defined by the appended claims.

[0111] The following description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. Rather, the following description of the preferred exemplary embodiments provides those skilled in the art with a possible description for implementing various embodiments. It is understood that the functions and arrangements of the elements may be variously changed without departing from the spirit and scope as described in the appended claims.

[0112] In the following description, specific details are provided to provide a comprehensive understanding of the embodiments. However, it will be understood that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in the form of block diagrams so as not to obscure the embodiments with unnecessary details. In other cases, well-known circuits, processes, algorithms, structures, and technologies may be shown without unnecessary details in order to avoid obscuring the embodiments.

Claims

1. accessing an image; generating tiling elements for the image, wherein generating the tiling elements comprises: determining the number of downsampling layers implemented in a machine learning model used for processing the image; determining the size of a kernel applied during a convolutional operation in the machine learning model; generating a non-overlapping core region and an overlapping border region of the tiling elements; and the overlapping border region surrounds the non-overlapping core region, and the size of the non-overlapping core region is determined based on a dimension reduction attribute N calculated from the number of iterations of downsampling and the size of the kernel, generating tiling elements for the image; extracting tiles from the image using the tiling elements, wherein each tile has the same non-overlapping core region and the same overlapping border region as the tiling elements; inputting each tile into the machine learning model; for each tile, generating a convolutional portion of the image using at least a convolutional layer, the kernel, and the downsampling layer; generating a convolutional version of the image using the convolutional portion of the image; and outputting the convolutional version of the image comprising a computer-implemented method.

2. the machine learning model is configured to include the convolutional layer for each of one or more resolutions; each of the iterations of downsampling results in an evaluation of an image version with a different resolution; generating the tiling elements further comprises determining the number of convolutional layers implemented by the machine learning model for each level or each resolution; the dimension reduction attribute N is calculated from the number of iterations of downsampling, the size of the kernel, and the number of convolutional layers, the computer-implemented method according to claim 1.

3. the convolutional layer without padding or border processing reduces each tile such that only the non-overlapping core region is used to generate each convolutional portion of the image, the computer-implemented method according to claim 2.

4. extracting the tiles from the image using the tiling elements comprises sliding the tiling element through the image until the tiling element crosses the entire image comprising for each portion of the image where the tiling elements are aligned as the tiling element slides through the image, corresponding tiles are extracted from the portion of the image, each tile having the same dimensions as the tiling element including the overlapping border region and the non-overlapping core region, the computer-implemented method of claim 1 **Claim 5** when the tiling element slides through the image, the non-overlapping core regions of the tiles extracted from adjacent portions of the image are adjacent to each other but do not overlap each other, and the overlapping border regions of the tiles extracted from adjacent portions of the image overlap each other, the computer-implemented method of claim 4 **Claim 6** for each tile, generating the convolutional portion of the image comprises (i) sliding the kernel one step at a time over the tile (ii) for each step, positioning the center of the kernel at a specific location on the tile (iii) in the sliding process, the result is obtained by multiplying the value of the kernel by the corresponding value in the input region on the tile covered by the kernel on an element-by-element basis (iv) the results of the element-by-element multiplications are summed to obtain a single value representing the output of the kernel at the specific location (v) repeating (i) to (iv) for all possible locations on the tile where the kernel can fit comprising each convolutional layer outputs a new feature map, and each element of the new feature map represents the output of the application of the kernel to the specific location on the tile, the computer-implemented method of claim 1 **Claim 7** no padding or border processing is applied to each tile, the computer-implemented method of claim 6 **Claim 8** one or more data processors, a non-transitory computer-readable storage medium containing instructions comprising the instructions, when executed on the one or more data processors, access an image, generate a tiling element for the image, wherein generating the tiling element Determining the number of downsampling layers implemented in the machine learning model used for processing the image; Determining the size of the kernel applied during the convolution operation in the machine learning model; Generating the non-overlapping core region and the overlapping border region of the tiling element; including; The overlapping border region surrounds the non-overlapping core region, and the size of the non-overlapping core region is determined based on the dimension reduction attribute N calculated from the number of iterations of downsampling and the size of the kernel, generating a tiling element for the image; Extracting tiles from the image using the tiling element, wherein each tile has the same non-overlapping core region and the same overlapping border region as the tiling element; Inputting each tile into the machine learning model; For each tile, generating the convolutional part of the image using at least a convolutional layer, the kernel, and the downsampling layer; Generating a convolutional version of the image using the convolutional part of the image; Outputting the convolutional version of the image; A system that causes the one or more data processors to execute a set of operations including.

9. The machine learning model is configured to include the convolutional layer for each of one or more resolutions, Each of the iterations of downsampling results in an evaluation of different resolution versions of the image, Generating the tiling element further includes determining the number of convolutional layers implemented by the machine learning model for each level or each resolution, The system according to claim 8, wherein the dimension reduction attribute N is calculated from the number of iterations of downsampling, the size of the kernel, and the number of convolutional layers.

10. The system according to claim 9, wherein the convolutional layer without padding or border processing shrinks each tile such that only the non-overlapping core region is used to generate each convolutional part of the image.

11. Extracting the tile from the image using the tiling element; includes sliding the tiling element through the image until the tiling element crosses the entire image; including; For each portion of the image in which the tiling elements are aligned when the tiling elements slide through the image, a corresponding tile is extracted from the portion of the image, and each tile has the same dimensions as the tiling element including the overlapping border region and the non-overlapping core region, the system according to claim 8.

12. When the tiling elements slide through the image, the non-overlapping core regions of the tiles extracted from adjacent portions of the image are adjacent to each other but do not overlap each other, and the overlapping border regions of the tiles extracted from adjacent portions of the image overlap each other, the system according to claim 11.

13. For each tile, generating the convolutional portion of the image comprises (i) sliding the kernel one step at a time over the tile, (ii) for each step, positioning the center of the kernel at a specific location on the tile, (iii) in the sliding process, the value of the kernel is multiplied element-wise by the corresponding value in the input region on the tile covered by the kernel to obtain a result, (iv) the results of the element-wise multiplications are summed to obtain a single value representing the output of the kernel at the specific location, (v) repeating (i) to (iv) for all possible locations on the tile where the kernel can fit and each convolutional layer outputs a new feature map, and each element of the new feature map represents the output of the application of the kernel to the specific location on the tile, the system according to claim 8.

14. Neither padding nor border processing is applied to each tile, the system according to claim 13.

15. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, accessing an image, generating tiling elements for the image, and generating the tiling elements comprises determining the number of downsampling layers implemented in a machine learning model used for processing the image, determining the size of the kernel applied during convolutional operations in the machine learning model, generating the non-overlapping core region and the overlapping border region of the tiling elements and The repeating border region surrounds the non-repeating core region, and the size of the non-repeating core region is determined based on the dimension reduction attribute N calculated from the number of iterations of downsampling and the size of the kernel, and generates a tiling element for the image. Extracting tiles from the image using the tiling element, wherein each tile has the same non-repeating core region and the same repeating border region as the tiling element. Inputting each tile into the machine learning model. For each tile, generating a convolutional portion of the image using at least a convolutional layer, the kernel, and the downsampling layer. Generating a convolutional version of the image using the convolutional portion of the image. Outputting the convolutional version of the image. A computer program product including instructions configured to cause one or more data processors to perform a set of operations including the above.

16. The machine learning model is configured to include the convolutional layer for each of one or more resolutions. Each of the iterations of downsampling results in an evaluation of an image version with a different resolution. Generating the tiling element further includes determining the number of convolutional layers implemented by the machine learning model for each level or each resolution. The dimension reduction attribute N is calculated from the number of iterations of downsampling, the size of the kernel, and the number of convolutional layers. The computer program product according to claim 15.

17. The convolutional layer without padding or border processing shrinks each tile so that only the non-repeating core region is used to generate each convolutional portion of the image. The computer program product according to claim 16.

18. Extracting the tile from the image using the tiling element includes sliding the tiling element through the image until the tiling element crosses the entire image. ​ For each portion of the image in which the tiling elements are aligned when the tiling elements slide through the image, a corresponding tile is extracted from the portion of the image, and each tile has the same dimensions as the tiling element including the overlapping border region and the non-overlapping core region, the computer program product according to claim 15.

19. When the tiling elements slide through the image, the non-overlapping core regions of the tiles extracted from adjacent portions of the image are adjacent to each other but do not overlap each other, and the overlapping border regions of the tiles extracted from adjacent portions of the image overlap each other, the computer program product according to claim 18.

20. For each tile, generating the convolution portion of the image comprises (i) sliding the kernel one step at a time over the tile, (ii) for each step, placing the center of the kernel at a specific position on the tile, (iii) in the sliding process, the value of the kernel is multiplied element-by-element with the corresponding value in the input region on the tile covered by the kernel to obtain a result, (iv) the results of the element-by-element multiplications are summed to obtain a single value representing the output of the kernel at the specific position, (v) repeating (i) to (iv) for all possible positions on the tile where the kernel can fit and each convolutional layer outputs a new feature map, and each element of the new feature map represents the output of the application of the kernel to the specific position on the tile, the computer program product according to claim 15.

Images