Deep multiscale network for multi-class image segmentation

Abstract

To provide a novel deep multiscale network (DMMN).SOLUTION: The multi-class tissue segmentation architecture processes a set of patches from multiple magnifications to make more accurate predictions. For supervised training, partial annotations can be used to reduce the burden on the annotator. The multi-encoder, multi-decoder, and multi-concatenation segmentation architecture outperforms other segmentation architectures on breast data sets and can be used to help pathologists assess breast cancer in margin specimens.SELECTED DRAWING: FIG. 20 (a)

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 909,401, filed October 2, 2019, entitled "Deep Multi-Magnification Networks for Multi-Class Breast Cancer Image Segmentation," which is incorporated by reference in its entirety. [Background technology]

[0002] Computer vision algorithms can be used to recognize and detect various features on digital images. Feature detection in biomedical images can consume large amounts of computing resources and time due to the potentially large resolution and size of biomedical images. Summary of the Invention

[0003] Breast cancer is one of the most common cancers among women in the United States. Analyzing the margin status of surgical procedures is important for evaluating surgical outcomes and has implications for future treatment of breast cancer patients. Tissue analysis is performed manually by pathologists examining glass slides containing margins of interest. Digital pathology has provided a means to digitize glass slides and generate whole-slide images. Computational pathology, particularly with advances in deep learning, allows whole-slide images to be automatically analyzed to assist pathologists. Whole-slide images typically contain gigapixels of data, making it impractical to process images at the whole-slide level. Most current deep learning techniques process images at the patch level, which can produce poor results by viewing individual patches with a narrow field of view at a single magnification.

[0004] Presented herein is a deep multi-magnification network (DMMN) that mimics the way pathologists view slides under their microscope. The multi-class tissue segmentation architecture processes sets of patches from multiple magnifications to make more accurate predictions. For supervised training, partial annotation can be used to reduce the burden on the annotator. A multi-encoder, multi-decoder, and multi-connection segmentation architecture outperforms other segmentation architectures on breast datasets and can be used to facilitate pathologists' assessment of breast cancer in margin specimens.

[0005] At least one aspect of the present disclosure is directed to a system and method for segmenting a biomedical image using a multi-magnification encoder-decoder concatenated network. A computing system having one or more processors can identify a biomedical image derived from a histopathological image generator. The biomedical image can be divided into a plurality of tiles. Each tile of the plurality of tiles can correspond to a portion of the biomedical image. The computing system can create a plurality of patches from at least one tile of the plurality of tiles of the biomedical image using a corresponding plurality of magnification factors. The plurality of patches can include a first patch at a first magnification factor of the plurality of magnification factors, a second patch at a second magnification factor of the plurality of magnification factors, and a third patch at a third magnification factor of the plurality of magnification factors.

[0006] Additionally, the computing system can apply a segmentation model to multiple patches from at least one tile. The segmentation model can include multiple networks for corresponding multiple augmentation factors. The multiple networks can include a first network for patches of a first augmentation factor. The first network can have a first set of encoders and a first set of decoders and can convert the first patch into a first set of feature maps for the first augmentation factor. Each decoder in the first set can have a concatenator that combines feature maps from the successive networks. The multiple networks can include a second network for patches of a second augmentation factor. The second network can have a second set of encoders and a second set of decoders and can convert the second patch into a second set of feature maps for the first augmentation factor. Each encoder in the second set can feed an output feature map to a concatenator of a corresponding decoder in the first set of the first network. The multiple networks can include a third network for patches of a third augmentation factor. The third network can have a third set of encoders and a third set of decoders and can convert the second patch into a third set of feature maps for the third augmentation factor. At least one of the third set of encoders can feed output feature maps to a concatenator of a corresponding first set of decoders in the first network. The computing system can generate segmented tiles corresponding to at least one tile of the first upscaling factor using the first set of feature maps output by the first network of the plurality of networks of the segmentation model.

[0007] At least one aspect of the present disclosure is directed to training a multi-magnification encoder-decoder concatenated network for segmenting biomedical images. A computing system having one or more processors can identify a training dataset. The training dataset can include a sample biomedical image derived from a histopathological image generator. The sample biomedical image can be divided into a plurality of tiles. Each tile of the plurality of tiles can correspond to a portion of the sample biomedical image. The sample biomedical image can have a region of interest. The training dataset can include an annotation labeling the portion of the region of interest. The annotation can indicate that at least a portion of the region of interest in the sample biomedical image is to be segmented. The computing system can create a plurality of patches from each tile of the plurality of tiles of the sample biomedical image using a corresponding plurality of magnification factors. The plurality of patches can include a first patch at a first magnification factor of the plurality of magnification factors, a second patch at a second magnification factor of the plurality of magnification factors, and a third patch at a third magnification factor of the plurality of magnification factors.

[0008] Additionally, the computing system can apply a segmentation model to multiple patches from at least one tile. The segmentation model can include multiple networks for corresponding multiple augmentation factors. The multiple networks can include a first network for patches of a first augmentation factor. The first network can have a first set of encoders and a first set of decoders and can convert the first patches into a first set of feature maps for the first augmentation factor. Each decoder in the first set can have a concatenator that combines feature maps from the successive networks. The multiple networks can include a second network for patches of a second augmentation factor. The second network can have a second set of encoders and a second set of decoders and can convert the second patches into a second set of feature maps for the first augmentation factor. Each encoder in the second set can feed an output feature map to a concatenator of a corresponding decoder in the first set of the first network. The multiple networks can include a third network for patches of a third augmentation factor. The third network can have a third set of encoders and a third set of decoders and can convert the second patches into a third set of feature maps for the third augmentation factor.

[0009] Further, the computing system can generate a segmented biomedical image using a first set of feature maps output by a first network of the plurality of networks of the segmentation model across a plurality of tiles of the biomedical image. The computing system can determine an error metric between the segmented biomedical image and the sample biomedical image based on annotations labeling portions of regions of interest in the sample biomedical image. The computing system can modify at least one parameter in the plurality of networks of the segmentation model based on the error metric.

[0010] At least one aspect of the present disclosure is directed to a system and method for segmenting a biomedical image. A computing system having one or more processors can identify a first patch at a first magnification factor and a second patch at a second magnification factor for at least one tile of the biomedical image. The computing system can apply a trained segmentation model to the first patch and the second patch to generate a segmented tile. The trained segmentation model can include multiple networks. The multiple networks can include a first network for generating multiple first feature maps using the first patch at the first magnification factor. The multiple networks can include a second network for generating a second feature map using the second patch at the second magnification factor and one or more first feature maps from the first network. The computing system can store the segmented tile identifying a region of interest within the at least one tile of the biomedical image.

[0011] In some embodiments, the plurality of networks of the segmentation model may include a third network. The third network may generate a plurality of third feature maps using a third patch of at least one tile at a third expansion factor. The third network may provide the plurality of third feature maps to corresponding plurality of decoders of the second network to generate a second feature map.

[0012] In some embodiments, the second network may include a plurality of decoders arranged across a corresponding plurality of columns. Each of the plurality of decoders may process a corresponding feature map of the plurality of first maps from the first network. In some embodiments, the first network may include a plurality of encoders arranged across a corresponding plurality of columns. Each of the plurality of encoders may provide a corresponding feature map of the plurality of first networks to a respective decoder in the second network. In some embodiments, the second network may include a plurality of concatenators that combine the plurality of first feature maps from the first network with a corresponding plurality of intermediate feature maps in generating the second feature map.

[0013] In some embodiments, the computing system can generate a segmented biomedical image using a plurality of segmented tiles by applying a segmentation model applied to a plurality of patches at a corresponding plurality of magnification factors. Each patch can be identified from a corresponding tile of the plurality of tiles of the biomedical image. In some embodiments, the computing system can obtain a biomedical image derived from a histopathological image generator. The biomedical image can be divided into a plurality of tiles. Each tile of the plurality of tiles corresponds to a portion of the biomedical image.

[0014] At least one aspect of the present disclosure is directed to a system and method for training a network for segmenting biomedical images. A computing system having one or more processors can identify a training dataset. The training dataset can include at least one sample tile from a sample biomedical image. The sample biomedical image can have a region of interest. The training dataset can include annotations labeling at least a portion of the region of interest. The annotations can indicate at least a portion of the region of interest in the at least one sample tile. The computing system can generate a first patch at a first magnification factor and a second patch at a second magnification factor for the at least one sample tile of the sample biomedical image. The computing system can train a segmentation model using the first patch, the second patch, and the annotations of the at least one sample tile. The segmentation model can include multiple networks. The multiple networks can include a first network for generating multiple first feature maps using the first patch at the first magnification factor. The plurality of networks can include a second network for generating a second feature map using the second patch at the second dilation factor and one or more first feature maps from the first network. The segmented tiles corresponding to the second feature map can be compared with the annotation.

[0015] In some embodiments, the computing system can train the segmentation model by determining an error metric between the segmented tile and the sample tile based on the annotations labeling the portion of the region of interest. In some embodiments, the computing system can train the segmentation model by updating at least one parameter in a plurality of networks of the segmentation model using the error metric.

[0016] In some embodiments, the plurality of networks of the segmentation model may include a third network. The third network may generate a plurality of third feature maps using a third patch of at least one tile at a third expansion factor. The third network may provide the plurality of third feature maps to corresponding plurality of decoders of the second network to generate a second feature map.

[0017] In some embodiments, the second network may include a plurality of decoders arranged across a corresponding plurality of columns, each of which may process a corresponding feature map of the plurality of first maps from the first network. In some embodiments, the first network may include a plurality of encoders arranged across a corresponding plurality of columns, each of which may provide a corresponding feature map of the plurality of first networks to a respective decoder in the second network.

[0018] In some embodiments, the second network can include a plurality of concatenators that combine a plurality of first feature maps from the first network with a corresponding plurality of intermediate feature maps in generating the second feature map. In some embodiments, the annotations in the training dataset can label less than an entire region of interest in the sample biomedical images. The annotations can be separated from the entire edge of the region of interest.

[0019] In some embodiments, the computing system can generate a segmented biomedical image using a plurality of segmented tiles by applying a segmentation model applied to a plurality of patches at a corresponding plurality of magnification factors. Each patch can be identified from a corresponding tile of the plurality of tiles of the sample biomedical image. In some embodiments, the sample biomedical image can be derived from a histopathological image generator. The sample biomedical image can be divided into a plurality of tiles. Each tile of the plurality of tiles can correspond to a portion of the sample biomedical image.

[0020] The above and other objects, aspects, features and advantages of the present disclosure will become more apparent and will be better understood by referring to the following description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0021] [Figure 1(a)-(b)] Comparison between deep single-magnification networks (DSMN) and deep multiple-magnification networks (DMMN). (a) A deep single-magnification network only sees patches from a single magnification with a limited field of view. (b) A deep multiple-magnification network can see a set of patches from multiple magnifications to have a broader field of view. [Figure 2]Block diagram of the proposed method using a deep multi-magnification network. The first step of the method is to partially annotate training whole slide images. After extracting training patches from the partial annotations and balancing the number of pixels between classes, a deep multi-magnification network is trained. The trained network is used for multi-class tissue segmentation of whole slide images. [Figure 3(a)-(b)] Examples of partial annotation: (a) Whole slide image from the breast cancer dataset. (b) Partially annotated image of the whole slide image in (a), where multiple tissue subtypes are annotated with distinct colors and white regions are unlabeled. [Figure 4(a)-(d)] CNN architectures for multi-class tissue segmentation using deep single-scale networks in (a) and deep multiple-scale networks in (b)-(d). (a) Single-Encoder Single-Decoder (SESD) is an architecture that uses patches from a single scale to generate segmentation prediction patches. (b) Multiple-Encoder Single-Decoder (MESD) is an architecture that uses multiple patches of various scales but has only one decoder to generate segmentation prediction patches. (c) Multiple-Encoder Multiple-Decoder Single-Concatenation (MEMDSC) is an architecture that uses multiple patches of various scales, but feature maps are concatenated only in the final layer to generate segmentation prediction patches. (d) Multiple-Encoder Multiple-Decoder Multiple-Concatenation (MEMDMC) is an architecture that uses multiple patches of various scales, and feature maps are concatenated between intermediate layers to strengthen the feature maps for the 20x decoder. [Figure 5(a)-(b)] Class equilibrium with elastic deformation in the training breast dataset. (a) Number of annotated pixels between classes before elastic deformation. (b) Number of annotated pixels between classes after elastic deformation. [Figure 6(a)-(l)]Segmentation predictions for the TNBC dataset. (a)–(f) are thumbnail versions of the entire slide image, and (g)–(l) are zoomed-in images with a size of 1024 × 1024 pixels at 10× magnification. (a) and (g) are original images, (b) and (h) are partial annotations, (c) and (i) are segmentation predictions using the single-encoder-single-decoder (SESD) architecture, (d) and (j) are segmentation predictions using the multiple-encoder-single-decoder (MESD) architecture, (e) and (k) are segmentation predictions using the multiple-encoder-multiple-decoder-single-concatenation (MEMDSC) architecture, and (f) and (l) are segmentation predictions using the multiple-encoder-multiple-decoder-multiple-concatenation (MEMDMC) architecture. [Figure 7(a)-(l)] Segmentation predictions for the TNBC dataset. (a)–(f) are thumbnail versions of the entire slide image, and (g)–(l) are zoomed-in images with a size of 1024 × 1024 pixels at 10× magnification. (a) and (g) are original images, (b) and (h) are partial annotations, (c) and (i) are segmentation predictions using the single-encoder-single-decoder (SESD) architecture, (d) and (j) are segmentation predictions using the multiple-encoder-single-decoder (MESD) architecture, (e) and (k) are segmentation predictions using the multiple-encoder-multiple-decoder-single-concatenation (MEMDSC) architecture, and (f) and (l) are segmentation predictions using the multiple-encoder-multiple-decoder-multiple-concatenation (MEMDMC) architecture. [Figure 8(a)-(l)]Segmentation predictions for the breast margin dataset. (a)–(f) are thumbnail versions of the whole slide image, and (g)–(l) are zoomed-in images with a size of 1024 × 1024 pixels at 10× magnification. (a) and (g) are original images, (b) and (h) are global annotations, (c) and (i) are segmentation predictions using a single-encoder, single-decoder (SESD) architecture, (d) and (j) are segmentation predictions using a multiple-encoder, single-decoder (MESD) architecture, (e) and (k) are segmentation predictions using a multiple-encoder, multiple-decoder, single-concatenation (MEMDSC) architecture, and (f) and (l) are segmentation predictions using a multiple-encoder, multiple-decoder, multiple-concatenation (MEMDMC) architecture. [Figure 9(a)-(l)] Segmentation predictions for the breast margin dataset. (a)–(f) are thumbnail versions of the whole slide image, and (g)–(l) are zoomed-in images with a size of 1024 × 1024 pixels at 10× magnification. (a) and (g) are original images, (b) and (h) are global annotations, (c) and (i) are segmentation predictions using a single-encoder, single-decoder (SESD) architecture, (d) and (j) are segmentation predictions using a multiple-encoder, single-decoder (MESD) architecture, (e) and (k) are segmentation predictions using a multiple-encoder, multiple-decoder, single-concatenation (MEMDSC) architecture, and (f) and (l) are segmentation predictions using a multiple-encoder, multiple-decoder, multiple-concatenation (MEMDMC) architecture. [Figure 10(a)-(b)]Comparison between deep single magnification network (DSMN) and deep multiple magnification network (DMMN). (a) DSMN views patches from a single magnification with a limited field of view. (b) DMMN views a set of patches from multiple magnifications to have a broader field of view. DMMN can learn both cellular features from higher magnifications and structural growth patterns from lower magnifications for more accurate predictions. [Figure 11] Block diagram of the proposed method using a deep multi-magnification network. The first step of the method is to partially annotate training whole slide images. After extracting training patches from the partial annotations and balancing the number of pixels between classes, a deep multi-magnification network is trained. The trained network is used for multi-class tissue segmentation of whole slide images. [Figure 12(a)-(b)] Examples of partial annotation: (a) Whole slide image from breast tissue. (b) Partially annotated image, where multiple tissue subtypes are annotated with distinct colors and white regions are unlabeled. [Figure 13(a)-(e)]CNN architectures for multi-class tissue segmentation using deep single-magnification networks (DSMNs), using single-magnification and deep multiple-magnification networks (DMMNs) in (a) and multiple patches at various magnifications in (b)–(e). (a) U-Net is used as the DSMN architecture. (b) Single-encoder, single-decoder (DMMN-S2) is a DMMN architecture in which multiple patches are concatenated and used as input to the U-Net architecture. (c) Multiple-encoder, single-decoder (DMMN-MS) is a DMMN architecture with only one decoder. (d) Multiple-encoder, multiple-decoder, single-concatenation (DMMN-M2S) is a DMMN architecture in which feature maps from multiple magnifications are concatenated only in the final layer. (e) The proposed multiple-encoder, multiple-decoder, multiple-concatenation (DMMN-M3) is a DMMN architecture in which feature maps are concatenated between hidden layers to strengthen the feature maps in the decoder with the highest magnification. [Figure 14] Class balancing using elastic deformations in a training breast dataset. [Figure 15(a)-(p)] Segmentation predictions for Dataset-I using two deep single-magnification networks (DSMNs), SegNet and U-Net, and four deep multiple-magnification networks (DMMNs), single-encoder single-decoder (DMMN-S2), multiple-encoder single-decoder (DMMN-MS), multiple-encoder multiple-decoder single concatenation (DMMN-M2S), and the proposed multiple-encoder multiple-decoder multiple concatenation (DMMN-M3). (a)-(h) are thumbnail versions of the whole slide images, and (i)-(p) are zoomed-in images of invasive ductal carcinoma (IDC) with a size of 1024 x 1024 pixels at 10x magnification. [Figure 16(a)-(p)]Segmentation predictions for Dataset-I using two deep single-magnification networks (DSMNs), SegNet and U-Net, and four deep multiple-magnification networks (DMMNs), single-encoder single-decoder (DMMN-S2), multiple-encoder single-decoder (DMMN-MS), multiple-encoder multiple-decoder single concatenation (DMMN-M2S), and the proposed multiple-encoder multiple-decoder multiple concatenation (DMMN-M3). (a)-(h) are thumbnail versions of the whole slide images, and (i)-(p) are zoomed-in images of benign epithelial tissue with a size of 1024 × 1024 pixels at 10× magnification. [Figure 17(a)-(p)] Segmentation predictions for Dataset-II using two deep single-magnification networks (DSMNs), SegNet and U-Net, and four deep multiple-magnification networks (DMMNs): single-encoder single-decoder (DMMN-S2), multiple-encoder single-decoder (DMMN-MS), multiple-encoder multiple-decoder single concatenation (DMMN-M2S), and the proposed multiple-encoder multiple-decoder multiple concatenation (DMMN-M3). (a)-(h) are thumbnail versions of the whole slide images, and (i)-(p) are zoomed-in images of ductal carcinoma in situ (DCIS) with a size of 1024 × 1024 pixels at 10× magnification. [Figure 18] Confusion matrices evaluating the segmentation of carcinoma, benign epithelium, stroma, necrosis, fat, and background for Dataset-I based on two deep single-magnification networks (DSMNs), SegNet and U-Net, and four deep multiple-magnification networks (DMMNs), single-encoder single-decoder (DMMN-S2), multiple-encoder single-decoder (DMMN-MS), multiple-encoder multiple-decoder single concatenation (DMMN-M2S), and the proposed multiple-encoder multiple-decoder multiple concatenation (DMMN-M3). [Figure 19]Confusion matrices for evaluating the segmentation of carcinoma, benign epithelium, and stroma on Dataset-II based on two deep single-magnification networks (DSMNs), SegNet and U-Net, and four deep multiple-magnification networks (DMMNs): single-encoder single-decoder (DMMN-S2), multiple-encoder single-decoder (DMMN-MS), multiple-encoder multiple-decoder single concatenation (DMMN-M2S), and the proposed multiple-encoder multiple-decoder multiple concatenation (DMMN-M3). Necrosis, fat, and background are excluded from the evaluation on Dataset-II due to the lack of evaluated pixels. [Figure 20(a)] FIG. 1 shows a block diagram of an exemplary system for segmenting biomedical images using a multi-scale encoder-decoder concatenation network. [Figure 20(b)] 1 shows an example biomedical image with partial annotations for training a segmentation model. [Figure 20(c)] FIG. 1 shows a block diagram of a convolutional block of a network in a segmentation model for segmenting biomedical images. [Figure 20(d)] FIG. 1 shows a block diagram of the deconvolution block of a network in a segmentation model for segmenting biomedical images. [Figure 20(e)] 1 shows a block diagram of the terminal convolution block of the network in the segmentation model. [Figure 20(f)] FIG. 1 shows a block diagram of a network in a segmentation model for segmenting biomedical images. [Figure 20(g)] FIG. 1 shows a block diagram of a segmentation model for segmenting biomedical images. [Figure 20(h)] 1 shows a flow diagram of a method for segmenting biomedical images using a multi-scale encoder-decoder concatenated network. [Figure 20(i)]1 shows a flow diagram of a method for training a model for segmenting biomedical images using a multi-scale encoder-decoder concatenated network. [Figure 21(a)] 1 is a block diagram illustrating an embodiment of a network environment including a client device in communication with a server device. [Figure 21(b)] FIG. 1 is a block diagram illustrating a cloud computing environment including a client device in communication with a cloud service provider. [Figure 21(c)-(d)] FIG. 1 is a block diagram illustrating an embodiment of a computing device useful in connection with the methods and systems described herein. DETAILED DESCRIPTION OF THE INVENTION

[0022] The following is a more detailed description of various concepts related to systems and methods for segmenting biomedical images and embodiments thereof. It should be understood that the various concepts introduced above and discussed in further detail below can be implemented in any of numerous ways, as the disclosed concepts are not limited to any particular implementation manner. Examples of specific implementations and applications are provided primarily for illustrative purposes.

[0023] Section A describes the first approach for deep multi-magnification networks for multi-class breast cancer image segmentation.

[0024] Section B describes a second approach for deep multi-magnification networks for multi-class breast cancer image segmentation.

[0025] Section C describes a system and method for segmenting biomedical images using multiple scaling factors, multiple encoders, multiple decoders and multiple connected networks.

[0026] Section D describes network and computing environments that may be useful in implementing various computing-related embodiments described herein.

[0027] A. First approach for deep multi-magnification networks for multi-class breast cancer image segmentation I. Introduction Breast cancer is the most commonly diagnosed cancer and the second leading cause of cancer deaths in women in the United States. Approximately 12% of women in the United States will be diagnosed with breast cancer during their lifetime. Patients with early-stage breast cancer often undergo breast-conserving surgery, or lumpectomy, which removes a portion of the breast tissue containing the cancer. Lumpectomy, usually combined with adjuvant radiation therapy, has been shown to be equivalent to mastectomy in terms of survival rate and to provide improved cosmetic outcomes. During surgery, the surgeon's goal is to remove the entire cancerous tumor and a rim of benign tissue surrounding the tumor. A common method for assessing surgical margins for lumpectomy specimens involves the surgeon resecting an additional segment of tissue from the wall of the lumpectomy cavity after the main mass containing the cancer has been removed. This "cavity shave" method, which allows the surgeon to specify specific margins intraoperatively, is associated with lower rates of positive margins (tumor present in the inked margins) and lower rates of re-excisions. To determine the completeness of surgical resection, the edges of tumor resection specimens, or margins, are evaluated microscopically by pathologists. Achieving negative margins (no cancer found touching the margins) is crucial to minimize the risk of local cancer recurrence. Pathological analysis of margin specimens involves pathologists examining approximately 20–40 histological slides per case, a process that can be time-consuming and tedious. With the increasing ability to digitally scan histological glass slides, computational pathology techniques could potentially improve the efficiency and accuracy of this process by evaluating whole slide images (WSIs) of margin specimens. Ultimately, accurate margin analysis by pathologists is crucial to determining the need for additional surgery.

[0028] Various approaches have been used to analyze WSI. Most models involve localization, detection, classification, and segmentation of objects (i.e., histological features) in digital slides. Histopathological features include pattern-based identification information such as nuclear features, cellular / stromal structures, or tissues. Computational pathology has been used in nuclear segmentation to extract nuclear features, such as size, shape, and their relationships. Nuclear segmentation is performed using adaptive thresholding and morphological operations to find areas of high nuclear density. Breast cancer grading methods can be developed by segmenting glands and nuclei using Bayesian classifiers and structural constraints from domain knowledge. Integrated active contours based on region, boundary, and shape can be proposed to segment overlapping nuclei and lymphocytes. These nuclear segmentation-based approaches are challenging because the shapes of nuclei and structures in cancerous regions can have large variations in the tissue captured in WSI.

[0029] Due to the availability of large training datasets and advances in graphics processing units (GPUs), deep learning, a type of machine learning, can be used for automated image analysis. Deep learning models, composed of deep layers with nonlinear activation functions, enable the learning of more refined features. In particular, convolutional neural networks (CNNs) learning spatial features in images have shown excellent results in image classification, object detection, and semantic segmentation. Fully convolutional networks (FCNs) can be used for semantic segmentation, also known as pixel-wise classification, and can understand the location, size, and shape of objects within an image. FCNs consist of an encoder and a decoder, where the encoder extracts low-dimensional features from the input image, and the decoder uses the low-dimensional features to generate segmentation predictions. Semantic segmentation has been used in medical imaging to automatically segment biological structures. For example, U-Nets are used to segment cells in microscopic images. The U-Net architecture has connections that transfer feature maps from the encoder to the decoder to preserve spatial information. This architecture demonstrated more accurate segmentation predictions for biomedical images.

[0030] Deep learning can be used in the computational pathology community. Researchers have demonstrated automated identification of invasive breast cancer in WSIs using a simple three-layer CNN. A method for classifying breast tissue slides as invasive cancer or benign by analyzing stromal regions can include using a CNN. A multi-instance learning-based CNN achieved 100% sensitivity when the CNN was trained with 44,732 WSIs from 15,187 patients. The availability of public pathology datasets has encouraged the development of many deep learning methods for computational pathology. For example, a breast cancer dataset for detecting lymph node metastasis was released for the CAMELYON challenge, and several deep learning techniques have been developed to analyze breast cancer datasets.

[0031] One drawback of using deep learning for WSI is that the overall size of a single WSI is too large to fit into a GPU. Images can be downsampled for processing by a pre-trained CNN, but crucial details necessary for clinical diagnosis in WSI are lost. To address this issue, patch-based approaches are typically used instead of slide-level approaches. Here, patches are extracted from the WSI and processed by a CNN. A patch-based process similar to multiclass logistic regression can be used for slide-level classification. The winner of the CAMELYON 16 challenge used the Otsu thresholding technique to extract tissue regions and train a patch-based model to classify tumor and non-tumor patches. To increase performance, class balancing between tumor and non-tumor patches and data augmentation techniques such as rotation, flip, and color jittering can be used. In addition, the winner of the CAMELYON 17 challenge developed a patch overlap strategy for more accurate prediction. Patches can be processed with additional larger patches containing boundary regions of the same magnification to segment subtypes in breast WSI. Alternatively, representation-aggregation CNNs have been developed to collect features generated from patches in WSI to share representations between patches. Patch-based approaches are impractical because (1) pathologists do not view patch-level slides with a narrow field of view, and (2) they frequently switch zoom levels to view slides at multiple magnifications to accurately analyze them.

[0032] To develop a more realistic CNN, it is necessary to input a set of patches at multiple magnifications, increasing the field of view and providing additional information from other magnifications. Figure 1 illustrates the difference between a deep single-magnification network (DSMN) and a deep multiple-magnification network (DMMN). The input to the DSMN in Figure 1(a) is a single patch with a size of 256 x 256 pixels at a single magnification of 20x, which limits the field of view. The input to the DMMN in Figure 1(b) is a set of patches with a size of 256 x 256 pixels at multiple magnifications of 20x, 10x, and 5x, which allows for a wider field of view. By providing multiple magnifications with a wider field of view, the DMMN can mimic the way a pathologist views a slide using a microscope, thereby producing a more accurate analysis.

[0033] Several approaches exist for analyzing whole-slide images using multiple magnifications. A binary segmentation CNN can be used to segment tumor regions in the CAMELYON dataset. In this work, four encoders with different magnifications are implemented, but only one decoder is used to generate the final segmentation prediction. A CNN architecture consisting of three expert networks with different magnifications, a weighting network to automatically select weights to emphasize specific magnifications based on the input patches, and an aggregation network to generate the final segmentation prediction can also be used. However, intermediate feature maps are not shared among the three expert networks, which may limit the use of feature maps from multiple magnifications.

[0034] This disclosure presents a deep multi-magnification network (DMMN) for accurately segmenting multiple subtypes in breast tissue images, with the goal of identifying breast cancers found in margin specimens. The DMMN architecture has multiple encoders, multiple decoders, and multiple connections between the decoders to have a richer feature map in the intermediate layers. To train the DMMN, WSIs can be partially annotated to reduce the annotation burden. A DMMN model trained with partial annotation can not only learn the features of each subtype but also the morphological relationships between subtypes, leading to superior segmentation performance. The multi-magnification model is tested on two breast datasets, and it is observed that the model consistently outperforms other architectures. This method can be used to automatically segment cancerous regions on breast margin images, assist in diagnosing patients' margin status, and determine future treatment. A deep multi-magnification network can be developed to combine feature maps from various magnifications for more accurate segmentation predictions. Partial annotation can be used to save pathologists' annotation time while still achieving high performance.

[0035] II. Method Figure 2 shows a block diagram of the method. The goal is to segment cancer regions in breast margin images using a deep multi-magnification network (DMMN). Breast margin images do not contain large cancer regions. Therefore, another breast cancer dataset containing large cancer regions can be used as the training dataset. First, manual annotation is performed on the C-class training dataset. Note that this annotation is performed in part for an efficient and fast process. To train the multi-class segmentation DMMN, patches are extracted from the whole slide image and the corresponding annotations. Before training the DMMN with the extracted patches, elastic deformation can be used to multiply patches belonging to rare classes and balance the number of pixels between classes. After the training step is performed, the model can be used for multi-class segmentation of breast cancer images.

[0036] Partial annotation Supervised learning requires a large set of annotations, which is typically a costly step requiring the time and effort of pathologists. In particular, due to the gigapixel scale of image sizes, comprehensive annotation that labels every pixel in a whole slide image is impractical. While much work has been done using public datasets such as the CAMELYON dataset, public datasets are designed for specific applications and may not generalize to other uses. To segment multiple tissue subtypes on the breast training dataset, images can be partially segmented.

[0037] In the case of partial annotation, while annotating the closely spaced boundary regions between subtypes, the thickness of these unlabeled regions can be minimized, avoiding the annotating of entire subtype components without cropping. Comprehensive annotation without overlaps and subsequent inaccurate labeling, especially in boundary regions, can be challenging given that the regions seamlessly merge into each other. In addition, the time required for complete and comprehensive labeling is enormous. By minimizing the thickness of these unlabeled boundary regions, a CNN model trained with partial annotation can learn the spatial relationships between subtypes and generate accurate segmentation boundaries. This differs from partial annotation, in which the annotated regions of different subtypes are too widely spaced and therefore not suitable for training the spatial relationships between them. This approach also demonstrates comprehensive annotation of subregions of the entire slide image to reduce annotation effort; however, if the subtype components are cropped, the CNN model cannot learn the growth patterns of different subtypes. Here, each subtype component can be globally annotated to allow the CNN model to learn the growth patterns of all subtypes. Figures 3(a) and 3(b) show an example of partial annotation, where an experienced pathologist may spend approximately 30 minutes partially annotating one whole slide image. Note that the white areas in Figure 3(b) are not labeled.

[0038] Training patch extraction Whole slide images are typically too large to process at the slide level using convolutional neural networks. For example, the dimensions of the smallest margin WSI are 43,824 pixels by 31,159 pixels, which is over 1.3 billion pixels. Patch-based methods are used to analyze WSIs. In these methods, patches extracted from the image are processed by a CNN, and the outputs are then combined for slide-level analysis. One limitation of patch-based methods is that they do not mimic a pathologist who switches zoom levels while examining a slide. In contrast, patch-based methods only view patches at a single magnification with a limited field of view.

[0039] To mimic what a pathologist does with a microscope, a set of multi-magnification patches can be extracted to train the DMMN. In this work, the size of the target patch can be set to be analyzed in the WSI and can be 256 x 256 pixels at 20x magnification. To analyze the target patch, an input patch with a size of 1024 x 1024 pixels at 20x is extracted from the image, with the target patch located at the center of the input patch. From this input patch, a set of three multi-magnification patches is extracted. The first patch is extracted from the center of the input patch with a size of 256 x 256 pixels at 20x, which is the same location and magnification as the target patch. The second patch is extracted from the center of the input patch with a size of 512 x 512 pixels and downsampled by a factor of 2 to a size of 256 x 256 pixels at 10x. Finally, the third patch is generated by downsampling the input patch by a factor of 4 to a size of 256 x 256 pixels at 5x. A set of three patches at different multiples serves as input to the DMMN for cancer segmentation in a target patch with a size of 256 × 256 pixels. Input patches are extracted from training images if more than 1% of the pixels in the corresponding target patch are annotated. The stride in the x and y directions is 256 pixels to avoid overlapping of target patches.

[0040] Class Equilibrium Class balancing is an essential step in training a CNN for accurate performance. If the number of training patches in one class is greater than the number of training patches in another class, the CNN cannot properly learn features from the minor class. In this work, class imbalance is observed in the annotation. For example, the number of annotated pixels in cancer regions is greater than the number of annotated pixels in benign epithelial regions. To achieve class balancing, we multiply the training patches belonging to the minor class using elastic deformation.

[0041] Due to the tortuous shape of biological structures, elastic deformation is widely used as a data augmentation technique in biomedical images. To perform elastic deformation on a patch, a set of grid points in the patch is selected and randomly displaced by a normal distribution with a standard deviation of σ. According to the displacement of the grid points, all pixels in the patch are displaced by bicubic interpolation. The grid points can be set by 17 × 17 and σ = 4.

[0042] The number of patches multiplied needs to be carefully chosen to balance the number of pixels between classes, where r c The elastic deformation rate for class c, denoted as , can be defined as the number of patches multiplied for class c and the class rank to determine the class rank when multiplying the patches. The rate can be selected based on the number of pixels in each class. The rate is a non-negative integer, and if the rate is 0, no elastic deformation is performed. The class rank can be determined based on the application. For example, if accurate segmentation in cancer regions is desired, the cancer class has a higher rank than other classes. To multiply patches, each patch needs to be classified into class c if the patch contains pixel labels classified into c. If a patch contains pixels in multiple classes, the class with the higher class rank becomes the class of the patch. After the patches are classified, the r of the patch is cThe number of patches is multiplied for each patch in class c using elastic deformation. Once class balancing is achieved, we train a CNN using all patches.

[0043] CNN Architecture Figures 4(a)-4(d) show various CNN architectures for cancer segmentation. Note that the input patch size is 256 × 256 pixels, and the output prediction size is 256 × 256 pixels. CONV_BLOCK contains two sets of convolutional layers with a 3 × 3 kernel size with 1-padded padding, followed by a series of rectified linear unit (ReLU) activation functions. CONV_TR_u contains a transposed convolutional layer followed by a ReLU activation function, where u is the upsampling rate. Note that CONV_TR_4 consists of a series of two CONV_TR_2. CONV_FINAL contains a convolutional layer with a 3 × 3 kernel size with 1-padded padding, a ReLU activation function, and a convolutional layer with a 1 × 1 kernel size to output C channels. The final segmentation prediction is generated using a softmax operation.

[0044] The green arrows are max pooling operations by a factor of 2, and the red arrows are center crop operations, where the crop rate is written in red. The center crop operation crops the center region of the feature maps in all channels by the crop rate to adapt the size and magnification of the feature maps for the next operation. During the center crop operation, if the crop rate is 2 and 4, respectively, the width and height of the cropped feature map will be half and one-quarter of the width and height of the input feature map.

[0045] The single-encoder, single-decoder (SESD) architecture in Figure 4(a) uses a single scaled patch at 20x to generate the corresponding segmentation prediction. Note that this implementation is the same as U-Net, except that the number of channels is reduced by a factor of two. The multiple-encoder, single-decoder (MESD) architecture in Figure 4(b) uses multiple encoders for scales of 20x, 10x, and 5x, but only a single decoder to generate the segmentation prediction. The multiple-encoder, multiple-decoder, single-concatenation (MEMDSC) architecture in Figure 4(c) has multiple encoders and corresponding decoders for scales of 20x, 10x, and 5x, but concatenation is only performed at the edge of the encoder-decoder architecture. Note that weighted CNNs are excluded for fair comparison with other architectures. Finally, the multiple-encoder, multiple-decoder, multiple-concatenation (MEMDMC) architecture in Figure 4(d) has multiple encoders and decoders, with concatenation between multiple layers in the decoder to strengthen the feature map for the 20x decoder.

[0046] CNN Training We train a multi-class segmentation CNN using the set of balanced patches from Section II-C. We can use weighted cross-entropy as the training loss function with N pixels in a patch and C classes.

[0047]

number

number

number

number

number

[0048]

number

[0049] Multi-class segmentation Multiclass tissue segmentation of breast images can be performed using a trained CNN. The final label for each pixel is selected as the class with the greatest predictive value among the C classes. An input patch with a size of 1024 × 1024 pixels is extracted from the whole slide image, and three sets of patches with a size of 256 × 256 pixels at magnifications of 20×, 10×, and 5× are generated by the process described in Section II-B. The three sets of patches are processed by the trained CNN. The segmentation prediction with a size of 256 × 256 pixels is located at the center location of the input patch. The input patch is extracted from the upper left corner of the WSI using a stride of 256 pixels in the x and y directions to process the entire WSI. Zero padding is performed at the boundary of the WSI before extracting the input patch. To accelerate the segmentation process, the Otsu thresholding technique can be used before extracting the patches as an option to remove background regions.

[0050] III. Experimental Results The goal of this architecture is to segment cancerous regions in a breast margin dataset. Images from the breast margin dataset contain either small or no cancerous regions. A triple-negative breast cancer (TNBC) dataset, which contains large invasive ductal carcinoma (IDC) regions, can be used to train the CNN model. All whole-slide images in the TNBC and breast margin datasets were hematoxylin-eosin (H&E) stained and digitized from Memorial Sloan Kettering Cancer Center. The TNBC dataset was scanned with an Aperio XT, where the microns per pixel (MPP) at 20x was 0.4979, and the breast margin dataset was scanned with an Aperio AT2, where the MPP at 20x was 0.5021.

[0051] Thirty-eight images were partially annotated from the TNBC dataset. The TNBC dataset was divided into 26 training images, 6 validation images, and 6 test images. Six classes (C = 6) were available in the TNBC dataset: cancer, benign epithelium, background, stroma, necrosis, and fat. Note that background is defined as non-tissue areas. Here, 5.48% of the pixels in the entire slide image were annotated. To balance the number of annotated pixels between classes, the values ​​r2 = 10, r1 = 2, r5 = 3, r3 = 1, r4 = 0, and r6 = 0 were experimentally set, where r1, r2, r3, r4, r5, and r6 are the elastic deformation rates of cancer, benign epithelium, background, stroma, necrosis, and fat, respectively. To accurately segment cancer regions and separate benign epithelium to reduce erroneous segmentation, benign epithelium was selected as the highest class ranking, followed by cancer, necrosis, and background. Figure 5 shows that elastic deformation is used to balance the number of annotated pixels between classes. Using a single NVIDIA GeForce GTX TITAN X GPU, the training process took approximately 3 days.

[0052] Figures 6 and 7 show multiclass segmentation predictions at both the slide level and 10x magnification for the single-encoder-single-decoder (SESD), multiple-encoder-single-decoder (MESD), multiple-encoder-multiple-decoder-single-concatenation (MEMDSC), and proposed multiple-encoder-multiple-decoder-multiple-concatenation (MEMDMC) architectures. Here, cancer is labeled red, benign epithelium is labeled blue, background is labeled yellow, stroma is labeled green, necrosis is labeled gray, and fat is labeled orange. Note that the white regions in Figures 6(b), 6(h), and 7(b), 7(h) are not labeled. The Otsu thresholding technique is not used for segmentation on the TNBC dataset because fat regions are predicted as background due to their pixel intensity. Without the Otsu thresholding technique, segmentation on one WSI takes approximately 15 minutes using a single GPU. SESD is observed to produce inaccurate boundaries between subtypes due to its narrow field of view. MESD, in some cases, is unable to distinguish between cancer and benign epithelium. MEMDSC and MEMDMC produce accurate segmentation predictions for TNBC images.

[0053] Breast margin images were processed using the same multi-class segmentation model. All cancer regions were comprehensively annotated for accurate assessment. Figures 8 and 9 show segmentation predictions for breast margin images at both the slide level and at 10x magnification. Cancer segmentation for breast margin images can be performed such that cancer regions are labeled red and non-cancerous regions, including benign epithelium, background, stroma, necrosis, and fat, are labeled yellow. The Otsu thresholding technique was used to reduce the processing time for segmenting cancer regions for breast margin images before tilting the whole slide image. Using the Otsu thresholding technique, segmentation for one WSI takes approximately 2 minutes using a single GPU. SESD still produces segmentation predictions with inaccurate boundaries. It has been observed that large non-cancerous regions are incorrectly segmented as cancer by MEMDSC. MESD and MEMDMC produce accurate segmentations for cancerous regions for breast margin images.

[0054] The predictions were evaluated numerically using intersection-over-union (IOU), recall, and precision, which are defined as follows:

[0055]

number

number

number

[0056] [Table 1] Table II. Recall on the TNBC dataset.

[0057] [Table 2] Table III Accuracy on TNBC dataset

[0058] [Table 3] Table IV. Intersection Over Union (IOU), recall and precision for cancer regions in the breast margin dataset.

[0059] [Table 4]

[0060] IV. Conclusion This paper describes a deep multi-magnification network (DMMN) for accurate multi-class tissue segmentation in whole slide images. The model is trained on partially annotated images to reduce the time and effort required for the annotator. Although the annotation was partial, the model was able to learn not only intra-class spatial characteristics but also inter-class spatial relationships. The DMMN architecture looks at all magnifications: 20x, 10x, and 5x, providing a broader field of view for more accurate predictions. We were also able to improve on the previous DMMN by transferring intermediate feature maps from the 10x and 5x decoders to the 20x decoder to enhance the feature maps.

[0061] Our implementation achieved excellent segmentation performance on multiple breast datasets. In particular, automated cancer segmentation in breast margin images can be used to determine future treatment for patients. We observed that because the model was primarily trained on invasive ductal carcinoma, it may not be successful in segmenting low-grade, well-differentiated cancers present in breast images. In the future, we can develop more accurate DMMN models that include a variety of cancer structures during training.

[0062] B. Second approach for deep multi-magnification networks for multi-class breast cancer image segmentation I. Introduction Breast cancer is the most common cancer diagnosed in women. Approximately 12% of women in the United States will be diagnosed with breast cancer during their lifetime. Pathologists diagnose breast cancer based on a wide variety of morphological features, including tumor growth patterns and nuclear cytologic characteristics. Pathological evaluation of breast tissue guides the patient's clinical management and provides prognostic information. Breast tissue from a wide variety of biopsies and surgical specimens is evaluated by pathologists. For example, patients with early-stage breast cancer often undergo breast-conserving surgery, or lumpectomy, which removes a portion of the breast tissue containing cancer. To determine the completeness of the surgical resection, the edges of the lumpectomy specimen, or margins, are evaluated microscopically by pathologists. Achieving negative margins (no cancer found touching the margins) is important to minimize the risk of local cancer recurrence. Accurate analysis of margins by pathologists is crucial to determining the need for additional surgery. Pathological analysis of margin specimens involves pathologists examining approximately 20–40 histologic slides per case, a process that can be time-consuming and tedious. With the increasing ability to digitally scan histological glass slides, computational pathology techniques can potentially improve the efficiency and accuracy of this process by evaluating whole slide images (WSI) of specimens.

[0063] Various techniques have been used to analyze WSI. Most models involve localization, detection, classification, and segmentation of objects (i.e., histological features) in digital slides. Histopathological features include pattern-based identification information such as nuclear features, cellular / stromal structures, or tissues. Computational pathology has been used in nuclear segmentation to extract nuclear features, such as size, shape, and their relationships. Nuclear segmentation is performed using adaptive thresholding and morphological operations to find regions with high nuclear density. A breast cancer grading method can be developed by segmenting glands and nuclei using a Bayesian classifier and structural constraints from domain knowledge. A unified active contour based on region, boundary, and shape is presented to segment overlapping nuclei and lymphocytes. A gland segmentation and classification method in prostate tissue is introduced. Here, structural and contextual features from the nucleus, cytoplasm, and lumen are used to classify artifacts, normal glands, and adenocarcinoma. These nuclear segmentation-based approaches are challenging because the shapes of nuclei and structures in cancer regions can have large variations in the tissue captured in WSI.

[0064] In recent years, due to the availability of large training datasets and the evolution of graphics processing units (GPUs), deep learning, a type of machine learning, has been used for automatic image analysis. Deep learning models, composed of deep layers with nonlinear activation functions, enable the learning of more refined features. In particular, convolutional neural networks (CNNs) learning spatial features in images have shown excellent results in image classification, object detection, and semantic segmentation. Fully convolutional networks (FCNs), developed for semantic segmentation, also known as pixel-wise classification, can understand the location, size, and shape of objects within an image. FCNs consist of an encoder and a decoder, where the encoder extracts low-dimensional features from the input image, and the decoder uses the low-dimensional features to generate segmentation predictions. To improve segmentation predictions, SegNet introduces a maximum unpooling layer, where the maximum pooling index in the encoder is stored and used in the corresponding upsampling layer in the decoder. Semantic segmentation has been used in medical images to automatically segment biological structures. For example, U-Net has been used to segment cells in microscopic images. The U-Net architecture has connections that transfer feature maps from the encoder to the decoder to preserve spatial information. This architecture has shown more accurate segmentation predictions for biomedical images.

[0065] Deep learning has received significant attention in the computational pathology community in recent years. Researchers have demonstrated automated identification of invasive breast cancer in WSIs using a simple three-layer CNN. A method for classifying breast tissue slides as invasive cancer or benign by analyzing stromal regions using a CNN is described. More recently, a multi-instance learning-based CNN achieved 100% sensitivity when the CNN was trained on 44,732 WSIs from 15,187 patients. The availability of public pathology datasets has encouraged the development of many deep learning methods for computational pathology. For example, a breast cancer dataset for detecting lymph node metastasis was released for the CAMELYON challenge, and several deep learning techniques have been developed to analyze breast cancer datasets.

[0066] One drawback of using deep learning for WSI is that the overall size of a single WSI is too large to process on a GPU. Images can be downsampled for processing by a pre-trained CNN, but crucial details necessary for clinical diagnosis in WSI are lost. To address this issue, patch-based approaches are typically used instead of slide-level approaches. Here, patches are extracted from the WSI and processed by a CNN. The patch-based process followed by multiclass logistic regression for slide-level classification is described. The winner of the CAMELYON 16 challenge used the Otsu thresholding technique to extract tissue regions and train a patch-based model to classify tumor and non-tumor patches. To increase performance, class balancing between tumor and non-tumor patches and data augmentation techniques such as rotation, flip, and color jittering were used. In addition, the winner of the CAMELYON 17 challenge developed a patch overlap strategy for more accurate prediction. Patches are processed with additional larger patches containing boundary regions of the same magnification to segment subtypes in breast WSI. Alternatively, a representation-aggregation CNN has been developed to collect features generated from patches in WSI to share representations between patches. Patch-based approaches are impractical because (1) pathologists do not view patch-level slides with a narrow field of view, and (2) they frequently switch zoom levels to extract various features on slides from multiple magnifications.

[0067] To develop a more realistic CNN, a set of patches at multiple magnifications is required as input, increasing the field of view and providing additional information from other magnifications. High-magnification patches provide cellular-level details, such as nuclear features, while low-magnification patches demonstrate the distribution of tissue types and architectural growth patterns of benign and malignant processes. Figure 1 illustrates the difference between a deep single-magnification network (DSMN) and a deep multiple-magnification network (DMMN). The input to the DSMN in Figure 10(a) is a single patch with a size of 256 x 256 pixels at a single magnification of 20x, which limits the field of view. The input to the DMMN in Figure 10(b) is a set of patches with a size of 256 x 256 pixels at multiple magnifications: 20x, 10x, and 5x, which allows for a wider field of view. By providing multiple magnifications with a wider field of view, the DMMN can mimic the way a pathologist views a slide using a microscope, thereby producing a more accurate analysis.

[0068] Some work has analyzed images from tissue samples using multiple magnifications. A multi-input, multi-output CNN has been introduced by analyzing input images at multiple resolutions to segment cells in fluorescence microscopy images. Similarly, a stain-aware multiscale CNN has also been designed for cell segmentation, for example, in histology images. A binary segmentation CNN is described to segment tumor regions in the CAMELYON dataset. In this work, four encoders with different magnifications are implemented, but only one decoder is used to generate the final segmentation prediction. More recently, a CNN architecture composed of three expert networks with different magnifications, a weighting network that automatically selects weights to emphasize specific magnifications based on the input patch, and an aggregation network that generates the final segmentation prediction, has been developed. Here, feature maps are not shared among the three expert networks until the final layer, which can limit the use of feature maps from multiple magnifications. The architecture is designed at a lower magnification, with center-cropped feature maps at the lower magnifications. The cropped feature maps are then upsampled to match size and magnification during concatenation. This may also limit the utility of feature maps in cropped boundary regions at lower magnifications.

[0069] This paper presents a deep multi-magnification network (DMMN) for accurately segmenting multiple subtypes in breast tissue images. The DMMN architecture has multiple encoders, multiple decoders, and multiple connections between the decoders to achieve richer feature maps in intermediate layers. To fully utilize feature maps at lower magnifications, intermediate feature maps can be center-cropped during concatenation. By concatenating intermediate feature maps at each layer, accurate segmentation predictions can be generated using feature maps from multiple magnifications. To train the DMMN, WSIs can be partially annotated, similar to reducing the annotation burden. A DMMN model trained with partial annotation can not only learn the features of each subtype, but also the morphological relationships between subtypes, especially the transition from one subtype to another in the boundary region, leading to superior segmentation performance. The multi-magnification model can be tested on two breast datasets, and it can be observed that the model consistently outperforms other architectures. This method can be used to automatically segment cancerous regions on breast images, aiding in the diagnosis of patient status and determining future treatment. The main contributions of this work are: (1) a deep multi-magnification network that combines feature maps at different magnifications for more accurate segmentation prediction, and (2) the introduction of partial annotation to save pathologists' annotation time and still achieve high performance.

[0070] II. Proposed Method Figure 11 shows a block diagram of the proposed method. The goal is to segment multiple subtypes in breast images using a deep multi-magnification network (DMMN). First, manual annotation is performed on a training dataset of C classes. This annotation is performed in part for an efficient and fast process. To train the multi-class segmentation DMMN, patches are extracted from whole slide images (WSIs) and their corresponding annotations. Before training the DMMN with the extracted patches, elastic deformation can be used to multiply patches belonging to rare classes to balance the number of annotated pixels between classes. After the training step is performed, the model can be used for multi-class segmentation of breast cancer images.

[0071] Partial annotation Supervised learning requires a large set of annotations, which is typically a costly step requiring the time and effort of pathologists. In particular, due to the gigapixel scale of image sizes, comprehensive annotation that labels every pixel in a WSI is impractical. While much work has been done using public datasets such as the CAMELYON dataset, public datasets are designed for specific applications and may not generalize to other uses. To segment multiple tissue subtypes on the breast training dataset, images can be partially annotated.

[0072] Partial annotation (1) annotates closely spaced boundary regions between subtypes while minimizing the thickness of these unlabeled regions and (2) avoids annotating entire subtype components without cropping. Comprehensive annotation without overlaps and subsequent inaccurate labeling, especially in boundary regions, can be challenging given that regions seamlessly merge into each other. In addition, the time required for complete and comprehensive labeling is enormous. By minimizing the thickness of these unlabeled boundary regions, a CNN model trained with partial annotation can learn spatial relationships between subtypes, such as the transition from one subtype to another, and generate accurate segmentation boundaries. This differs from partial annotation, which is performed when annotated regions of different subtypes are too widely spaced and therefore not suitable for training spatial relationships between them. This work also suggests comprehensive annotation of WSI subregions to reduce annotation effort, but if subtype components are cropped, the CNN model cannot learn the growth patterns of different subtypes. To allow the CNN model to learn the growth patterns of all subtypes, each subtype component can be globally annotated. Figure 12(a) shows an example of partial annotation, where an experienced pathologist may spend approximately 30 minutes partially annotating one WSI. Note that the white areas in Figure 12(b) are not labeled.

[0073] Training patch extraction Whole slide images are typically too large to process at the slide level using convolutional neural networks. To analyze WSI, patch-based methods are used, where patches extracted from the image are processed by a CNN, and then the outputs are combined for slide-level analysis. One limitation of patch-based methods is that they only view patches at a single magnification with a limited field of view.

[0074] To obtain a wider field of view, a set of multi-magnification patches is extracted to train the DMMN. The size of the target patch can be set to be analyzed in the WSI and can be 256 x 256 pixels at 20x magnification. To analyze the target patch, an input patch with a size of 1024 x 1024 pixels at 20x is extracted from the image, with the target patch located at the center of the input patch. From this input patch, a set of three multi-magnification patches is extracted. The first patch is extracted from the center of the input patch with a size of 256 x 256 pixels at 20x, which is the same location and magnification as the target patch. The second patch is extracted from the center of the input patch with a size of 512 x 512 pixels and downsampled by a factor of 2 to a size of 256 x 256 pixels at 10x. Finally, the third patch is generated by downsampling the input patch by a factor of 4 to a size of 256 x 256 pixels at 5x. The set of three patches at different multiples serves as input to the DMMN for cancer segmentation in a target patch with a size of 256 × 256 pixels. Input patches are extracted from training images if more than 1% of the pixels in the corresponding target patch are annotated. The stride in the x and y directions is 256 pixels to avoid overlapping target patches. Note that a target patch may have multiple class labels.

[0075] Class Equilibrium Class balancing is an essential step in training a CNN for accurate performance. If the number of training patches in one class is greater than the number of training patches in another class, the CNN cannot properly learn features from the minor class. In this work, class imbalance is observed in the annotation. For example, the number of annotated pixels in cancer regions is greater than the number of annotated pixels in benign epithelial regions. To achieve class balancing, we multiply the training patches belonging to the minor class using elastic deformation.

[0076] Due to the tortuous shape of biological structures, elastic deformation is widely used as data augmentation in biomedical images. To perform elastic deformation on a patch, a set of grid points in the patch is selected and randomly displaced by a normal distribution with a standard deviation of σ. According to the displacement of the grid points, all pixels in the patch are displaced by bicubic interpolation. The grid points can be empirically set to 17 × 17 and σ = 4 to avoid losing features due to excessive distortion of the kernel.

[0077] The number of patches to be multiplied must be carefully selected to balance the number of pixels between classes. To determine the class ranking when multiplying patches, an elastic deformation rate can be defined for class c, denoted as rc, to be the number of patches multiplied for class c and the class rank. The rate can be selected based on the number of pixels in each class. The rate is a non-negative integer, and if the rate is 0, elastic deformation is not performed. The class rank can be determined based on the application. For example, if accurate segmentation of cancer regions is desired, the cancer class has a higher rank than other classes. To multiply patches, each patch needs to be classified into class c if it contains pixel labels classified as c. If a patch contains pixels in multiple classes, the class with the higher class rank becomes the class of the patch. After the patches are classified, the rc number of patches is multiplied for each patch in class c using elastic deformation. Once class balancing is achieved, all patches are used to train a CNN.

[0078] CNN Architecture Figure 13 shows the architectures of a deep single-magnification network (DSMN) and a deep multiple-magnification network (DMMN) for multi-class tissue segmentation. The input patch size is 256 × 256 pixels, and the output prediction size is 256 × 256 pixels. CONV_BLOCK includes two sets of convolutional layers with a 3 × 3 kernel size with 1-padding, followed by a series of rectified linear unit (ReLU) activation functions. CONV_TR_u includes a transposed convolutional layer followed by a ReLU activation function, where u is the upsampling rate. Note that CONV_TR_4 consists of a series of two CONV_TR2s. CONV_FINAL includes a convolutional layer with a 3 × 3 kernel size with 1-padding, a ReLU activation function, and a convolutional layer with a 1 × 1 kernel size to output C channels. The final segmentation prediction is generated using a softmax operation. The green arrows are max pooling operations by a factor of 2, and the red arrows are center crop operations, where the crop rate is written in red. The center crop operation crops the center region of the feature maps in all channels by the crop rate to adapt the size and magnification of the feature maps for the next operation. During the center crop operation, if the crop rate is 2 and 4, respectively, the width and height of the cropped feature map will be half and one-quarter of the width and height of the input feature map.

[0079] The original U-Net architecture in Figure 13(a) uses a single 20x scale patch to generate corresponding segmentation predictions. The single-encoder, single-decoder (DMMN-S2) architecture in Figure 13(b) uses multiple patches at 20x, 10x, and 5x scales, but these are concatenated and used as input to the U-Net architecture. The multiple-encoder, single-decoder (DMMN-MS) architecture in Figure 13(c), which was motivated by the work, uses multiple encoders at 20x, 10x, and 5x scales, but only a single decoder at 20x by transferring feature maps at 10x and 5x from the encoder. The multiple-encoder, multiple-decoder, single-concatenation (DMMN-M2S) architecture in Figure 13(d), which was also motivated by the work, has multiple encoders and corresponding decoders for 20x, 10x, and 5x scales, but concatenation is only performed at the ends of the encoder-decoder pairs. Here, weighted CNNs are excluded for a fair comparison with other architectures. Finally, the proposed multiple-encoder, multiple-decoder, multiple-concatenation (DMMN-M3) architecture in Figure 13(e) has multiple encoders and decoders and has connections between decoders at multiple layers to strengthen the feature map from the decoder at 10x and 5x to the decoder at 20x. In addition, a center crop operation can be used while transferring the feature map from the decoder at 10x and 5x to the decoder at 20x to extract as many features as possible at 10x and 5x. Note that DMMN-MS and DMMN-M2S use a center crop operation at the 10x and 5x levels, where the crop region before concatenation may limit the feature extraction process at lower magnifications.

[0080] CNN Training The set of balanced patches from above is used to train a multi-class segmentation CNN. Weighted cross-entropy is used as the training loss function with N pixels in the patch and C classes.

[0081]

number

number

number

number

number

[0082]

number

[0083] Multi-class segmentation Multiclass tissue segmentation of breast images can be performed using a trained CNN. The final label for each pixel is selected as the class with the greatest predictive value among the C classes. An input patch with a size of 1024 x 1024 pixels is extracted from the WSI, and three sets of patches with a size of 256 x 256 pixels at 20x, 10x, and 5x magnifications are generated using the process described above. The three sets of patches are processed by the trained CNN. A segmentation prediction with a size of 256 x 256 pixels is located at the center location of the input patch. The input patch is extracted from the upper left corner of the WSI using a stride of 256 pixels in the x and y directions to process the entire WSI. Zero padding is performed at the boundary of the WSI before extracting the input patch. To accelerate the segmentation process, the Otsu thresholding technique can be optionally used before extracting the patch to remove background regions. No preprocessing step is used during segmentation.

[0084] III. Experimental Results Two breast datasets, Dataset-I and Dataset-II, were used to train and evaluate various multiclass tissue segmentation methods. Dataset-I consists of whole slide images (WSIs) with triple-negative breast cancer (TNBC), including invasive ductal carcinoma (IDC). Dataset-II consists of WSIs from lumpectomy and breast margins, including IDC and ductal carcinoma in situ (DCIS) of various histological grades. All WSIs in Datasets I and II are from different patients, were hematoxylin-eosin (H&E) stained, and digitized from Memorial Sloan Kettering Cancer Center. Dataset-I was digitized using an Aperio XT, where the microns per pixel (MPP) at 20x is 0.4979, and Dataset-II was digitized using an Aperio AT2, where the MPP at 20x is 0.5021. The WSIs in Dataset-I were partially annotated by two pathologists, and the WSIs in Dataset-II were partially annotated by another pathologist.

[0085] Thirty-two training images from Dataset-I were used to train the model. Six images were separated for validation. Images from Dataset-II were not used during training. In our work, only 5.34% of the pixels in the training WSI were annotated. Our model can predict six classes (C = 6): cancer, benign epithelium, background, stroma, necrosis, and fat. Note that background is defined as non-tissue areas. To balance the number of annotated pixels between classes, the following parameters can be set: r2 = 10, r1 = 2, r5 = 3, r3 = 1, r4 = 0, and r6 = 0, where r1, r2, r3, r4, r5, and r6 are the elastic deformation rates of cancer, benign epithelium, background, stroma, necrosis, and fat, respectively. Because it may be desirable to accurately segment cancerous regions and separate benign epithelium to reduce mis-segmentation, benign epithelium was selected as the highest class ranking, followed by cancer, necrosis, and background. Figure 14 shows that the number of annotated pixels between classes is balanced using elastic deformation. Two deep single-magnification networks (DSMNs), the SegNet and U-Net architectures, and four deep multiple-magnification networks (DMMNs), the single-encoder single-decoder (DMMN-S2) architecture, the multiple-encoder single-decoder (DMMN-MS) architecture, the multiple-encoder multiple-decoder single-concatenation (DMMN-M2S) architecture, and the proposed multiple-encoder multiple-decoder multiple-concatenation (DMMN-M3) architecture, can be trained. The number of convolutional layers, the number of downsampling and upsampling layers, and the number of channels remain the same between the SegNet and original U-Net architectures used in this experiment. Additionally, the number of channels in DMMN-MS, DMMN-M2S, and DMMN-M3 is reduced by a factor of two from the original U-Net architecture.Table V lists the compared models, the number of trainable parameters, and the segmentation time, where the segmentation time was measured in Fig. 15 for a whole slide image of size 53,711 × 38,380 pixels with 31,500 patches using a single NVIDIA GeForce GTX TITAN X GPU. Table V. Number of trainable parameters and computation time for multi-class segmentation models.

[0086] [Table 5]

[0087] Fifty-five test images from Dataset-I and 34 test images from Dataset-II were processed to evaluate various models. Figures 15, 16, and 17 show multi-class segmentation predictions of the SegNet, U-Net, DMMN-S2, DMMN-MS, DMMN-M2S, and our proposed DMMN-M3 architectures at both slide level and 10x magnification. Tissue subtypes are labeled with distinct colors: red for cancer, blue for benign epithelium, yellow for background, green for stroma, gray for necrosis, and orange for fat. White areas in Figures 15(b), 15(j), 16(b), 16(j), and 17(b) and 17(j) are unlabeled. Since the model is observed to be sensitive to background noise, leading to incorrect segmentation in the background regions, we used the Otsu thresholding technique to extract patches only in the foreground regions of the WSI from Dataset-II, digitized from a different scanner. The white areas in Figures 17(c) to 17(f) and 17(i) to 17(l) are removed by the Otsu technique.

[0088] The predictions were evaluated numerically using the intersection-over-union (IOU), recall, and precision, which are defined as follows:

[0089]

number

number

number

[0090] [Table 6] Table VII. Average IOU, recall and precision on Dataset-II.

[0091] [Table 7]

[0092] Based on our visual and numerical evaluation of Dataset-I, both DSMNs had blocky boundaries between subtypes, as shown in Figures 15(k) and 15(l) and Figures 11(k) and 11(l), due to their narrow field of view. DMMN-S2 also had blocky boundaries between subtypes, as shown in Figures 15(m) and 16(m), due to the inability to fully extract various features from multiple magnifications due to patches from multiple magnifications being concatenated early in the model. These blocky predictions led to low mIOU, mRecall, and mPrecision in Table VI. DMMN-MS and DMMN-M2S had smoother boundaries between subtypes but did not have consistent predictions across subtypes. For example, DMMN-MS and DMMN-M2S failed to successfully predict necrosis, as shown in Figures 18(d) and 18(e). Our proposed DMMN-M3 showed accurate prediction across all subtypes shown in Figure 18(f), leading to the best mIOU, mRecall and mPrecision in Table VI.

[0093] Our model was trained on Dataset-I, and for our test set, we reserved images in Dataset-II annotated by a different pathologist. We still observed blocky boundaries in the predictions made by SegNet, U-Net, and DMMN-S2 for Dataset-II, as shown in Figures 17(k), 17(l), and 17(m). We noticed that predictions by DMMN-M2S were unsuccessful, with large regions incorrectly segmented as benign epithelium in Figures 17(g) and 17(o). DMMN-MS and DMMN-M3 produced the best predictions, as shown in Figures 17(n) and 17(p). While numerical analysis based on mIOU, mRecall, and mPrecision is comparable between U-Net and DMMN-M3, visual analysis in Figures 17(a)–17(p) reveals dramatic differences that are not captured by the partially annotated ground truth. Figures 17(a)-17(p) demonstrate that the proposed DMMN-M3, with smoother and clearer boundaries between subtypes, outperforms the state-of-the-art with comprehensively annotated ground truth.

[0094] IV. Conclusion A deep multi-magnification network (DMMN) is described for accurate multi-class tissue segmentation in whole slide images. The model is trained on partially annotated images to reduce the time and effort required for the annotator. Although the annotation was partial, our model was able to learn not only intra-class spatial characteristics but also inter-class spatial relationships. Our DMMN architecture looked at all magnifications: 20x, 10x, and 5x, to have a broader field of view for more accurate predictions based on feature maps from multiple magnifications. DMMN could be improved by transferring intermediate feature maps from the decoders at 10x and 5x to the decoder at 20x to enhance the feature maps. Our implementation achieved excellent segmentation performance on a breast dataset, which can be used to determine future treatment for patients. One major challenge is that our model was trained primarily on invasive ductal carcinoma, and therefore may not be successful in segmenting low-grade, well-differentiated carcinomas present in breast images. We also observe that our model may be sensitive to background noise, which may lead to incorrect segmentation in background regions when whole slide images are digitized by other scanners.More accurate DMMN models can be developed when various cancer structures and background noise patterns are included during training.

[0095] C. Systems and methods for segmenting biomedical images and training networks using a concatenated multi-scale encoder-decoder network to segment biomedical images Referring to FIG. 20( a), a block diagram of an environment or system 2000 for segmenting biomedical images using a multi-magnification encoder-decoder concatenated network is shown, according to an exemplary embodiment. In overview, the system 2000 may comprise at least one image segmentation system 2002, at least one imager 2004, and at least one display 2006. The image segmentation system 2002, the imager 2004, and the display 2006 may be communicatively coupled to one another. The image segmentation system 2002 may comprise, among other things, at least one patch generator 2008, at least one model applicator 2010, at least one model trainer 2012, at least one segmentation model 2014, and at least one training dataset 2016. The segmentation model 2014 may include a set of networks 2018A-N (hereinafter generally referred to as networks 2018).

[0096] Each of the components in system 2000 listed above can be implemented using hardware (e.g., one or more processors coupled with memory) or a combination of hardware and software as detailed herein in Section B. Each of the components in system 2000 can implement or perform the functions detailed herein in Section A as described in conjunction with Figures 1(a)-4(d) and 5(a)-9(l), and Section B as described in conjunction with Figures 10(a)-13(e). For example, segmentation model 2014 can implement or have the functionality of the architecture discussed herein in conjunction with Figure 4(d) or 13(e).

[0097] In further detail, the imager 2004 (sometimes referred to herein as a microscope imager or imaging device) can acquire or generate at least one biomedical image 2020A (also referred to herein as an acquired image). The biomedical image 2020A can be, for example, a histological section having a hematoxylin-eosin (H&E) stain, a hemosiderin stain, a Sudan stain, a Schiff stain, a Congo red stain, a Gram stain, a Ziehl-Neelsen stain, an auramine-rhodamine stain, a trichrome stain, a silver stain, and a Wright stain, among others. The imager 2004 can acquire the biomedical image 2020A from a tissue section from a subject (e.g., human, animal, or plant) for histopathological investigation. The tissue sample can be from any part of the subject, such as muscle tissue, connective tissue, epithelial tissue, or nervous tissue, in the case of a human or animal subject. The tissue sample used to acquire the biomedical image 2020A can have an area of ​​interest such as a tumor growth, a lesion, or a wound, among others. In some embodiments, the imager 2004 can acquire the biomedical image 2020A according to a microscopy technique such as using an optical microscope, a confocal microscope, a fluorescent microscope, a phosphorescent microscope, an electron microscope, among others. Upon acquisition or generation, the imager 2004 can transmit or provide the biomedical image 2020A to the image segmentation system 2002.

[0098] The imager 2004 can use the biomedical image 2020A to generate a sample biomedical image 2020B. Upon generation of the sample biomedical image 2020B, the imager 2004 can provide the sample biomedical image 2020B to the image segmentation system 2002 for storage as part of the training dataset 2016. The sample biomedical image 2020B can be a histological section stained using one of the techniques discussed above in connection with the biomedical image 2020A. Additionally, the sample biomedical image 2020B can be included as part of the training dataset 2016 for training and updating the segmentation model 2014 according to supervised learning techniques.

[0099] Referring to FIG. 20(b), a diagram of an example sample biomedical image 2020B is shown in accordance with an illustrative embodiment. The sample biomedical image 2020B may be associated with at least one annotation 2024A-N (hereinafter collectively referred to as annotation 2024). In some embodiments, the annotation 2024 may be manually prepared by a pathologist reviewing the sample biomedical image 2020B. The annotation 2024 may indicate or otherwise label at least a portion of each region of interest 2026A-N (hereinafter collectively referred to as region of interest 2026) within the sample biomedical image 2020B. In some embodiments, the annotation 2024 may specify or define the area, dimensions, or coordinates (e.g., pixel coordinates) of the region of interest 2026 within the sample biomedical image 2020B.

[0100] Each region of interest 2026 may correspond to a tumor, lesion, or other lesion in the imaged tissue sample used to generate sample biomedical image 2020B. In some embodiments, each region of interest 2026 may correspond to a different type (sometimes referred to herein as a subtype) of feature in the imaged tissue sample. Each annotation 2024 may also correspond to a different type of feature in the region of interest 2026. For example, the types of features in the imaged tissue sample may include cancerous tissue, benign epithelial tissue, stromal tissue, necrotic tissue, and adipose tissue, among others. In the illustrated example, sample biomedical image 2020B may include a first region of interest 2026A that is partially labeled with a first annotation 2024A. Additionally, sample biomedical image 2020B may include a second region of interest 2026B that is partially labeled with a second annotation 2024B.

[0101] The annotations 2024 may completely label or partially label (e.g., as shown) the corresponding region of interest 2026 in the sample biomedical image 2020B. When completely labeled, the annotations 2024 may cover or substantially (e.g., 90% or more) cover the corresponding region of interest 2026 in the sample biomedical image 2020B. Conversely, when partially labeled, the annotations 2024 may define a portion (less than the entirety) of the region of interest 2026 in the sample biomedical image 2020B. Additionally, the portion defined by each annotation 2024 may be separated from at least one edge 2028 of the region of interest 2026. The edge 2028 may define the perimeter or boundary of two or more regions of interest 2026. The annotations 2024 may be separated or spaced apart from the edge 2028 by separation distances 2030A-N (hereinafter collectively referred to as separation distances 2030). In the illustrated example, a first annotation 2024A that partially labels a region of interest 2026A can be separated from the edge 2028 by a first distance 2030A. Furthermore, a second annotation 2024B that partially labels a region of interest 2026B can be separated from the edge 2028 by a second distance 2030B. By partially labeling the region of interest 2026 with annotations 2024, the time consumed in generating annotations 2024 can be reduced. Furthermore, segmentation model 2014 can learn spatial relationships between different types of features in region of interest 2026 and produce more accurate and precise results.

[0102] 20(a), the image segmentation system 2002 itself, as well as components therein such as the patch generator 2008, the model applicator 2010, and the model trainer 2012, can have a runtime mode and a training mode. Under the training mode, the image segmentation system 2002 can train a segmentation model 2014 using a training dataset 2016 compared to a segmented image 2020′. Under the runtime mode, the image segmentation system 2020 can apply the segmentation model 2014 to an acquired biomedical image 2020A to generate a segmented image 2020′.

[0103] A patch generator 2008 executing on the image segmentation system 2002 can receive, retrieve, or otherwise identify an acquired biomedical image 2020A or a sample biomedical image 2020B (together referred to collectively as a biomedical image 2020). Under a runtime mode, the patch generator 2008 can identify an acquired biomedical image 2020A from the imager 2004. In contrast to the sample biomedical image 2020B, the acquired biomedical image 2020A may lack any associated annotations 2024. The acquired biomedical image 2020A may differ from the sample biomedical image 2020B of the training dataset 2016. Conversely, under a training mode, the patch generator 2008 can access the training dataset 2016 to identify the sample biomedical image 2020B. The sample biomedical image 2020B may be associated with at least one of a set of annotations 2024 that label a region of interest 2026.

[0104] Upon identification of the biomedical image 2020, the patch generator 2008 can process the biomedical image 2020 for feeding into the segmentation model 2014. The patch generator 2008 can generate a set of tiles 2022A-N (hereinafter collectively referred to as tiles 2022) from the biomedical image 2020. The patch generator 2008 can partition or divide the biomedical image 2020 into the set of tiles 2022. Each tile 2022 can correspond to a portion of the biomedical image 2020. The portions may be mutually exclusive or may partially overlap. In some embodiments, the portions of the biomedical image 2020 corresponding to the tiles 2022 can be non-overlapping. In some embodiments, the portions of the biomedical image 2020 corresponding to the tiles 2022 can partially overlap. Adjacent tiles 2022 generated from the biomedical image 2020 can overlap each other by an overlap ratio (e.g., 10-90%).

[0105] In some embodiments, the patch generator 2008 can identify or detect one or more regions of the biomedical image 2020 that correspond to negative space. Identification of negative space can be performed according to a feature detection algorithm. Negative space regions of the biomedical image 2020 may be devoid of any portion of a microdissected sample or specimen. For example, negative space can correspond to regions of the biomedical image 2020 that are null or white, or substantially null or white (e.g., at least 90%). Upon detection of negative space, the patch generator 2008 can remove regions of the biomedical image 2020 that correspond to negative space from generating the set of tiles 2022.

[0106] For each tile 2022 from the biomedical image 2002, the patch generator 2008 can generate or create a set of patches 2036A-N (hereinafter collectively referred to as patches 2036) using a corresponding set of magnification factors. Each patch 2036 can correspond to a portion of the biomedical image 2020 for which the tile 2022 was generated at one of the magnification factors. Each magnification factor can specify an amount, number, or ratio by which to magnify the tile 2022 (e.g., by increasing the resolution of the image within the tile 2022) or shrink the tile 2022 (e.g., by decreasing the resolution of the image within the tile 2022). To create each patch 2036, the patch generator 2008 can apply a corresponding magnification factor to the tile 2022. The magnification factor used to generate the patch 2036 can range, for example, from 1× to 300×. The number of patches 2036 generated can range from two to several hundred. For example, using the same tile 2022 from the biomedical image 2020, the first patch 2036A can be at a magnification factor of 20x, the second patch 2036A can be at a magnification factor of 10x, and the third patch 2036C can be at a magnification factor of 20x.

[0107] In some embodiments, the patch generator 2008 can generate one patch 2036 at one magnification factor using another patch 2036 at another magnification factor for the same tile 2022 of the biomedical image 2020. The patch generator 2008 can generate a first patch 2036A by applying a first magnification factor from one tile 2022 of the biomedical image 2020A. During generation, the patch generator 2008 can identify a portion of the first patch 2036A. The portion can correspond to, for example, a center portion, a left portion, a right portion, an upper portion, a lower portion, an upper left portion, an upper right portion, a lower left portion, a lower right portion, or any other subsection of the patch 2036. In the identified portion, the patch generator 2008 can apply a second magnification factor to generate a second patch 2036B. In some embodiments, the patch generator 2008 can downsample (or upsample) the identified portion of the first patch 2036A to generate a second patch 2036B at a second magnification factor. The patch generator 2008 can iteratively identify and sample portions to generate a set of patches 2036. In some embodiments, the iteration can be defined for the number of patches 2036 generated by the patch generator 2008 for a tile 2022 of the biomedical image 2020.

[0108] A model applicator 2010 executing in the image segmentation system 2002 can maintain and establish a segmentation model 2014 in the image segmentation system 2002. The model applicator 2010 can apply the segmentation model 2014 to each patch 2036. The application of the segmentation model 2014 can occur in both runtime mode and training mode. When applying, the model applicator 2010 can feed or input a set of patches 2036 from each tile 2022 of the biomedical image 2020 to the segmentation model 2014. In some embodiments, the model applicator 2010 can feed each patch 2036 to a corresponding network 2018 of the segmentation model 2014 for an expansion factor attributed to the patch 2036. For example, the model applicator 2010 may feed a first patch 2036A at a first magnification factor to a first network 2018A, a second patch 2036B at a second magnification factor to a second network 2018B, and a third patch 2036C at a third magnification factor to a third network 2018C, and so on. By applying each tile 2022, the model applicator 2010 may generate a corresponding tile 2022' for the segmented image 2020'. The segmented tile 2022' may identify a region of interest 2026 within the tile 2022. Details of the functionality and structure of the segmentation model 2014 and network 2018 are provided herein in conjunction with Figures 20(c)-20(g).

[0109] Referring to FIG. 20(c), a block diagram of a convolution block 2032 (sometimes referred to herein as an encoder) is shown in one of the networks 2018 of the segmentation model 2014 according to an example embodiment. One or more instances of the convolution block 2032 may be included in each network 2018 of the segmentation model 2014. In overview, the convolution block 2032 may include a set of transform layers 2034A-N. The set of transform layers 2034A-N may include, among other things, one or more convolution layers 2034A and one or more normalization layers 2034B, as well as one or more activation layers 2034C (also referred to herein as rectified linear units). In some embodiments, the set of transform layers 2034A-N may lack one or more normalization layers 2034B. The convolution block 2032 may have as input a set of feature maps (as shown) or one of the patches 2036. The set of transform layers 2034A-N of the convolution block 2032 can be applied in any order (such as that shown) to an input, such as a set of feature maps 2038 or one of the patches 2036. The set of input feature maps 2038 can have the same resolution as the patch 2036 that is input to the network 2018 in which the convolution block 2032 resides. The resolution can correspond to or define the number of pixels in each dimension of the feature map 2038 or patch 2036. The set of input feature maps 2038 can be the resulting output from processing one of the patches 2036 of another component of the same or a different network 2018.

[0110] The convolutional layer 2034A of the convolutional block 2032 may include one or more filters (sometimes referred to as kernels or feature detectors). The convolutional layer 2034A may apply one or more filters to the input set of the feature map 2038. Each filter may be a function that is applied to the input of the convolutional layer 2034A over a predetermined size at a predetermined stride (e.g., ranging from 1 to 64) to generate an output. The filter's function may include one or more parameters (sometimes referred to as weights) that are applied to the input. The one or more parameters may be set, adjusted, or changed through training. Each filter may have a predetermined size (e.g., ranging from 3x3x1 to 1024x1024x3). The size and number of filters may vary depending on which network 2018 includes an instance of the convolutional layer 2034A. The parameters of the convolutional layer 2034A may be repeatedly applied to the input (e.g., the input set of the feature map 2038) according to a predetermined stride to generate an output.

[0111] The normalization layer 2034B of the convolution block 2032 may include at least one function applied to the output of the previous transform layer 2034A-N (e.g., convolution layer 2034A as shown). The function of the normalization layer 2034B may include one or more parameters applied to the input. The function may be set, adjusted, or changed through training. The normalization layer 2034B may identify a range of values ​​for the input. From the range of values, the normalization layer 2034B may identify a minimum value, a maximum value, and a difference between the minimum and maximum values ​​for the input. The normalization layer 2034B may determine transform coefficients based on the minimum value, maximum value, and difference between the minimum and maximum values ​​(e.g., as a linear function). The normalization layer 2034B may then apply (e.g., multiply) the transform coefficients to all inputs to form the output.

[0112] The activation layer 2034C of the convolution block 2032 may include at least one function applied to the output of the previous transform layer 2034A-N (e.g., the convolution layer 2034A or the normalization layer 2034B as shown). The function of the activation layer 2034C may be an activation function such as the identity function, the unit step function, the hyperbolic function, the inverse trigonometric function (arcus function), or the rectified linear function (max(0,x)), among others. The function may be set, adjusted, or changed through training. The activation function may be nonlinear. The activation layer 2034C may traverse all s inputs, each of which corresponds to the output of one of the filters in the previous transform layer 2034A-N, such as the convolution layer 2034A as shown. During traversal, the activation layer 2034C may apply the activation function to the input to generate an output set of feature maps 2038′. Due to the activation function, the output of the activation layer 2034C may be nonlinear with respect to the input.

[0113] By applying the set of transform layers 2034A-N to the set of feature maps 2038 or one of the patches 2036, the convolution block 2032 can generate a set of feature maps 2038' as output. By doing so, the convolution block 2032 can feed or provide the output set of feature maps 2038' along the same network 2018 of the segmentation model 2014 (e.g., as indicated by the horizontal arrow). The convolution block 2032 can feed or provide the output set of feature maps 2038' to the network 2018 for different upscaling factors of the segmentation model 2014 (e.g., as indicated by the downward vertical arrow). The output set of feature maps 2038' can have the same resolution as the input set of feature maps 2038 and patches 2036 that are input to the network 2018 in which the convolution block 2032 is located.

[0114] Referring to FIG. 20(d), a block diagram of a deconvolution block 2040 (sometimes referred to herein as a decoder) is shown in one of the networks 2018 of the segmentation model 2014 according to an example embodiment. One or more instances of the deconvolution block 2040 may be included in each network 2018 of the segmentation model 2014. In overview, the deconvolution block 2040 may include a set of transform layers 2042A-N. The set of transform layers 2042A-N may include, among other things, one or more upsampling layers 2042A, one or more convolution layers 2042B, one or more normalization layers 2042C, and one or more activation layers 2042D (also referred to herein as normalized linear units). In some embodiments, the set of transform layers 2042A-N may lack one or more normalization layers 2042B. The deconvolution block 2040 may have as input a set of feature maps 2038. The set of transform layers 2042A-N of the deconvolution block 2040 can be applied in any order (such as that shown) to an input such as a set of feature maps 2038. The set of input feature maps 2038 can have the same resolution or size as the patches 2036 input to the network 2018 in which the deconvolution block 2040 resides. The set of input feature maps 2038 can be the resulting output of another component of the same or a different network 2018 from processing one of the patches 2036 input to the network 2018.

[0115] The upsampling layer 2042A of the deconvolution block 2040 can increase the resolution of the input set of the feature map 2038 to a target resolution. The target resolution can be equal to the resolution of another network 2018 in the segmentation model 2014 to which the resulting output set of the feature map 2038′ is fed. In some embodiments, the upsampling layer 2042A can identify the resolution of the input set of the feature map 2038. Upon identification, the upsampling layer 2042A can determine the difference between the resolution and the target resolution. Upon determining the difference, the upsampling layer 2042A can zero-pad the input set of the feature map 2038 to increase the resolution by the difference from the target resolution. The upsampling layer 2042A can apply an interpolation filter to the zero-padded input set of the feature map 2038 to smooth discontinuities resulting from the zero-padding. The interpolation filter can be the same size as the target resolution.

[0116] The transposed convolutional layer 2042B of the deconvolution block 2040 may include one or more filters (sometimes called kernels or feature detectors). The convolutional layer 2042B may apply one or more filters to the input set of the feature map 2038. Each filter may be a function applied to the input of the transposed convolutional layer 2042B over a predetermined size at a predetermined stride (e.g., ranging from 1 to 64) to generate an output. The filter function may include one or more parameters (sometimes called weights) that are applied to the input. Relative to the parameters in the convolutional layer 2034A of the convolution block 2032, the parameters of the filters in the transposed convolutional layer 2042B of the deconvolution block 2040 may be transposed or inverted (pseudo-inverted). The one or more parameters of the filters may be set, adjusted, or changed through training. The adjustment of parameters of filters in transposed convolutional layer 2042B of deconvolution block 2040 may depend on the adjustment of parameters of filters in convolutional layer 2034A of convolution block 2032. Each filter may be of a predetermined size (e.g., ranging from 3x3x1 to 1024x1024x3). The size and number of filters may vary depending on which network 2018 includes an instance of convolutional layer 2042B. The parameters of convolutional layer 2042B may be repeatedly applied to an input (e.g., an input set of feature maps 2038) according to a predetermined stride to generate an output.

[0117] The normalization layer 2042C of the deconvolution block 2040 may include at least one function applied to the output of the previous transform layer 2042A-N (e.g., the transposed convolution layer 2042B as shown). The function of the normalization layer 2042C may include one or more parameters applied to the input. The function may be set, adjusted, or changed through training. The normalization layer 2042C may identify a range of values ​​for the input. From the range of values, the normalization layer 2042C may identify a minimum value, a maximum value, and a difference between the minimum and maximum values ​​for the input. The normalization layer 2042C may determine transform coefficients based on the minimum value, the maximum value, and the difference between the minimum and maximum values ​​(e.g., as a linear function). The normalization layer 2042C may then apply (e.g., multiply) the transform coefficients to all inputs to form the output.

[0118] The activation layer 2042D of the deconvolution block 2040 may include at least one function applied to the output of the previous transform layer 2042A-N (e.g., the transposed convolution layer 2042B or the normalization layer 2042C as shown). The function of the activation layer 2042D may be an activation function such as the identity function, the unit step function, the hyperbolic function, the Arcus function, or the rectified linear function (max(0,x)), among others. The function may be set, adjusted, or changed through training. The activation function may be nonlinear. The activation layer 2042D may traverse all inputs, each of which corresponds to the output of one of the filters in the previous transform layer 2042A-N, such as the convolution layer 2042B. During traversal, the activation layer 2042D may apply the activation function to the input to generate an output set of feature maps 2038′. Due to the activation function, the output of the activation layer 2042B may be nonlinear with respect to the input.

[0119] By applying the set of transform layers 2042A-N to the set of input feature maps 2038, the deconvolution block 2040 can generate a set of feature maps 2038' as output. By doing so, the deconvolution block 2040 can feed or provide the output set of feature maps 2038' to the network 2018 for different upscaling factors of the segmentation model 2014 (e.g., as indicated by the upper vertical arrows). The output set of feature maps 2038' can have the same resolution as the input set of feature maps 2038 of the network 2018 to which the deconvolution block 2040 feeds the output set of feature maps 2038'.

[0120] Referring to FIG. 20(e), a block diagram of a terminal convolution block 2046 in a segmentation model 2014 is shown, according to an example embodiment. One or more instances of the terminal convolution block 2046 may be included in the segmentation model 2014. In overview, the terminal convolution block 2046 may include a set of transform layers 2048A-N. The set of transform layers 2048A-N may include, among other things, one or more initial convolutional layers 2048A, one or more activation layers 2048B, one or more final convolutional layers 2048C, and one or more recurrent layers 2048D. The terminal convolution block 2046 may have as input a set of feature maps 2038. The set of transform layers 2048A-N of the terminal convolution block 2046 may be applied in any order (such as that shown) to an input, such as a set of feature maps 2038′, output by one of the networks 2018. The set of feature maps 2038′ may be the resulting output of one of the networks 2018 from processing one of the patches 2036 and the other input feature maps 2038 that are input to the network 2018.

[0121] The initial convolutional layer 2048A of the terminal convolutional block 2046 may include one or more filters (sometimes called kernels or feature detectors). The convolutional layer 2048A may apply one or more filters to the set of feature maps 2038′ input to the terminal convolutional block 2046. Each filter may be a function applied to the input of the convolutional layer 2048A over a predetermined size at a predetermined stride (e.g., ranging from 1 to 64) to generate an output. The filter's function may include one or more parameters (sometimes called weights) that apply to the input. The one or more parameters may be set, adjusted, or changed through training. Each filter may have a predetermined size (e.g., ranging from 3×3×1 to 1024×1024×3). The size and number of filters may vary depending on the network 2018 that provides the set of feature maps 2038′ as input to the terminal convolutional block 2046. The parameters of the convolutional layer 2048A can be repeatedly applied to the input (e.g., the set of feature maps 2038′) to generate the output according to a predetermined stride.

[0122] The activation layer 2048B of the terminal convolution block 2046 may include at least one function applied to the output of the previous transform layer 2048A-N (e.g., the initial convolution layer 2048A). The function of the activation layer 2048B may be an activation function such as the identity function, unit step function, hyperbolic function, or Arcus function, or a rectified linear function (max(0,x)), among others. The function may be set, adjusted, or changed through training. The activation function may be nonlinear. The activation layer 2048B may traverse all inputs, each of which corresponds to the output of one of the filters in the previous transform layer 2048A-N, such as the initial convolution layer 2048A. During traversal, the activation layer 2048B may apply the activation function to the input to generate an output set of feature maps. Due to the activation function, the output of the activation layer 2048B may be nonlinear with respect to the input.

[0123] The final convolutional layer 2048C of the terminal convolutional block 2046 may include one or more filters (sometimes called kernels or feature detectors). The convolutional layer 2048C may apply one or more filters to the output of the activation layer 2048B. Each filter may be a function applied to the input of the convolutional layer 2048C over a predetermined size at a predetermined stride (e.g., ranging from 1 to 64) to generate an output. The filter's function may include one or more parameters (sometimes called weights) applied to the input. The one or more parameters may be set, adjusted, or changed through training. Each filter may have a predetermined size (e.g., ranging from 3x3x1 to 1024x1024x3). The size and number of filters may vary depending on the network 2018 that provides the set of feature maps 2038′ as input to the terminal convolutional block 2046. The parameters of the convolutional layer 2048C may be repeatedly applied to the input (e.g., the set of feature maps 2038′) to generate an output according to a predetermined stride.

[0124] The recurrent layer 2048D of the terminal convolution block 2046 may include at least one function applied to the output of the previous transform layer 2048A-N (e.g., the final convolution layer 2048C). The function of the recurrent layer 2048D may be, among other things, a logistic activation function such as a softmax function (sometimes referred to herein as a normalized exponential function or polynomial logistic regression). The function may be set, adjusted, or changed through training. The activation function may be nonlinear. The recurrent layer 2048D may traverse all inputs, each of which corresponds to the output of one of the filters in the previous transform layer 2048A-N, such as the final convolution layer 2048C. During traversal, the recurrent layer 2048D may apply the activation function to the input to generate an output set of feature maps. Due to the logistic activation function, the output of the recurrent layer 2048D may be nonlinear with respect to the input.

[0125] By applying the set of transform layers 2048A-N to the set of feature maps 2038', the terminal convolution block 2046 can generate corresponding tiles 2022' for the segmented image 2020'. The output tiles 2022' can have the same resolution as the set of feature maps 2038' provided as input to the terminal convolution block 2046. The output tiles 2022' can also have the same resolution and magnitude coefficients as the original patches 2036 output by the network 2018 that provides the set of feature maps 2038' as input. The generation of additional tiles 2022' can use the segmented image 2020' corresponding to the biomedical image 2020.

[0126] Referring to FIG. 20(f), a block diagram of one of the networks 2018 of the segmentation model 2014 is shown, according to an example embodiment. An instance of the network 2018 can be included in the segmentation model 2014. Additionally, different instances of the network 2018 may include or lack at least some of the components or input / output connections shown in the example. In overview, the network 2018 and its components can be divided, organized, or arranged into a set of rows 2054A-N (hereinafter collectively referred to as rows 2054) and a set of columns 2056A-N (hereinafter collectively referred to as columns 2056). The network 2018 can have one or more sets of feature maps 2038′ (generally shown above) as outputs. The network 2018 can have as input one of the patches 2036 of the tile 2022 in the biomedical image 2020 (shown generally on the left) and a set of feature maps 2038' output from another network 2018 (shown generally below).

[0127] The network 2018 may have one or more convolution blocks 2032 and one or more deconvolution blocks 2040 arranged across different rows 2054 (sometimes referred to herein as channels). Each row 2054 may correspond to the feedforward of processed feature maps of the same resolution or size. The size may correspond to or define the number of pixels in each dimension of the feature map. For example, a first row 2054A may have two convolution blocks 2032 (one shown generally to the left and the other generally to the right). The feature maps processed and output by the convolution blocks 2032 in the first row 2054A may be of the same resolution. Similarly, a second row 2054B may feedforward feature maps of the same resolution, a third row 2054C may feedforward feature maps of the same resolution, and so on. Conversely, the feature maps in different rows 2054 may be of different resolutions. For example, the feature map fed forward along the second row 2054B may have a different resolution than the feature map fed forward along the first row 2054A. Each row 2054 may include at least one convolution block 2032. At least some of the rows 2054 below the first row 2054A, such as rows 2054B-N, may include at least one deconvolution block 2040.

[0128] The network 2018 may have one or more convolution blocks 2032 and one or more deconvolution blocks 2040 arranged across different columns 2056. Each column 2056 may correspond to feeding a processed feature map upward or downward from one resolution (or one row 2054) to another resolution (or another row 2054) after processing by one of the convolution blocks 2032 or one of the deconvolution blocks 2040. For example, the first column 2056A may correspond to a first instance in which a processed feature map is passed from one row to another after being processed by the first convolution block 2032 in each row 2054. Similarly, the second column 2056B may correspond to a second instance in which a feature map is passed from one row to another after being processed by the convolution block 2032 in the third row 2054C or by the deconvolution block 2040 in the fourth row 2054D.

[0129] In addition to the convolution block 2032 and the deconvolution block 2040, the network 2018 may include one or more pooling units 2058 (sometimes referred to herein as pooling operators). Each pooling unit 2058 may span multiple rows 2054 or may be located in one of the rows 2054 (e.g., as shown) and feed from one row 2054 to another. Each pooling unit 2058 may retrieve, receive, or otherwise identify a feature map to be processed by one of the convolution blocks 2032 or deconvolution blocks 2040 (e.g., as shown) in the same row 2054. The pooling unit 2058 may apply a pooling operation to the identified feature maps. The pooling operation may include, for example, max pooling by a predetermined factor (e.g., 2 to 10) to select the highest value in each set patch in the feature map, or average pooling to determine the average value in the set patches in the feature map. By applying the pooling operation, the pooling unit 2058 can generate feature maps of the same resolution as the rows 2054 to which the feature maps are provided. The size of the resulting set of feature maps can have a smaller size than the input set of feature maps.

[0130] Additionally, the network 2018 may include one or more cropping units 2060 (sometimes referred to herein as crop operators or croppers). Each cropping unit 2060 may retrieve, receive, or otherwise identify a feature map to be processed by the convolution block 2032 (e.g., as shown) or one of the deconvolution blocks 2040 in the same row 2054. The cropping unit 2060 may apply or perform a cropping operation on the identified feature map at a predetermined cropping rate (e.g., between 2 and 10). When performing the cropping operation, the cropping unit 2060 may identify a portion of the feature map. This portion may correspond to, for example, a center portion, a left portion, a right portion, a top portion, a bottom portion, a top-left portion, a top-right portion, a bottom-left portion, a bottom-right portion, or any other subsection of the feature map. The portion identified by the cropping unit 2060 may be the same portion identified by the patch generator 2008 to generate the patch 2036. By applying the cropping operation, cropping unit 2060 can generate feature maps 2038' at the same resolution as network 2018 to which it is provided (e.g., as shown generally toward the top). The size of the resulting set of feature maps can have a larger size than the input set of feature maps.

[0131] Additionally, the network 2018 may include one or more concatenation units 2062 (sometimes referred to herein as concatenators or aggregators). Each concatenation unit 2062 may concatenate, combine, or otherwise add two or more feature maps before being processed by a subsequent deconvolution block 2040 (e.g., as shown) or a convolution block 2032 in the same row 2054. In some embodiments, the concatenation unit 2062 may be part of a deconvolution block 2040 that will process the resulting set of feature maps in the same row 2054. Each received feature map may be from another network 2018 in the segmentation model 2014. Upon receiving the input feature maps, the concatenation unit 2062 may combine the feature maps to generate a resulting set of feature maps for feeding forward along the row 2054. The combination of feature maps (e.g., feature map 2038′) by the concatenation unit 2062 may include concatenation, weighted summation, and addition, among others. The resulting set of feature maps can be fed forward to a deconvolution block 2040 (e.g., as shown) for further processing in the network 2018 for input.

[0132] Due to the arrangement of the components of the network 2018 across multiple rows 2054 and columns 2056, the network 2018 may also have one or more inputs and outputs arranged across the rows 2054 and columns 2056. The first row 2054A may have as an input one of the patches 2036 from the tile 2022 of the biomedical image 2020 and as an output a set of resulting feature maps 2038′. The second and subsequent columns 2056B-N (e.g., the second column 2056B, the third column 2056C, the fourth column 2056D) may each have as an input a set of resulting feature maps 2038′ from another network 2018, such as a network 2018 associated with a lower dilation factor. Additionally, the second and subsequent columns 2056B-N may each have as output a set of resulting feature maps 2038' output by an internal component as output to another network 2018, such as a network 2018 associated with a higher magnification factor.

[0133] Referring to FIG. 20(g), a block diagram of a segmentation model 2014 is shown in accordance with an example embodiment. The segmentation model 2014 may include a set of networks 2018, such as a first network 2018A, a second network 2018B, and a third network 2018C, as shown. Each network 2018 may include or exclude the components and connections described herein in conjunction with FIG. 20(f). As shown, the first network 2018A may lack an optional cropping unit 2060, while the second network 2018B and the third network 2018C may each lack a concatenation unit 2062. Although three networks 2018A-C are shown, the segmentation model 2014 may have any number of networks 2018.

[0134] From outside the segmentation model 2014, each network 2018 can have as an input one of the patches 2036 at a corresponding magnification factor from one of the tiles 2022 of the biomedical image 2020. The patches 2036 can be fed to a first row 2054A of each network 2018. For example, as shown, a patch 2036A at a first magnification factor (e.g., 20×) can be fed as an input to the first row 2054A of a first network 2018A, a patch 2036B at a second magnification factor (e.g., 10×) can be fed as an input to the first row 2054A of a second network 2018B, and a patch 2036 at a third magnification factor (e.g., 20×) can be fed as an input to the first row 2054A of a third network 2018C. Additionally, each network 2018 may itself correspond to or be associated with one of the magnification factors. For example, a first network 2018A may be associated with a first magnification factor (e.g., 20×), a second network 2018B may be associated with a second magnification factor (e.g., 10×), a third network 2018C may be associated with a third magnification factor (e.g., 20×), and so on.

[0135] In the segmentation model 2014, the inputs and outputs of the networks 2018 can be fed into each other. The outputs of the networks 2018 with lower augmentation factors can be fed as inputs to the network 2018 with the highest augmentation factor. For example, as shown, the resulting set of feature maps 2038′ from the second network 2018B (associated with an augmentation factor of 10×) from each column 2056 can be provided as inputs to the concatenation unit 2062 of the corresponding column 2056 of the first network 2018A (associated with an augmentation factor of 20×). In some embodiments, the outputs of the networks 2018 with even lower augmentation factors can be fed as inputs to the network 2018 with an augmentation factor having a shift 2064. The shift 2064 can define the transfer of the output set of feature maps 2038′ of one column 2056 from the network 2018 with the lower augmentation factor to another column 2056 of the network 2018 with the higher augmentation factor as input. The shift 2064 can account for differences in resolution or size of the next component to which the output set of feature maps 2038' is fed. For example, the resulting set of feature maps 2038' from each column 2056 of the third network 2018C can be fed as inputs to the concatenation unit 2062 in one column 2056 across the first network 2018A. As shown, the resulting set of feature maps 2038' from the third column 2056C in the third network 2018C can be fed as inputs to the second column 2056B of the first network 2018A.

[0136] Additionally, at the output of the first row 2054A of the first network 2018A, the segmentation model 2014 may have a terminal convolution block 2046. The output of the first row 2054A of the first network 2018A may be fed as input to the terminal convolution block 2046. Transform layers 2048A-N of the terminal convolution block 2046 may be applied to generate tiles 2022' for the segmented biomedical image 2020'. The tiles 2022' may be of the same resolution and magnification factor as the patches 2036A fed to the first network 2018A. Additional tiles 2022' for the segmented image 2020' may be generated by applying the network 2018 of the segmentation model 2014 to further tiles 2022-to-patches 2036 of the biomedical image 2020.

[0137] Referring again to FIG. 20( a), the model applicator 2010 may apply the segmentation model 2014 to patches 2036 generated from tiles 2022 of the biomedical image 2020 to generate a segmented image 2020′. The model applicator 2010 may use the output of the terminal convolution block 2046 as the tiles 2022′ for the segmented image 2020′. Through repeated application of the segmentation model 2014 to the patches 2036, the model applicator 2010 may aggregate the tiles 2022′ to construct the segmented image 2020′. The segmented image 2020′ may have one or more indications of regions of interest 2026 within the biomedical image 2020. Each indication may correspond to at least one of the regions of interest 2026 in the biomedical image 2020. In some embodiments, the segmented image 2020' may have different indications (e.g., indicated by color, shading, or another visual marker) for each type of region of interest 2026, such as cancer, benign epithelium, background, stroma, necrosis, and adipose tissue, among others.

[0138] In some embodiments, the model applicator 2010 can provide the biomedical image 2020 or the segmented image 2020′ to the display 2006 for presentation. The display 2006 can present or render an image output by the image segmentation system 2002 (e.g., biomedical image 2020A or 2020B or segmented image 2020′ as shown) provided by the model applicator 2010. The display 2006 can include any monitor, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) monitor, and a cathode ray tube (CRT), among others. The display 2006 can be communicatively coupled to the image segmentation system 2002 and can render and output images from the image segmentation system 2002. In some embodiments, the model applicator 2010 may store the biomedical image 2020 ′ (or individual tiles 2022 ′ of the biomedical image 2020 ′) in a database accessible to the image segmentation system 2002 .

[0139] A model trainer 2012 executing in the image segmentation system 2002 can train a segmentation model 2014 using a training dataset 2016. Training of the segmentation model 2014 can occur when the image segmentation system 2002 is in training mode. Training of the segmentation model 2014 can occur in conjunction with segmented images 2020′ generated by the segmentation model 2014 using sample biomedical images 2020B. The model trainer 2012 can identify segmented images 2020′ (or segmented tiles 2022′) generated by applying the segmentation model 2014 to the sample biomedical images 2020B. From the training dataset 2016, the model trainer 2012 can also identify the sample biomedical images 2020B and a set of annotations 2024. In some embodiments, the model trainer 2012 can identify a tile 2022 in the sample biomedical image 2020B that corresponds to the segmented tile 2022' to be compared. Upon identification, the model trainer 2012 can compare the segmented image 2020' to the annotations 2024 for the sample biomedical image 2020B that were used to generate the segmented image 2020'. As discussed above, the annotations 2024 can be full or partial for the region of interest 2026 in the sample biomedical image 2020. In some embodiments, the model trainer 2021 can compare the indication for the region of interest 2026 in the segmented image 2020' to the annotation 2024 for the type of region of interest.

[0140] By comparing the segmented image 2020′ to the set of annotations 2024, the model trainer 2012 can calculate or otherwise determine at least one loss metric between the segmented image 2020′ and the annotations 2024. The loss metric may indicate at least one difference between the segmented image 2020′ and the annotations 2024 and may be, for example, a root mean square error, a relative root mean square error, and a weighted cross entropy, among others. The comparison between the segmented image 2020′ and the annotations 2024 may be pixel-by-pixel, and the loss metric may also be calculated pixel-by-pixel. In some embodiments, the model trainer 2012 may determine a loss metric across all or at least a portion of the pixels of the segmented image 2020′ and the annotations 2024. In some embodiments, the model trainer 2012 can determine a loss metric between the segmented image 2020′ and the annotations 2024 for each type of region of interest 2026.

[0141] Calculating the loss metric may cause the model trainer 2012 to update at least one of the parameters in the network 2018 of the segmentation model 2014. In some embodiments, the model trainer 2012 may use the loss metric calculated for each type of region of interest 2026 to update at least one of the parameters in the network 2018. The parameters updated using the loss metric may be in the transform layers 2034A-N of the convolution block 2032, the transform layers 2042A-N of the deconvolution block 2040, and the transform layers 2048A-N of the terminal convolution block 2046, among others.

[0142] In some embodiments, the model trainer 2012 may determine whether the segmentation model 2014 has converged based on a comparison of a current determination of a loss metric with a previous determination of the loss metric. If the difference between the two loss metrics is greater than a threshold, the model trainer 2012 may continue training the segmentation model 2014. Otherwise, if the difference is equal to or less than the threshold, the model trainer 2012 may stop training. In some embodiments, the training of the segmentation model 2014 is performed at a set learning rate (e.g., 10) over multiple iterations (e.g., ranging from 10 to 100). -6 ~10 -4 ), momentum (e.g., 0.5 to 1), and weight decay (e.g., 10 -6 ~10 -4 It can be defined using an objective function (e.g., stochastic gradient descent (SGD)) with a range of

[0143] Referring to Figure 20(h), a flowchart of a method 2070 for segmenting a biomedical image using a multi-magnification encoder-decoder concatenated network is shown, according to an example embodiment. The method 2070 may be implemented, practiced, or performed using any of the components described in conjunction with Figures 20(a)-20(g) or 16(a)-16(d). In overview, a computing system may identify a biomedical image (572). The computing system may create patches at multiple magnification factors (574). The computing system may apply a segmentation model (576). The computing system may generate a segmented image (578).

[0144] In further detail, a computing system (e.g., image segmentation system 2002) can identify (572) a biomedical image (e.g., acquired biomedical image 2020A). The biomedical image 2020 can be, for example, a histological section stained with hematoxylin-eosin (H&E) stain or the like. An imaging device (e.g., imager 2004) can acquire the biomedical image from a tissue section from a subject to perform a histopathological examination. The tissue sample used to acquire the biomedical image can have an area of ​​interest such as a tumor growth, a lesion, or a trauma, among other things. The biomedical image identified by the computing system may lack any associated annotations (e.g., annotation 2024). The computing system can divide the biomedical image into a set of tiles (e.g., tile 2022). Each tile can correspond to a portion of the biomedical image.

[0145] The computing system can create 574 patches (e.g., patch 2036) at multiple magnification factors. Each patch can correspond to a portion of the biomedical image at a different magnification factor. The magnification factor can define the amount, number, or ratio by which the corresponding tile is magnified. To create each pile from the tiles, the computing system can apply a corresponding magnification factor to the same tile.

[0146] The computing system may apply (576) a segmentation model (e.g., segmentation model 2014). The application of the segmentation model may be to a set of patches created from each tile of the biomedical image. The segmentation model may include a set of networks (e.g., network 2018) corresponding to a set of augmentation factors used to create the patches. Each network may include a set of encoders (e.g., convolution block 2032) and a set of decoders (e.g., deconvolution block 2040). Each encoder may have a set of transform layers (e.g., transform layers 2034A-N) for transforming feature maps. Each decoder may have a set of transform layers (e.g., transform layers 2042A-N) for transforming feature maps. In the network corresponding to the highest augmentation factor (e.g., first network 2018A), each decoder may also have a concatenator (e.g., concatenation unit 2062) that combines feature maps from other networks. Using an encoder and decoder, each network can transform input patches at the corresponding expansion factor into a set of resulting feature maps (e.g., feature map 2038′).

[0147] The computing system can generate a segmented image (e.g., biomedical image 2020') (578). The segmentation model can also have a final encoder (e.g., terminal convolution block 2046) to convert the resulting set of feature maps from the network with the largest expansion factor into tiles (e.g., tiles 2022') for the segmented image (e.g., segmented image 2020'). By applying the segmented image, the computing system can generate the segmented image. The computing system can render the segmented image for display. The computing system can store the segmented image in a database.

[0148] Referring to FIG. 20(i), a flow diagram of a method 2080 for training a model for segmenting biomedical images using a multi-magnification encoder-decoder concatenated network is shown, according to an example embodiment. The method 2080 may be implemented, practiced, or performed using any of the components described in conjunction with FIGS. 20(a)-20(g) or 16(a)-16(d). In overview, a computing system may identify a sample biomedical image (582). The computing system may create patches at multiple magnification factors (584). The computing system may apply a segmentation model (586). The computing system may generate a segmented image (588). The computing system may determine a loss for partial annotations (590). The computing system may update the segmentation model (592).

[0149] In further detail, a computing system (e.g., image segmentation system 2002) can identify 582 a sample biomedical image (e.g., sample biomedical image 2020B). The function of 582 can be similar to the function of 570 of method 2070, except with respect to the sample biomedical image. The sample biomedical image can be associated with a set of annotations (e.g., annotations 2024). The annotations can indicate or label at least a portion of a region of interest (e.g., region of interest 2026) in the sample biomedical image. Each annotation can correspond to a different type of region of interest, such as cancerous tissue, benign epithelial tissue, stromal tissue, necrotic tissue, and adipose tissue, among others. When partially annotated, the annotation can be separated from a boundary of the region of interest by a separation distance (e.g., distance 2030).

[0150] The computing system may create patches (e.g., patch 2036) at multiple magnification factors (584). The functionality of (584) may be similar to the functionality of (574) of method 2070, described in detail above. The computing system may apply a segmentation model (e.g., segmentation model 2014) (586). The functionality of (586) may be similar to the functionality of (576) of method 2070, described in detail above. The computing system may generate a segmented image (e.g., segmented image 2020′) (588). The functionality of (588) may be similar to the functionality of (578) of method 2070, described in detail above.

[0151] The computing system may determine a loss for the partial annotation (590). The computing system may compare a segmented image (e.g., segmented image 2020′) generated by applying the segmentation model to the sample biomedical image with the partial annotation for the sample biomedical image. In the comparison, the computing system may determine or calculate a loss between the indications in the segmented image and the labeling in the annotation for the sample biomedical image. The comparison may be pixel-by-pixel, and the determined loss metric may be pixel-by-pixel. The computing system may update the segmentation model (592). Using the determined loss, the computing system may modify or update at least one parameter of the network of the segmentation model, such as in one of the encoder or decoder.

[0152] D. Computing and Networking Environment It may be useful to describe aspects of the operating environment and associated system components (e.g., hardware elements) relevant to the methods and systems described in Section A. Referring to FIG. 21(a), an embodiment of a network environment is shown. In overview, the illustrated discovery network environment includes one or more clients 2102a-2102n (also referred to collectively as local machines 2102, clients 2102, client nodes 2102, client machines 2102, client computers 2102, client devices 2102, endpoints 2102, or endpoint nodes 2102) communicating with one or more servers 2106a-2106n (also referred to collectively as servers 2106, nodes 2106, or remote machines 2106) via one or more networks 2104. In some embodiments, a client 2102 has the capability to function both as a client node seeking access to resources provided by a server and as a server providing access to hosted resources for other clients 2102a-2102n.

[0153] 21(a) shows a network 2104 between the client 2102 and the server 2106, but the client 2102 and the server 2106 can be on the same network 2104. In some embodiments, there are multiple networks 2104 between the client 2102 and the server 2106. In one of these embodiments, the network 2104' (not shown) can be a private network, and the network 2104 can be a public network. In another of these embodiments, the network 2104 can be a private network, and the network 2104' can be a public network. In yet another of these embodiments, both the networks 2104 and 2104' can be private networks.

[0154] The network 2104 can be connected via wired or wireless links. Wired links can include digital subscriber lines (DSL), coaxial cable lines, or fiber optic lines. Wireless links can include Bluetooth, Wi-Fi, NFC, RFID, worldwide interoperable microwave access (WiMAX), infrared channels, or satellite bands. Wireless links can also include any cellular network standard used for communication between mobile devices, including standards qualified as 1G, 2G, 3G, or 4G. Network standards can qualify as one or more generations of mobile telecommunications standards by meeting specifications or standards such as those maintained by the International Telecommunications Union. For example, the 3G standard can correspond to the International Mobile Telecommunications-2000 (IMT-2000) specification, and the 4G standard can correspond to the International Mobile Telecommunications-Advanced (IMT-Advanced) specification. Examples of cellular network standards include AMPS, GSM, GPRS, UMTS, LTE, LTE Advanced, Mobile WiMAX, and WiMAX-Advanced. Cellular network standards may use various channel access methods, such as FDMA, TDMA, CDMA, or SDMA. In some embodiments, different types of data may be transmitted over different links and standards. In other embodiments, the same type of data may be transmitted over different links and standards.

[0155] The network 2104 can be any type and / or form of network. The geographic scope of the network 2104 can vary widely, and the network 2104 can be a body area network (BAN), a personal area network (PAN), a local area network (LAN), e.g., an intranet, a metropolitan area network (MAN), a wide area network (WAN), or the Internet. The topology of the network 2104 can take any form and can include, for example, any of the following: point-to-point, bus, star, ring, mesh, or tree. The network 2104 can be virtual, an overlay network that sits on top of one or more layers of other networks 2104′. The network 2104 can be any such network topology known to those skilled in the art that is capable of supporting the operations described herein. The network 2104 can utilize different technologies and layers or protocol stacks, including, for example, Ethernet protocols, Internet Protocol Suite (TCP / IP), ATM (Asynchronous Transfer Mode) techniques, Synchronous Optical Network (SONET) protocols, or Synchronous Digital Hierarchy (SDH) protocols. The TCP / IP Internet protocol suite may include an application layer, a transport layer, an internet layer (including, for example, IPv6), or a link layer. Network 2104 may be some type of broadcast network, telecommunications network, data communications network, or computer network.

[0156] In some embodiments, the system may include logically grouped servers 2106. In one of these embodiments, the logical grouping of servers may be referred to as a server farm 2107 or a machine farm 2107. In another of these embodiments, the servers 2106 may be geographically distributed. In other embodiments, the machine farm 2107 may be managed as a single entity. In yet other embodiments, the machine farm 2107 includes multiple machine farms 38. The servers 2106 within each machine farm 2107 may be heterogeneous. One or more of the servers 2106 or machines 2106 may operate according to one type of operating system platform (e.g., WINDOWS NT®, manufactured by Microsoft Corp. of Redmond, Washington), while one or more of the other servers 2106 may operate according to another type of operating system platform (e.g., Unix, Linux®, or Mac OS X).

[0157] In one embodiment, the servers 2106 in the machine farm 2107, along with their associated storage systems, may be housed in a high-density rack system and located in an enterprise data center. In this embodiment, centralizing the servers 2106 in this manner may improve system manageability, data security, system physical security, and system performance by locating the servers 2106 and high-performance storage systems on a localized, high-performance network. Centralizing the servers 2106 and storage systems and coupling them with advanced system management tools allows for more efficient use of server resources.

[0158] The servers 2106 of each machine farm 2107 need not be physically close to another server 2106 within the same machine farm 2107. Thus, a group of servers 2106 logically grouped as a machine farm 2107 can be interconnected using wide area network (WAN) or metropolitan area network (MAN) connections. For example, a machine farm 2107 can include servers 2106 physically located on different continents, or in different regions of a continent, country, state, city, campus, or room. Data transmission rates between servers 2106 within a machine farm 2107 can be increased if the servers 2106 are connected using local area network (LAN) connections or some form of direct connection. Additionally, a heterogeneous machine farm 2107 can include one or more servers 2106 that operate according to a type of operating system, while one or more other servers 2106 run one or more types of hypervisors rather than an operating system. In these embodiments, a hypervisor can be used to emulate virtual hardware, partition physical hardware, virtualize physical hardware, and run virtual machines that provide access to computing environments, allowing multiple operating systems to run simultaneously on a host computer. A native hypervisor can run directly on a host computer. Hypervisors can include VMware ESX / ESXi, manufactured by VMWare, Inc. of Palo Alto, California; the Xen hypervisor, an open source product developed by Citrix Systems, Inc.; and the HYPER-V hypervisor, offered by Microsoft and others. A host hypervisor can run within an operating system at a second software level. Examples of host hypervisors include VMware Workstation and VirtualBox.

[0159] Management of the machine farm 2107 may be decentralized. For example, one or more servers 2106 may include components, subsystems, and modules for supporting one or more management services for the machine farm 2107. In one of these embodiments, the one or more servers 2106 provide functionality for managing dynamic data, including techniques for failover, data replication, and increasing the robustness of the machine farm 2107. Each server 2106 may communicate with a persistent store, and in some embodiments, a dynamic store.

[0160] Server 2106 may be a file server, application server, web server, proxy server, appliance, network appliance, gateway, gateway server, virtualization server, deployment server, SSL VPN server, or firewall. In one embodiment, server 2106 may be referred to as a remote machine or node. In another embodiment, multiple nodes may be in the path between any two communication servers.

[0161] Referring to FIG. 21(b), a cloud computing environment is illustrated. The cloud computing environment can provide a client 2102 with one or more resources provided by a network environment. The cloud computing environment can include one or more clients 2102a-2102n that communicate with a cloud 2108 via one or more networks 2104. The clients 2102 can include, for example, thick clients, thin clients, and zero clients. A thick client can provide at least some functionality even when disconnected from the cloud 2108 or server 2106. A thin client or zero client can provide functionality by relying on a connection to the cloud 2108 or server 2106. A zero client can rely on the cloud 2108 or other networks 2104 or servers 2106 to retrieve operating system data for the client device. The cloud 2108 can include a backend platform, for example, a server 2106, storage, a server farm, or a data center.

[0162] The cloud 2108 can be public, private, or hybrid. A public cloud can include a public server 2106 maintained by a third party to the client 2102 or the client's owner. As disclosed above and elsewhere, the server 2106 may be located off-site in a remote geographic location. A public cloud can be connected to the server 2106 through a public network. A private cloud can include a private server 2106 physically maintained by the client 2102 or the client's owner. A private cloud can be connected to the server 2106 through a private network 2104. A hybrid cloud 2108 can include both a private network 2104 and a public network 2104, as well as a server 2106.

[0163] Cloud 2108 can also include cloud-based delivery, such as software as a service (SaaS) 2110, platform as a service (PaaS) 2112, and infrastructure as a service (IaaS) 2114. IaaS can refer to users paying a fee to use the infrastructure resources they need for a specific period of time. IaaS providers can offer storage, networking, server, or virtualization resources from a large pool, allowing users to quickly scale up by accessing more resources as needed. PaaS providers can offer the functionality offered by IaaS, including, for example, storage, networking, server, or virtualization, as well as additional resources, such as operating systems, middleware, or runtime resources. Examples of PaaS include WINDOWS AZURE, offered by Microsoft Corporation of Redmond, Washington; Google App Engine, offered by Google Inc.; and HEROKU, offered by Heroku, Inc. of San Francisco, California. SaaS providers can offer the resources offered by PaaS, including storage, networking, server, virtualization, operating systems, middleware, or runtime resources. In some embodiments, the SaaS provider may provide additional resources, including, for example, data and application resources.

[0164] Client 2102 can access IaaS resources through one or more IaaS standards, including, for example, Amazon Elastic Compute Cloud (EC2), Open Cloud Computing Interface (OCCI), Cloud Infrastructure Management Interface (CIMI), or OpenStack standards. Some IaaS standards allow clients to access resources through HTTP and can use the Representational State Transfer (REST) ​​protocol or the Simple Object Access Protocol (SOAP). Client 2102 can access PaaS resources with different PaaS interfaces. Some PaaS interfaces use the HTTP package, standard Java APIs, JavaMail APIs, Java Data Objects (JDO), Java Persistence API (JPA), Python APIs, web integration APIs for different programming languages, including Rack for Ruby, WSGI for Python, or PSGI for Perl, or other APIs that can be built on REST, HTTP, XML, or other protocols. Client 2102 can access SaaS resources through the use of a web-based user interface provided by a web browser. The client 2102 can also access the SaaS resources through a smartphone or tablet application that the client 2102 has. The client 2102 can also access the SaaS resources through a client operating system.

[0165] In some embodiments, access to IaaS, PaaS, or SaaS resources can be authenticated. For example, a server or authentication server can authenticate a user via a security certificate, HTTPS, or API key. The API key can include various encryption standards, such as the Advanced Encryption Standard (AES). Data resources can be transmitted over Transport Layer Security (TLS) or Secure Sockets Layer (SSL).

[0166] The client 2102 and server 2106 may be deployed as and / or executed on any type and form of computing device, e.g., a computer, network device, or appliance, capable of communicating over any type and form of network and performing the operations described herein. FIGS. 21(c) and 21(d) show block diagrams of computing devices 2100 useful for implementing embodiments of the client 2102 or server 2106. As shown in FIGS. 21(c) and 21(d), each computing device 2100 includes a central processing unit 2121 and a main memory unit 2122. As shown in FIG. 21(c), the computing device 2100 may include a storage device 2128, an installation device 2116, a network interface 2118, an I / O controller 2123, display devices 2124a-n, a keyboard 2126, and a pointing device 2127, e.g., a mouse. The storage device 2128 may include, but is not limited to, an operating system and / or software 2120. As shown in FIG. 21(d), each computing device 2100 may further include additional optional elements, such as a memory port 2103, a bridge 2170, one or more input / output devices 2130a-1130n (collectively referred to with reference numeral 2130), and a cache memory 2140 in communication with the central processing unit 2121.

[0167] The central processing unit 2121 is any logic circuitry that responds to and processes instructions fetched from the main memory unit 2122. In many embodiments, the central processing unit 2121 is provided by a microprocessor unit. The computing device 2100 can be based on any of these processors or any other processor capable of operating as described herein. The central processing unit 2121 can utilize instruction-level parallelism, thread-level parallelism, different levels of caches, and multi-core processors. A multi-core processor can include two or more processing units in a single computing component.

[0168] The main memory unit 2122 can include one or more memory chips that can store data and allow the microprocessor 2121 to directly access memory locations. The main memory unit 2122 can be volatile and faster than the storage 2128 memory. The main memory unit 2122 can be dynamic random access memory (DRAM) or any variant including static random access memory (SRAM), burst SRAM or synchronous burst SRAM (BSRAM), fast page mode DRAM (FPM DRAM), enhanced DRAM (EDRAM), extended data output RAM (EDO RAM), extended data output DRAM (EDO DRAM), burst extended data output DRAM (BEDO DRAM), single data rate synchronous DRAM (SDR SDRAM), double data rate SDRAM (DDR SDRAM), direct Rambus DRAM (DRDRAM), or extreme data rate DRAM (XDR DRAM). In some embodiments, the main memory 2122 or storage 2128 is non-volatile and may be, for example, non-volatile read-access memory (NVRAM), flash memory non-volatile static RAM (nvSRAM), ferroelectric RAM (FeRAM), magnetoresistive RAM (MRAM), phase-change memory (PRAM), conductive bridge RAM (CBRAM), silicon-oxide-nitride-oxide-silicon (SONOS), resistive RAM (RRAM), racetrack, nanoRAM (NRAM), or Millipede memory. The main memory 2122 may be based on any of the memory chips previously described or any other available memory chip capable of operating as described herein. In the embodiment shown in FIG. 21(c), the processor 2121 communicates with the main memory 2122 via a system bus 2150 (described in more detail below). FIG. 21(d) shows an embodiment of a computing device 2100 in which the processor communicates directly with the main memory 2122 via a memory port 2103. For example, in FIG. 21(d), the main memory 2122 can be a DRDRAM.

[0169] FIG. 21(d) illustrates an embodiment in which the main processor 2121 communicates directly with the cache memory 2140 through a secondary bus, sometimes referred to as a backside bus. In other embodiments, the main processor 2121 communicates with the cache memory 2140 using a system bus 2150. The cache memory 2140 typically has a faster response time than the main memory 2122 and is typically provided by SRAM, BSRAM, or EDRAM. In the embodiment illustrated in FIG. 21(d), the processor 2121 communicates with various I / O devices 2130 through a local system bus 2150. Various buses can be used to connect the central processing unit 2121 to any of the I / O devices 2130, including a PCI bus, a PCI-X bus, a PCI-Express bus, or a NuBus. In an embodiment in which the I / O device is a video display 2124, the processor 2121 can communicate with the display 2124 or an I / O controller 2123 for the display 2124 using an Advanced Graphics Port (AGP). Figure 21(d) shows an embodiment of a computer 2100 in which the main processor 2121 communicates directly with an I / O device 2130b or another processor 2121' via HYPERTRANSPORT, RAPIDIO, or INFINIBAND communication techniques. Figure 21(d) also shows an embodiment that combines local bus and direct communication, in which the processor 2121 communicates with an I / O device 2130a using a local interconnect bus, but communicates directly with an I / O device 2130b.

[0170] Various I / O devices 2130a-2130n may be present in computing device 2100. Input devices may include a keyboard, mouse, trackpad, trackball, touchpad, touch mouse, multi-touch touchpad and touch mouse, microphone, multi-array microphone, drawing tablet, camera, single lens reflex camera (SLR), digital SLR (DSLR), CMOS sensor, accelerometer, infrared light sensor, pressure sensor, magnetometer sensor, angular rate sensor, depth sensor, proximity sensor, ambient light sensor, gyro sensor, or other sensor. Output devices may include a video display, a graphics display, speakers, headphones, inkjet printer, laser printer, and 3D printer.

[0171] The devices 2130a-1130n may include a combination of multiple input or output devices. Some devices 2130a-1130n enable gesture recognition input by combining some of the inputs and outputs. Some devices 2130a-1130n offer facial recognition, which can be used as input for different purposes, including authentication and other commands. Some devices 2130a-1130n offer voice recognition and input. Additional devices 2130a-1130n have both input and output capabilities, including, for example, haptic feedback devices, touchscreen displays, or multi-touch displays. Touchscreens, multi-touch displays, touchpads, touch mice, or other touch-sensing devices can sense touch using different technologies, including, for example, capacitive, surface capacitive, projected capacitive touch (PCT), in-cell capacitive, resistive, infrared, waveguide, distributed signal touch (DST), in-cell optical, surface acoustic wave (SAW), bending wave touch (BWT), or force-based sensing technologies. Some multi-touch devices allow for more than two points of contact with a surface, thereby enabling advanced functionality including, for example, pinch, spread, rotate, scroll, or other gestures. Some touchscreen devices, including those on a tabletop or wall, can also interact with other electronic devices. Some I / O devices 2130a-1130n, display devices 2124a-1124n, or groups of devices, may be augmented reality devices. As shown in FIG. 21(c), the I / O devices may be controlled by an I / O controller 2123. The I / O controller may control one or more I / O devices, such as, for example, a keyboard 2126 and a pointing device 2127, e.g., a mouse or optical pen. Additionally, the I / O devices may provide storage and / or installation media 2116 for the computing device 2100. In yet another embodiment, the computing device 2100 may provide a USB connection (not shown) to accept a portable USB storage device.In further embodiments, I / O device 2130 may be a bridge between system bus 2150 and an external communication bus, such as a USB bus, a SCSI bus, a Firewire bus, an Ethernet bus, a Gigabit Ethernet bus, a Fibre Channel bus, or a Thunderbolt bus.

[0172] In some embodiments, the display devices 2124a-1124n can be connected to the I / O controller 2123. The display devices can include, for example, a liquid crystal display (LCD), a thin film transistor LCD (TFT-LCD), a blue-phase LCD, an electronic paper (e-ink) display, a flexible display, a light-emitting diode display (LED), a digital light processing (DLP) display, a liquid crystal on silicon (LCOS) display, an organic light-emitting diode (OLED) display, an active matrix organic light-emitting diode (AMOLED) display, a liquid crystal laser display, a time-domain optical shutter (TMOS) display, or a 3D display. Examples of 3D displays can use, for example, stereoscopic vision, polarized filters, active shutters, or autostereoscopic vision. The display devices 2124a-1124n can be head-mounted displays (HMDs). In some embodiments, the display devices 2124a-1124n or corresponding I / O controllers 2123 can be controlled by, or have hardware support for, the OPENGL or DIRECTX API or other graphics libraries.

[0173] In some embodiments, the computing device 2100 may include or be connected to multiple display devices 2124a-1124n, each of which may be the same or different types and / or forms. In this manner, any of the I / O devices 2130a-1130n and / or I / O controller 2123 may include any type and / or form of suitable hardware, software, or combination of hardware and software to support, enable, or provide for the connection and use of multiple display devices 2124a-1124n by the computing device 2100. For example, the computing device 2100 may include any type and / or form of video adapter, video card, drivers, and / or libraries for interfacing, communicating, connecting, or otherwise using the display devices 2124a-1124n. In one embodiment, the video adapter may include multiple connectors for interfacing to multiple display devices 2124a-1124n. In other embodiments, computing device 2100 may include multiple video adapters, each connected to one or more of display devices 2124a-1124n. In some embodiments, any portion of the operating system of computing device 2100 may be configured to use multiple displays 2124a-1124n. In other embodiments, one or more of display devices 2124a-1124n may be provided by one or more other computing devices 2100a or 2100b connected to computing device 2100 via network 2104. In some embodiments, software may be designed and configured to use a display device of another computer as a second display device 2124a of computing device 2100.

[0174] Referring again to FIG. 21(c), computing device 2100 may include storage 2128 (e.g., one or more hard disk drives or redundant array of independent disks) for storing an operating system or other related software, and for storing application software programs, such as any programs associated with software 2120. Examples of storage 2128 include, for example, a hard disk drive (HDD), an optical drive, a solid-state drive (SSD), a USB flash drive, or any other device suitable for storing data. Some storage devices may include multiple volatile and nonvolatile memories, including, for example, a solid-state hybrid drive that combines a hard disk and a solid-state cache. Some storage devices 2128 may be nonvolatile, alterable, or read-only storage devices. Some storage devices 2128 are internal storage devices and may be connected to computing device 2100 via bus 2150. Some storage devices 2128 are external storage devices and may be connected to computing device 2100 via I / O device 2130, which provides an external bus. Some storage devices 2128 may be connected to the computing device 2100 via a network interface 2118 over the network 2104. Some client devices 2100 may not require non-volatile storage 2128 and may be thin clients or zero clients 2102. Some storage devices 2128 may also be used as installation devices 2116 and may be suitable for installing software and programs.

[0175] The client device 2100 can also install software or applications from an application distribution platform. The application distribution platform can facilitate the installation of software on the client device 2102. The application distribution platform can include a repository of applications on a server 2106 or a cloud 2108 that is accessible to the clients 2102a-2102n over the network 2104. The application distribution platform can include applications developed and provided by various developers. A user of the client device 2102 can select, purchase, and / or download applications via the application distribution platform.

[0176] Additionally, computing device 2100 may include a network interface 2118 that interfaces to network 2104 through a variety of connections, including, but not limited to, standard telephone line LAN or WAN links (e.g., 802.11, T1, T3, Gigabit Ethernet, InfiniBand), broadband connections (e.g., ISDN, Frame Relay, ATM, Gigabit Ethernet, Ethernet over SONET, ADSL, VDSL, BPON, GPON, fiber optics including FiOS), wireless connections, or any combination of any or all of the above. Connections may be established using a variety of communication protocols (e.g., TCP / IP, Ethernet, ARCNET, SONET, SDH, Fiber Distributed Data Interface (FDDI), IEEE 802.11a / b / g / n / ac CDMA, GSM, WiMax, and direct asynchronous connection). In one embodiment, computing device 2100 communicates with other computing devices 2100′ via any type and / or form of gateway or tunnel protocol, such as Secure Socket Layer (SSL) or Transport Layer Security (TLS). Network interface 2118 may include an internal network adapter, a network interface card, a PCMCIA network card, an EXPRESS CARD network card, a card bus network adapter, a wireless network adapter, a USB network adapter, a modem, or any other device suitable for interfacing computing device 2100 capable of communicating and performing the operations described herein to any type of network.

[0177] 21(b) and 21(c) may operate under the control of an operating system that controls the scheduling of tasks and access to system resources. Computing device 2100 may operate with any operating system, such as any version of the MICROSOFT WINDOWS operating system, different releases of Unix and Linux operating systems, any version of MAC OS for Macintosh computers, any embedded operating system, any real-time operating system, any open source operating system, any dedicated operating system, any operating system for mobile computing devices, or any other operating system capable of operating on a computing device and performing the operations described herein. Common operating systems include, but are not limited to, WINDOWS 2000, WINDOWS Server 2012, WINDOWS CE, WINDOWS Phone, WINDOWS XP, WINDOWS VISTA, WINDOWS 7, WINDOWS RT, and WINDOWS 8, all manufactured by Microsoft Corporation of Redmond, Washington; MAC OS and iOS manufactured by Apple, Inc. of Cupertino, California; Linux, a freely available operating system, such as the Linux Mint distribution ("distro") or Ubuntu distributed by Canonical Ltd. of London, England, or Unix or other Unix-derived operating systems; and Android, designed by Google of Mountain View, California. Several operating systems, including, for example, CHROME OS by Google, may be used in zero or thin clients, including, for example, CHROMEBOOKS.

[0178] The computer system 2100 may be any workstation, telephone, desktop computer, laptop or notebook computer, netbook, tablet, server, portable computer, mobile phone, smartphone or other portable telecommunications device, media playing device, gaming system, mobile computing device, or any other type and / or form of communications-capable computing device, telecommunications device, or media device. The computer system 2100 has sufficient processor power and memory capacity to perform the operations described herein. In some embodiments, the computing device 2100 can have different processors, operating systems, and input devices consistent with the device.

[0179] In some embodiments, the computing device 2100 is a gaming system. In some embodiments, the computing device 2100 is a digital audio player. Some digital audio players can have other functionality, including, for example, a gaming system or any functionality made available by applications from a digital application distribution platform. In some embodiments, the computing device 2100 is a portable media player or digital audio player that supports the included file formats. In some embodiments, the computing device 2100 is a tablet. In other embodiments, the computing device 2100 is an e-reader. In some embodiments, the communication device 2102 includes a combination of devices, such as a smartphone combined with a digital audio player or portable media player. For example, one of these embodiments is a smartphone. In yet another embodiment, the communication device 2102 is a laptop or desktop computer with a web browser and a microphone and speaker system, such as a telephony headset. In these embodiments, the communication device 2102 is web-enabled and can receive and make calls. In some embodiments, the laptop or desktop computer also includes a webcam or other video capture device that enables video chat and video calling. In some embodiments, the communication device 2102 is a wearable mobile computing device.

[0180] In some embodiments, typically as part of network management, the status of one or more machines 2102, 2106 in the network 2104 is monitored. In one of these embodiments, the status of a machine may include determining load information (e.g., the number of processes on the machine, CPU and memory utilization), port information (e.g., the number of available communication ports and port addresses), or session status (e.g., the duration and type of process, whether the process is active or idle). In another of these embodiments, this information may be determined in terms of multiple metrics, which may be applied at least in part to load balancing decisions, network traffic management, network failure recovery, and any aspect of the operation of the present solutions described herein. Aspects of the operating environment and components described above will become apparent in the context of the systems and methods disclosed herein.

[0181] Descriptions herein of modules emphasize the structural independence of aspects of the image segmentation system and indicate one grouping of operations and responsiveness of the image segmentation system. Other groups that perform similar overall operations are understood to be within the scope of this application. Modules can be embodied in hardware and / or as computer instructions on a non-transitory computer-readable storage medium, and modules can be distributed across various hardware or computer-based components.

[0182] Exemplary, non-limiting module implementation elements include sensors providing any of the values ​​determined herein, sensors providing any value preceding the values ​​determined herein, data links and / or network hardware including communication chips, vibrating crystals, communication links, cables, twisted pair wiring, coaxial wiring, shielded wiring, transmitters, receivers and / or transceivers, logic circuits, hardwired logic circuits, reconfigurable logic circuits, particularly in a non-transient state configured according to the module specifications, any actuators including at least electric, hydraulic or pneumatic actuators, solenoids, operational amplifiers, analog control elements (springs, filters, integrators, adders, dividers, gain elements) and / or digital control elements.

[0183] Non-limiting examples of various embodiments are disclosed herein, and features from one embodiment disclosed herein can be combined with features of another embodiment disclosed herein, as will be understood by those skilled in the art.

[0184] As used herein, the terms "generally," "about," "substantially," and similar terms are intended to have a broad meaning consistent with common and accepted usage by those of ordinary skill in the art to which the subject matter of this disclosure pertains. Those skilled in the art who consider this disclosure will understand that these terms are intended to enable description of the particular features being described, and are not intended to limit the scope of those features to the precise numerical ranges provided. Thus, these terms should be interpreted to indicate that insubstantial or minor changes or modifications of the subject matter being described are considered to be within the scope of this disclosure.

[0185] For purposes of this disclosure, the term "coupled" refers to the direct or indirect joining of two parts to one another. Such joining may be of a static or movable nature. Such joining may be achieved by the two parts, or the two parts and any additional intermediate parts, being integrally formed with one another as a single body, or by the two parts, or the two parts and any additional intermediate parts, being attached to one another. Such joining may be of a permanent nature, or of a removable or releasable nature.

[0186] It should be noted that the orientation of various elements may differ according to other exemplary embodiments, and such variations are intended to be encompassed by the present disclosure. It is recognized that features of the disclosed embodiments may be incorporated into other disclosed embodiments.

[0187] It is important to note that the construction and arrangement of the devices or their components as shown in the various exemplary embodiments are illustrative only. While only a few embodiments have been described in detail in this disclosure, those skilled in the art reviewing this disclosure will readily appreciate that numerous modifications (e.g., variations in the size, dimensions, structure, shape, and proportions of the various elements, parameter values, mounting arrangements, use of materials, color, orientation, etc.) are possible without materially departing from the novel teachings and advantages of the disclosed subject matter. For example, elements shown as integrally formed may be constructed from multiple parts or elements, element positions may be reversed or otherwise changed, and the nature or number of separate elements or positions may be modified or changed. The order or sequence of any process or method steps may be changed or re-ordered in accordance with alternative embodiments. Other substitutions, alterations, substitutions, and omissions may also be made to the design, operating conditions, and arrangements of the various exemplary embodiments without departing from the scope of the present disclosure.

[0188] While various embodiments of the invention have been described and illustrated herein, those skilled in the art will readily envision a variety of other mechanisms and / or structures for performing the functions described herein and / or for obtaining the results and / or one or more of the advantages described herein, and each of such variations and / or modifications is deemed to fall within the scope of the embodiments of the invention described herein. More generally, those skilled in the art will readily understand that any parameters, dimensions, materials, and configurations described herein are intended to be exemplary unless otherwise specified, and that the actual parameters, dimensions, materials, and / or configurations will vary depending on the specific application for which the teachings of the present invention are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, numerous equivalents to the specific embodiments of the invention described herein. Accordingly, it should be understood that the above-described embodiments are presented by way of example only, and that within the scope of the appended claims and their equivalents, embodiments of the invention may be practiced other than as specifically described and claimed. The inventive embodiments of the present disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more of such features, systems, articles, materials, kits, and / or methods is within the inventive scope of the present disclosure, provided that such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent.

[0189] Additionally, the techniques described herein may be implemented as a method, at least one example of which is provided. Acts performed as part of a method may be ordered in any suitable way unless otherwise specified. Thus, embodiments may be constructed such that acts are performed in an order different from that illustrated, which may include performing some acts simultaneously even though they are shown as sequential acts in the exemplary embodiments.

[0190] As used in this specification and claims, the indefinite articles "a" and "an" should be understood to mean "at least one" unless expressly stated otherwise. When used in this specification and claims, "or" should be understood to have the same meaning as "and / or," as defined above. For example, when separating items in a list, "or" or "and / or" should be interpreted as inclusive, i.e., including at least one, but also including more than one of a plurality of elements or a list of elements, and may include additional unlisted items. Only terms expressly stated otherwise, such as "only one of" or "exactly one of," mean the inclusion of exactly one element of a plurality of elements or a list of elements. As used herein as a whole, the term "or" should be interpreted as indicating exclusive alternatives (i.e., "one or the other, but not both") only when preceded by exclusive terms, such as "either," "one of," "only one of," or "exactly one of."

[0191] As used herein and in the claims, the phrase "at least one" in reference to a list of one or more elements should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of every element specifically listed in the list of elements, and not excluding any combination of elements in the list of elements. This definition also allows for the optional presence of elements different from those specifically identified in the list of elements to which the phrase "at least one" refers, whether related to those elements specifically identified or not. Thus, as a non-limiting example, "at least one of A and B" (or, equivalently, "at least one of A or B" or, equivalently, "at least one of A and / or B") can refer, in one embodiment, to at least one A, which may include more than one, and no B (and which may include elements other than B); in another embodiment, to at least one B, which may include more than one, and no A (and which may include elements other than A); in yet another embodiment, to at least one A, which may include more than one, and at least one B (and which may include other elements), which may include more than one, etc.

Claims

1. 1. A method for segmenting a biomedical image, comprising: Identifying, by a computing system having one or more processors, for at least one tile of a biomedical image, a first patch at a first magnification factor and a second patch at a second magnification factor; applying, by the computing system, a trained segmentation model to the first patch and the second patch to generate segmented tiles, the trained segmentation model including a plurality of networks, the plurality of networks comprising: a first network for generating a plurality of first feature maps using the first patches at the first expansion factor; and a second network for generating a second feature map using the second patch at the second expansion factor and the one or more first feature maps from the first network; generating a storing, by the computing system, the segmented tiles identifying a region of interest within the at least one tile of the biomedical image; A method comprising:

2. The plurality of networks of the segmentation model include: generating a plurality of third feature maps using third patches of the at least one tile at a third expansion factor; providing the third feature maps to corresponding decoders of the second network to generate the second feature maps; The method of claim 1 , further comprising a third network for performing:

3. 2. The method of claim 1 , wherein the second network comprises a plurality of decoders arranged across corresponding columns, each of the plurality of decoders for processing a corresponding feature map of the plurality of first maps from the first network.

4. 2. The method of claim 1 , wherein the first network comprises a plurality of encoders arranged across corresponding columns, each of the plurality of encoders for providing a corresponding feature map of the plurality of first networks to a respective decoder in the second network.

5. 2. The method of claim 1, wherein the second network comprises a plurality of concatenators that combine the plurality of first feature maps from the first network with a corresponding plurality of intermediate feature maps in generating the second feature maps.

6. 2. The method of claim 1, further comprising generating, by the computing system, a segmented biomedical image using a plurality of segmented tiles by applying the segmentation model applied to a plurality of patches at a corresponding plurality of magnification factors, each patch being identified from a corresponding tile of the plurality of tiles of the biomedical image.

7. 10. The method of claim 1, further comprising acquiring, by the computing system, the biomedical image derived from a histopathological image generator, wherein the biomedical image is divided into a plurality of tiles, each tile of the plurality of tiles corresponding to a portion of the biomedical image.

8. 1. A method of training a network for segmenting biomedical images, comprising: A computing system having one or more processors, at least one sample tile from a sample biomedical image, the sample biomedical image having a region of interest; and an annotation labeling at least a portion of the region of interest, the annotation indicating at least a portion of the region of interest within the at least one sample tile; identifying a training dataset comprising: generating, by the computing system, a first patch at a first magnification factor and a second patch at a second magnification factor for the at least one sample tile of the sample biomedical image; training, by the computing system, a segmentation model using the annotations of the first patch, the second patch, and the at least one sample tile, the segmentation model comprising a plurality of networks, the plurality of networks comprising: a first network for generating a plurality of first feature maps using the first patches at the first expansion factor; and a second network for generating a second feature map using the second patch at the second expansion factor and the one or more first feature maps from the first network, wherein segmented tiles correspond to the second feature map compared to the annotation; training, including A method comprising:

9. Training the segmentation model includes: determining, by the computing system, an error metric between the segmented tile and the sample tile based on the annotations labeling the portion of the region of interest; updating, by the computing system, at least one parameter in the plurality of networks of the segmentation model using the error metric; The method of claim 8 further comprising:

10. The plurality of networks of the segmentation model include: generating a plurality of third feature maps using third patches of the at least one tile at a third expansion factor; providing the third feature maps to corresponding decoders of the second network to generate the second feature maps; The method of claim 8 , further comprising a third network for performing:

11. 9. The method of claim 8, wherein the second network comprises a plurality of decoders arranged across corresponding columns, each of the plurality of decoders for processing a corresponding feature map of the plurality of first maps from the first network.

12. 9. The method of claim 8, wherein the first network comprises a plurality of encoders arranged across corresponding columns, each of the plurality of encoders for providing a corresponding feature map of the plurality of first networks to a respective decoder in the second network.

13. 9. The method of claim 8, wherein the annotations of the training dataset label less than an entire portion of the region of interest in the sample biomedical image, the annotations being separated from the entire edge of the region of interest.

14. 9. The method of claim 8, further comprising generating, by the computing system, a segmented biomedical image using a plurality of segmented tiles by applying the segmentation model applied to a plurality of patches at a corresponding plurality of magnification factors, each patch identified from a corresponding tile of the plurality of tiles of the biomedical image.

15. 9. The method of claim 8, wherein the sample biomedical image is derived from a histopathological image generator, and the sample biomedical image is divided into a plurality of tiles, each tile of the plurality of tiles corresponding to a portion of the sample biomedical image.

16. 1. A system for segmenting a biomedical image, comprising: a computing system having one or more processors coupled to a memory, the computing system comprising: identifying, for at least one tile of the biomedical image, a first patch at a first magnification factor and a second patch at a second magnification factor; applying a trained segmentation model to the first patch and the second patch to generate segmented tiles, the trained segmentation model including a plurality of networks, the plurality of networks comprising: a first network for generating a plurality of first feature maps using the first patches at the first expansion factor; and a second network for generating a second feature map using the second patch at the second expansion factor and the one or more first feature maps from the first network; generating a storing the segmented tiles identifying a region of interest within the at least one tile of the biomedical image; A system configured to:

17. The plurality of networks of the segmentation model include: generating a plurality of third feature maps using third patches of the at least one tile at a third expansion factor; providing the third feature maps to corresponding decoders of the second network to generate the second feature maps; 17. The system of claim 16, further comprising a third network for performing:

18. 17. The system of claim 16, wherein the second network comprises a plurality of decoders arranged across corresponding columns, each of the plurality of decoders for processing a corresponding feature map of the plurality of first maps from the first network.

19. 17. The system of claim 16, wherein the first network comprises a plurality of encoders arranged across corresponding columns, each of the plurality of encoders for providing a corresponding feature map of the plurality of first networks to a respective decoder in the second network.

20. 17. The system of claim 16, wherein the computing system is further configured to generate a segmented biomedical image using a plurality of segmented tiles by applying the segmentation model applied to a plurality of patches at a corresponding plurality of magnification factors, each patch identified from a corresponding tile of the plurality of tiles of the sample biomedical image.

21. 17. The system of claim 16, wherein the computing system is further configured to acquire the biomedical image derived from a histopathological image generator, the biomedical image being divided into a plurality of tiles, each tile of the plurality of tiles corresponding to a portion of the biomedical image.

Images