Multi-class interactive segmentation graphical user interface

Abstract

Methods and systems are provided for performing operations including receiving an image and displaying the image using a graphical user interface; receiving at least one first image annotation provided by a user via the graphical user interface; generating a first segmented image using a deep learning model, where the deep learning model generates the first segmented image using the digital pathology image and the at least one first image annotation; displaying the first segmented image using the graphical user interface; receiving at least one second image annotation provided by a user via the graphical user interface; generating a second segmented image using the deep learning model, where the deep learning model generates the second segmented image using the digital pathology image, the at least one first image annotation, and the at least one second image annotation; and displaying the second segmented image using the graphical user interface.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 269,833, filed March 23, 2022, which is incorporated by reference herein in its entirety for all purposes. [Technical Field]

[0002] The present disclosure relates to digital pathology, and in particular to techniques for segmenting images. [Background technology]

[0003] Digital pathology involves scanning a slide (e.g., a histopathology or cytopathology glass slide) to generate a digital image. The slide may contain a biosample (e.g., a tissue slide or biofluid) that has been stained using one or more stains (i.e., dyes) that selectively bind to specific cellular components or tissue types. The digital image may then be processed using digital pathology imaging techniques and / or interpreted by a pathologist. This subsequent processing may be used for a variety of reasons, such as to predict or facilitate the diagnosis of disease, estimate the degree to which a given treatment has been effective for a given subject, predict the degree to which a given treatment will be effective for a given subject, and / or facilitate the development of new treatments (e.g., new active agents, dosages, compositions, treatment schedules, etc.).

[0004] The traditional approach for analyzing digital pathology images is for a trained human pathologist to inspect a highly magnified portion (i.e., "field of view") of the slide, manually segment the image to identify one or more areas of interest (e.g., to eliminate background, artifacts, macrophages, etc.), and then detect and classify signals within each portion of the image. Segmentation and classification are typically performed at 40x-400x magnification. Therefore, when standard approaches are used, generating digital pathology results is a highly labor-intensive, time-intensive, and economically expensive endeavor.

[0005] Another approach is to use machine learning models to process digital pathology images. However, this typically involves training the model using a training dataset that includes a large number of manually labeled images. Given the level of accuracy required for segmentation (which typically involves delineating region boundaries with pixel-level accuracy) and the amount of data required to train a model equipped to process digital pathology images, generating annotations for the training data can require a time investment of six months or more from a highly trained pathologist.

[0006] Deep learning models are a type of machine learning model with an architecture sophisticated enough to learn complex data signatures frequently. However, as the complexity of the model increases, the size of the required training dataset also increases. Therefore, despite the power of deep learning systems, developing such systems and promoting their widespread use in the clinical field is challenging, especially given the variability in data collection, signal variability, and privacy considerations that affect clinical data.

[0007] Digital pathology presents a particularly challenging environment for training any model, let alone a deep learning model. In digital pathology, the stains used are highly variable. Furthermore, multiple biomarker dyes (e.g., three or more dyes) are frequently used, which means that even fewer images are available showing samples stained with a particular combination of dyes (especially when privacy constraints are considered). Furthermore, models trained on one tissue type cannot be easily transferred to other tissue types. Deep learning models built for a specific dataset (i.e., a specific image domain) often perform poorly on similar or even related datasets (other, different image domains).

[0008] Thus, in digital pathology, models designed for a specific diagnostic assay cannot be easily reused for other assays due to performance issues. This is also true when applying the model to images from the same slide but scanned from a different scanner. (See Aubreville M et al. (2021). MItosis DOmain Generalization Challenge. 24th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI 2021). Zenodo. https: / / doi.org / 10.5281 / zenodo.4573978.) This lack of model generalizability in digital pathology leads to unavoidable ongoing investment in developing new models, even for related tasks.

[0009] Furthermore, platforms that use machine learning models are often built end-to-end and therefore remain a black box to end users. This can lead to poor interpretability and poor model performance (due to users not being aware of logical issues or how best to modify the model to address detected problems). Furthermore, this inflexible implementation can prevent the use of models trained in one domain from being used in other domains.

[0010] Therefore, processing digital pathology images to accurately and efficiently segment regions of interest is particularly challenging. Moreover, even when models are configured to perform this type of segmentation, existing systems are not configured to receive responsive user input that does not lead to overfitting or to cause the utility of the model to be discarded. Summary of the Invention

[0011] In some embodiments, a computer-implemented method is provided, the computer-implemented method including receiving an image and displaying the image using a graphical user interface; receiving at least one first image annotation provided by a user via the graphical user interface; generating a first segmented image using a deep learning model, wherein the deep learning model generates the first segmented image using the digital pathology image and the at least one first image annotation; displaying the first segmented image using the graphical user interface; receiving at least one second image annotation provided by the user via the graphical user interface; generating a second segmented image using the deep learning model, wherein the deep learning model generates the second segmented image using the digital pathology image, the at least one first image annotation, and the at least one second image annotation; and displaying the second segmented image using the graphical user interface.

[0012] The deep learning model may be a three-class model, and the at least one first image annotation may include at least one annotation for each class.

[0013] At least one image annotation may be provided by the user using a mouse click on the displayed image.

[0014] The mouse click may include a left mouse click associated with the first segmentation class.

[0015] The mouse click may include a right mouse click associated with a second segmentation class.

[0016] The mouse clicks may include combination mouse clicks associated with a third segmentation class.

[0017] The combination mouse click may include a left mouse double click.

[0018] The deep learning model may have been trained using a first dataset that includes training images with one or more annotations.

[0019] The training images may include images from a first image domain.

[0020] The training images may further include images from a second image domain.

[0021] The second image domain may be different from the first image domain, and the first image domain and the second image domain each include one of a natural scene image, a digital pathology image, an immunohistochemistry image, an X-ray image, and a hematoxylin and eosin image.

[0022] The one or more annotations may include one or more simulated click annotations.

[0023] The images may be whole slide pathology images.

[0024] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods disclosed herein.

[0025] In some embodiments, a computer program product is provided that is tangibly embodied in a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0026] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied on a non-transitory machine-readable storage medium, the computer program product including instructions configured to cause one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0027] The terms and expressions which have been employed are used as terms of description rather than limitation, and there is no intention in the use of such terms and expressions to exclude all equivalents of the features or portions thereof shown and described, recognizing that various modifications are possible within the scope of the claimed invention. Thus, although the claimed invention has been specifically disclosed by embodiments and optional features, it should be understood that modifications and variations of the concepts disclosed herein may be reclassified by those skilled in the art, and that such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims. [Brief explanation of the drawings]

[0028] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0029] Aspects and features of various embodiments will become more apparent from the following detailed description of the preferred embodiments taken in conjunction with the accompanying drawings.

[0030] [Figure 1]1 shows an exemplary network for generating digital pathology images and detecting and labeling regions within the digital images using a deep learning model and a labeling GUI.

[0031] [Figure 2] 1 illustrates one embodiment of a multi-class DL-based interactive segmentation workflow.

[0032] [Figure 3] 1 illustrates an example pipeline for model building according to various embodiments.

[0033] [Figure 4] 1 illustrates an example workflow of a multi-class deep learning system according to various embodiments.

[0034] [Figure 5] 1 illustrates an embodiment of applying interactive binary segmentation to multi-class segmentation of DP images.

[0035] [Figure 6] 1 illustrates one embodiment of a three-class deep learning model and training.

[0036] [Figure 7] 1 illustrates an example workflow for providing annotations according to various embodiments.

[0037] [Figure 8] 1 illustrates an interactive GUI according to various embodiments.

[0038] [Figure 9] 10A-10C illustrate visualizations of radial expansion strategies for full-slide interactive segmentation according to various embodiments.

[0039] [Figure 10]10A-10C illustrate visualizations of interactive segmentation results from four image domains according to various embodiments.

[0040] In the accompanying drawings, similar components and / or features may have the same reference label. Furthermore, various components of the same type may be distinguished by following the reference label with a dash and a second label that distinguishes between the similar components. If only a first reference label is used herein, the description is applicable to any one of the similar components having the same first reference label, regardless of the second reference label. DETAILED DESCRIPTION OF THE INVENTION

[0041] While specific embodiments are described, these embodiments are presented by way of example only and are not intended to limit the scope of protection. The apparatus, methods, and systems described herein may be embodied in various other forms. Furthermore, various omissions, substitutions, and changes in form of the exemplary methods and systems described herein may be made without departing from the scope of protection.

[0042] overview Various embodiments of the present invention relate to training a deep learning model for interactive segmentation of digital pathology images to identify distinct regions within the images. Additionally, a graphical user interface (GUI) may be utilized that may enable a user to provide images, provide interactions to facilitate segmentation, and / or provide region marker updates (subject to segmentation). Thus, the deep learning model provides near real-time segmentation results, and the GUI is configured to enable a human reviewer to identify region markers very efficiently and quickly.

[0043] Training images and click annotations The deep learning model may be trained using a multi-class training dataset having images from one or more domains and corresponding annotations. The annotations may include segmentation annotations that identify boundaries or areas of depictions of different things. For each depicted segment, the annotations may (but need not) include a class (or label, as used interchangeably herein) for the segment. The annotations may include or be based on click annotations. For example, for a given image, multiple points may each be identified, each point within a segment that depicts a particular type of object, person, or entity. For each point, the annotation may indicate which of multiple classes is assigned to the corresponding segment.

[0044] A multi-class training dataset is multi-class in that it identifies segmentations for multiple types of representations. For example, in the case of digital pathology images, classes may include stromal regions, tumor regions, and background regions. As another example, in the case of natural scene images, classes may include each vehicle, each traffic light, and the background (all other parts of the image). The background class may be defined to include (a) representations of other objects not selected from the original mask, and (b) representations of pure background where no objects are annotated in the original mask.

[0045] However, multiple classes need not have semantic meaning. For example, in a three-class instance, the classes may correspond to the following: a first type of region, a second type of region, and a background region (which may, but need not, include one or more other types of regions). What constitutes a first and second type of region may be arbitrary. By way of example, in an image of a vehicle at a traffic light, the first type of region may be a vehicle, a traffic light, a person, a crosswalk, etc. If an image depicts multiple objects of a given selected type (e.g., multiple vehicles, multiple people, etc.), all such depictions may be considered to be regions of the same selected type.

[0046] The number of click annotations identified for a given training image may, but need not be, pre-defined. For example, an implementation may be configured such that each training image is associated with one click annotation per class, three click annotations per class, six click annotations total, ten click annotations total, etc.

[0047] In some cases, click annotations are automatically identified with respect to training images. For example, some training images may be associated with ground truth masks that indicate (or can be used to determine) which label should be assigned to each pixel. Such ground truth masks may vary with different click annotation targets. For example, two of multiple image domains in an input image can be used as segmentation targets, and a corresponding ground truth mask can be generated to indicate the target image regions and ignore the remaining image regions. The same image can also be paired with click annotations that target other sets of image regions and corresponding ground truth masks in these specific image regions. For each ground truth mask specific to a set of click annotations, the image region can belong to one class or two classes. The number of unique labels in the ground truth mask may differ from the number of classes for which the deep learning model is to be trained. If there are more unique labels than classes, a subset of labels can be selected, with the number of labels in the subset equal to the number of classes minus one (if a background class can be used). The selection can be (for example) random, arbitrary, or biased towards labels associated with the most pixel assignments. If there are fewer unique labels than classes, the corresponding image can be discarded from the training dataset.

[0048] The click annotations can be used to generate a ground truth mask for each image, and the ground truth mask can be used to train a deep learning model. The click annotations can be used to generate an additional input map for each image, and the additional input map can be used to train a deep learning model. The input map (which may be the same size as the input image) may be generated by encoding the click annotations using a map such as a disk map or Euclidean distance map. For example, a disk map can be generated by starting with an image with a value of zero, changing the pixel value to 1 (or other positive value) at clicked pixels, and then changing the values ​​of pixels surrounding these clicked pixels to 1, expanding the click neighborhood into a disk-shaped image region with a value of 1. A click map can be generated by setting the value of clicked pixels (and only clicked pixels) to be different from other pixels. A square map can be generated to include a square region surrounding the clicked pixel location. In some cases, a single map indicates the click locations of multiple classes. In some cases, a separate map is generated for each class.

[0049] The mask can be used along with the input image as model training and / or inference input to inform the model where the user input is, so that the model can generate a segmentation mask according to where the user clicks. In training, the model can be trained how to respond to any object in the image that the user wants to target and provide click input for. In inference, the model can predict the target image region / object according to where the user clicks (there can be many objects in an image, and the model can use the click location to predict which object / image region the user is trying to segment).

[0050] These encodings are further detailed in K. Sofiiuk, I. Petrov, O. Barinova, A. Konushin, F-BRS: Rethinking backpropagating refinement for interactive segmentation, in: 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2020, pp. 8623-8632. doi:10.1109 / cvpr42600.2020.00865, and Sofiiuk, Konstantin, Ilia A. Petrov, and Anton Konushin. "Reconstructing Iterative Training with Mask Guidance for Interactive Segmentation," arXiv preprint arXiv:2102.06583 (2021), each of which is incorporated by reference in its entirety for all purposes.

[0051] Because the classes in the training data do not need to have semantic meaning, a deep learning model trained on the training data can be configured to provide segmentation in a manner separate from region labeling. Thus, the deep learning model can be configured to predict which parts of an image correspond to different things without predicting what type of thing a given part will depict and / or whether multiple parts depict the same type of thing. To this end, multi-class training datasets containing images and annotations from multiple domains can be used.

[0052] The domains may include, for example, a digital pathology dataset, a natural scene image dataset, an immunohistochemistry dataset, an H&E dataset, or any other suitable image-annotation pair dataset as would be understood by one of ordinary skill in the art. Training data from one or more domains other than digital pathology (e.g., potentially in addition to training data from the digital pathology domain) can be used to collect a larger training set. However, by continuing to include digital pathology images in a multi-class training dataset, a deep learning model can be easily applied to process multiple types of digital pathology images without substantial further model development. However, in some cases, the training dataset does not even include training data from the domain in which the trained deep learning model will later be used for image annotation.

[0053] Deep Learning Models and Training A deep learning model can include a neural network with more than three layers, more than four layers, or more than five layers (including an input layer and an output layer). A deep learning model can include a convolutional neural network.

[0054] A deep learning model can be trained using the training data described herein (e.g., including multi-class images and possibly including ground truth masks and / or click annotations) to learn how to segment a specific number of classes (e.g., two target classes and one background class). The training data effectively trains a network of representations to match segmentation predictions corresponding to specific pixels (or specific regions in the ground truth mask) indicated via click annotations. For example, such a model learns to segment any target region indicated by these pixel locations. Thus, the model is trained to group together image pixels with unified labels (i.e., similar / identical network representations) as "annotated" image pixels, regardless of their precise underlying semantic meaning. In other words, the model need not learn to distinguish specific semantic classes within an image, but rather can learn to identify semantically similar image regions with "annotated" pixels and group only these pixels together to generate a segmented target region. Due to such design, the interactive deep learning model can identify any target object or region for which a user provides click annotations, and does not require domain-specific training to identify the exact class of region. Thus, such embodiments provided herein impose few restrictions on whether the test and training images come from the same domain. This feature is key to its strong ability to generalize across image domains.

[0055] The trained deep learning model can then be used to process other images to generate predicted segmentation annotations.

[0056] In some cases, a user may provide click annotations along with the input image. For example, a user may select or upload an image using a GUI. The GUI may then present the image and utilize tools for adding click annotations. For each set of classes, the click annotations may identify one or more pixels in the image that correspond or do not correspond to the class. The deep learning model may use the click annotations and trained model parameters to predict which portions of the image correspond to each of the set of classes.

[0057] In some cases, the user provides an input image but no click annotations. The deep learning model may then use the learned model parameters to identify a set of segments. The model may attempt to assign labels based on similarity across various regions, or may simply provide segmentation data (e.g., identifying pixels on the boundaries of various segmentation regions).

[0058] In some cases, the output of a trained deep learning model may include one or more values ​​for each pixel. The value may include a class assignment (e.g., class 1, 2, or 3). In some cases, the value may include an error indicating that no class was assigned. The output may include a set of masks, each of which corresponds to a given predetermined class. Each mask may include a value for each pixel, with a particular value (e.g., 1) indicating that the pixel is predicted to be part of the class's region (or the region's perimeter) and another particular value (e.g., 0) indicating that the pixel is predicted not to be part of the class's region (or the region's perimeter). Alternatively, the value at each pixel may represent a predictability or confidence that the pixel is part of the class's region.

[0059] Interactive GUI for efficient sign identification After the deep learning model processes the input image, a GUI can be generated or a GUI (e.g., the same GUI used to provide click annotations) can be updated to represent the segmentation produced by the processing.

[0060] Specifically, the GUI may be configured to receive any marker modifications and / or identify, for each of one or more regions, a region marker to be assigned to the region. For example, an input-transformation definition may be created that indicates what the updated region marker will be for each of a number of specific types of mouse clicks. Illustratively, a single left click may correspond to a tumor region, a single right click may correspond to a stromal region, and a left double click may correspond to a background region. Then, when a user performs any of these types of clicks within a depiction of a given segmented region, the region's region marker may be updated or defined accordingly.

[0061] As another example, the GUI may be configured so that a pull-down menu (identifying different potential region markers) appears after the user clicks within a given region. The pull-down menu may identify the potential region markers, and the user may select which region marker to assign to the region.

[0062] In this manner, regions may be identified and classified very quickly and in a manner requiring little time to be invested by the user (e.g., compared to traditional manual segmentation and manually identifying a region marker for each region). For example, an image tile 600x600 pixels in size may be segmented in less than one minute, and the GUI may facilitate rapid identification of region markers, resulting in real-time updates of the region markers, as well as updates of the GUI that are intuitive to the user and facilitate further identification of region markers.

[0063] A sign modification may quickly or immediately trigger a sign update.

[0064] Typical Practical Applications The region markers may be used (for example) to generate one or more metrics or recommendations to assist a care provider in determining a diagnosis, prognosis, or treatment recommendation for a subject. Typical metrics may include the absolute cumulative area of ​​each of one or more types of regions (e.g., tumor and / or stromal regions) in one or more digital pathology images and / or the relative cumulative area of ​​tumor regions (e.g., relative to the sum of the areas of tumor and stromal regions) in one or more digital pathology images. In some cases, a change metric is generated that indicates the degree to which the cumulative absolute area or cumulative relative area has changed relative to a similar metric generated by processing a digital pathology image corresponding to the same subject but at a previous time point. The change metric may identify a difference or rate of change, or may consist of a loss function of the typical type used in optimization and artificial neural networks.

[0065] The region markers may also or alternatively be used to facilitate the performance of digital pathology analysis, including (for example) characterizing cells depicted in the image. For example, a background color representing the region markers may be provided in a presented digital pathology interface to facilitate segmentation and / or labeling of individual cells. As another example, the region markers may be used to evaluate one or more rules that determine whether there is a potential (or actual) inconsistency between the cell markers and the region markers. If so, a warning may be presented or sent to request re-examination of the inconsistency.

[0066] Typical Benefits Thus, the embodiments disclosed herein provide a platform that supports simultaneous segmentation of more than two types of regions and demonstrates success in DL-based interaction segmentation applied across different imaging modalities. The techniques disclosed herein have clear performance advantages over alternative approaches.

[0067] For example, active learning uses a heuristic scoring strategy to query a small subset of unlabeled examples (those most informative in the dataset), and the model is retrained by iteratively adding only this small set of selected examples. While this eliminates the need to annotate many images, models developed with active learning are only applied to test images with the same or identical distribution as the training images. Therefore, active learning cannot be used to generate generalizable models that are well-suited to the high variability of dyes, scanners, tissues, etc. present in digital pathology.

[0068] As another example, the technology disclosed herein provides a GUI configured to enable users to provide labeling inputs very efficiently. While other annotation tools have been developed based on machine learning or deep features, such approaches require models pre-trained on segmentation tasks similar to the input image, which significantly limits the generalizability of the models. Furthermore, such approaches are limited to binary segmentation, whereas most applications in digital pathology require the simultaneous identification of three or more different types of target regions.

[0069] As a more extreme example, transfer learning and domain adaptation improve model generalization to some extent, but require further model training with additional labeled examples from the image domain of interest. Unsupervised domain adaptation aims to avoid using annotations from the target domain, but still relies on a validation set from the target domain for model selection, which inevitably tends to fit the validation set. Furthermore, a prerequisite for these techniques to work well is the existence of similarities between image domains. Therefore, such techniques do not provide real-time prediction changes or cross-domain generalization without making assumptions about the image domain or further model retraining.

[0070] Furthermore, many existing segmentation models (both optimization-based and learning-based approaches) are designed to segment individual objects one at a time, thereby using binary segmentation to simultaneously identify two different types of regions: foreground objects and non-target background. Foreground objects (or "foreground" for short) are defined as target object instances in an image, while non-target background is anything other than the target object. In contrast, embodiments of the present invention provide segmentation of target objects, regardless of whether the target object is a single connected region or multiple disconnected regions of the same class.

[0071] Exemplary Network for Digital Pathology Image Generation and Processing Figure 1 illustrates an exemplary network for generating digital pathology images and labeled regions of interest. It can be understood that each of one or more of the components illustrated in Figure 1 (e.g., model training system 105, segmentation controller 120, imaging system 150, labeling GUI controller 160, etc.) may include one or more processors and computer-readable media with instructions that, when executed, perform the operations identified herein. In some cases, a single physical or virtual computing system includes one or more processors and computer-readable media with instructions that, when executed, perform the operations described as being performed by multiple components of Figure 1. For example, a single computing system may perform the operations described herein as being performed by model training system 105, segmentation controller 120, and labeling GUI controller 160.

[0072] The network includes a model training system 105 that retrieves multi-class images 110a-110n from one or more data sources. The one or more data sources may include one or more remote data sources and / or one or more local data sources. In some cases, at least some of the multi-class images 110a-110n are retrieved by sending an online request for the image and downloading the image. Each of the multi-class images 110a-110n may include annotation data that indicates boundaries or areas of depiction of different segments within the image. Each segment of the multi-class images 110a-110n may correspond to a different type of object or object. For example, in addition to identifying where vehicles are depicted in the image, the annotation data may also identify the depiction of each vehicle, each stop light, each crosswalk, each person, etc.

[0073] The multi-class images 110a-110n may, but need not, be from different domains. The multi-class images 110a-110n may include (for example) one or more natural scene images, one or more digital pathology images, one or more immunohistochemistry images, one or more H&E images, one or more MRI images, and / or images from one or more other domains. The multi-class images 110a-110n may, but need not, include one or more digital pathology images. The multi-class images 110a-110n may also include one or more images that are not digital pathology images.

[0074] Each of the multi-class images 110a-110n may be accompanied by segmentation annotation data. The segmentation annotation data may include annotations disclosed herein. For example, the segmentation annotation data may include one or more click annotations, each of which corresponds to a particular pixel in the image and a particular positive indicator (e.g., identifying a particular class) of a negative indicator. As another example, the segmentation annotation data map includes a transformed version of one or more click annotations (e.g., where each click annotation has been transformed using a map such as a click map or Euclidean map). As another example, the segmentation annotation data may include one or more ground truth masks that indicate, for each pixel, an identification or prediction as to whether the pixel corresponds to a particular indicator. The segmentation annotation data may be identified by or generated based on input from a user interacting with a GUI showing each training image. It may be understood that the user providing the segmentation annotation data input need not be the same user as the user providing subsequent images (and / or other related information) to be processed by the trained model.

[0075] The model training system 105 may use the multi-class images 110a-n (e.g., and any accompanying segmentation annotation data) to train a deep learning model 115. The deep learning model 115 may include a neural network with more than three, more than four, more than five, or more than six layers. The deep learning model 115 may have a convolutional neural network architecture. Training the deep learning model 115 may include learning values ​​for a set of learnable parameters of the model (e.g., weights assigned to connections between various nodes).

[0076] After the deep learning model is trained, segmentation controller 120 can process each of one or more digital pathology (DP) images 125a-125n using the trained deep learning model. Each DP image 125 may have been generated by image generation system 130, which may be remote from segmentation controller 120 or local with the segmentation controller. It may be understood that some operations performed by image generation system 130 (and / or by one or more components thereof) are performed automatically, while other operations are (in some cases) performed using at least some human action.

[0077] As part of the imaging system 130, a fixation / embedding system 135 fixes and / or embeds tissue samples (e.g., samples containing at least a portion of at least one tumor) using a fixative (e.g., a liquid fixative such as a formaldehyde solution) and / or an embedding substance (e.g., a histological wax such as paraffin wax and / or one or more resins such as styrene or polyethylene). Each slice may be fixed by exposing the slice to the fixative for a predetermined period of time (e.g., at least 3 hours) and then dehydrating the slice (e.g., via exposure to an ethanol solution and / or a clearing intermediate agent). The embedding substance can penetrate the slice when it is in a liquid state (e.g., when heated).

[0078] A tissue slicer 140 then slices the fixed and / or embedded tissue sample (e.g., a tumor sample) to obtain a series of sections, each having a thickness of, for example, 4-5 microns. Such sectioning may be performed by first chilling the sample and then slicing the sample in a warm water bath. The tissue may be sliced ​​using (for example) a vibratome or compressome.

[0079] Because tissue sections and the cells therein are largely transparent, slide preparation typically involves staining (e.g., automatically staining) the tissue sections to make relevant structures more visible. In some cases, staining is performed manually. In other cases, staining is performed semi-automatically or automatically using a staining system 145.

[0080] Staining can involve exposing individual sections of tissue to one or more different stains (e.g., sequentially or simultaneously) to reveal different properties of the tissue. For example, each section can be exposed to a predetermined amount of stain for a predetermined period of time.

[0081] One typical type of tissue stain is histochemical staining, which uses one or more chemical dyes (e.g., acidic dyes, basic dyes) to stain tissue structures. Histochemical stains may be used to reveal general aspects of tissue morphology and / or cellular microanatomy (e.g., distinguishing cell nuclei from cytoplasm, revealing lipid droplets, etc.). One example of a histochemical stain is hematoxylin and eosin (H&E). Other examples of histochemical stains include trichrome stains (e.g., Masson's trichrome), periodic acid-Schiff (PAS), silver stains, and iron stains. The molecular weight of histochemical stains (e.g., dyes) is generally about 500 kilodaltons (kD) or less, although some histochemical stains (e.g., Alcian blue, phosphomolybdic acid (PMA)) may have molecular weights of up to 2000 or 3000 kD. An example of a high molecular weight histochemical stain is α-amylase (approximately 55 kD) which may be used to demonstrate glycogen.

[0082] Another type of tissue staining is immunohistochemistry (IHC, also called "immunostaining"), which uses a primary antibody that specifically binds to a target antigen of interest (also called a biomarker). IHC can be direct or indirect. In direct IHC, the primary antibody is directly conjugated to a label (e.g., a chromophore or fluorophore). In indirect IHC, the primary antibody first binds to the target antigen, and then a secondary antibody conjugated to a label (e.g., a chromophore or fluorophore) binds to the primary antibody. Because antibodies have a molecular weight of approximately 150 kD or greater, the molecular weight of IHC reagents is much larger than that of histochemical staining reagents.

[0083] The sections may then be individually mounted onto corresponding slides, which can then be scanned or imaged by imaging system 150 to generate raw digital pathology images 125a-125n. Each section may be mounted onto a slide, which may then be scanned to create a digital image, which may then be reviewed by digital pathology image analysis and / or by a human pathologist (e.g., using image viewer software).

[0084] The labeling GUI controller 160 can generate, configure, or update a GUI for presenting a given digital pathology image associated with a user. The given digital pathology image may be one queued for segmentation and labeling or one identified in a request for segmentation and labeling. The GUI may be configured to accept one or more annotations. For example, the GUI may include an input component that enables a user to select sets of pixels in the digital pathology image and indicate, for each set of pixels, a label (e.g., a first type region, a second type region, or background) to be assigned (or not assigned) to the pixels.

[0085] The segmentation controller 120 can process each digital pathology image 125a-125n and any annotations using the trained deep learning model 115 to generate segmentation data 155. For each segment detected by the deep learning model 115, the segmentation data identifies the location of the segment within the corresponding digital pathology image 125. For example, for a given segmented region, the segmentation data 155 may identify pixels within the region's boundary and / or pixels within the segmented region. In some cases, the segmentation data 155 includes an annotation image for each one or more pixels indicating whether the pixel is assigned to a certain type of region or segmented region, and if so, to which region. For example, the annotation image may indicate to which of a predetermined set of classes the pixel is assigned (e.g., a first type region, a second type region, or background). Although the classes may lack semantic meaning, training may be performed such that any class assignment is likely to match any annotation provided in each digital pathology image 125a-125n. As another example, each segmented region may be assigned a unique identifier relative to other segmented regions in the image, and for each segmented region, pixels that are within the segmented region (or alternatively or additionally, that are part of the boundary of the segmented region) may be assigned a value equal to the unique identifier.

[0086] The labeling GUI controller 160 can generate, configure, or update a GUI to present a given DP image (e.g., at reduced intensity) to indicate the boundaries of each segmented portion. If a deep learning model is configured to assign labels, the labels may be represented (e.g., via shading, background color, symbol color, etc.).

[0087] The GUI may also be configured to receive input that identifies or updates the region indicator for each region. The GUI may also be configured to allow the user to zoom in and out of the image and / or move the field of view, allowing the user to better see details such as cell morphology and size within a given segmented region.

[0088] The GUI may be configured to allow a user to select from among a predetermined set of region labels for each segmented region. If label data was provided as part of the training data, the deep learning model may have generated a predicted class for each region. In that case, an indication of the predicted class may be presented for each segmented region (e.g., via shading, background color, symbol color, symbol shape, etc.). If label data was not provided as part of the training data (or if insufficient label data was provided), all regions may be represented as having an indication that a region label has not yet been assigned, but a region label may be subsequently assigned to each segmented region (and the display may be updated accordingly) based on label selection input provided by the user. In another example, an initial region label may be automatically assigned to each region, and the user may optionally provide input to change the label if the user believes the initial label is incorrect. In this case, all regions may be assigned the same initial region label (e.g., the statistically most common label), or one or more rules may be evaluated to determine which initial region label to assign to each region (e.g., based on the area of ​​the region, the perimeter of the region, the ratio of the perimeter to the area of ​​the region, the number of cells detected within the region, etc.).

[0089] In either case, the GUI may be configured to receive input identifying a label to be assigned to one or more portions of the image (e.g., as an initial display or as an updated display). For example, a user may click a color-coded marker to identify an initial label using a radio button list, and then provide input identifying a different label. As another example, a user may click within a segmentation region depiction and then use a defined mouse click signature to identify an initial (e.g., and final) label for the region.

[0090] In some cases, a distinct type of mouse click and / or touchscreen interaction may be assigned to each of a predetermined set of area indicators. For example, (1) a single left mouse click, (2) a single right mouse click, and (3) a double left mouse click may each be assigned to one of the area indicator options. (FIG. 7 and the associated discussion provide an illustration of the mapping of mouse clicks to indicators.) As another example, (1) a short hold followed by a slight downward swipe, (2) a short hold followed by a slight upward swipe, and (3) a short hold followed by a slight left swipe may each be assigned to one of the area indicator options.

[0091] The GUI may also include an option to assign (for example) each and every currently unlabeled segmentation region in the image, or all currently unlabeled segmentation regions that are entirely within the field of view, to a given region marker (which may be selected by the user). Thus, for example, within a given field of view, a user may select to indicate that after all stromal and background regions are labeled, all other regions should be labeled as tumor regions. Alternatively or additionally, the GUI may be configured to allow a user to select a tool to define a rectangular area within the depicted image and then assign all regions within the area to the selected region marker.

[0092] As each label is assigned or updated, the GUI may be updated to reflect the labeling (e.g., by adding a background color to the areas that may represent the assigned labeling) so that the user can easily see the current labeling (and, if applicable, which areas remain unlabeled). The GUI may include a submit or complete input option that can be selected by the user when the user is satisfied with the labeling. Further details of the GUI are provided herein, such as when describing FIG. 8.

[0093] Marker GUI controller 160 may store and / or output region markers 160a-160k. Region markers 160a-160k may (for example) include a mapping associating each unique region identifier with a region marker (or region marker identifier). Region markers 160a-160k may alternatively or additionally include, for each of some or all pixels in the image, a mapping associating the pixel with a region marker (or region marker identifier).

[0094] A typical interactive segmentation workflow Referring to Figure 2, one embodiment of a multi-class interactive segmentation workflow 200 is shown. Various operations in the multi-class interactive segmentation workflow 200 may be performed by the model training system 105, the segmentation controller 120, and / or the signage GUI controller 160. Figure 3 shows an example of a corresponding pipeline for model building according to various embodiments.

[0095] In various embodiments, the multi-class interactive segmentation workflow 200 designs and builds a deep learning model to perform segmentation for multiple classes and receive user input identifying a label for each of one or more pixels and / or segmented regions. An interactive GUI may be configured to receive the input image and potentially initially receive annotation input identifying the class to which a given point or segment should be assigned, and / or receive post-training input identifying or updating the label to be assigned to a given segmented region.

[0096] For example, the first dataset 210 may include multiple images (e.g., multi-class images 110a-110n) from one or more classes (e.g., DP images, natural scene images, immunohistochemistry images, H&E images, or any other related images) and may include corresponding annotations (e.g., click annotations, identifying distinct segments within the images, ground truth maps, etc.). Some or all of the first dataset 210 may have been received via an interactive GUI from a user facilitating training of a deep learning model (e.g., deep learning model 115) connected to the interactive GUI.

[0097] The plurality of images may include digital pathology images featuring different tissue types and their corresponding segmentation annotations. In other embodiments, a first dataset 210 from another domain may be used, such as natural scene images. The first dataset 210 may include annotation data (for example) associating each of one or more pixels or areas with a given class. The class may be one of a plurality of predetermined non-semantic classes.

[0098] A deep learning model may be designed and built (in part 1 of FIG. 3) by specifying the model's architecture, hyperparameters, and loss function. At 220, a first dataset 210 may be utilized (e.g., by the model training system 105) to train the deep learning model. By using datasets from multiple domains, the resulting model trained using the data may be more generalizable compared to using training data from a single domain.

[0099] A GUI can also be developed that displays the digital pathology image and segmentation data (e.g., by overlaying segmentation boundaries on the image) and is configured to receive user input identifying landmarks (e.g., a predetermined set of landmarks) in each section or to modify the predicted landmarks. (See part 2 of the pipeline in Figure 3.) This interactive GUI may be, but need not be, the GUI that received some or all of the first data set. Even if the GUI is the same, the GUI may be configured to present a first interface to a first user for receiving some or all of the first data set and a second interface to a second user for receiving landmark definitions or updates.

[0100] The GUI of Part 2 may be connected to the model (see Part 3 of the pipeline in Figure 3). This connection allows a user to identify (e.g., upload, select, etc.) one or more digital pathology images and / or checkpoint annotation data (e.g., a second dataset of images) and request processing via the GUI. The request may automatically trigger processing by the deep learning model to identify segmentations, and once segmentations are identified, the GUI may be configured to automatically display the image and segmentation data and receive label identification or updates. In some embodiments, an initial label (e.g., a non-semantic label) may be assigned to each of the one or more segmented regions. In some embodiments, at least one segmented region lacks an initial label.

[0101] Using interactive GUI 230 (e.g., controlled by labeling GUI controller 160), a user may efficiently identify and / or update labels for at least some of the segmented regions. For example, in various embodiments, a user may use a mouse to click on an area to provide annotations. The interface may be configured so that annotations (e.g., identifying new or updated labels) can be created quickly and efficiently, and the GUI is updated in real time to reflect the most current labels in the region. For example, in various embodiments, different colors may be used to annotate different classes. For example, blue dots may be used to annotate stromal regions, green dots may be used to annotate tumor regions, and red dots may be used to annotate background regions. Furthermore, multiple annotations may be used for one or more classes.

[0102] In various embodiments, the trained deep learning model 235 is not further modified after the initial training (using multi-class data). Instead, the deep learning model 235 outputs a segmentation for the image, and the interactive GUI is configured to receive input that quickly identifies landmarks in each of some of the segmented regions.

[0103] In one embodiment, a deep learning model is used to segment a 600x600 pixel image tile in less than one minute. Thus, users may quickly and efficiently annotate images or perform image segmentation. Furthermore, the model does not require retraining or fine-tuning during use, and the model does not over-fit to a specific dataset, maintaining generalizability. Furthermore, the system offers users confidence because the process and output are not a black box like traditional deep learning systems, but instead allow for user-guided modification of model predictions, providing real-time feedback and thus being interactive, intuitive, and flexible for end users.

[0104] Typical segmentation methods using limited annotations Referring to FIG. 4, an exemplary process 400 for interactive segmentation is shown. In various embodiments, a multi-class deep learning system receives an image to be segmented (step 410). In some embodiments, the image may be a complete image (e.g., a full-slide digital pathology image) or a portion of an image (e.g., a field of view of the complete image, a region of interest of the complete image, a tile of the complete image, etc.). In various embodiments, a user may provide one or more annotations to the image (step 415). Such annotations may include (for example) click annotations that may be further processed to potentially be converted into other annotations using a conversion algorithm. In various other embodiments, annotations may not be provided by the user prior to initial segmentation.

[0105] Segmentation may be performed on the image (and its annotations, if available) using a deep learning model (step 420). In various embodiments, the model may be an n-class model. Thus, the model may be configured to segment the input image into up to n types of different areas at each round of user interaction, thereby generating a single segmentation mask with up to n classes of areas. For example, in various embodiments, the deep learning model is a 3-class model configured to classify the input image into areas including tumor cells, stromal cells, and background tissue. In other embodiments, the deep learning model is a 4-class model configured to classify the input image into four different areas including immune cells, tumor cells, stromal cells, and background cells. As will be appreciated by those skilled in the art, deep learning models may be configured for any number of classes and to segment any type of tissue or cell.

[0106] The deep learning model output provides the segmented image to an interactive GUI, and the results are displayed to the user (step 430). The user may then choose to provide annotations using the interactive GUI (step 440). Once the user provides annotations, the interactive GUI provides the annotations as further input to the deep learning model (step 450), and the process may be repeated as many times as necessary to achieve a satisfactory result. [Example]

[0107] Example 1 Traditional interactive segmentation models, both optimization-based and learning-based algorithms, are designed to segment individual objects one at a time, which uses binary segmentation to simultaneously identify two different types of regions: foreground objects and non-target background. Foreground objects (or "foreground" for short) are defined as target object instances in an image, while non-target background is anything other than the target object.

[0108] The techniques disclosed herein can produce robust results that can segment target objects with reasonable success, whether they are a single connected region or multiple disconnected regions of the same class. Interactive binary segmentation was used for multi-class segmentation of DP images.

[0109] Figures 5A-5H illustrate the steps performed when binary segmentation is used to segment a digital pathology image to identify three types of regions. Figure 5A shows a typical colon cancer H&E image tile. Figure 5B shows the first round of binary segmentation to identify tumor (red masked regions) and non-tumor (remaining unmasked regions). Figure 5B shows the same image tile after the first round of binary segmentation has been performed using a deep learning model to identify tumor (red masked regions) and non-tumor (remaining unmasked regions). In this example, a user used the interactive GUI 130 to annotate the image with green dots using positive clicks on tumor regions and red dots using negative clicks on non-tumor regions. Figure 5C shows a typical updated tumor region segmentation result (red masked regions). Thus, tumor regions are identified despite being disconnected in this image.

[0110] Figure 5D shows the image after a second round of segmentation to identify stromal and non-stromal regions. At this point, the green dot indicates the first positive click to identify stromal regions. The green masked areas are stromal regions identified in the second round of segmentation after the first positive green click, and the red masked areas are tumor regions identified by the first round of tumor vs. non-tumor segmentation. In the example shown, after the first green click, the prediction of stromal regions extends into the previously identified tumor regions, likely because the second round of segmentation is unrelated to the first round.

[0111] Figure 5E shows an example of modifying the stroma segmentation results with the addition of red dots (negative clicks). Figure 5F shows a series of click annotations that modify the second round of segmentation. Figure 5G shows the final segmentation results in tumor, stroma, and background regions, with light blue corresponding to stroma regions, gray corresponding to tumor regions, and black corresponding to background regions. Figure 5H shows a zoomed-in view of the red rectangular area from Figure 5G. The top image contains the segmentation results overlaid on the original image, and the bottom image shows the segmentation mask used. In this example, the arrows point to gaps after combining the masks generated from the second round of segmentation.

[0112] Because each round of binary segmentation is independent, users must correct already segmented objects from the previous round (Figure 5D-5E). This extends the analysis time due to the necessary corrections to previously identified regions in addition to the target class in the current round of segmentation (Figure 5F). Such corrections to repair already annotated regions are redundant and time-consuming. Furthermore, users must carefully provide click annotations via clicking on the boundaries of the target class; otherwise, gaps often occur between segmented objects from different rounds of segmentation (Figure 5G-5H), which inevitably leads to confusion and errors.

[0113] Example 2 6 shows a typical representation of the use of a three-class deep learning model 600. For a three-class model, the ground truth mask contains up to three different types of regions, and the user annotations contain up to three sets of user annotations, each set corresponding to one type of region.

[0114] The ground truth mask for each segmentation task is an instance-level annotation where multiple object instances are annotated. For example, during training, up to two objects are sampled from the mask to generate a new mask, and the rest of the image is considered as non-target background. Such non-target background can include (a) other objects not selected from the original mask, and (b) pure background where no objects are annotated in the original mask.

[0115] The training data for the three-class deep learning model includes images with click annotations 610. Colored dots represent mouse clicks on target regions. The location and number of dots are simulated for model training to mimic user input at test time, where a user provides mouse clicks to guide the segmentation of the target region.

[0116] Simulated click annotations are used during training. For example, mouse click annotations are encoded into a disk map or Euclidean distance map, and these maps form a further mask that can be conveniently fed to a deep learning model along with the training images. Various strategies of click simulation may be utilized for binary interactive segmentation, including positive click sampling and negative click sampling. Positive click sampling mimics a user's mouse click behavior on a target object by randomly sampling a variable number of pixel locations within the target object region in the mask. Negative click sampling mimics a user's behavior to identify the background (i.e., any location not on the target object). In various embodiments, three strategies of negative click sampling may be used. For example, a first strategy may include using a variable number of pixel locations randomly sampled from a pure background region. A second strategy may include using a variable number of pixel locations sampled from unselected objects in the ground truth mask. A third strategy may include using a variable number of pixel locations from the pure background sampled near the boundary of the target object.

[0117] In this example, to train a three-class deep learning model, negative click sampling is used for one type of region and positive click sampling is used for the other two types of regions. It is understood that negative clicks can be better for effectively correcting incorrectly segmented regions. Therefore, maintaining such negative click sampling for one of the classes provides users with the flexibility to perform negative clicks for the class of regions that are most difficult to segment.

[0118] An image with annotations 610 is divided into images 620, and the annotations for each object / class are segmented by a deep learning model 600. For example, a three-class deep learning model includes a first class annotation input 630, a second class annotation input 640, and a background annotation input 650. The images 620 and annotations 630, 640, and 650 are used to generate a ground truth mask 660 that is used to train a deep learning model 670, which can then generate predictions 680.

[0119] Example 3 7 illustrates an example workflow for providing annotations via an interactive GUI configured to provide user input capabilities for multiple segmentation classes. Annotations are provided by a user using various user inputs 710, and the GUI is generated on a display 720. User inputs 710 may include any computer input device, such as a mouse, touchscreen, keyboard, or other input device. User inputs 710 include a mouse with at least two buttons, including a left-click button and a right-click button.

[0120] In a typical binary object segmentation GUI, a left mouse click is used as a positive click (i.e., the location is part of a class) and a right mouse click is used as a negative click (i.e., the location is not part of a class). Using a multi-class interactive GUI, the left and right click buttons may be mapped to classes. For example, in a three-class system, a left mouse click 712 may be associated with a first class, a right mouse click 714 may be associated with a second class, and a combination mouse click 716 may be associated with a third class. Each mouse click may cause the interactive GUI to overlay a dot on the image 728 corresponding to the class associated with the click.

[0121] Left mouse clicks 712 may be associated with a first type of region (e.g., tumor region), and each left mouse click triggers the display of a green dot 722 on the interactive GUI via display 720. The left mouse clicks 712 may function as a positive click. Right mouse clicks 714 may be associated with a second type of region (e.g., stromal region), and each right mouse click triggers the display of a red dot 724 by the GUI and may also function as a positive click. A left mouse double click 716 may be associated with a third type of region and may generate a yellow dot 726. In various embodiments, the left mouse double click 716 may function as a negative click. To connect the trained model to the interactive GUI, the GUI backend is configured to convert the pixel locations clicked by the user in the three classes into three disk maps and use these disk maps along with the image as deep learning model inputs to generate predictions in the corresponding three classes.

[0122] Using the interactive GUI, users can initiate segmentation by adding a first positive click on a target region of interest. This click triggers an initial network prediction. Additional clicks trigger changes to the network input, which leads to corresponding changes in the segmentation prediction. Segmentation results can be displayed as a colored overlay on the input image.

[0123] Example 4 FIG. 8 shows an example of an interactive GUI for defining click annotations. The interactive GUI includes an image display area 800 and a control panel 810. The image display area 800 is configured to show the image being segmented and provides an area for the user to add annotations. The controls enable various options for operating the system. For example, the interactive GUI may include a button 820 for quitting, a button 830 for canceling the click, and a button 840 for resetting the click. The GUI controls 810 may also include various zoom options 850. In various embodiments, the user can adjust various model parameters using the GUI controls. For example, the GUI controls 810 may include a backpropagation refinement (BRS) option 860 and sliders for changing the prediction threshold 870 and the alpha blending coefficient 880. Other control options include a slider for the dot radius 890.

[0124] Example 5 A title-by-title approach is used for whole-slide image segmentation. Tile-by-tile predictions for WSI. Using a tile-by-tile approach, the user performs interactive segmentation on one image tile at a time. In this way, the user segments one tile to their satisfaction as described above and moves on to the next. A platform of interest (web application, GUI, etc.) is selected to display the whole-slide image and support interactive segmentation. Once all tiles have been segmented, the platform can automatically assemble all predictions and display the whole-slide segmentation results.

[0125] Example 6 A radial expansion scheme was used for whole-slide image segmentation. FIG. 8 shows a visualization of a radial expansion strategy for whole-slide interactive segmentation according to various embodiments. The radial expansion approach leverages predictions in overlapping tiles to expand predictions radially, thus facilitating segmentation in large images such as WSIs ( FIG. 9 ). For example, when a user starts analysis by clicking anywhere on the WSI (e.g., blue dot in FIG. 9 ), multiple tiles covering this click are selected to maximize total tile coverage, and predictions for all these tiles can be generated (blue tiles in FIG. 9 ). In the next step, the user can click anywhere within the predicted region that needs to be corrected (an exemplary second click shown as an orange dot in FIG. 9 ). Correcting the click helps to expand predictions to tiles covering these clicks (orange tiles in FIG. 9 ). Because one user click triggers segmentation in multiple tiles, performing such predictions with the designed radial expansion scheme enables much faster propagation of segmented regions in the whole-slide image, thus accelerating whole-slide image segmentation. Another advantage of this approach is that the system can leverage multiple tile parallelism or distributed model inference to further accelerate whole slide image segmentation.

[0126] Example 7 Domain-specific and dataset-specific whole-slide image segmentation is provided. In use cases where segmentation is required for a particular dataset involving multiple whole-slide images, interactive segmentation can serve as both an initial annotation generation strategy and a fast annotation expansion strategy. For example, initially, a subset of slide image regions can be selected manually, randomly, or via an active learning-based approach. Segmentation masks of these selected large image regions in the selected WSI can then be efficiently generated using one of the aforementioned techniques (i.e., tile prediction or tile by radial expansion strategy), followed by iterative model training and fine-tuning. In this way, a model can be rapidly built for WSI segmentation of the dataset of interest.

[0127] Example 8 The deep learning model was trained on the Semantic Boundaries Dataset (SBD), a commonly used benchmark for interactive segmentation with instance-level annotations from the natural scene image domain. The SBD contains 11,355 images, of which 8,498 were used for training and 2,857 for validation. In each image, one or more foreground objects (i.e., instances) are annotated.

[0128] An HRNet with an OCR module was pre-trained on ImageNet as the backbone, an architecture recently proposed to enable fine detail identification in several vision tasks, including image segmentation and human pose estimation. To incorporate simulated user clicks into the model training, convolutional blocks were coded separately for the image and disk map from the user input. Note that such a backbone can be replaced with other models selected to address specific use cases. Furthermore, domain-specific annotated datasets, such as IHC images with corresponding masks if available, can be utilized to train such interactive systems.

[0129] Implementation Details: Three-class click sampling was performed on the fly during training. Specifically, for each image, up to two foreground instances (i.e., individual connected objects in the ground truth mask) were randomly selected from the segmentation mask. Along with the pure background annotations, a new mask with up to three distinct regions was generated. Two of the regions corresponded to the target objects and one to the non-target background. From such a new mask, positive clicks were randomly placed inside each foreground object region, and negative clicks were placed in the background region using the same strategy. Images were cropped to 320x480 patches for training and augmented with random rescaling, flipping, and color jittering.

[0130] As shown in Figure 10, we used the interactive platform to segment four different classes of images. Specifically, Figure 10 shows model performance on (A) an exemplary colon cancer H&E image tile, (B) an exemplary IHC image from breast cancer tissue targeted with Ki67 (brown DAB stain) and counterstained with hematoxylin (blue stain), (C) an exemplary IHC image from lung cancer tissue targeted with PDL1 (pink Tumbler stain) and CK7 (yellow Dabsyl stain) and counterstained with hematoxylin (blue), and (D) an exemplary chest x-ray image. In the bottom panels of A–C, the light blue regions represent stroma, the gray regions represent tumor, and the black regions represent non-target background. In the bottom panel of D, the light blue regions represent the right lung, the gray regions represent the left lung, and the black regions represent non-target background. The predictions demonstrate that the proposed model design and training strategy successfully segments images into three classes of regions interactively, as shown in Figure 10.

[0131] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product including instructions configured to cause one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0132] The terms and expressions which have been employed are used as terms of description rather than of limitation, and there is no intention in the use of such terms and expressions to exclude all equivalents of the features shown and described or portions thereof, recognizing that various modifications are possible within the scope of the invention as claimed. Thus, although the invention as claimed has been specifically disclosed by embodiments and optional features, it is to be understood that modifications and variations of the concepts disclosed herein may be employed by those skilled in the art, and that such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims.

[0133] The description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the description of preferred exemplary embodiments will provide those skilled in the art with an enabling description for implementing various embodiments. It will be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope as set forth in the appended claims.

[0134] In the following description, specific details are given to provide a comprehensive understanding of the embodiments. However, it will be understood that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

Claims

1. Receiving a digital pathology image and displaying the digital pathology image using a graphical user interface; receiving a first set of at least one first image annotation provided by a user via the graphical user interface; generating a first segmented image using a deep learning model, the deep learning model generating the first segmented image using the digital pathology image and the first set of at least one first image annotation; displaying the first segmented image using the graphical user interface; receiving a second set of at least one first image annotation and at least one second image annotation provided by the user via the graphical user interface; generating a second segmented image using the deep learning model, wherein the deep learning model generates the second segmented image using the digital pathology image, the first set of the at least one first image annotation, the second set of the at least one first image annotation, and the at least one second image annotation; displaying the second segmented image using the graphical user interface; 11. A computer-implemented method comprising:

2. 2. The method of claim 1 , wherein the deep learning model is a three-class model, and the first set of at least one first image annotation and the second set of at least one first image annotation include at least one annotation for each class.

3. The method of claim 1 or 2, wherein the at least one image annotation provided by the user is provided using a mouse click on the displayed digital pathology image.

4. The method of claim 3 , wherein the mouse click comprises a left mouse click associated with a first segmentation class.

5. The method of claim 4 , wherein the mouse click comprises a right mouse click associated with a second segmentation class.

6. The method of claim 4 , wherein the mouse clicks include combination mouse clicks associated with a third segmentation class.

7. The method of claim 6 , wherein the combination mouse clicks include a left mouse double click.

8. The method of claim 1 , wherein the deep learning model is trained using a first dataset including training images with one or more annotations.

9. The method of claim 8 , wherein the training images include images from a first image domain.

10. The method of claim 9 , wherein the training images further include images from a second image domain.

11. 11. The method of claim 10, wherein the second image domain is different from the first image domain, and the first image domain and the second image domain each comprise one of a natural scene image, a digital pathology image, an immunohistochemistry image, an X-ray image, and a hematoxylin and eosin image.

12. The method of claim 8 , wherein the one or more annotations include one or more simulated click annotations.

13. 10. The method of claim 1, wherein the digital pathology image is a whole slide pathology image.

14. one or more data processors; 1. A non-transitory computer-readable storage medium comprising instructions that, when executed on the one or more data processors, cause the one or more data processors to perform a set of operations, the set of operations comprising: receiving a digital pathology image and displaying the digital pathology image using a graphical user interface; receiving a first set of at least one first image annotation provided by a user via the graphical user interface; generating a first segmented image using a deep learning model, the deep learning model generating the first segmented image using the digital pathology image and the first set of at least one first image annotation; displaying the first segmented image using the graphical user interface; receiving a second set of at least one first image annotation and at least one second image annotation provided by the user via the graphical user interface; generating a second segmented image using the deep learning model, wherein the deep learning model generates the second segmented image using the digital pathology image, the first set of the at least one first image annotation, the second set of the at least one first image annotation, and the at least one second image annotation; displaying the second segmented image using the graphical user interface; a non-transitory computer-readable storage medium, A system comprising:

15. 1. A computer program comprising instructions configured to cause one or more data processors to perform a set of operations, said set of operations comprising: receiving a digital pathology image and displaying the digital pathology image using a graphical user interface; receiving a first set of at least one first image annotation provided by a user via the graphical user interface; generating a first segmented image using a deep learning model, the deep learning model generating the first segmented image using the digital pathology image and the first set of at least one first image annotation; displaying the first segmented image using the graphical user interface; receiving a second set of at least one first image annotation and at least one second image annotation provided by the user via the graphical user interface; generating a second segmented image using the deep learning model, wherein the deep learning model generates the second segmented image using the digital pathology image, the first set of the at least one first image annotation, the second set of the at least one first image annotation, and the at least one second image annotation; displaying the second segmented image using the graphical user interface; a computer program comprising:

Images