Multi-Class Interactive Segmentation Graphical User Interface

Abstract

Methods and systems are provided for performing operations including receiving an image and displaying the image using a graphical user interface; receiving at least one first image annotation provided by a user via the graphical user interface; generating a first segmented image using a deep learning model, where the deep learning model generates the first segmented image using the digital pathology image and the at least one first image annotation; displaying the first segmented image using the graphical user interface; receiving at least one second image annotation provided by a user via the graphical user interface; generating a second segmented image using the deep learning model, where the deep learning model generates the second segmented image using the digital pathology image, the at least one first image annotation, and the at least one second image annotation; and displaying the second segmented image using the graphical user interface.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 269,833, filed March 23, 2022, which is incorporated by reference in its entirety for all purposes. [Technical field]

[0002] The present disclosure relates to digital pathology, and in particular to techniques for segmenting images. [Background technology]

[0003] Digital pathology involves the scanning of a slide (e.g., a histopathology or cytopathology glass slide) to generate a digital image. The slide may contain a biosample (e.g., a tissue slide or a biofluid) that has been stained using one or more stains (i.e., dyes) that selectively bind to specific cellular components or tissue types. The digital image may then be processed by using digital pathology image processing techniques and / or interpreted by a pathologist. This subsequent processing may be used for a variety of reasons, such as predicting or facilitating the diagnosis of a disease, estimating the degree to which a given treatment has been effective for a given subject, predicting the degree to which a given treatment will be effective for a given subject, and / or facilitating the development of new treatments (e.g., new active agents, dosages, compositions, treatment schedules, etc.).

[0004] The traditional approach for analyzing digital pathology images is for a trained human pathologist to inspect a highly magnified portion (i.e., a "field of view") of the slide, manually segment the image to identify one or more portions of interest (e.g., to exclude background, artifacts, macrophages, etc.), and then detect and classify signals within each portion of the image. Segmentation and classification is typically performed at 40x-400x magnification. Thus, when standard approaches are used, generating digital pathology results is a highly labor-intensive, time-intensive, and economically expensive endeavor.

[0005] Another approach is to use machine learning models to process digital pathology images. However, this typically involves training the model using a training data set that includes a large number of manually labeled images. Given the level of accuracy required for segmentation (which typically involves delineating the boundaries of regions with pixel-level accuracy) and the amount of data required to train a model equipped to process digital pathology images, generating annotations for the training data may require a time investment of six months or more from a highly trained pathologist.

[0006] Deep learning models are a type of machine learning model with an architecture so advanced that they can frequently learn complex data signatures. However, as the complexity of the model increases, the size of the required training dataset also increases. Thus, despite the power of deep learning systems, developing such systems to facilitate their widespread application in the clinical field is challenging, especially given the variability of data collection, signal variability, and privacy considerations that affect clinical data.

[0007] Digital pathology is a particularly challenging context for training any model, let alone a deep learning model. In digital pathology, the stains used are highly diverse. Furthermore, multiple biomarker dyes (e.g., three or more dyes) are frequently used, which means that there are even fewer images available that show samples stained with a particular combination of dyes (especially when privacy constraints are taken into account). Furthermore, models trained on one tissue type cannot be easily transferred to other tissue types. Deep learning models built for a particular dataset (i.e., a particular image domain) often do not perform well on similar or even related datasets (other, different image domains).

[0008] Thus, in digital pathology, models designed for a specific diagnostic assay are not easily reused for other assays due to performance issues. This is also true when applying models to images from the same slide but scanned from a different scanner. (See Aubreville M et al. (2021). MItosis DOmain Generalization Challenge. 24th International Conference on Medical Image Computing and Computer-Aided Intervention (MICCAI 2021). Zenodo. https: / / doi.org / 10.5281 / zenodo.4573978.) This lack of model generalizability in digital pathology results in unavoidable ongoing investments in developing new models even in related tasks.

[0009] Furthermore, platforms that use machine learning models are often built end-to-end and therefore remain a black box for end users. This can lead to poor interpretability and also poor model performance (due to users not being aware of logical issues or how best to modify the model to address detected issues). Furthermore, this inflexible implementation can prevent the use of models trained in one domain from being used in other domains.

[0010] Thus, processing digital pathology images to accurately and efficiently segment regions of interest is particularly challenging. Moreover, even when models are configured to perform this type of segmentation, existing systems are not configured to receive responsive user input that does not lead to overfitting or to result in the utility of the model being discarded. Summary of the Invention

[0011] In some embodiments, a computer-implemented method is provided that includes receiving an image, displaying the image using a graphical user interface, receiving at least one first image annotation provided by a user via the graphical user interface, generating a first segmented image using a deep learning model, where the deep learning model generates the first segmented image using the digital pathology image and the at least one first image annotation, displaying the first segmented image using the graphical user interface, receiving at least one second image annotation provided by the user via the graphical user interface, generating a second segmented image using the deep learning model, where the deep learning model generates the second segmented image using the digital pathology image, the at least one first image annotation, and the at least one second image annotation, and displaying the second segmented image using the graphical user interface.

[0012] The deep learning model may be a three-class model, and the at least one first image annotation may include at least one annotation for each class.

[0013] At least one image annotation may be provided by a user using a mouse click on the displayed image.

[0014] The mouse click may include a left mouse click associated with the first segmentation class.

[0015] The mouse click may include a right mouse click associated with a second segmentation class.

[0016] The mouse clicks may include combination mouse clicks associated with a third segmentation class.

[0017] The combination mouse click may include a left mouse double click.

[0018] The deep learning model may have been trained using a first dataset that includes training images with one or more annotations.

[0019] The training images may include images from a first image domain.

[0020] The training images may further include images from a second image domain.

[0021] The second image domain may be different from the first image domain, and the first image domain and the second image domain each include one of a natural scene image, a digital pathology image, an immunohistochemistry image, an X-ray image, and a hematoxylin and eosin image.

[0022] The one or more annotations may include one or more simulated click annotations.

[0023] The images may be whole slide pathology images.

[0024] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0025] In some embodiments, a computer program product is provided that is tangibly embodied in a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0026] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product including instructions configured to cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0027] The terms and expressions used are used as terms of description and not of limitation, and there is no intention in the use of such terms and expressions to exclude any equivalents of the features or portions thereof shown and described, recognizing that various modifications are possible within the scope of the invention claimed. Thus, although the invention described in the claims has been specifically disclosed by embodiments and optional features, it should be understood that modifications and variations of the concepts disclosed herein may be reclassified by those skilled in the art, and such modifications and variations are considered to be within the scope of the invention as defined by the appended claims. [Brief description of the drawings]

[0028] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0029] BRIEF DESCRIPTION OF THE DRAWINGS Aspects and features of the various embodiments will become more apparent from the following detailed description of the invention taken in conjunction with the accompanying drawings.

[0030] [Figure 1]1 shows an exemplary network for generating digital pathology images and detecting and labeling regions within the digital images using a deep learning model and a labeling GUI.

[0031] [Diagram 2] 1 illustrates one embodiment of a multi-class DL-based interactive segmentation workflow.

[0032] [Diagram 3] 1 illustrates an example pipeline for model building according to various embodiments.

[0033] [Figure 4] 1 illustrates an example workflow of a multi-class deep learning system according to various embodiments.

[0034] [Diagram 5] 1 illustrates an embodiment of applying interactive binary segmentation to multi-class segmentation of DP images.

[0035] [Figure 6] 1 illustrates one embodiment of a three-class deep learning model and training.

[0036] [Figure 7] 1 illustrates an example workflow for providing annotations according to various embodiments.

[0037] [Figure 8] 1 illustrates an interactive GUI according to various embodiments.

[0038] [Figure 9] 1 illustrates a visualization of a radial expansion strategy for full-slide interactive segmentation according to various embodiments.

[0039] [Figure 10]1 illustrates visualization of interactive segmentation results from four image domains according to various embodiments.

[0040] In the accompanying drawings, similar components and / or features may have the same reference label. Furthermore, various components of the same type may be distinguished by following the reference label with a dash and a second label that distinguishes between the similar components. If only a first reference label is used herein, the description is applicable to any one of the similar components having the same first reference label, regardless of the second reference label. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0041] Although specific embodiments are described, these embodiments are presented by way of example only and are not intended to limit the scope of protection. The devices, methods, and systems described herein may be embodied in various other forms. Furthermore, various omissions, substitutions, and changes in form of the exemplary methods and systems described herein may be made without departing from the scope of protection.

[0042] overview Various embodiments of the present invention relate to training a deep learning model in interactive segmentation of digital pathology images to identify different regions within the image. Additionally, a graphical user interface (GUI) may be utilized that may enable a user to provide an image, provide interaction to facilitate segmentation, and / or provide region marker updates (following segmentation). Thus, the deep learning model provides near real-time segmentation results, and the GUI is configured to enable a human reviewer to identify region markers very efficiently and quickly.

[0043] Training images and click annotations The deep learning model may be trained using a multi-class training dataset having images and corresponding annotations from one or more domains. The annotations may include segmentation annotations that identify boundaries or areas of depictions of different things. For each depicted segment, the annotations may (but need not) include a class (or label, as used interchangeably herein) for the segment. The annotations may include or be based on click annotations. For example, for a given image, each of a number of points may be identified, each point being within a segment that depicts a particular type of object, person, or entity. For each point, the annotations may indicate which of a number of classes is assigned to the corresponding segment.

[0044] A multi-class training dataset is multi-class in that it identifies segmentations for multiple types of representations. For example, in the case of digital pathology images, the classes may include stromal regions, tumor regions, and background regions. As another example, in the case of natural scene images, the classes may include each vehicle, each traffic light, and the background (all other parts of the image). The background class may be defined to include (a) representations of other objects not selected from the original mask, and (b) representations of pure background where no objects are annotated in the original mask.

[0045] However, the classes need not have semantic meaning. For example, in a three-class instance, the classes may correspond to the following: a first type of region, a second type of region, and a background region (which may, but need not, include one or more other types of regions). What constitutes a first and second type of region may be arbitrary. By way of illustration, for an image of a vehicle at a traffic light, the first type of region may be a vehicle, a traffic light, a person, a crosswalk, etc. If an image depicts multiple objects of a given selected type (e.g., multiple vehicles, multiple people, etc.), all such depictions may be considered to be regions of the same selected type.

[0046] The number of click annotations identified for a given training image may, but need not be, predefined. For example, an implementation may be configured such that each training image is associated with one click annotation per class, three click annotations per class, six click annotations total, ten click annotations total, etc.

[0047] In some cases, click annotations are automatically identified with respect to training images. For example, some training images may be associated with ground truth masks that indicate (or may be used to determine) which label should be assigned to each pixel. Such ground truth masks may vary with different click annotation targets. For example, two of multiple image domains in an input image may be used as segmentation targets, and corresponding ground truth masks may be generated to indicate the target image regions and ignore the remaining image regions. The same image may also be paired with click annotations that target other sets of image regions and corresponding ground truth masks in these specific image regions. For each ground truth mask specific to a set of click annotations, an image region may belong to one class or two classes. The number of unique labels in the ground truth mask may differ from the number of classes for which the deep learning model is to be trained. If there are more unique labels than classes, a subset of labels may be selected, with the number of labels in the subset equal to the number of classes minus one (if a background class can be used). The selection can be (for example) random, arbitrary, or biased towards labels associated with the most pixel allocations. If there are fewer unique labels than classes, the corresponding image can be discarded from the training dataset.

[0048] The click annotations can be used to generate a ground truth mask for each image, and the ground truth mask can be used to train the deep learning model. The click annotations can be used to generate a further input map for each image, and the further input map can be used to train the deep learning model. The input map (which may be the same size as the input image) may be generated by encoding the click annotations using a map such as a disk map or Euclidean distance map. For example, a disk map can be generated by starting with an image with a value of zero, changing the pixel value to 1 (or other positive value) at the clicked pixels, and then changing the values ​​of the pixels surrounding these clicked pixels to 1, expanding the click neighborhood into a disk-shaped image region of value 1. A click map can be generated by setting the value of the clicked pixel (and only the clicked pixel) to be different from other pixels. A square map can be generated to include a square region surrounding the clicked pixel location. In some cases, a single map indicates the location of clicks for multiple classes. In some cases, a separate map is generated for each class.

[0049] The mask can be used along with the input image as model training and / or inference input to inform the model where the user input is, so that the model can generate a segmentation mask according to where the user clicks. In training, the model can be trained how to respond to any object in the image that the user wants to target and provide click input for. In inference, the model can predict the target image region / object according to where the user clicks (there can be many objects in an image and the model can use the click location to predict which object / image region the user is trying to segment).

[0050] These encodings are further detailed in K. Sofiiuk, I. Petrov, O. Barinova, A. Konushin, F-BRS: Rethinking backpropagating refinement for interactive segmentation, in: 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2020, pp. 8623-8632. doi:10.1109 / cvpr42600.2020.00865, and Sofiiuk, Konstantin, Ilia A. Petrov, and Anton Konushin. "Rethinking backpropagating refinement for interactive segmentation with mask guidance," arXiv preprint arXiv:2102.06583 (2021), each of which is incorporated by reference in its entirety for all purposes.

[0051] Deep learning models trained on the training data can be configured to provide segmentation in a manner other than region labeling, since the classes in the training data do not need to have semantic meaning. Thus, deep learning models can be configured to predict which parts of an image correspond to different things, without predicting what type of thing a given part depicts and / or without predicting whether multiple parts depict the same type of thing. For this reason, multi-class training datasets that include images and annotations from multiple domains can be used.

[0052] The domain may include, for example, a digital pathology dataset, a natural scene image dataset, an immunohistochemistry dataset, an H&E dataset, or any other suitable image-annotation pair dataset as understood by those skilled in the art. Training data from one or more domains other than digital pathology (e.g., potentially in addition to training data from the digital pathology domain) may be used to collect a larger training set. However, by continuing to include digital pathology images in the multi-class training dataset, the deep learning model may be easily applied to process multiple types of digital pathology images without substantial further model development. However, in some cases, the training dataset does not even include training data from the domain in which the trained deep learning model is later used for image annotation.

[0053] Deep Learning Models and Training The deep learning model may include a neural network with more than three layers, more than four layers, or more than five layers (including an input layer and an output layer). The deep learning model may include a convolutional neural network.

[0054] A deep learning model may be trained using training data described herein (e.g., including multi-class images and which may include ground truth masks and / or click annotations) to learn how to segment a particular number of classes (e.g., two target classes and one background class). The training data performs network learning of effective representations to match segmentation predictions corresponding to particular pixels (or particular regions in the ground truth mask) indicated via click annotations. For example, such a model learns to segment any target region pointed to by these pixel locations. Thus, the model is trained to group together image pixels of unified indications (i.e., similar / identical network representations) as "annotated" image pixels, regardless of what exact underlying semantic meaning they may have. In other words, the model need not learn to distinguish particular semantic classes in an image, but rather can learn to identify semantically similar image regions with "annotated" pixels and group only these pixels together to generate segmented target regions. Due to such design, the interactive deep learning model can identify any target object or region for which a user provides click annotation, and does not require domain-specific training to identify the exact class of region. Thus, such embodiments provided herein impose few restrictions on whether the test images and training images come from the same domain. This feature is key to its strong ability to generalize across image domains.

[0055] The trained deep learning model can then be used to process other images to generate predicted segmentation annotations.

[0056] In some cases, a user may provide click annotations along with the input image. For example, a user may select or upload an image using a GUI. The GUI may then utilize tools to present the image and add click annotations. For each set of classes, the click annotations may identify one or more pixels in the image that correspond or do not correspond to the class. The deep learning model may use the click annotations and the learned model parameters to predict which portions of the image correspond to each of the set of classes.

[0057] In some cases, the user provides an input image but no click annotations. The deep learning model may then use the learned model parameters to identify a set of segments. The model may attempt to assign labels based on similarity across various regions, or may simply provide segmentation data (e.g., identifying pixels on the boundaries of various segmentation regions).

[0058] In some cases, the output of one trained deep learning model may include one or more values ​​for each pixel. The values ​​may include a class assignment (e.g., class 1, 2, or 3). In some cases, the values ​​may include an error indicating that a class was not assigned. The output may include a set of masks, each of which corresponds to a given predefined class. Each mask may include a value for each pixel, with a particular value (e.g., 1) indicating that the pixel is predicted to be part of the region of the class (or the perimeter of the region) and another particular value (e.g., 0) indicating that the pixel is predicted not to be part of the region of the class (or the perimeter of the region). Alternatively, the value at each pixel may represent a predictability or confidence that the pixel is part of the region of the class.

[0059] Interactive GUI for efficient sign identification After the deep learning model processes the input image, a GUI can be generated or a GUI (e.g., the same GUI used to provide click annotations) can be updated to represent the segmentation produced by the processing.

[0060] In particular, the GUI may be configured to receive any marker modifications and / or identify, for each of one or more regions, a region marker to be assigned to the region. For example, an input-transformation definition may be created that indicates what the updated region marker will be for each of a number of particular types of mouse clicks. To illustrate, a single left click may correspond to a tumor region, a single right click may correspond to a stromal region, and a left double click may correspond to a background region. Then, when a user performs any of these types of clicks within the depiction of a given segmented region, the region's region marker may be updated or defined accordingly.

[0061] As another example, the GUI may be configured such that a pull-down menu (identifying different potential region markers) appears after the user clicks within a given region. The pull-down menu may identify the potential region markers, and the user may select which region marker to assign to the region.

[0062] In this manner, regions may be identified and classified very quickly in a manner that requires little time to be invested by the user (e.g., compared to traditional manual segmentation and manually identifying a region marker for each region). For example, an image tile 600×600 pixels in size may be segmented in less than one minute, and the GUI may facilitate rapid identification of region markers, resulting in real-time updates of the region markers, as well as updates to the GUI that are intuitive to the user and facilitate further identification of region markers.

[0063] A sign modification may trigger a sign update quickly or immediately.

[0064] Typical Practical Applications The region markers may be used to generate (for example) one or more metrics or recommendations to assist a care provider in determining a diagnosis, prognosis, or treatment recommendation for a subject. Exemplary metrics may include the absolute cumulative area of ​​each of one or more types of regions (e.g., tumor regions and / or stromal regions) in one or more digital pathology images and / or the relative cumulative area (e.g., relative to the sum of the areas of tumor regions and stromal regions) of tumor regions in one or more digital pathology images. In some cases, a change metric is generated that indicates the degree to which the cumulative absolute area or cumulative relative area has changed relative to a comparable metric generated by processing a corresponding digital pathology image of the same subject but at a previous time point. The change metric may identify a difference or rate of change, or may consist of a loss function of the typical kind used in optimization and artificial neural networks.

[0065] The region markers may also or alternatively be used to facilitate the performance of digital pathology analysis, including (for example) characterizing cells depicted in the image. For example, a background color representative of the region markers may be provided in the presented digital pathology interface to facilitate segmentation and / or labeling of individual cells. As another example, the region markers may be used to evaluate one or more rules to determine whether there is a potential inconsistency (or actual inconsistency) between the cell markers and the region markers. If so, a warning may be presented or sent to request re-examination of the inconsistency.

[0066] Typical Benefits Thus, the embodiments disclosed herein provide a platform that supports simultaneous segmentation of more than two types of regions and demonstrates success in DL-based dialogue segmentation applied across different imaging modalities. The techniques disclosed herein have clear performance advantages over alternative approaches.

[0067] For example, active learning uses a heuristic scoring strategy to query a small subset of unlabeled examples (those that are most informative in the dataset), and the model is retrained by iteratively adding only a small set of such selected examples. This eliminates the need to annotate many images, but models developed with active learning are only applied to test images with the same or identical distribution as the training images. Therefore, active learning cannot be used to generate generalized models that are suitable for the high variability of dyes, scanners, tissues, etc. present in digital pathology.

[0068] As another example, the technology disclosed herein provides a GUI that is configured to allow users to provide labeling inputs very efficiently. Other annotation tools have been developed based on machine learning or deep features, but such approaches require models to be pre-trained on segmentation tasks similar to the input image, which severely limits the generalizability of the models. Moreover, such approaches are limited to binary segmentation, whereas most applications in digital pathology require the simultaneous identification of three or more different types of target regions.

[0069] As a more extreme example, transfer learning and domain adaptation improve model generalization to some extent, but require further model training with additional labeled examples from the image domain of interest. Unsupervised domain adaptation aims to not use annotations from the target domain, but still relies on a validation set from the target domain for model selection, which inevitably tends to fit the validation set. Furthermore, a prerequisite for these techniques to work well is the existence of similarities between image domains. Thus, such techniques do not provide real-time prediction changes or cross-domain generalization without assumptions about the image domain or without further model retraining.

[0070] Furthermore, many existing segmentation models (both optimization-based and learning-based approaches) are designed to segment individual objects one at a time, thereby using binary segmentation to simultaneously identify two different types of regions: foreground objects and non-target background. Foreground objects (or "foreground" for short) are defined as target object instances in an image, while non-target background is anything other than the target object. On the other hand, embodiments of the present invention provide segmentation of target objects, regardless of whether the target object is a single connected region or multiple disconnected regions of the same class.

[0071] Exemplary Network for Digital Pathology Image Generation and Processing FIG. 1 illustrates an exemplary network for generating digital pathology images and labeled regions of interest. It can be understood that each of one or more of the components illustrated in FIG. 1 (e.g., model training system 105, segmentation controller 120, imaging system 150, labeling GUI controller 160, etc.) may include one or more processors and computer-readable media with instructions that, when executed, perform the operations identified herein. In some cases, a single physical or virtual computing system includes one or more processors and computer-readable media with instructions that, when executed, perform the operations described as being performed by multiple components of FIG. 1. For example, a single computing system may perform the operations described herein as being performed by model training system 105, segmentation controller 120, and labeling GUI controller 160.

[0072] The network includes a model training system 105 that retrieves the multi-class images 110a-110n from one or more data sources. The one or more data sources may include one or more remote data sources and / or one or more local data sources. In some cases, at least some of the multi-class images 110a-110n are retrieved by sending an online request for the image and downloading the image. Each of the multi-class images 110a-110n may include annotation data indicating boundaries or areas of depiction of different segments within the image. Each segment of the multi-class images 110a-110n may correspond to a different type of object or thing. For example, in addition to identifying where vehicles are depicted in the image, the annotation data may identify the depiction of each vehicle, each stop light, each crosswalk, each person, etc.

[0073] The multi-class images 110a-110n may, but need not, be from different domains. The multi-class images 110a-110n may include (for example) one or more natural scene images, one or more digital pathology images, one or more immunohistochemistry images, one or more H&E images, one or more MRI images, and / or images from one or more other domains. The multi-class images 110a-110n may, but need not, include one or more digital pathology images. The multi-class images 110a-110n may include one or more images that are not digital pathology images.

[0074] Each of the multi-class images 110a-110n may be accompanied by segmentation annotation data. The segmentation annotation data may include annotations disclosed herein. For example, the segmentation annotation data may include one or more click annotations, each of which corresponds to a particular pixel in the image and a particular positive label of a negative label (e.g., identifying a particular class). As another example, the segmentation annotation data map includes a transformed version of the one or more click annotations (e.g., where each click annotation has been transformed using a map such as a click map or Euclidean map). As another example, the segmentation annotation data may include one or more ground truth masks indicating, for each pixel, an identification or prediction as to whether the pixel corresponds to a particular label. The segmentation annotation data may be identified by or generated based on input from a user interacting with a GUI showing each training image. It may be understood that the user providing the segmentation annotation data input need not be the same user as the user providing subsequent images (and / or other related information) to be processed by the trained model.

[0075] The model training system 105 may use the multi-class images 110a-n (e.g., and any accompanying segmentation annotation data) to train a deep learning model 115. The deep learning model 115 may include a neural network with more than three, more than four, more than five, or more than six layers. The deep learning model 115 may have a convolutional neural network architecture. Training the deep learning model 115 may include learning values ​​for a set of learnable parameters of the model (e.g., weights assigned to connections between various nodes).

[0076] After the deep learning model is trained, the segmentation controller 120 can process each of one or more digital pathology (DP) images 125a-125n using the trained deep learning model. Each DP image 125 may have been generated by an image generation system 130, which may be remote from the segmentation controller 120 or may be local with the segmentation controller. It may be appreciated that some operations performed by the image generation system 130 (and / or by one or more components thereof) are performed automatically, while other operations are (in some cases) performed using at least some human action.

[0077] As part of the imaging system 130, a fixation / embedding system 135 fixes and / or embeds tissue samples (e.g., samples containing at least a portion of at least one tumor) using a fixative (e.g., a liquid fixative such as a formaldehyde solution) and / or an embedding substance (e.g., a histological wax such as paraffin wax and / or one or more resins such as styrene or polyethylene). Each slice may be fixed by exposing the slice to the fixative for a predetermined period of time (e.g., at least 3 hours) and then dehydrating the slice (e.g., via exposure to an ethanol solution and / or a clearing intermediate agent). The embedding substance can penetrate the slice when it is in a liquid state (e.g., when heated).

[0078] A tissue slicer 140 then slices the fixed and / or embedded tissue sample (e.g., a tumor sample) to obtain a series of sections, each having a thickness of, for example, 4-5 microns. Such sectioning may be performed by first chilling the sample and then slicing the sample in a warm water bath. The tissue may be sliced ​​using (for example) using a vibratome or compressome.

[0079] Because tissue sections and the cells therein are largely transparent, slide preparation typically involves staining (e.g., automatically staining) the tissue sections to make the relevant structures more visible. In some cases, the staining is performed manually. In some cases, the staining is performed semi-automatically or automatically using a staining system 145.

[0080] Staining can include exposing individual sections of tissue to one or more different stains (e.g., sequentially or simultaneously) to reveal different properties of the tissue. For example, each section can be exposed to a predetermined amount of stain for a predetermined period of time.

[0081] One typical type of tissue stain is histochemical staining, which uses one or more chemical dyes (e.g., acid dyes, basic dyes) to stain tissue structures. Histochemical stains may be used to show general aspects of tissue morphology and / or cell microanatomy (e.g., distinguishing cell nuclei from cytoplasm, showing lipid droplets, etc.). One example of a histochemical stain is hematoxylin and eosin (H&E). Other examples of histochemical stains include trichrome stains (e.g., Masson's trichrome), periodic acid Schiff (PAS), silver stains, and iron stains. The molecular weight of histochemical stains (e.g., dyes) is generally about 500 kilodaltons (kD) or less, although some histochemical stains (e.g., Alcian blue, phosphomolybdic acid (PMA)) may have a molecular weight of up to 2000 or 3000 kD. An example of a high molecular weight histochemical stain is α-amylase (approximately 55 kD) which may be used to demonstrate glycogen.

[0082] Another type of tissue staining is immunohistochemistry (IHC, also called "immunostaining"), which uses a primary antibody that specifically binds to a target antigen (also called a biomarker) of interest. IHC may be direct or indirect. In direct IHC, the primary antibody is directly conjugated to a label (e.g., a chromophore or fluorophore). In indirect IHC, a primary antibody is first bound to the target antigen, and then a secondary antibody conjugated to a label (e.g., a chromophore or fluorophore) is bound to the primary antibody. The molecular weight of IHC reagents is much larger than that of histochemical staining reagents, as antibodies have a molecular weight of about 150 kD or more.

[0083] The sections may then be individually mounted onto corresponding slides, which can then be scanned or imaged by imaging system 150 to generate raw digital pathology images 125a-125n. Each section may be mounted onto a slide, which may then be scanned to create a digital image, which may then be reviewed by digital pathology image analysis and / or by a human pathologist (e.g., using image viewer software).

[0084] The labeling GUI controller 160 can generate, configure, or update a GUI for presenting a given digital pathology image associated with a user. The given digital pathology image may be one queued for segmentation and labeling or one identified in a request for segmentation and labeling. The GUI may be configured to receive one or more annotations. For example, the GUI may include an input component that allows a user to select sets of pixels in the digital pathology image and indicate for each set of pixels a label (e.g., a first type of region, a second type of region, or background) that should (or should not) be assigned to the pixels.

[0085] The segmentation controller 120 may process each digital pathology image 125a-125n and any annotations using the trained deep learning model 115 to generate segmentation data 155. For each segment detected by the deep learning model 115, the segmentation data may identify where the segment is located within the corresponding digital pathology image 125. For example, for a given segmented region, the segmentation data 155 may identify pixels that are within the boundary of the region and / or pixels that are within the segmented region. In some cases, the segmentation data 155 includes an annotation image that indicates, for each one or more pixels, whether the pixel is assigned to a type of region or segmented region, and if so, to which region. For example, the annotation image may indicate which of a predefined set of classes the pixel has been assigned to (e.g., a first type of region, a second type of region, or background). Although the classes may lack semantic meaning, training may be performed such that any class assignment is likely to match any annotation provided in the respective digital pathology image 125a-125n. As another example, each segmented region may be assigned a unique identifier with respect to other segmented regions in the image, and for each segmented region, pixels that are within the segmented region (or alternatively or additionally that are part of the boundary of the segmented region) may be assigned a value equal to the unique identifier.

[0086] The labeling GUI controller 160 can generate, configure, or update a GUI to present a given DP image (e.g., at reduced intensity) to indicate the boundaries of each segmented portion. If a deep learning model is configured to assign labels, the labels may be represented (e.g., via shading, background color, symbol color, etc.).

[0087] The GUI may also be configured to receive input that identifies or updates the region marker for each region. The GUI may also be configured to allow the user to zoom in and out of the image and / or move the field of view, allowing the user to better see details such as cellular morphology and size within a given segmented region.

[0088] The GUI may be configured to allow a user to select from among a predefined set of region labels for each segmented region. If label data was provided as part of the training data, the deep learning model may have generated a predicted class for each region. In that case, an indication of the predicted class may be presented for each segmented region (e.g., via shading, background color, symbol color, symbol shape, etc.). If label data was not provided as part of the training data (or if insufficient label data was provided), all regions may be represented as having an indication that a region label has not yet been assigned, but a region label may be subsequently assigned to each segmented region based on a label selection input provided by the user (and the display may be updated accordingly). In other examples, an initial region label may be automatically assigned to each region, and the user may selectively provide input to change the label if the user believes the initial label is incorrect. In this case, all regions may be assigned the same initial region label (e.g., the statistically most common label), or one or more rules may be evaluated to determine which initial region label to assign to each region (e.g., based on the area of ​​the region, the perimeter of the region, the perimeter-to-area ratio of the region, the number of cells detected within the region, etc.).

[0089] In either case, the GUI may be configured to receive input identifying a label to be assigned to one or more portions of the image (e.g., as an initial display or as an updated display). For example, a user may click a color-coded marker to identify an initial label using a radio button list, and then provide input identifying a different label. As another example, a user may click within a segmentation region depiction, and then use a defined mouse click signature to identify an initial (e.g., and final) label for the region.

[0090] In some cases, a separate type of mouse click and / or touch screen interaction may be assigned to each of a given set of area indicators. For example, (1) a single left mouse click, (2) a single right mouse click, and (3) a double left mouse click may each be assigned to one of the area indicator options. (FIG. 7 and the associated discussion provide an illustration of the mapping of mouse clicks to indicators.) As another example, (1) a short hold followed by a slight downward swipe, (2) a short hold followed by a slight upward swipe, and (3) a short hold followed by a slight left swipe may each be assigned to one of the area indicator options.

[0091] The GUI may also include an option to assign (for example) any and all currently unlabeled segmentation regions in the image, or all currently unlabeled segmentation regions that are entirely within the field of view, to a given region marker (which may be selected by the user). Thus, for example, within a given field of view, a user may select to indicate that after labeling all stromal and background regions, all other regions should be labeled as tumor regions. The GUI may alternatively or additionally be configured to allow a user to select a tool to define a rectangular area within the depicted image and then assign all regions within the area to a selected region marker.

[0092] As each label is assigned or updated, the GUI may be updated to reflect the labeling (e.g., by adding a background color to the regions that may represent the assigned labeling) so that the user can easily see the current labeling (and, if applicable, which regions remain unlabeled). The GUI may include a submit or complete input option that can be selected by the user when the user is satisfied with the labeling. Further details of the GUI are provided herein, such as when describing FIG. 8.

[0093] Marker GUI controller 160 may store and / or output region markers 160a-160k. Region markers 160a-160k may (for example) include a mapping associating each unique region identifier with a region marker (or region marker identifier). Region markers 160a-160k may alternatively or additionally include a mapping for each of some or all pixels in an image that associates the pixel with a region marker (or region marker identifier).

[0094] A typical interactive segmentation workflow Referring to Figure 2, one embodiment of a multi-class interactive segmentation workflow 200 is shown. Various operations in the multi-class interactive segmentation workflow 200 may be performed by the model training system 105, the segmentation controller 120, and / or the signage GUI controller 160. Figure 3 shows an example of a corresponding pipeline for model building according to various embodiments.

[0095] In various embodiments, the multi-class interaction segmentation workflow 200 designs and builds a deep learning model to perform segmentation for multiple classes and receive user input identifying a label for each of one or more pixels and / or segmented regions. The interactive GUI may be configured to receive the input image and potentially initially receive annotation input identifying a class to which a given point or segment should be assigned and / or receive post-training input identifying or updating a label to be assigned to a given segmented region.

[0096] For example, the first dataset 210 may include multiple images (e.g., multi-class images 110a-110n) from one or more classes (e.g., DP images, natural scene images, immunohistochemistry images, H&E images, or any other relevant images) and may include corresponding annotations (e.g., click annotations, identifying distinct segments within the images, ground truth maps, etc.). Some or all of the first dataset 210 may have been received via an interactive GUI from a user facilitating training of a deep learning model (e.g., deep learning model 115) that is connected to the interactive GUI.

[0097] The multiple images may include digital pathology images featuring different tissue types and their corresponding segmentation annotations. In other embodiments, a first dataset 210 from another domain may be used, such as natural scene images. The first dataset 210 may include annotation data (for example) associating each of one or more pixels or areas with a given class. The class may be one of a number of predefined non-semantic classes.

[0098] A deep learning model may be designed and built (in part 1 of FIG. 3) by specifying the model's architecture, hyperparameters, and loss function. At 220, a first dataset 210 may be utilized (e.g., by the model training system 105) to train the deep learning model. By using datasets from multiple domains, the resulting model trained using the data may be more generalizable compared to using training data from a single domain.

[0099] A GUI can also be developed that is configured to display the digital pathology images and segmentation data (e.g., by overlaying segmentation boundaries on the images) and receive user input identifying landmarks (e.g., a predefined set of landmarks) in the respective sections or to modify predicted landmarks. (See part 2 of the pipeline in FIG. 3.) This interactive GUI may be, but need not be, the GUI that received some or all of the first data set. Even if the GUI is the same, the GUI may be configured to present a first interface to a first user for receiving some or all of the first data set and a second interface to a second user for receiving landmark definitions or updates.

[0100] The GUI of part 2 may be connected to the model (see part 3 of the pipeline in FIG. 3). This connection allows a user to identify (e.g., upload, select, etc.) one or more digital pathology images and / or checkpoint annotation data (e.g., a second dataset of images) and request processing via the GUI. The request may automatically trigger processing by the deep learning model to identify segmentations, and once segmentations are identified, the GUI may be configured to automatically display the image and segmentation data and receive identification or updates of labels. In some embodiments, an initial label (e.g., a non-semantic label) may be assigned to each of the one or more segmented regions. In some embodiments, at least one segmented region lacks an initial label.

[0101] Using the interactive GUI 230 (e.g., controlled by the labeling GUI controller 160), a user may efficiently identify and / or update labels for at least some of the segmented regions. For example, in various embodiments, a user may use a mouse to click on an area to provide annotations. The interface may be configured such that annotations (e.g., identifying new or updated labels) can be made quickly and efficiently, and the GUI is updated in real-time to reflect the most current labels in the region. For example, in various embodiments, different colors may be used to annotate different classes. For example, blue dots may be used to annotate stromal regions, green dots may be used to annotate tumor regions, and red dots may be used to annotate background regions. Additionally, multiple annotations may be used for one or more classes.

[0102] In various embodiments, the trained deep learning model 235 is not further modified after the initial training (using multi-class data). Instead, the deep learning model 235 outputs a segmentation for the image, and the interactive GUI is configured to receive input to quickly identify landmarks in each of some of the segmented regions.

[0103] In one embodiment, a deep learning model is used to segment a 600x600 pixel image tile in less than a minute. Thus, users may quickly and efficiently annotate images or perform image segmentation. Furthermore, the model does not over-fit to a specific dataset and maintains generalizability without the need for retraining or fine-tuning the model during use. Furthermore, the system provides confidence to the user as the process and output are not black boxes as in traditional deep learning systems, but instead allow for user-guided modification of model predictions, which provide real-time feedback and are therefore interactive, intuitive and flexible for end users.

[0104] Typical segmentation methods using limited annotations Referring to FIG. 4, an exemplary process 400 of interactive segmentation is shown. In various embodiments, a multi-class deep learning system receives an image to be segmented (step 410). In some embodiments, the image may be a complete image (e.g., a full slide digital pathology image) or a portion of an image (e.g., a field of view of a complete image, a region of interest of a complete image, a tile of a complete image, etc.). In various embodiments, a user may provide one or more annotations to the image (step 415). Such annotations may include (for example) click annotations that may be further processed to potentially be converted into other annotations using a conversion algorithm. In various other embodiments, annotations may not be provided by the user prior to the initial segmentation.

[0105] Segmentation may be performed on the image (and its annotations, if available) using the deep learning model (step 420). In various embodiments, the model may be an n-class model. Thus, the model may be configured to segment the input image into up to n types of different areas at each round of user interaction, thereby generating a single segmentation mask with up to n classes of areas. For example, in various embodiments, the deep learning model is a 3-class model configured to classify the input image into areas including tumor cells, stromal cells, and background tissue. In other embodiments, the deep learning model is a 4-class model configured to classify the input image into four different areas including immune cells, tumor cells, stromal cells, and background cells. As will be appreciated by those skilled in the art, the deep learning model may be configured for any number of classes and configured to segment any type of tissue or cell.

[0106] The deep learning model output provides the segmented image to an interactive GUI, and the results are displayed to the user (step 430). The user may then choose to provide annotations using the interactive GUI (step 440). Once the user provides annotations, the interactive GUI provides the annotations as further inputs to the deep learning model (step 450), and the process may be repeated as many times as necessary to achieve a satisfactory result. EXAMPLES

[0107] Example 1 Traditional interactive segmentation models, both optimization-based and learning-based algorithms, are designed to segment individual objects one at a time, which uses binary segmentation to simultaneously identify two different types of regions: foreground objects and non-target background. Foreground objects (or "foreground" for short) are defined as target object instances in an image, while non-target background is anything other than the target object.

[0108] The techniques disclosed herein can provide robust results that can segment target objects with reasonable success, whether they are a single connected region or multiple disconnected regions of the same class. Interactive binary segmentation was used for multi-class segmentation of DP images.

[0109] 5A-5H show the steps performed when binary segmentation is used to segment a digital pathology image to identify three types of regions. FIG. 5A shows a typical colon cancer H&E image tile. FIG. 5B shows a first round of binary segmentation to identify tumor (red masked regions) and non-tumor (remaining unmasked regions). FIG. 5B shows the same image tile after a first round of binary segmentation has been performed using a deep learning model to identify tumor (red masked regions) and non-tumor (remaining unmasked regions). In this example, a user used the interactive GUI 130 to annotate the image with green dots using positive clicks on tumor regions and red dots using negative clicks on non-tumor regions. FIG. 5C shows a typical updated tumor region segmentation result (red masked regions). Thus, the tumor region is identified, even though it is cut off in this image.

[0110] FIG. 5D shows the image after a second round of segmentation identifying stromal and non-stromal regions. At this point, the green dot indicates the first positive click to identify stromal regions. The green masked regions are stromal regions identified in the second round of segmentation after the first positive green click, and the red masked regions are tumor regions identified by the first round of tumor vs. non-tumor region segmentation. In the illustrated example, after the first green click, the prediction of the stromal regions extends into the previously identified tumor regions, because the second round of segmentation is likely unrelated to the first round.

[0111] FIG. 5E shows an example of correcting the stroma segmentation result with the addition of a red dot (negative click). FIG. 5F shows a series of click annotations that modify the second round of segmentation. FIG. 5G shows the final segmentation results in tumor, stroma and background regions, with light blue corresponding to stroma regions, grey corresponding to tumor regions and black corresponding to background regions. FIG. 5H shows a zoomed in view from the red rectangular region from FIG. 5G. The top image contains the segmentation results overlaid on the original image and the bottom image shows the segmentation masks used. In this example, the arrows point to the gaps after combining the masks generated from the second round of segmentation.

[0112] Since each round of binary segmentation is independent, users are required to correct already segmented objects from the previous round (Figure 5D-E). This extends the analysis time due to the necessary corrections to previously identified regions in addition to the target class in the current round of segmentation (Figure 5F). Such corrections to repair already annotated regions are redundant and time-consuming. Furthermore, users need to carefully provide click annotations via clicks at the boundaries of the target classes, otherwise gaps often occur between segmented objects from different rounds of segmentation (Figure 5G-H), thus inevitably leading to confusion and errors.

[0113] Example 2 6 shows an exemplary representation of the use of a three-class deep learning model 600. For a three-class model, the ground truth mask contains up to three different types of regions, and the user annotations contain up to three sets of user annotations, each of the sets corresponding to one type of region.

[0114] The ground truth mask in each segmentation task is an instance-level annotation where multiple object instances are annotated. For example, during training, up to two objects are sampled from the mask to generate a new mask, and the rest of the image is considered as non-target background. Such non-target background can include (a) other objects not selected from the original mask, and (b) pure background where no objects are annotated in the original mask.

[0115] The training data for the three-class deep learning model includes images with click annotations 610. The colored dots represent mouse clicks on the target region. The location and number of dots are simulated for model training to mimic user input at test time, where the user provides mouse clicks to guide the segmentation of the target region.

[0116] The simulated click annotations are used during training. For example, mouse click annotations are encoded into a disk map or Euclidean distance map, and these maps form a further mask that can be conveniently fed to the deep learning model along with the training images. Various strategies of click simulation may be utilized for binary interactive segmentation, including positive click sampling and negative click sampling. Positive click sampling mimics the user's behavior of mouse clicking on a target object by randomly sampling a variable number of pixel locations within the target object region in the mask. Negative click sampling mimics the user's behavior to identify the background (i.e., any location not on the target object). In various embodiments, three strategies of negative click sampling may be used. For example, the first strategy may include using a variable number of pixel locations randomly sampled from a pure background region. The second strategy may include using a variable number of pixel locations sampled from non-selected objects in the ground truth mask. The third strategy may include using a variable number of pixel locations from the pure background sampled near the boundary of the target object.

[0117] In this example, negative click sampling is used for one type of region and positive click sampling is used for the other two types of regions to train a three-class deep learning model. It is understood that negative clicks can be better for effectively correcting incorrectly segmented regions. Therefore, maintaining such negative click sampling for one of the classes provides the flexibility for users to perform negative clicks for the class of regions that are most difficult to segment.

[0118] An image with annotations 610 is split into images 620 and the annotations in each object / class are segmented by the deep learning model 600. For example, a three-class deep learning model includes a first class annotation input 630, a second class annotation input 640, and a background annotation input 650. The images 620 and annotations 630, 640, 650 are used to generate a ground truth mask 660 that is used to train a deep learning model 670 that can then generate predictions 680.

[0119] Example 3 7 illustrates an example of a workflow for providing annotations via an interactive GUI configured to provide user input capabilities for multiple segmentation classes. Annotations are provided by a user using various user inputs 710, and the GUI is generated on a display 720. User inputs 710 may include any computer input device, such as a mouse, touch screen, keyboard, or other input device. User inputs 710 include a mouse with at least two buttons, including a left click button and a right click button.

[0120] In a typical binary object segmentation GUI, a left mouse click is used as a positive click (i.e., the location is part of a class) and a right mouse click is used as a negative click (i.e., the location is not part of a class). Using a multi-class interactive GUI, the left and right click buttons may be mapped to classes. For example, in a three-class system, a left mouse click 712 may be associated with a first class, a right mouse click 714 may be associated with a second class, and a combination mouse click 716 may be associated with a third class. Each mouse click may cause the interactive GUI to overlay a dot on the image 728 corresponding to the class associated with the click.

[0121] A left mouse click 712 may be associated with a first type of region (e.g., tumor region), and each left mouse click triggers the display of a green dot 722 on the interactive GUI via the display 720. The left mouse click 712 may function as a positive click. A right mouse click 714 may be associated with a second type of region (e.g., stromal region), and each right mouse click triggers the display of a red dot 724 by the GUI and may also function as a positive click. A left mouse double click 716 may be associated with a third type of region, and may generate a yellow dot 726. In various embodiments, the left mouse double click 716 may function as a negative click. To connect the trained model to the interactive GUI, the GUI backend is configured to convert the user-clicked pixel locations in the three classes into three disk maps, and use these disk maps together with the image as deep learning model inputs to generate predictions in the corresponding three classes.

[0122] Using the interactive GUI, a user may initiate segmentation by adding a first positive click on the target region of interest. This click triggers an initial network prediction. Adding further clicks triggers changes to the network inputs, which lead to corresponding changes in the segmentation predictions. The segmentation results may be displayed as a colored overlay on the input image.

[0123] Example 4 FIG. 8 shows an example of an interactive GUI for defining click annotations. The interactive GUI includes an image display area 800 and a control panel 810. The image display area 800 is configured to show the image being segmented and provides an area for the user to add annotations. The controls allow various options for operating the system. For example, the interactive GUI may include a button 820 for quitting, a button 830 for canceling the click, and a button 840 for resetting the click. The GUI controls 810 may also include various zoom options 850. In various embodiments, the user may adjust various model parameters using the GUI controls. For example, the GUI controls 810 may include a backpropagation refinement (BRS) option 860 and sliders for changing the prediction threshold 870 and the alpha blending factor 880. Other control options include a slider for the dot radius 890.

[0124] Example 5 A title-by-title approach is used for full slide image segmentation. Tile-by-tile predictions for WSI. With a tile-by-tile approach, the user performs interactive segmentation one image tile at a time. In this way, the user segments one tile to their satisfaction as described above and moves on to the next. A platform of interest (web application, GUI, etc.) is selected to display full slide images and support interactive segmentation. Once all tiles have been segmented, the platform can automatically assemble all predictions and display the full slide segmentation results.

[0125] Example 6 A radial expansion scheme was used for full slide image segmentation. FIG. 8 shows a visualization of a radial expansion strategy for full slide interactive segmentation according to various embodiments. The radial expansion approach takes advantage of predictions in overlapping tiles to expand predictions in a radial direction, thus facilitating segmentation in large images such as WSI (FIG. 9). For example, when a user starts the analysis by clicking anywhere in the WSI (e.g., blue dot in FIG. 9), multiple tiles covering this click can be selected to maximize the total tile coverage, so that predictions for all these tiles can be generated (blue tiles in FIG. 9). In the next step, the user can click anywhere in the predicted area that needs to be corrected (an exemplary second click shown as an orange dot in FIG. 9). Correcting the click helps to expand predictions to tiles covering these clicks (orange tiles in FIG. 9). Since one user click triggers segmentation in multiple tiles, performing such predictions with the designed radial expansion scheme allows much faster propagation of segmented regions in the full slide imaging, thus speeding up full slide image segmentation. Another advantage of this approach is that the system can leverage parallelism or distributed model inference across multiple tiles to further expedite whole-slide image segmentation.

[0126] Example 7 Domain-specific and dataset-specific whole slide image segmentation is provided. In use cases where segmentation is required for a particular dataset with multiple whole slide images, interactive segmentation can serve as both an initial annotation generation strategy and a fast annotation expansion strategy. For example, initially, a subset of slide image regions can be selected manually, randomly, or via active learning-based techniques. Segmentation masks of these selected large image regions in the selected WSI can then be efficiently generated using one of the aforementioned techniques (i.e., tile prediction or tile by radial expansion strategy), followed by iterative model training and fine-tuning. In this way, a model can be rapidly built for WSI segmentation of the dataset of interest.

[0127] Example 8 The deep learning model was trained on the Semantic Boundaries dataset (SBD), a commonly used benchmark for interactive segmentation with instance-level annotations from natural scene image domains. SBD contains 11,355 images, of which 8,498 were used for training and 2,857 were used for validation. In each image, one or more foreground objects (i.e., instances) are annotated.

[0128] HRNet with OCR module was pre-trained on ImageNet as a backbone, an architecture recently proposed to enable fine detail discrimination in several vision tasks including image segmentation and human pose estimation. To incorporate simulated user clicks into the model training, convolutional blocks were coded separately on the image and disk maps from the user input. Note that such a backbone can be replaced with other models chosen to address specific use cases. Furthermore, domain-specific annotated datasets, e.g., IHC images with corresponding masks if available, can be utilized to train such interactive systems.

[0129] Implementation details: We performed on-the-fly three-class click sampling during training. Specifically, for each image, at most two foreground instances (i.e., individual connected objects in the ground truth mask) were randomly selected from the segmentation mask. Along with the pure background annotations, we generated a new mask with at most three distinct regions. Two of the regions correspond to the target objects and one to the non-target background. From such a new mask, positive clicks were randomly placed inside each foreground object region, and negative clicks were placed in the background region with the same strategy. Images were cropped to 320x480 patches for training and augmented with random rescaling, flipping, and color jittering.

[0130] The interactive platform was used to segment four different classes of images, as shown in Figure 10. Specifically, Figure 10 shows images including (A) model performance on an exemplary colon cancer H&E image tile, (B) model performance on an exemplary IHC image from breast cancer tissue targeted to Ki67 (brown DAB stain) and counterstained with hematoxylin (blue stain), (C) exemplary IHC image from lung cancer tissue targeted to PDL1 (pink tumbler stain) and CK7 (yellow dabsyl stain) and counterstained with hematoxylin (blue), and (D) an exemplary chest x-ray image. In the bottom panels of A-C, the light blue regions are stroma, the gray regions are tumor, and the black regions are non-target background. In the bottom panel of D, the light blue regions are right lung, the gray regions are left lung, and the black regions are non-target background. Predictions demonstrate that the proposed model design and training strategy successfully segments images into three classes of regions interactively, as shown in FIG.

[0131] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium that includes instructions configured to cause one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0132] The terms and expressions employed are used as terms of description and not of limitation, and there is no intention in the use of such terms and expressions to exclude any equivalents of the features shown and described or portions thereof, recognizing that various modifications are possible within the scope of the claimed invention. Thus, although the claimed invention has been specifically disclosed by embodiments and optional features, it should be understood that modifications and variations of the concepts disclosed herein may be employed by those skilled in the art, and such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims.

[0133] The description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Instead, the description of the preferred exemplary embodiments provides one of ordinary skill in the art with an enabling description for implementing various embodiments. It will be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope as set forth in the appended claims.

[0134] In the following description, specific details are given to provide a comprehensive understanding of the embodiments. However, it will be understood that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

Claims

1. receiving an image and displaying said image using a graphical user interface; receiving at least one first image annotation provided by a user via the graphical user interface; generating a first segmented image using a deep learning model, the deep learning model generating the first segmented image using the digital pathology image and the at least one first image annotation; Displaying the first segmented image using the graphical user interface; and receiving at least one second image annotation provided by the user via the graphical user interface; generating a second segmented image using the deep learning model, the deep learning model generating the second segmented image using the digital pathology image, the at least one first image annotation, and the at least one second image annotation; Displaying the second segmented image using the graphical user interface; and 4. A computer-implemented method comprising:

2. The method of claim 1 , wherein the deep learning model is a three-class model and the at least one first image annotation includes at least one annotation for each class.

3. The method of claim 1 or 2, wherein the at least one image annotation provided by the user is provided using a mouse click on the displayed image.

4. The method of claim 3 , wherein the mouse click comprises a left mouse click associated with a first segmentation class.

5. The method of claim 4 , wherein the mouse click comprises a right mouse click associated with a second segmentation class.

6. The method of claim 4 , wherein the mouse clicks include combination mouse clicks associated with a third segmentation class.

7. The method of claim 6 , wherein the combination mouse clicks include a left mouse double click.

8. The method of claim 1 , wherein the deep learning model is trained using a first dataset including training images with one or more annotations.

9. The method of claim 8 , wherein the training images include images from a first image domain.

10. The method of claim 9 , wherein the training images further include images from a second image domain.

11. 10. The method of claim 8 or 9, wherein the second image domain is different from the first image domain, and the first image domain and the second image domain each comprise one of a natural scene image, a digital pathology image, an immunohistochemistry image, an X-ray image, and a hematoxylin and eosin image.

12. The method of claim 8 , wherein the one or more annotations include one or more simulated click annotations.

13. The method of claim 1 , wherein the images are whole slide pathology images.

14. one or more data processors; 11. A non-transitory computer-readable storage medium comprising instructions that, when executed on the one or more data processors, cause the one or more data processors to perform a set of operations, the set of operations comprising: receiving an image and displaying said image using a graphical user interface; receiving at least one first image annotation provided by a user via the graphical user interface; generating a first segmented image using a deep learning model, the deep learning model generating the first segmented image using the digital pathology image and the at least one first image annotation; Displaying the first segmented image using the graphical user interface; and receiving at least one second image annotation provided by the user via the graphical user interface; generating a second segmented image using the deep learning model, the deep learning model generating the second segmented image using the digital pathology image, the at least one first image annotation, and the at least one second image annotation; Displaying the second segmented image using the graphical user interface; and A non-transitory computer readable storage medium, A system comprising:

15. A computer program product tangibly embodied in a non-transitory machine-readable storage medium that includes instructions configured to cause one or more data processors to perform a set of operations, the set of operations comprising: receiving an image and displaying said image using a graphical user interface; receiving at least one first image annotation provided by a user via the graphical user interface; generating a first segmented image using a deep learning model, the deep learning model generating the first segmented image using the digital pathology image and the at least one first image annotation; Displaying the first segmented image using the graphical user interface; and receiving at least one second image annotation provided by the user via the graphical user interface; generating a second segmented image using the deep learning model, the deep learning model generating the second segmented image using the digital pathology image, the at least one first image annotation, and the at least one second image annotation; Displaying the second segmented image using the graphical user interface; and 2. A computer program product comprising:

Images