Multi-class interactive segmented graphical user interface

Abstract

The present disclosure provides methods and systems to perform an action, the action comprising: receiving an image and displaying the image using a graphical user interface; receiving at least one first image annotation provided by a user via the graphical user interface; generating a first segmented image using a deep learning model, wherein the deep learning model uses the digital pathological image and the at least one first image annotation; and displaying the first segmented image using the graphical user interface; receiving at least one second image annotation provided by the user via the graphical user interface; generating a second segmented image using the deep learning model, wherein the deep learning model uses the digital pathological image, the at least one first image annotation, and the at least one second image annotation; and displaying the second segmented image using the graphical user interface.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 269,833, filed on March 23, 2022, which is hereby incorporated by reference in its entirety for all purposes. Technical Field

[0003] The present disclosure relates to digital pathology, and more particularly to techniques for segmenting images. Background Art

[0004] Digital pathology involves scanning a slide (e.g., a histopathology or cytopathology slide) to produce a digital image. The slide may include a biological sample (e.g., a tissue slide or a biological fluid) that has been stained with one or more stains (i.e., dyes) that selectively bind to a specific cell component or tissue type. The digital image may be subsequently processed using digital pathology image processing techniques and / or by interpretation by a pathologist. This subsequent processing may be used for a variety of reasons such as: predicting or facilitating the diagnosis of a disease, estimating the extent to which a given therapy is effective for a given subject, predicting the extent to which a given therapy will be effective for a given subject, and / or facilitating new treatments (e.g., the development of new active agents, dosages, compositions, treatment plans, etc.).

[0005] The traditional method for analyzing digital pathology images is for trained human pathologists to examine a highly magnified portion of a slide (i.e., a "field of view") and manually segment the image to identify one or more portions of interest (e.g., to exclude background, artifacts, macrophages, etc.), and then detect and classify the signal within each portion of the image. Segmentation and classification are typically performed at magnifications of 40x to 400x. Therefore, generating digital pathology results is a very labor-intensive, time-consuming, and costly endeavor when using standard methods.

[0006] Another approach is to use machine learning models to process digital pathology images. However, this generally involves training the model using a training dataset that includes a large number of manually labeled images. Given the level of accuracy required for segmentation (which generally involves delineating the boundaries of a region with pixel-level accuracy) and the amount of data required to train a model equipped to process digital pathology images, generating annotations for the training data may require a half-year or more investment of time from a trained pathologist.

[0007] Deep learning models are a class of machine learning models that have such sophisticated architectures that they can frequently learn complex data features. However, as the complexity of the model increases, the size of the required training dataset also increases. Therefore, despite the power of deep learning systems, developing such systems and promoting their widespread use in the clinical field remains challenging, especially given the data collection variations, signal variations, and privacy considerations that affect clinical data.

[0008] Digital pathology is a particularly challenging environment for training any model, let alone a deep learning model. In digital pathology, the stains used vary widely. In addition, the frequent use of multiple biomarker dyes (e.g., three or more dyes) means that even fewer images depicting samples stained with a specific combination of dyes are available (especially when considering privacy constraints). In addition, models trained on one tissue type are not easily transferable to other tissue types. It is often the case that a deep learning model built for a specific dataset (i.e., a specific image domain) fails to perform well even on a similar or related dataset (another different image domain).

[0009] Therefore, in digital pathology, models designed for a specific diagnostic assay are not easily reusable for another assay due to performance issues. The same is true when the model is applied to images from the same slide but scanned from different scanners. (See Aubreville M et al. (2021). MItosis DOmain Generalization Challenge. 24th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2021). Zenodo. https: / / doi.org / 10.5281 / zenodo.4573978 ). This lack of model generalization in digital pathology leads to an inevitable and continuous investment in developing new models, even for related tasks.

[0010] Furthermore, platforms that use machine learning models are often built end-to-end and therefore remain a black box to the end user. This can lead to poor interpretability and poor model performance (due to not making the user aware of logical problems or how best to modify the model to resolve detected problems). Furthermore, this inflexible implementation can prevent a model trained in one domain from being used in another domain.

[0011] Therefore, processing digital pathology images to accurately and efficiently segment the region of interest is particularly challenging. Furthermore, even if the model is configured to perform this type of segmentation, existing systems are not configured to receive responsive user input that does not lead to overfitting or reduce the utility of the model. Summary of the invention

[0012] In some embodiments, a computer-implemented method is provided, the method comprising: receiving an image and displaying the image using a graphical user interface; receiving at least one first image annotation provided by a user via the graphical user interface; using a deep learning model to generate a first segmented image, wherein the deep learning model uses a digital pathology image and at least one first image annotation; and displaying the first segmented image using the graphical user interface; receiving at least one second image annotation provided by a user via the graphical user interface; using a deep learning model to generate a second segmented image, wherein the deep learning model uses the digital pathology image, at least one first image annotation, and at least one second image annotation; and displaying the second segmented image using the graphical user interface.

[0013] The deep learning model may be a three-category model, and the at least one first image annotation may include at least one annotation for each category.

[0014] The at least one image annotation that may have been provided by the user was provided using a mouse click on the displayed image.

[0015] The mouse click may include a left mouse click associated with the first segmentation category.

[0016] The mouse click may include a right mouse click associated with the second segmentation category.

[0017] The mouse clicks may include combined mouse clicks associated with a third segmentation category.

[0018] A combination mouse click may include a double left mouse click.

[0019] The deep learning model may have been trained using a first dataset comprising training images having one or more annotations.

[0020] The training images may include images from a first image domain.

[0021] The training images may further include images from a second image domain.

[0022] The second image domain may be different from the first image domain, and wherein the first image domain and the second image domain each include one of: a natural scene image, a digital pathology image, an immunohistochemistry image, an x-ray image, and a hematoxylin and eosin image.

[0023] The one or more annotations may include one or more simulated click annotations.

[0024] The image may be a whole slide pathology image.

[0025] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein.

[0026] In some embodiments, a computer program product is provided, which is tangibly embodied in a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein.

[0027] Some embodiments of the present disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions, which, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium, which includes instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes disclosed herein.

[0028] The terms and expressions that have been adopted are used as terms of description and not of limitation, and when such terms and expressions are used, there is no intention to exclude any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the claimed invention. Therefore, it should be understood that although the claimed invention has been specifically disclosed through embodiments and optional features, modifications and variations of the concepts disclosed herein may be adopted by those skilled in the art, and such modifications and variations are considered to be within the scope of the invention as defined by the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] This patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with one or more color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0030] Aspects and features of various embodiments will become more apparent by describing examples with reference to the accompanying drawings, in which:

[0031] Figure 1 An exemplary network for generating a digital pathology image and detecting and labeling regions within the digital image using a deep learning model and a labeling GUI is shown.

[0032] Figure 2 An embodiment of an interactive segmentation workflow based on multi-class DL is depicted.

[0033] Figure 3 An example of a pipeline for model building is depicted, in accordance with various embodiments.

[0034] Figure 4 Depicted is an example workflow of a multi-class deep learning system in accordance with various embodiments.

[0035] Figure 5 An embodiment of applying interactive binary segmentation to multi-class segmentation of DP images is depicted.

[0036] Figure 6 Depicted are three categories of deep learning models and training embodiments.

[0037] Figure 7 Depicted is an example workflow for providing annotations in accordance with various embodiments.

[0038] Figure 8 Depicted are example interactive GUIs in accordance with various embodiments.

[0039] Fig. 9 Depicted is a visualization of a radial expansion strategy for whole-slide interactive segmentation in accordance with various embodiments.

[0040] Fig.10 Depicted are visualizations of interactive segmentation results from four image domains in accordance with various embodiments.

[0041] In the drawings, similar parts and / or features may have the same reference numeral. In addition, various parts of the same type may be distinguished by following the reference numeral with a dash and a second reference numeral that distinguishes the similar parts. If only the first reference numeral is used in the specification, the description applies to any similar part having the same first reference numeral, regardless of the second reference numeral. DETAILED DESCRIPTION

[0042] Although certain embodiments are described, these embodiments are presented only by way of example and are not intended to limit the scope of protection. The devices, methods, and systems described herein may be embodied in a variety of other forms. In addition, various omissions, substitutions, and changes may be made to the forms of the example methods and systems described herein without departing from the scope of protection.

[0043] Overview

[0044] Various embodiments of the present invention relate to training a deep learning model for interactive segmentation of digital pathology images to identify different regions within an image. In addition, a graphical user interface (GUI) can be utilized that can allow a user to provide images, provide interactions to facilitate segmentation, and / or provide region labeling updates (after segmentation). Thus, the deep learning model provides near real-time segmentation results, and the GUI is configured to allow a human reviewer to identify region labels very efficiently and quickly.

[0045] Training images and click annotations

[0046] The deep learning model can be trained using a multi-class training dataset having images and corresponding annotations from one or more domains. The annotations may include segmentation annotations that identify boundaries or regions of depictions of different things. For each indicated segment, the annotation may (but need not) include the category (or label, used interchangeably herein) of the segment. The annotation may include or may be based on click annotations. For example, for a given image, each of a plurality of points may be identified, each of which is within a segment depicting a particular type of object, person, or being. For each point, the annotation may indicate which of a plurality of categories the corresponding segment is assigned to.

[0047] The multi-class training dataset is multi-class because it identifies segmentations for multiple types of depictions. For example, for digital pathology images, the categories may include stroma regions, tumor regions, and background regions. As another example, for natural scene images, the categories may include each vehicle, each stop light, and background (all other parts of the image). The background category may be defined to include depictions of (a) other objects not selected from the original mask and (b) pure background (where no objects were annotated in the original mask).

[0048] That is, the multiple categories need not have semantic meaning. For example, in the three-category example, the categories may correspond to: regions of a first type; regions of a second type; and background regions (which may, but need not, include regions of one or more other types). The content that constitutes regions of the first type and regions of the second type may be arbitrary. For example, for an image of a vehicle at a traffic light, a region of the first type may be a vehicle, a stop light, a person, a crosswalk, etc. In the case where an image depicts multiple objects of a given selected type (e.g., multiple vehicles, multiple people, etc.), all such depictions may be considered regions of the same selected type.

[0049] The number of click annotations identified for a given training image may be predefined, but need not be. For example, an implementation may be configured such that each training image is associated with one click annotation per category, three click annotations per category, six click annotations total, ten click annotations total, etc.

[0050] In some cases, click annotations are automatically identified for training images. For example, some training images may be associated with a reference truth mask that indicates (or can be used to determine) which label each pixel is to be assigned to. Such reference truth masks may change with different click annotation targets. For example, two of multiple image regions in an input image may be used as segmentation targets, and corresponding reference truth masks may be generated to indicate the target image regions and ignore the remaining image regions. The same image may also be paired with click annotations targeting another set of image regions and corresponding reference truth masks for these specific image regions. For each reference truth mask specific to a set of click annotations, the image region may belong to one category or two categories. The number of unique labels in the reference truth mask may be different from the number of categories for which the deep learning model is to be trained. When there are more unique labels than categories, a subset of labels may be selected, where the number of labels in the subset is equal to the number of categories minus one (assuming that background categories can be used). The selection may be, for example, random, arbitrary, or biased toward labels associated with the most pixel assignments. When there are fewer unique labels than categories, the corresponding image may be discarded from the training data set.

[0051] Using click annotations, a reference truth mask can be generated for each image, which can be used to train a deep learning model. Using click annotations, an additional input map can be generated for each image, which can be used to train a deep learning model. The input map (which can be the same size as the input image) can be generated by encoding the click annotations using a map (such as a disk map or a Euclidean distance map). The disk map can be generated, for example, by starting with an image with a value of zero, changing the pixel value to 1 (or another positive value) in the clicked pixel, and then changing the value of the pixels around these clicked pixels to 1 to expand the click neighborhood into a disk-shaped image area with a value of 1. The click map can be generated by setting the value of the clicked pixel (and only the clicked pixel) to be different from the rest of the pixels. A square map can be generated to include a square area around the clicked pixel location. In some cases, a single map indicates the location of clicks of multiple categories. In some cases, a separate map is generated for each category.

[0052] The mask can be used as model training and / or inference input along with the input image to tell the model where the user input is so that the model can generate a segmentation mask based on where the user clicked. In training, the model can be trained to respond to any object in the image that the user wants to target and provide click input for. In inference, the model can predict the target image region / object based on where the user clicked (there can be many objects in the image, and the model can use the click location to predict which object / image region the user intended to segment).

[0053] These encodings are further described in: K. Sofiiuk, I. Petrov, O. Barinova, A. Konushin, F-BRS: Rethinking backpropagating refinement for interactive segmentation, in: 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2020, pp. 8623–8632. doi:10.1109 / cvpr42600.2020.00865; and Sofiiuk, Konstantin, Ilia A. Petrov, and Anton Konushin. "Reviving Iterative Training with Mask Guidance for Interactive Segmentation" arXiv preprint arXiv:2102.06583 (2021), each of which is hereby incorporated by reference in its entirety for all purposes.

[0054] Since the categories in the training data do not need to have semantic meaning, a deep learning model trained on the training data can be configured to provide segmentation in a manner separate from region labeling. Thus, a deep learning model can be configured to predict which parts of an image correspond to different things without predicting what type of thing is depicted in a given part and / or without predicting whether multiple parts depict the same type of thing. As such, a multi-class training dataset can be used that includes images and annotations from multiple domains.

[0055] The domains may include, for example, digital pathology datasets, natural scene image datasets, immunohistochemistry datasets, H&E datasets, or any other appropriate image-annotation pair datasets as will be appreciated by those skilled in the art. Using training data from one or more domains other than digital pathology (e.g., possibly plus training data from the digital pathology domain), a larger training set may be collected. However, by continuing to include digital pathology images in the multi-class training dataset, the deep learning model may be easily applied to process multiple types of digital pathology images without requiring a large amount of additional model development. However, in some cases, the training dataset does not even include training data from the domain in which the trained deep learning model is later used for image annotation.

[0056] Deep learning models and training

[0057] A deep learning model may include a neural network having more than three, more than four, or more than five layers (including an input layer and an output layer). A deep learning model may include a convolutional neural network.

[0058] A deep learning model can be trained using the training data described herein (e.g., which includes multi-class images and can include ground truth masks and / or click annotations) so that the deep learning model learns how to segment a specific number of categories (e.g., two target categories and one background category). The training data implements network learning of effective representations to match segmentation predictions corresponding to specific pixels indicated via click annotations (or to specific regions within the ground truth mask). For example, such a model will learn to segment any target region pointed to by these pixel locations. Therefore, the model is trained to group image pixels that are uniformly labeled (i.e., similar / same network representations) together as "annotated" image pixels, regardless of the exact underlying semantic meaning of the image pixels. In other words, this model does not need to learn to distinguish specific semantic categories in an image, but can learn to identify semantically similar image regions with "annotated" pixels, and only group these pixels together to generate segmented target regions. Due to this design, the interactive deep learning model is able to recognize any target object or region for which a user provides click annotations, and does not require domain-specific training to identify the exact category of the region. Therefore, such embodiments provided herein have almost no restriction on whether the test image and the training image are from the same domain. This feature is the key to the powerful ability to generalize across image domains.

[0059] Therefore, the trained deep learning model can be used to process other images to generate predicted segmentation annotations.

[0060] In some cases, a user can provide click annotations using an input image. For example, a user can use a GUI to select or upload an image. The GUI can then present the image and use tools to add click annotations. For each category in a set of categories, the click annotation can identify one or more pixels in the image that correspond or do not correspond to the category. The deep learning model can use the click annotations and the learned model parameters to predict which parts of the image correspond to each category in the set of categories.

[0061] In some cases, the user provides an input image but does not provide click annotations. The deep learning model can then use the learned model parameters to identify a set of segments. The model can attempt to assign labels based on similarities across regions, or can provide only segmentation data (e.g., identifying pixels on the boundaries of individual segmented regions).

[0062] In some cases, the output of the trained deep learning model may include one or more values ​​for each pixel. One or more values ​​may include a category assignment (e.g., category 1, 2, or 3). In some cases, the value may include an error indicating an unassigned category. The output may include a set of masks, each mask in the set of masks corresponding to a given predefined classification. Each mask may include a value for each pixel, wherein a particular value (e.g., 1) indicates that the pixel is predicted to be part of an area (or perimeter of an area) of the category, and another particular value (e.g., 0) indicates that the pixel is not predicted to be part of an area (or perimeter of an area) of the category. Alternatively, the value of each pixel may represent the likelihood or confidence of the prediction that the pixel is part of an area of ​​the category.

[0063] Interactive GUI for efficient marker recognition

[0064] After the deep learning model processes the input image, a GUI can be generated, or a GUI (e.g., the same GUI used to provide click-through annotations) can be updated to represent the segmentation generated via the processing.

[0065] Specifically, the GUI can be configured to receive corrections for any label and / or to identify, for each of one or more regions, a region label to be assigned to the region. For example, an input translation definition can be created that indicates what the updated region label will be for each of a plurality of specific types of mouse clicks. For example, a single left click can correspond to a tumor region; a single right click can correspond to a stroma region; and a double left click can correspond to a background region. If the user then performs any of these types of clicks within a depiction of a given segmented region, the region label for that region can be updated or defined accordingly.

[0066] As another example, the GUI may be configured so that after a user clicks within a given region, a drop-down menu (identifying different potential region labels) appears. The drop-down menu may identify the potential region labels, and the user may select which region label to assign to the region.

[0067] In this way, regions can be quickly identified and classified very quickly in a manner that requires little time investment by the user (e.g., compared to traditional manual segmentation and manual identification of region labels for each region). For example, an image tile of size 600x600 pixels can be segmented in less than a minute, and the GUI can facilitate rapid identification of region labels, which can enable real-time updates of region labels that are intuitive to the user and to the GUI and facilitate further region label identification.

[0068] A marker correction may quickly or immediately trigger an update of the marker.

[0069] Exemplary practical applications

[0070] The region marking can be used, for example, to generate one or more metrics or recommendations to support a care provider in determining a diagnosis, prognosis, or treatment recommendation for a subject. Exemplary metrics can include an absolute cumulative area of ​​each of one or more types of regions (e.g., tumor regions and / or stromal regions) in one or more digital pathology images and / or a relative cumulative area of ​​a tumor region in one or more digital pathology images (e.g., relative to the sum of the areas of the tumor region and the stromal region). In some cases, a change metric is generated that indicates the degree to which a cumulative absolute area or cumulative relative area has changed relative to a comparable metric generated by processing a digital pathology image corresponding to the same subject but at a previous time point. The change metric can identify a difference or a percentage change, or consist of a typical type of loss function used in optimization and artificial neural networks.

[0071] Region markings may further or alternatively be used, for example, to facilitate digital pathology analysis that includes characterizing cells depicted in an image. For example, a background color representing region markings may be provided in a digital pathology interface to facilitate segmentation and / or marking of individual cells. As another example, region markings may be used to assess whether there are one or more rules that have a potential inconsistency (or actual inconsistency) with respect to cell markings and region markings. If so, an alert may be presented or sent to request a review of the inconsistency.

[0072] Exemplary Advantages

[0073] Thus, embodiments disclosed herein provide a platform that supports simultaneous segmentation of more than two types of regions and demonstrates success in DL-based interactive segmentation across different imaging modality applications. The techniques disclosed herein have significant performance advantages over alternative approaches.

[0074] For example, active learning uses a heuristic scoring strategy to query a smaller subset of unlabeled examples (which are the most informative in the dataset), and the model is retrained by iteratively adding only a smaller set of selected examples from this set. While this avoids the need to annotate many images, models developed with active learning are only applicable to test images that have the same or identical distribution as the training images. Therefore, active learning cannot be used to produce a generalized model suitable for the high variability of dyes, scanners, tissues, etc. present in digital pathology.

[0075] As another example, the technology disclosed herein provides a GUI configured to allow a user to provide labeling input very efficiently. Although other annotation tools have been developed based on machine learning or deep features, such methods require pre-trained models for segmentation tasks similar to the input image, which greatly limits model generalization. In addition, such methods are limited to binary segmentation, while most applications in digital pathology require simultaneous identification of three or more different types of target regions.

[0076] As another example, while transfer learning and domain adaptation improve model generalization to some extent, they require additional model training using additional labeled examples from the target image domain. Although unsupervised domain adaptation aims to not use annotations from the target domain, it still relies on a validation set from the target domain for model selection, which often inevitably overfits to the validation set. In addition, a prerequisite for these methods to work well is that there is similarity between image domains. Therefore, without assumptions about the image domain or additional model retraining, such methods neither provide real-time prediction modification nor cross-domain generalization.

[0077] In addition, many existing segmentation models (both optimization-based and learning-based) are designed to segment a single object at a time, and these models simultaneously identify two different types of regions with binary segmentation: foreground objects and non-target background. A foreground object (referred to as "foreground") is defined as an instance of a target object in an image, while a non-target background is any other object other than the target object. At the same time, embodiments of the present invention achieve segmentation of a target object, regardless of whether the target object is a single connected region or multiple unconnected regions of the same category.

[0078] An exemplary network for digital pathology image generation and processing

[0079] Figure 1An exemplary network for generating a digital pathology image and a labeled region of interest is shown. It should be understood that Figure 1 Each of one or more of the depicted components (e.g., model training system 105, segmentation controller 120, imaging system 150, labeling GUI controller 160, etc.) in the example embodiment may include one or more processors and a computer-readable medium having instructions that, when executed, perform the actions identified herein. In some cases, a single physical or virtual computing system includes one or more processors and a computer-readable medium having instructions that, when executed, perform the actions described as being performed by Figure 1 For example, a single computing system may perform the actions described herein as being performed by model training system 105, segmentation controller 120, and label GUI controller 160.

[0080] The network includes a model training system 105 that retrieves multi-category images 110a to 110n from one or more data sources. The one or more data sources may include one or more remote data sources and / or one or more local data sources. In some cases, at least some of the multi-category images 110a to 110n are retrieved by sending an online request for the image and downloading the image. Each of the multi-category images 110a to 110n may include annotation data indicating the boundaries or regions of the depiction of different segments in the image. The segments in each of the multi-category images 110a to 110n may correspond to different types of objects or things. For example, instead of only identifying the location of the depicted vehicle in the image, the annotation data may also identify the depiction of each vehicle, each stop light, each crosswalk, each individual, and the like.

[0081] The multi-category images 110a to 110n may, but need not, be from different domains. The multi-category images 110a to 110n may include, for example, one or more natural scene images, one or more digital pathology images, one or more immunohistochemistry images, one or more H&E images, one or more MRI images, and / or images from one or more other domains. The multi-category images 110a to 110n may, but need not, include one or more digital pathology images. The multi-category images 110a to 110n may include one or more images that are not digital pathology images.

[0082] Each of the multi-category images 110a to 110n may be accompanied by segmentation annotation data. The segmentation annotation data may include annotations as disclosed herein. For example, the segmentation annotation data may include one or more click annotations, each of which corresponds to a specific pixel in the image and a specific positive label of a negative label (e.g., identifying a specific category). As another example, the segmentation annotation data map includes a transformed version of one or more click annotations (e.g., where each click annotation is transformed using a map such as a click map or a Euclidean map). As another example, the segmentation annotation data may include one or more ground truth masks that indicate, for each pixel, an identification or prediction of whether the pixel corresponds to a specific label. The segmentation annotation data may be identified by or generated based on input from a user interacting with a GUI that displays each training image. It should be understood that the user who provides input for the segmentation annotation data does not need to be the same user as the user who provides subsequent images (and / or other related information) to be processed by the trained model.

[0083] The model training system 105 can use the multi-class images 110a to 110n (e.g., and any accompanying segmentation annotation data) to train a deep learning model 115. The deep learning model 115 can include a neural network with more than three, more than four, more than five, or more than six layers. The deep learning model 115 can have a convolutional neural network architecture. Training the deep learning model 115 can include learning values ​​for a set of learnable parameters of the model (e.g., weights assigned to connections between various nodes).

[0084] After training the deep learning model, the segmentation controller 120 can use the trained deep learning model to process each of the one or more digital pathology (DP) images 125a to 125n. Each DP image 125 may have been generated by an image generation system 130, which may be at a remote location or a local location of the segmentation controller 120. It should be understood that some actions performed by the image generation system 130 (and / or by one or more components thereof) are performed automatically, while other actions (in some cases) are performed using at least some human actions.

[0085] As part of the image generation system 130, the fixation / embedding system 135 uses a fixative (e.g., a liquid fixative, such as a formaldehyde solution) and / or an embedding substance (e.g., a histological wax such as paraffin, and / or one or more resins such as styrene or polyethylene) to fix and / or embed a tissue sample (e.g., a sample comprising at least a portion of at least one tumor). Each slice can be fixed by exposing the slice to the fixative for a predetermined period of time (e.g., at least 3 hours) and then dehydrating the slice (e.g., via exposure to an ethanol solution and / or a clarifying intermediate). The embedding substance can infiltrate the sample while the slice is in a liquid state (e.g., when heated).

[0086] The tissue slicer 140 then slices the fixed and / or embedded tissue sample (e.g., tumor sample) to obtain a series of slices, each slice having a thickness of, for example, 4 to 5 microns. Such slicing can be performed by first cooling the sample and slicing the sample in a warm water bath. The tissue can be sliced ​​using, for example, a vibrating microtome or a compression microtome.

[0087] Because tissue sections and cells therein are actually transparent, the preparation of sections usually includes staining the tissue sections (e.g., automatic staining) to make the relevant structures more visible. In some cases, staining is done manually. In some cases, staining is done semi-automatically or automatically using a staining system 145.

[0088] Staining can include exposing a single slice of tissue to one or more different stains (e.g., sequentially or simultaneously) to express different features of the tissue. For example, each slice can be exposed to a predefined volume of stain for a predefined period of time.

[0089] An exemplary type of tissue staining is a histochemical staining, which uses one or more chemical dyes (e.g., acid dyes, basic dyes) to stain tissue structures. Histochemical staining can be used to indicate general aspects of tissue morphology and / or cell histology (e.g., to distinguish between nuclei and cytoplasm, indicate lipid droplets, etc.). An example of a histochemical stain is hematoxylin and eosin (H&E). Other examples of histochemical stains include trichrome stains (e.g., Masson's trichrome staining), periodic acid-Schiff (PAS), silver stains, and iron stains. The molecular weight of a histochemical staining reagent (e.g., dye) is typically about 500 kilodaltons (kD) or less, although some histochemical staining reagents (e.g., Alcian blue, phosphomolybdic acid (PMA)) may have a molecular weight of up to two or three thousand kD. An example of a high molecular weight histochemical staining reagent is alpha-amylase (about 55kD), which can be used to indicate glycogen.

[0090] Another type of tissue staining is immunohistochemistry (IHC, also called "immunostaining"), which uses a primary antibody that specifically binds to a target antigen of interest (also called a biomarker). IHC can be direct or indirect. In direct IHC, the primary antibody is directly conjugated to a marker (e.g., a chromophore or fluorophore). In indirect IHC, the primary antibody first binds to the target antigen, and then a secondary antibody conjugated to a marker (e.g., a chromophore or fluorophore) binds to the primary antibody. The molecular weight of IHC reagents is much higher than that of histochemical staining reagents, as the molecular weight of antibodies is approximately 150 kD or higher.

[0091] The slices can then be individually mounted on corresponding slides, which can then be scanned or imaged by the imaging system 150 to generate raw digital pathology images 125a to 125n. Each slice can be mounted on a slide, which can then be scanned to create a digital image that can be subsequently reviewed by digital pathology image analysis and / or interpreted by a human pathologist (e.g., using image viewer software).

[0092] The tagging GUI controller 160 can generate, configure, or update a GUI to present a given digital pathology image associated with a user. The given digital pathology image can be an image queued for segmentation and tagging processing or an image that has been identified in a request for segmentation and tagging processing. The GUI can be configured to receive one or more annotations. For example, the GUI can include an input component that allows a user to select a group of pixels in a digital pathology image and indicate, for each pixel in the group of pixels, a tag (e.g., a first type of region, a second type of region, or background) that will (or will not) be assigned to the pixel.

[0093] The segmentation controller 120 can use the trained deep learning model 115 to process each digital pathology image 125a to 125n and any annotations to generate segmentation data 155. For each segmentation detected by the deep learning model 115, the segmentation data identifies the location of the segmentation within the corresponding digital pathology image 125. For example, for a given segmented region, the segmentation data 155 can identify pixels in the boundary of the region and / or pixels in the segmented region. In some cases, the segmentation data 155 includes an annotation image that indicates for each of one or more pixels whether the pixel has been assigned to a type of region or segmented region, and if so, which region. For example, the annotation image can indicate which category of a predefined set of categories (e.g., a first type of region, a second type of region, or a background) the pixel has been assigned to. Although the categories may lack semantic meaning, training can be performed so that any category assignment may be consistent with any annotation provided with the corresponding digital pathology images 125a to 125n. As another example, each segmented region can be assigned an identifier that is unique relative to other segmented regions in the image. For each segmented region, pixels within the segmented region (or alternatively or additionally being part of a boundary of the segmented region) may be assigned a value equal to the unique identifier.

[0094] The labeling GUI controller 160 can generate, configure, or update a GUI to present a given DP image (e.g., at reduced intensity) and show the boundaries of each segmented portion. When the deep learning model is configured to assign labels, the labels can also be represented (e.g., via shading, background color, symbol color, etc.).

[0095] The GUI may also be configured to receive input identifying or updating a region label for each region. The GUI may be configured to allow the user to zoom in and out of the image and / or move the field of view. This may allow the user to better review details such as the morphology and size of cells within a given segmented region.

[0096] The GUI can be configured so that for each segmented region, the user can select from a predefined set of region labels. When label data is provided as part of the training data, the deep learning model may have generated a predicted category for each region. In this case, an indication of the predicted category can be presented for each segmented region (e.g., via shading, background color, symbol color, symbol shape, etc.). When label data is not provided as part of the training data (or when insufficient label data is provided), all regions can be represented as having indications that region labels are to be assigned, but the region labels can be subsequently assigned to each segmented region based on the label selection input provided by the user (and the representation can be updated accordingly). In other cases, the initial region label is automatically assigned to each region, and when the user believes that the initial label is incorrect, the user can selectively provide input to change the label. In this case, the same initial region label (e.g., the most commonly used label statistically) can be assigned to all regions, or one or more rules can be evaluated to determine which initial region label to assign to each region (e.g., based on the area of ​​the region, the perimeter of the region, the perimeter of the region to the area ratio, the number of cells detected in the region, etc.).

[0097] In any case, the GUI can be configured to receive input that identifies (e.g., as an initial indication or an update indication) a label to be assigned to one or more portions of the image. For example, a user can click on a color-coded marker to identify an initial label using a radio button list, and can then provide input identifying a different label. As another example, a user can click within a segmented region description, and can then use defined mouse click characteristics to identify an initial (e.g., and final) label for the region.

[0098] In some cases, a different type of mouse click and / or touch screen interaction may be assigned to each of the predefined set of region markers. For example, each of the following may be assigned to one of the region marker options: (1) a single left mouse click; (2) a single right mouse click; and (3) a double left mouse click. Figure 7 and associated descriptions provide illustrations of the mapping of mouse clicks to labels. As another example, one of the area labeling options may be assigned each of the following: (1) briefly hold and then slide slightly downward; (2) briefly hold and then slide slightly upward; and (3) briefly hold and then slide slightly to the left.

[0099] GUI can also include the option of all current unlabeled segmented regions or all current unlabeled segmented regions completely in the visual field in order to (for example) assign a given area mark (which can be selected by the user).Therefore, for example, in a given visual field, the user can then choose to mark all matrix regions and background regions, and then indicate that all other regions will be marked as tumor regions.GUI can alternatively or additionally be configured to allow the user to select the tool in order to define the rectangular region in the depicted image and then all regions in the region are assigned to the regional mark of selection.

[0100] As each tag is assigned or updated, the GUI can be updated to reflect the tag (e.g., by adding a background color to the area that can represent the assigned tag) so that the user can easily see the current tags (and, if applicable, which areas are still untagged). The GUI can include a submit or complete input option that can be selected by the user when the user is satisfied with the tags. Further details of the GUI are provided herein, such as when describing Figure 8 Show.

[0101] The tag GUI controller 160 may store and / or output region tags 160a to 160k. The region tags 160a to 160k may, for example, include a mapping associating each unique region identifier with a region tag (or region tag identifier). The region tags 160a to 160k may alternatively or additionally include a mapping associating, for each pixel of some or all pixels in an image, the pixel with a region tag (or region tag identifier).

[0102] Example interactive segmentation workflow

[0103] refer to Figure 2 , depicting an embodiment of a multi-class interactive segmentation workflow 200. Various actions in the multi-class interactive segmentation workflow 200 may be performed by the model training system 105, the segmentation controller 120, and / or the labeling GUI controller 160. Figure 3 An example of a corresponding pipeline for model building is depicted, according to various embodiments.

[0104] In various embodiments, the multi-class interactive segmentation workflow 200 designs and builds a deep learning model to segment multiple classes and receives user input identifying a label for each of one or more pixels and / or segmented regions. The interactive GUI can be configured to receive an input image and possibly initially receive annotation input identifying a class to which a fixed point or segment is to be assigned and / or receive post-training input identifying or updating a label to be assigned to a given segmented region.

[0105] For example, the first dataset 210 may include multiple images (e.g., multi-category images 110a to 110n) from one or more categories (e.g., DP images, natural scene images, immunohistochemistry images, H&E images, or any other relevant images) and may include corresponding annotations (e.g., identifying click annotations, different segments within an image, ground truth maps, etc.). A portion or all of the first dataset 210 may have been received from a user via an interactive GUI that is facilitating training a deep learning model (e.g., deep learning model 115) connected to the interactive GUI.

[0106] The plurality of images may include digital pathology images featuring different tissue types and their corresponding segmentation annotations. In other embodiments, a first data set 210 from another domain (e.g., natural scene images) may be used. The first data set 210 may include, for example, annotation data associating each of one or more pixels or regions with a given category. The category may be one of a plurality of predefined non-semantic categories.

[0107] A deep learning model may have been designed and built by defining the model’s architecture, hyperparameters, and loss function ( Figure 3 1 ). At 220, a deep learning model can be trained (e.g., by the model training system 105) using the first dataset 210. By using datasets from multiple domains, the resulting model trained using the data can be more generalizable than using training data from a single domain.

[0108] A GUI can also be developed that is configured to display the digital pathology image and the segmentation data (e.g., by overlaying segmentation boundaries on the image) and receive user input identifying a label (e.g., a label from a predefined set of labels) for each slice or to correct the predicted label. (See Figure 3 The interactive GUI may be, but need not be, the GUI that receives part or all of the first data set. Even if the GUIs are the same, the GUIs may be configured to present a first interface to the first user for receiving part or all of the first data set, and present a second interface to the second user for receiving tag definitions or updates.

[0109] The GUI of Part 2 can be connected to the model (see Figure 33 of the pipeline). This connection can allow a user to identify (e.g., upload, select, etc.) one or more digital pathology images and / or checkpoint annotation data (e.g., a second data set of images) and request processing via the GUI. The request can automatically trigger processing by the deep learning model to identify the segmentation, and once the segmentation is identified, the GUI can automatically display the image and segmentation data and be configured to receive a label identification or update. In some embodiments, an initial label (e.g., a non-semantic label) can be assigned to each of the one or more segmented regions. In some embodiments, at least one segmented region lacks an initial label.

[0110] Using the interactive GUI 230 (e.g., controlled by the marking GUI controller 160), the user can efficiently identify and / or update the markings of at least some of the segmented regions. For example, in various embodiments, the user can use a mouse and click on the region to provide an annotation. The interface can be configured so that annotations can be made quickly and efficiently (e.g., identifying new or updated markings), and so that the GUI is updated in real time to reflect the latest markings of the region. For example, in various embodiments, different colors can be used to annotate different categories. For example, blue dots can be used to annotate matrix regions, green dots can be used to annotate tumor regions, and red dots can be used to annotate background regions. In addition, multiple annotations can be used for one or more of the categories.

[0111] In various embodiments, the trained deep learning model 235 is not further modified after the initial training (using multi-class data). Instead, the deep learning model 235 outputs a segmentation for the image, and the interactive GUI is configured to receive input that quickly identifies labels for each of some of all segmented regions.

[0112] In one embodiment, the deep learning model is used to segment image blocks of 600x600 pixels in less than one minute. Therefore, users can annotate images or perform image segmentation quickly and efficiently. In addition, the model will not overfit to a specific data set without the need for model retraining or fine-tuning during use, but will maintain generalization. In addition, the system provides confidence to the user because the process and output are not black boxes (such as traditional deep learning systems), but instead enable user-guided modifications to model predictions, which provides real-time feedback and is therefore interactive, intuitive, and flexible for the end user.

[0113] Exemplary segmentation method using limited annotations

[0114] refer to Figure 4, depicting an exemplary process 400 for interactive segmentation. In various embodiments, a multi-category deep learning system receives an image to be segmented (step 410). In some embodiments, the image can be a complete image (e.g., a full slide digital pathology image) or a portion of an image (e.g., a field of view of a complete image, a region of interest of a complete image, a partition of a complete image, etc.). In various embodiments, a user can provide one or more annotations for the image (step 415). Such annotations can include, for example, click annotations that can potentially be further processed to be transformed into other annotations using a transformation algorithm. In various other embodiments, the user may not provide annotations before the initial segmentation.

[0115] A deep learning model can be used to segment the image (and its annotations, if available) (step 420). In various embodiments, the model can be an n-category model. Thus, the model can be configured to segment the input image into up to n types of different regions in each round of user interaction, which generates a single segmentation mask with up to n categories of regions. For example, in various embodiments, the deep learning model is a three-category model configured to classify the input image into regions including tumor cells, stromal cells, and background tissue. In other embodiments, the deep learning model is a four-category model configured to classify the input image into four different regions including immune cells, tumor cells, stromal cells, and background cells. It will be appreciated by those skilled in the art that the deep learning model can be configured for any number of categories and to segment any type of tissue or cell.

[0116] The deep learning model output provides the segmented image to the interactive GUI, and the result is displayed to the user (step 430). The user can then choose to provide annotations using the interactive GUI (step 440). When the user provides annotations, the interactive GUI provides the annotations to the deep learning model as additional input (step 450), and the process can be repeated as many times as necessary to achieve a satisfactory result.

[0117] Examples

[0118] Example 1

[0119] Traditional interactive segmentation models (both optimization-based methods and learning-based algorithms) are designed to segment a single object at a time. These models use binary segmentation to simultaneously identify two different types of regions: foreground objects and non-target background. Foreground objects (referred to as "foreground") are defined as target object instances in an image, while non-target background is any other object except the target object.

[0120] The techniques disclosed herein can generate robust results, where a target object can be segmented quite successfully regardless of whether the target object is a single connected region or multiple unconnected regions of the same class.Interactive binary segmentation is used for multi-class segmentation of DP images.

[0121] Figure 5 A to 5H illustrate the steps performed when segmenting a digital pathology image using binary segmentation to identify three types of regions. Figure 5 A shows an exemplary colon cancer H&E image segmentation. Figure 5 B shows the first round of binary segmentation identifying tumor (red masked area) and non-tumor (the remaining unmasked area). Figure 5 B depicts the same image partition after a first round of binary segmentation using a deep learning model to identify tumors (red masked areas) and non-tumors (the remaining unmasked areas). In this example, the user has annotated the image using the interactive GUI 130 with green dots for positive clicks on tumor areas and red dots for negative clicks on non-tumor areas. Figure 5 C shows an exemplary updated tumor region segmentation result (red masked area). Therefore, although the tumor region is unconnected in this image, it is still identified.

[0122] Figure 5 D depicts the image after the second round of segmentation of the identification matrix region and non-matrix region. At this point, the green dots indicate the first positive click to identify the matrix region. The green masked region is the matrix region identified in the second round of segmentation after the first positive green click, and the red masked region is the tumor region identified by the first round of segmentation of tumor and non-tumor region. In the described example, after the first green click, the prediction of the matrix region extends to the tumor region previously identified, because the second round of segmentation may be independent of the first round.

[0123] Figure 5 E depicts an example of a corrected matrix segmentation result, where red dots (negative clicks) have been added. Figure 5 F depicts a series of click annotations that modified the second round of segmentation. Figure 5 G depicts the final segmentation results of the tumor, stroma, and background regions, where light blue corresponds to the stroma region, gray corresponds to the tumor region, and black corresponds to the background region. Figure 5 H depicts the red rectangular area (from Figure 5 G) is a magnified view of the segmentation result. The upper image includes the segmentation result superimposed on the original image, while the lower image shows the segmentation mask used. In this example, the arrow points to the gap after combining the masks generated from the two rounds of segmentation.

[0124] Since each round of binary segmentation is independent, the user needs to correct the already segmented objects from the previous round ( Figure 5 D to 5E). This prolongs the analysis time because previously identified regions need to be corrected in addition to the target class in the current round of segmentation ( Figure 5 F). Such corrections to remediate annotated regions are redundant and time-consuming. Furthermore, the user needs to carefully provide click annotations via clicks at the boundaries of the target classes, otherwise gaps often appear between segmented objects from different rounds of segmentation ( Figure 5 G to 5H), and thus inevitably leads to confusion and errors.

[0125] Example 2

[0126] Figure 6 An exemplary representation of the use of a three-class deep learning model 600 is shown. For the three-class model, the ground truth mask contains up to three different types of regions, and the user annotations contain up to three groups of user annotations, where each group of user annotations corresponds to one type of region.

[0127] The ground truth mask used for each segmentation task is an instance-level annotation in which multiple object instances are annotated. For example, during training, up to two objects are sampled from the mask to generate a new mask in which the rest of the image is considered as non-target background. This non-target background can contain (a) other objects not selected from the original mask and (b) pure background (in which no object is annotated in the original mask).

[0128] The training data for the three-class deep learning model includes images with click annotations 610. The colored dots represent mouse clicks on the target region. The location and number of dots are simulated for model training to mimic user input at test time, where the user provides mouse clicks to guide the segmentation of the target region.

[0129] Simulated click annotations are used during training. For example, mouse click annotations are encoded into disk maps or Euclidean distance maps, which form additional masks that can be conveniently fed into the deep learning model together with the training images. Various strategies for click simulation can be used for binary interactive segmentation, including positive click sampling and negative click sampling. Positive click sampling imitates the user's mouse click behavior on the target object by randomly sampling a variable number of pixel positions within the target object area in the mask. Negative click sampling imitates the user's behavior of identifying the background (i.e., anywhere not on the target object). In various embodiments, three strategies for negative click sampling can be used. For example, a first strategy can include using a variable number of pixel positions randomly sampled from a pure background area. A second strategy can include using a variable number of pixel positions sampled from an unselected object in a ground truth mask. A third strategy can include using a variable number of pixel positions sampled from the pure background close to the boundary of the target object.

[0130] In this example, to train a three-class deep learning model, negative click sampling is used for one type of region, while positive click sampling is used for the other two types of regions. It should be understood that negative clicks may be better for effectively correcting mis-segmented regions. Therefore, reserving this negative click sampling for one of the categories can provide users with the flexibility to perform negative clicks on the most difficult category of regions to segment.

[0131] The image with annotations 610 is split into an image 620 and annotations for each object / class segmented by the deep learning model 600. For example, a three-class deep learning model includes a first class annotation input 630, a second class annotation input 640, and a background annotation input 650. The image 620 and annotations 630, 640, 650 are then used to generate a ground truth mask 660, which is used to train a deep learning model 670, which can then generate predictions 680.

[0132] Example 3

[0133] Figure 7 An example workflow of providing annotations via an interactive GUI configured to provide user input functionality for a variety of segmentation categories is depicted. Annotations are provided by a user using various user inputs 710, and the GUI is generated on a display 720. The user input 710 can include any computer input device, such as a mouse, touch screen, keyboard, or other input device. The user input 710 includes a mouse with at least two buttons, including a left click button and a right click button.

[0134] In a typical binary object segmentation GUI, a left mouse click is used as a positive click (i.e., the location is part of a category), while a right click button is used as a negative click (i.e., the location is not part of a category). Using a multi-category interactive GUI, the left and right click buttons can be mapped to categories. For example, in a three-category system, a left mouse click 712 can be associated with a first category, a right mouse click 714 can be associated with a second category, and a combined mouse click 716 can be associated with a third category. Each of the mouse clicks can cause the interactive GUI to overlay a dot corresponding to the category associated with the click onto the image 728.

[0135] The left mouse click 712 can be associated with a first type of region (e.g., a tumor region), and each left mouse click triggers the display of a green dot 722 on the interactive GUI via the display 720. The left mouse click 712 can act as a positive click. The right mouse click 714 can be associated with a second type of region (e.g., a matrix region), and each right mouse click triggers the display of a red dot 724 by the GUI, and can also act as a positive click. Double left mouse clicks 716 can be associated with a third type of region and can produce a yellow dot 726. In various embodiments, double left mouse clicks 716 can act as negative clicks. In order to connect the trained model with the interactive GUI, the GUI backend is configured to convert the pixel locations of the user clicks for the three categories into three disk maps, and use these disk maps together with the image as deep learning model inputs to generate predictions for the corresponding three categories.

[0136] Using an interactive GUI, the user can initiate segmentation by adding a first positive click on the target destination region. This click triggers the initial network prediction. Adding additional clicks triggers changes in the network input, which results in corresponding changes in the segmentation prediction. The segmentation results can be displayed as a color overlay on the input image.

[0137] Example 4

[0138] Figure 8Depicted is an example interactive GUI for defining click annotations. The interactive GUI includes an image display area 800 and a control panel 810. The image display area 800 is configured to show a segmented image and provide a user with an area to add annotations. The control implements a variety of options for operating the system. For example, the interactive GUI may include buttons for completing 820, undoing clicks 830, and resetting clicks 840. The GUI control 810 may also include various zoom options 850. In various embodiments, the user may be able to use the GUI control to adjust various model parameters. For example, the GUI control 810 may include a back propagation refinement (BRS) option 860 and a slider to modify a prediction threshold 870 and an alpha blending coefficient 880. Other control options include a slider 890 for a dot radius.

[0139] Example 5

[0140] Whole slide image segmentation is performed using a block-by-block approach. Block-by-block prediction for WSI. Using the block-by-block approach, the user interactively segments one image block at a time. In this way, the user segments a block to their satisfaction in the manner described above, and then moves on to the next block. The destination platform (web application, GUI, etc.) is selected to display whole slide images and support interactive segmentation. Once all blocks are segmented, the platform can then automatically assemble all predictions and display the whole slide segmentation results.

[0141] Example 6

[0142] Whole-slide image segmentation was performed using a radial expansion scheme. Figure 8 Depicted is a visualization of a radial expansion strategy for whole slide interactive segmentation according to various embodiments. The radial expansion method exploits the benefit of predictions in terms of superimposed tiles to expand the predictions in a radial manner and thus speed up segmentation in larger images such as WSI ( Fig. 9 ). For example, when a user clicks anywhere in the WSI (e.g., Fig. 9 ), multiple tiles covering this hit are selected to maximize the total tile coverage, so that predictions can be generated for all of these tiles ( Fig. 9 In the next step, the user can click anywhere in the predicted area that needs correction (for example, the second click is in Fig. 9 Correcting clicks helps extend the prediction to the tiles covering these clicks ( Fig. 9). Since one user click triggers segmentation in multiple tiles, utilizing the designed radial expansion scheme for this prediction enables faster propagation of the segmented regions in the whole slide image and thus speeds up the whole slide image segmentation. As another advantage of this approach, the system can utilize parallelization or distributed model inference of multiple tiles to further speed up the whole slide image segmentation.

[0143] Example 7

[0144] Domain-specific and dataset-specific whole slide image segmentation is provided. In the use case where segmentation of a specific dataset with multiple whole slide images is required, interactive segmentation can serve as both an initial annotation generation strategy and a rapid annotation expansion strategy. For example, first, a subset of slide image regions can be selected manually, randomly, or via an active learning-based approach. Then, one of the above methods (i.e., block-by-block prediction or radial expansion strategy) can be utilized to efficiently generate segmentation masks for these selected larger image regions in the selected WSI, followed by iterative model training and fine-tuning. In this way, a model can be quickly built to perform WSI segmentation on the target dataset.

[0145] Example 8

[0146] The deep learning model is trained on the Semantic Boundary Dataset (SBD), a common benchmark for interactive segmentation that leverages instance-level annotations from the natural scene image domain. SBD contains 11,355 images, 8,498 of which are used for training and 2,857 for validation. In each image, one or more foreground objects (i.e., instances) are annotated.

[0147] HRNet with an OCR module is pre-trained on ImageNet as the backbone, a recently proposed architecture for recognizing fine details in many vision tasks including image segmentation and human pose estimation. To incorporate simulated user clicks into model training, convolutional blocks are encoded separately for images and disk maps from user input. Note that this backbone can be replaced with another model of choice to suit specific use cases. In addition, domain-specific annotated datasets (e.g., IHC images with corresponding masks if available) can be used to train this interactive system.

[0148] Implementation details: Three-category click sampling is performed dynamically during training. Specifically, for each image, up to two foreground instances (i.e., individual connected objects in the ground truth mask) are randomly selected from the segmentation mask. Together with the pure background annotations, a new mask with up to three different regions is generated. Two of the regions correspond to target objects, while one corresponds to non-target background. From this new mask, positive clicks are randomly placed within each foreground object region, while negative clicks are placed in the background region with the same strategy. Images are cropped into 320x480 patches for training and augmented by random rescaling, flipping, and color jittering.

[0149] Use an interactive platform to segment four different categories of images, such as Fig.10 shown. Specifically, Fig.10 Images depicted include: (A) model performance on example colon cancer H&E image segments, (B) model performance on example IHC images from breast cancer tissue targeted with Ki67 (brown DAB stain) and counterstained with hematoxylin (blue stain), (C) example IHC images from lung cancer tissue targeted with PDL1 (pink Tambra stain) and CK7 (yellow Dabsyl stain) counterstained with hematoxylin (blue), and (D) example chest X-ray images. In the bottom images of A to C, the light blue area is the stroma, the gray area is the tumor, and the black area is the non-target background. In the bottom image of D, the light blue area is the right lung, the gray area is the left lung, and the black area is the non-target background. Predictions show that the proposed model design and training strategy can successfully segment images interactively for three categories of regions, such as Fig.10 shown.

[0150] Some embodiments of the present disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions, which, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium, which includes instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes disclosed herein.

[0151] The terms and expressions that have been adopted are used as terms of description rather than limitation, and when such terms and expressions are used, there is no intention to exclude any equivalents of the features shown and described or parts thereof, but it should be recognized that various modifications are possible within the scope of the claimed invention. Therefore, it should be understood that although the claimed invention is specifically disclosed through embodiments and optional features, those skilled in the art may make modifications and changes to the concepts disclosed herein, and such modifications and changes are considered to be within the scope of the invention as defined in the appended claims.

[0152] This description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the description of the preferred exemplary embodiments will provide a feasible description for implementing various embodiments for those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope set forth in the appended claims.

[0153] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it should be understood that the embodiments can be practiced without these specific details. For example, circuits, systems, networks, processes, and other components can be shown as parts in block diagram form to avoid obscuring the embodiments in unnecessary details. In other cases, well-known circuits, processes, algorithms, structures, and techniques can be shown without unnecessary details to avoid obscuring the embodiments.

Claims

1. A computer-implemented method, the method comprising: receiving an image and displaying said image using a graphical user interface; receiving at least one first image annotation provided by a user via the graphical user interface; generating a first segmented image using a deep learning model, wherein the deep learning model uses the digital pathology image and the at least one first image annotation; as well as displaying the first segmented image using the graphical user interface; receiving at least one second image annotation provided by the user via the graphical user interface; generating a second segmented image using the deep learning model, wherein the deep learning model uses the digital pathology image, the at least one first image annotation, and the at least one second image annotation; and The second segmented image is displayed using the graphical user interface.

2. The method of claim 1, wherein the deep learning model is a three-category model and the at least one first image annotation comprises at least one annotation for each category.

3. The method of claim 1 or 2, wherein the at least one image annotation provided by the user is provided using a mouse click on the displayed image. The method of claim 3 , wherein the mouse click comprises a left mouse click associated with a first segmentation category. The method of claim 4 , wherein the mouse click comprises a right mouse click associated with a second segmentation category. The method of claim 4 , wherein the mouse clicks comprise combined mouse clicks associated with a third segmentation category.

7. The method of claim 6, wherein the combined mouse click comprises a double left mouse click.

8. The method of claim 1, wherein the deep learning model is trained using a first dataset comprising training images having one or more annotations.

9. The method of claim 8, wherein the training images comprise images from a first image domain.

10. The method of claim 9, wherein the training images further comprise images from a second image domain.

11. The method of claim 8 or 9, wherein the second image domain is different from the first image domain, and wherein the first image domain and the second image domain each include one of the following: natural scene images, digital pathology images, immunohistochemistry images, x-ray images, and hematoxylin and eosin images.

12. The method of claim 8, wherein the one or more annotations include one or more simulated click annotations.

13. The method of claim 1, wherein the image is a whole slide pathology image.

14. A system comprising: one or more data processors; as well as A non-transitory computer-readable storage medium comprising instructions that, when executed on the one or more data processors, cause the one or more data processors to perform a set of operations, the set of operations comprising: receiving an image and displaying said image using a graphical user interface; receiving at least one first image annotation provided by a user via the graphical user interface; generating a first segmented image using a deep learning model, wherein the deep learning model uses the digital pathology image and the at least one first image annotation; and displaying the first segmented image using the graphical user interface; receiving at least one second image annotation provided by the user via the graphical user interface; generating a second segmented image using the deep learning model, wherein the deep learning model uses the digital pathology image, the at least one first image annotation, and the at least one second image annotation; and The second segmented image is displayed using the graphical user interface.

15. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product comprising instructions configured to cause one or more data processors to perform a set of operations comprising: receiving an image and displaying said image using a graphical user interface; receiving at least one first image annotation provided by a user via the graphical user interface; generating a first segmented image using a deep learning model, wherein the deep learning model uses the digital pathology image and the at least one first image annotation; as well as displaying the first segmented image using the graphical user interface; receiving at least one second image annotation provided by the user via the graphical user interface; generating a second segmented image using the deep learning model, wherein the deep learning model uses the digital pathology image, the at least one first image annotation, and the at least one second image annotation; as well as The second segmented image is displayed using the graphical user interface.

Images