Image processing based object classification using self-bootstrapping with region level annotations

Abstract

The technology of this disclosure relates to object classification using self-bootstrapping with region-level annotation. For each image of a plurality of images, regions within the image can be identified. For each region, a region-specific label can be identified, a set of objects within the region can be detected, and an object-specific label can be assigned to each object. The object-specific label can be the same as the region-specific label assigned to the region in which the object is located. For each image of the plurality of images, a training data set can be defined to include object location data (indicating image- within location data for detected objects) and label data (indicating the object-specific label assigned to the objects). An image processing model can be trained using the training data. The training can include learning values of a set of parameters that define computations performed by the image processing model.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit and priority of U.S. Provisional Application No. 62 / 929,100, filed on October 31, 2019, which is incorporated herein by reference in its entirety for all purposes. Background Technology

[0003] Typically, images are processed to detect different objects within each image and to classify each of those different objects. This processing can involve manual work and / or automated methods. An exemplary use case is in a biological setting, where histological images are processed to identify individual cells and the cell type for each cell. Performing this processing manually can be very time-consuming and financially expensive, especially when reviewing large numbers of images (which is often the case). For example, a whole-slide histological image can be very large and contain descriptions of thousands of cells. Furthermore, this labeling may require significant domain expertise.

[0004] One way to speed up this process is to use automated techniques to process images. For example, deep learning models can be used to assign labels to individual objects (e.g., individual cells) within an image. However, the size of the training dataset can strongly influence the sensitivity and specificity of this approach. Training data typically consists of annotated images (which identify the location of objects within the image and object-specific classifications) traditionally produced as a result of manual image evaluation.

[0005] If manual evaluation incurs the aforementioned significant time and financial costs, obtaining a large training set can be costly and / or difficult. Therefore, training image processing models (e.g., deep learning models) to accurately perform these tasks can be challenging. Summary of the Invention

[0006] In some embodiments, bootstrapping techniques are used to process images for object detection and classification. More specifically, each of one or more images may be processed to initially identify higher-level regions within the image and associate each higher-level region with a region-specific label. For example, a higher-level region may be defined as a portion of an image defined by a user-input contour, and the region-specific label may include a cell type indicated by the user input. Lower-level objects within the region may then be detected (e.g., using edge detection techniques, contrast analysis, intensity analysis, color analysis, etc.). Lower-level objects may include (or) individual cells or cell nuclei.

[0007] An object-specific label can be assigned to each of the lower-level objects in the lower-level objects that are within the higher-level region, the object-specific label being the same as the region-specific label assigned to the higher-level region. An image processing model (e.g., a machine learning model such as a deep neural network and / or a convolutional neural network) can be trained. Post-processing can include a manual review of a subset of the automatically detected objects that identified the associated object-specific label. For example, one or more presentations can be generated and outputted that include some or all of the images and identification of the subset of objects and their assigned object-specific labels. Any input received can be used as an indication as to whether the object-specific labels were accurate (or whether other labels corresponded to the identified objects). The image processing model can be retrained with such feedback. Thus, the image processing model can be trained to have high specificity and sensitivity while requiring relatively less input to indicate manual review.

[0008] In some embodiments, a computer-implemented method is provided. For each image in a set of images, a plurality of regions within the image can be identified. For each region in the plurality of regions, a region-specific label can be identified for each region, a set of objects within the region can be detected, and an object-specific label can be assigned to each object of the set of objects. The object-specific label can be the same as the region-specific label assigned to the region in which the object is located. For each image of the set of images, a training data set can be defined to include object location data (indicating, for each object in the detected set of objects, location data within the image for the object) and label data (indicating, for each object in the detected set of objects, the object-specific label assigned to the object). An image processing model can be trained using the training data. The training can include learning a set of parameter values defining a set of computations performed by the image processing model.

[0009] Some embodiments of the disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed by the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes disclosed herein. Some embodiments of the disclosure include a computer program product tangibly embodied in a non-transitory machine- readable storage medium including instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes disclosed herein.

[0010] The terms and expressions employed herein are used as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the application claimed. Thus, it should be understood that although the present application has been specifically disclosed by embodiments and optional features, modification and variation of the concepts herein disclosed can be resorted to by those skilled in the art, and that such modifications and variations are considered to be within the scope of this application. BRIEF DESCRIPTION OF DRAWINGS

[0011] The present disclosure is described in relation to the following drawings:

[0012] FIG. 1 An image processing interaction system including multiple devices / systems configured to support inter-device communication and image processing is shown.

[0013] FIG. 2 How individual regions within an image can include multiple object types is shown.

[0014] FIG. 3 A process 300 for generating and training a machine learning model to detect and label objects within an image is shown.

[0015] FIG. 4A-4E An image with respect to region annotations is shown.

[0016] FIG. 5A-5F Results of applying a non-machine learning model and image processing heuristics to produce bootstrap, noise kernel annotations from region annotations are shown.

[0017] FIG. 6A-6C Object labels produced by a model trained with bootstrap annotations of three histology images are shown.

[0018] FIG. 7A-7C Object-specific labels and region-specific labels for a histology image are shown.

[0019] FIG. 8A-8D Specific metrics for determining which object-specific labels are available for a reviewer to use are shown.

[0020] FIG. 9A-9B Patch selection and representation for reviewing selected object-specific labels are shown.

[0021] FIG. 10 A process for assigning labels to objects in an image using machine learning is shown.

[0022] In the drawings, like parts and / or features can have the same reference label. Also, various parts of the drawings can be referred to, by the first reference label plus a dash and a second reference label that distinguishes the part. If only the first reference label is used in the text, the description is applicable to any one of the parts having the same first reference label irrespective of the second reference label. DETAILED DESCRIPTION

[0023] I. Image processing interactive system

[0024] FIG. 1 An image processing interaction system 100 is shown that includes a plurality of devices / systems configured to support inter-device communication and image processing. A user device 105 can send an image processing request to an image processing system 110. The image processing request can be sent through an interface, such as an online interface. The image processing request can identify one or more images to be processed. For example, the image processing request can include an identifier (e.g., of a subject and / or sample) that can be used to retrieve images from an image source 115. The image processing system 110 can then send a request to the image source 115 for images including the identifier and authorization data, which the image source can return. As another example, the image processing request can include images that have been uploaded at the client device 105 or associated with the images.

[0025] The images can include histology images, such as H&E images that depict H&E staining. The images can include another immunohistochemistry (IHC) image (by definition, which uses a stain other than H&E). A single sample can be associated with corresponding H&E and IHC images. The H&E and IHC images can be co-registered (using rigid or deformable image registration). The H&E and IHC images can then be used for different evaluations (e.g., where a region is detected with the H&E image and object information for a marker is identified with the IHC image). The images can include brightfield images, non-brightfield images (e.g., electron microscopy images), and / or scanned whole slide histology images.

[0026] The image processing system 110 can include a region-specific labeler that processes each image to identify one or more regions within the image. Each region can correspond to a histology region and can correspond to, for example, a tumor region, a stroma region, or a lymphocytic infiltrate region. Each region can correspond to, for example, an area within the region where cells predominantly and substantially have similar visual features (e.g., in size, shape, intensity, and / or color). The label for each region can indicate, for example, a biological structure that corresponds to the region and / or an important cell type of the region.

[0027] The region-specific labeler 120 can define and / or label each region automatically, semi-automatically, or based on user input. For example, a graphical user interface can be used by the user device 105. The graphical user interface can present a portion or all of the image and can include tools that can be used to identify boundaries of individual regions (e.g., lasso tool, pencil tool, box select tool, oval select tool, etc.). The interface can further accept input received at the user device 105 that identifies a label for a region. The input can be accepted at a window or pop-up component of the interface that can include a text box for accepting a free-form label for a region or a selection component for selecting a label from a set of potential labels. The selection component can include a drop-down menu or a plurality of selectable radio buttons.

[0028] As another example, an automated or semi-automated method can process the image to detect important edges and can identify and / or propose regions accordingly (e.g., via a representation in the interface). The automated or semi-automated method can further or alternatively identify image metrics (e.g., intensity, hue, and / or contrast) for each patch in a set of patches within the image. Individual regions can be defined to correspond to sets of image patches having similar image metrics.

[0029] The object detector 125 can detect objects within the image. Each of the detected objects can include (or be) a biological structure, such as a cell or a nucleus. The objects can be detected automatically using image processing. The image processing can include using connected component analysis, edge detection, color (or transformed color) channel-specific analysis, smoothing algorithms, thresholding algorithms, etc. For example, the image processing can include performing stain “separation” using an established method, such as a method based on Macenko principal component analysis (PCA) or a method relying on sparse non-negative matrix factorization (SNMF). For each region label, a stain channel (e.g., a hematoxylin channel) can then be used (e.g., multiplied) to transform a region prior (e.g., a binary region prior). One or more smoothing and / or thresholding processes can be performed to obtain boundaries of nuclei (e.g., by iteratively applying Gaussian smoothing and Otsu thresholding). Connected component analysis can then be performed to isolate the nuclei objects.

[0030] The object-specific labeler 130 can assign an object-specific label to each detected object. Initially, each object-specific label can be defined to equal the label associated with the respective region. The object-specific labeler 130 can identify an incomplete subset of the set of objects for which each object in the incomplete subset is to be reassigned or assigned to a non-cell label. The non-cell label can indicate that the object is predicted not to be and / or not to correspond to a cell. The incomplete subset of objects can include, for example, objects having an area that is below and / or above a respective predetermined threshold area and / or having an aspect ratio, circularity, and / or area that does not correspond to a benchmark instance.

[0031] The model controller 135 can then train a machine learning model using the object-specific labels. The machine learning model can include, for example, a deep neural network, a recurrent neural network, and / or a convolutional neural network. The training can use overlapping slices or patches for the center or middle of each slice or patch and / or restrict the model predictions, which can reduce edge artifact effects. In some cases, post-processing can be defined and / or trained using the object-specific labels, but need not be. The post-processing can use binary erosion, binary dilation, Gaussian smoothing, or other techniques to suppress segmentation artifacts.

[0032] After training the machine learning model (e.g., and post-processing techniques), the model controller 135 can use the model to process image data corresponding to the same images and / or different images used to train the model. The image data can include a portion or all of the images and object-specific data identifying which portions of the images correspond to different objects. The output of the machine learning model (and / or post-processing) can include an object-specific label for each detected object. It should be understood that the object-specific labels generated using the trained model can differ from the object-specific labels initially assigned based on the region-specific labels.

[0033] The object-specific labeler 130 can modify selected labels. An individual population-based approach can be used to identify selected objects associated with unusual (e.g., having a prevalence below a predetermined threshold) sizes, shapes, intensity distributions, contrast distributions, and / or color distributions. For each of these selected objects, the label assignment can be modified and / or a new label can be defined. For example, an unusual shape of an object can represent a cell that is necrotic. Thus, the shape itself does not necessarily indicate a cell type, but can indicate a physiological stage. As another example, the rarity of a given label in a particular context (e.g., vascular endothelial cells in an image corresponding to breast cancer) can trigger a reassignment of the label. The newly assigned label can include a label indicating that the label is not to be used in training the model to classify objects.

[0034] The model evaluator 140 can select objects to be reviewed. The objects can be selected based on a confidence metric associated with the label assignment, for example. The confidence can indicate a degree to which one or more features of the object (e.g., size, shape, intensity distribution, contrast distribution, and / or color distribution) correspond to or match one or more features of the object type associated with the assigned label. A trained machine learning model can output a confidence metric associated with each label output (corresponding to an individual output). Selecting objects for review can include some or all objects associated with a confidence metric below an absolute threshold or a relative threshold (e.g., within a lowest 5% confidence range of the object set).

[0035] The model evaluator 140 can further or alternatively select objects for review using a pseudo-random selection technique and / or a targeted selection technique. For example, a predetermined rule can indicate that a special number of objects are to be selected for review, and / or can identify an upper and / or lower limit to the number of objects to be identified for review. A targeted selection technique can be configured to bias and / or constrain object selection to objects having special features.

[0036] Selection of objects for review can involve, affect, and / or depend on selection of one or more images (e.g., corresponding to different fields of view). In instances where a machine learning model processes multiple images to detect and label objects, review need not include all of the multiple images. Instead, a subset of images (e.g., a single image) can be selected. Each image in the subset of images can be selected to have a representative field of view of at least a threshold number, at least a threshold percentage of the slides represented by the object-specific labels and / or the region-specific labels. Alternatively, each image in the subset of images can be selected based on which objects are labeled for review. For example, the subset can include images having a large number or high density (e.g., that is above a predetermined threshold and / or relative to other images) of objects labeled for review. As another example, the subset can be selected such that each of a plurality of object-specific labels is associated with an object represented in an image of the subset. To achieve such representation, for each of the object-specific labels, it can be determined which image(s) have the labeled object, and then a single image can be selected (e.g., using a pseudo-random selection technique). Density and / or variability of objects can also be used to improve stratified sampling of images for review. As yet another example, the subset can be selected to provide a large-scale review of hotspots or representative foci of objects labeled for review.

[0037] For selected objects and / or regions identified for review, the model evaluator 140 can generate and transmit a communication to the user device 105 (or to other user devices). Receipt of the communication can trigger presentation of an interface including the selected image, the one or more selected objects, and one or more object-specific labels associated with the selected objects. The interface can further identify the boundaries of the one or more regions and the region-specific labels. For example, for each of the selected objects and / or regions, a border or boundary defining the area of the object or region can be overlaid on the image. Upon mouse-over or clicking on the area, the associated label can be delineated or presented within the area.

[0038] The interface can include input components that can detect input identifying modifications or replacements of the boundaries and / or labels. For example, a pencil tool or an ellipse tool can be used to redefine or modify the boundaries and / or a text tool (or a radio button tool or a drop-down tool) can be used to identify the labels. The interface can further include tools that can be used to indicate whether the identified boundaries or labels are deemed to be appropriate.

[0039] The user device 105 can transmit a communication to the image processing system 110 (to be routed to the model evaluator 140) representing the inputs received at the interface.

[0040] The model evaluator 140 can generate an assessment quality metric representative of the quality of the speculation of the review. The metric can be generated based on which user is associated with the user device 105 (e.g., the number of reviews previously performed by the user, stored credentials associated with the user, etc.), the number of reviews previously performed at the user device 105, the degree to which the proportion or number of modified or replaced labels or boundaries corresponds to a typical proportion or number (e.g., for a given field of view, slide type, classifier confidence, or other heuristics), the amount of time during which the interface was presented and / or during which input activity was detected, the absolute number of modifications or replacements made and / or the degree to which the actual number of corrections matches the expected number of corrections (e.g., for a given field of view type, slide type, classifier confidence, etc.).

[0041] The assessment quality metric can include metrics applicable to the entire review or to the reviewer generally. Alternatively or additionally, assessment quality metrics can be generated that are applicable to more specific portions of the review and / or that correspond to more than one review and / or more than one reviewer. For example, an assessment quality metric can be generated for a given field of view. The metric can be determined based on, for example:

[0042] • the degree of consistency (e.g., in terms of corrections made or the number of special corrections) between reviews generated by multiple reviewers and / or subject matter experts (e.g., senior pathologists) on a common set of images corresponding to a given field of view;

[0043] • a degree of consistency between reviews generated by the same reviewer over multiple time periods;

[0044] • a degree of variability between reviews when displaying initial bootstrap cells compared to when displaying cells using model-generated labels (e.g., for the same field of view); and / or

[0045] a degree of variability between reviews for the same field of view when the reviewer provides labels from scratch (“ab initio”) compared to when the reviewer corrects model-generated labels.

[0046] The model controller 135 can then retrain the machine learning model using any modified or new labels or boundaries. The retraining can include reinforcement training. In instances where one or more evaluation quality metrics are generated, the retraining can be performed using the metrics. For example, a training data set can be defined to include a plurality of images, annotations of reviews, and weights associated with the images based on the evaluation quality metrics associated with the individual images. The training can be performed via a single stage or multiple stages of training. For example, a multi-step training can begin with a large-scale training performed using review data to provide model initialization. Such large-scale training can be particularly advantageous when the machine learning model includes a deep learning model (e.g., a mask-recursive convolutional neural network). The training can then continue using high-quality images that have been carefully examined (e.g., during reviews or by reviewers associated with high evaluation quality metrics).

[0047] The model controller 135 can then process new images using the retrained machine learning model. The new images can be provided by a different or the same entity as compared to the entity that provided the initial images and / or as compared to the reviewer. For example, the model controller 135 can determine to authorize a client device 145 to use the retrained machine learning model. Utilizing the machine learning model can include unlocking software functionality to operate locally at the client device 145 and / or to process images online using the model in response to a request (e.g., by performing automated labeling). The retrained model can then be used to generate accurate object polar annotations without relying on heuristics and / or subject expertise.

[0048] It should be appreciated that the image processing system 110 can operate to produce highly accurate cell localization and / or segmentation models. To enhance the accuracy of the models, multiple reviews of model generation metrics can be performed, the quality of the reviews can be evaluated, and the models can be repeatedly retrained based on the reviews and quality evaluations. Performing a large number of reviews, review evaluations, and retraining is a time-consuming effort. Thus, the result of this process is a moderate throughput of the system. Alternatively, the image processing system 110 can operate to produce a high throughput system by implementing a rapid prototyping framework. The framework can reduce the reliance on user input and / or reviewers. For example, the framework can include segmenting using simple heuristic-based algorithms, training machine learning models using bootstrap object annotations, and possibly facilitating higher level reviews of important labeling errors or inconsistencies.

[0049] II. Accuracy and efficiency advantages of bootstrap machine learning techniques

[0050] Conventional histology image processing relies heavily on manual input by users identifying annotations or regions and cells. In part due to the size of the images, annotating cells is a very time-consuming effort. Further, while regions are typically dominated by a single cell type, regions often include a large number of other types of cells. (See FIG. 2 which shows different regions and cell species in an image of a breast cancer slide.) Further, due to the variability of a given cell's appearance, the expertise required for cell annotation is high.

[0051] Depending on the financial cost, time cost, and logistical complexity, the types of annotations can be ordered (from easiest to most complex) as follows:

[0052] 1. Image-level classification (e.g., labeling a slide as "invasive ductal breast carcinoma")

[0053] 2. Region-level classification (e.g., labeling a tissue region as "dominated by lymphocytes")

[0054] 3. Object centroid localization (e.g., placing a seed on a lymphocyte)

[0055] 4. Bounding box localization (e.g., placing a box around a lymphocyte)

[0056] 5. Object segmentation (e.g., drawing a polygon around a lymphocyte)

[0057] The bootstrap technique provided herein, in which a machine learning model is trained based on an initial definition of cell labels for respective regions, can thus greatly improve annotation efficiency. Annotators do not have to spend time labeling individual cells. Instead, by focusing the annotation effort to higher level regions, efficiency can be improved. Further, selected and strategic reviews, as well as retraining, can improve accuracy. III. Process of bootstrap machine learning techniques

[0058] FIG. 3 A process 300 for generating and training a machine learning model to detect and label objects within images is shown. The process 300 begins at block 305 with performing region annotation for each image in a set of images. Each image in the set of images can correspond to a field of view of a histology slide. The fields of view can partially overlap one another across the set of images.

[0059] For each region of the one or more regions, the region annotation can include identifying boundaries (or other spatial features) of the region within the image. The region annotation can be performed based on user input, semi-automatically using computer vision techniques, or automatically using computer vision techniques. The region annotation can further include identifying a region-specific label for each region. The region-specific label can correspond to a characterization of a particular cell type and / or biological structure. For example, the region-specific label can include“tumor” or“tumor cell.” In cases where the region annotation is performed using user input, the user can identify the region-specific label by inputting free text or selecting from a set of pre-defined label options.

[0060] While not shown in FIG. 3 Object detection can be performed prior to or after block 305. The object detection can be performed automatically using computer vision techniques. The objects being detected can include biological cells as well as possibly other biological structures.

[0061] At block 310, for each image in the set of images, object annotation bootstrapping is performed. For each object of at least some of the objects, the bootstrapping can include assigning a label corresponding to the region in which the object is located. In some cases, for objects having spatial features corresponding to cell spatial features (e.g., having an area, contrast, and / or aspect ratio that corresponds to a comparable statistical profile of cells of a given image field of view), the bootstrapping can be selectively performed. Further, for objects that are not within any defined region, the bootstrapping need not be performed.

[0062] At block 315, a machine learning model is trained using training data that includes bootstrap annotations. Input to the machine learning model can include one or more images and object data, such as a center location of each detected object and / or a boundary of each detected object. In some cases, pre-processing can be performed to filter out a subset of the set of objects based on spatial features of the objects (e.g., to filter out objects that have a shape that is not typical of a cell and / or that are near an edge of the image and / or an edge of a region). Input to the machine learning model can include a label for each identified object and a confidence metric that can be associated with the label. The machine learning model can include a neural network, a convolutional neural network, a recurrent neural network, and / or a deep neural network. Training the model can include learning parameter values that define how to transform input data into output data.

[0063] At block 320, the trained machine learning model is executed to generate object-specific labels. The object-specific labels generated at block 320 can be generated by processing images in the training data or by processing other images.

[0064] At block 325, a subset of the object-specific labels generated at block 320 are modified using a region prior. More specifically, the trained machine learning model can have predicted multiple objects to correspond to an object-specific label that is different from a region-specific label for a region within the image in which the objects are located. For some of these cases, the object-specific label and the region-specific label can be defined to be the same (at block 325) despite the opposite prediction of the model. The region prior label can be assigned when, for example, the objects have atypical spatial features (e.g., heterogeneous and / or substantially asymmetric across axes) and / or when the object type associated with the model’s predicted label is unlikely to exist within the slide environment associated with the image. For example, it can be unlikely to observe vascular endothelial cells in a slide associated with a breast cancer sample.

[0065] At block 330, a number of the object-specific labels are selected for evaluation. The number of object-specific labels selected for evaluation can be less than the number of labels defined for a given slide, field of view, image, or data set. The selection can be made based on a confidence metric and / or a sampling technique (e.g., a pseudo-random selection technique). Information about the object-specific labels and / or the associated objects can be made available to a reviewer. For example, an interface can be generated that includes the image and identifies locations of the objects associated with the number of object-specific labels (e.g., by including a marker on a nucleus of a cell object and / or by including an outline of the object).

[0066] At block 335, enhanced labels for a subset of objects are detected. The subset of objects can correspond to some or all of the plurality of object-specific labels selected for evaluation. The enhanced labels can include labels that are different from the object-specific labels generated by the machine learning model and / or the same as the object-specific labels generated by the machine learning model. For example, an interface for use by the reviewer can be configured to receive input identifying an object-specific label for an object. The object-specific label can be identified explicitly (e.g., by receiving an explicit label identification that can be different or the same as the label generated by the machine learning model) or implicitly (e.g., by inferring that the presented label generated by the machine learning model is correct if no contrary label is provided). The object-specific label identified by the reviewer (and / or approved by the reviewer) can be labeled as an enhanced label.

[0067] At block 340, a quality metric can be generated for the enhanced labels. The quality metric can be generated based on the following: tracked reviewer experience, length of time during which the review interface is presented, percentage of identified label modifications, consistency of identified labels across multiple presentations of the same data (e.g., across users and / or for the same user), correlation between likelihood of correcting a label and model output confidence score, etc. The quality metric can be assigned for a given dataset, slide, image, reviewer, image type, and / or field of view.

[0068] At block 345, the machine learning model is retrained using the enhanced labels. In some cases, the quality metric is used to weight portions of the enhanced labels in the retraining. For example, enhanced labels from a reviewer and / or associated with a field of view can be given a higher weight relative to enhanced labels from another reviewer and / or associated with another field of view.

[0069] At block 350, new images are processed using the retrained machine learning model. In some cases, no region annotation is performed prior to processing the new images. In some cases, region annotation is performed as pre-processing. Post-processing can also be performed (e.g., using the region priors described herein to modify a subset of the object-specific labels). The output can be transmitted and / or presented at a user device. The output can include one or more images; indications of where objects are located within the images and a label for each object. The output can additionally or alternatively include higher level information such as a list of object-specific labels detected within the dataset, a number of labels associated with each of the object-specific labels, and / or spatial distribution information corresponding to each of the object-specific labels.

[0070] IV. Examples

[0071] The following examples relate to generating subject-specific labels using various image processing techniques, including the bootstrap techniques described herein. More specifically, 125 images were obtained from the TCGA dataset. Each of the images depicts a field of view of an H&E stained diagnostic breast cancer slide.

[0072] IV. A. Region annotation

[0073] FIG. 4A-4E An image is shown with respect to region annotations. FIG. 4A An original RGB image from the TCGA dataset is shown. A trained user then manually identified the boundaries of the histological regions in the image. The user also identified region-specific labels for each region. FIG. 4B A processed version of the original RGB image is shown, in which pixels are colored to indicate the region-specific label for the corresponding region in which the pixel lies.

[0074] In some cases, smoothing can be applied to the manual region annotations. FIG. 4C The leftmost image in FIG. 4D also shows an original RGB image from the TCGA dataset, and the rightmost image in FIG. 4C and FIG. 4D shows the histological regions identified by user input. FIG. 4C The middle image in FIG. 4D shows the results of applying a fully connected conditional random field smoothing to enhance the correspondence between the manual annotations and the tissue boundaries.

[0075] FIG. 4E A processed version of the image in FIG. 4B is shown. In this case, the image from FIG. 4B was processed by color thresholding the image in HSV space and then applying Gaussian smoothing and Otsu thresholding.

[0076] Thus, FIG. 4 shows various techniques by which regions can be defined (e.g., purely based on manual input, by applying a random field smoothing to the manually annotated input, by applying another type of smoothing (e.g., Gaussian smoothing), and / or by applying one or more thresholding).

[0077] IV. B. Generating bootstrap nucleus annotations from region annotations using non-machine learning models and graphics processing heuristics

[0078] FIG. 5A-5F The results of applying non-machine learning models and image processing heuristics to generate bootstrap, noise kernel annotations from region annotations are shown. Notably, performing thresholding and Gaussian smoothing to generate the images in FIG. 5 resulted in regions that were smaller than FIG. 4B , FIG. 4C and FIG. 4DThe regions in the middle are more discontinuous, which can indicate that this technique better detects non-homogenous cell types within the regions. The RGB images of FIG. 4A are processed using computer vision techniques to detect individual cells. Object-specific labels that match the region-specific labels for the regions of pixels are initially assigned to each of the detected cells. FIG. 5A An image is depicted with color differences that indicate the object-specific labels assigned based on the region-specific labels depicted. FIG. 4E An image is depicted with color differences that indicate the object-specific labels assigned based on the region-specific labels depicted. Red shading represents the tumor label, blue shading represents the lymphocyte label, green shading represents the fibroblast label, orange shading represents the necrotic cell label, and gray shading represents the "other" label. As shown, many of the cells within the stromal region are classified as fibroblasts. This represents an inconsistency between the labels. The lymphocyte label should likely be assigned to the cells.

[0079] A heuristic-based approach is applied to modify the label assignments. More specifically, shape and size constraints are applied to identify objects for which the initial label assignments are removed. FIG. 5B Object-specific label assignments are shown after the heuristic-based processing. Notably, the label modifications affect many of the objects that were initially defined as lymphocytes within the stromal region.

[0080] FIG. 5C-5F Each of the images in the middle shows how the initial label assignments are made to the initial RGB image (leftmost image) based on the bootstrap technique. In FIG. 5C and FIG. 5D the bootstrap technique achieves correct classification of most of the nuclei. (Compare the right two "transition" images to the left two "final" images.) Meanwhile, in FIG. 5E and FIG. 5F the bootstrap technique results in incorrect label assignments to nuclei. FIG. 5E A stromal region is depicted with lymphocytes, and FIG. 5F A stromal region is depicted with fibroblasts. FIG. 5E An example is depicted in which a lymphocyte receives an incorrect label "fibroblast" before the heuristic modification of the initial labels is applied.

[0081] IV. C. Predictions from trained machine learning models using bootstrap annotations

[0082] FIG. 6A-6C Object labels produced by a model trained with bootstrap annotations of three histology images are shown. In FIG. 6A-6CEach of the top row shows the original image (far right image), the region annotation (second right image), and the noise bootstrap nuclei (far left image), and the bottom row shows the prediction from the mask-recurrent convolutional neural network model trained using the information with bootstrap labels. Notably, the mask-recurrent convolutional neural network prediction is more extensive (does not conform to the imprecise region annotation boundaries). Further, the mask-recurrent convolutional neural network is more "round" and better conforms to the nuclei contours. Additionally, even though the bootstrap labels (used to train the model) are incorrect, the nuclei that are not dominant in the region are correctly classified in the mask-recurrent convolutional neural network output. This is most apparent for the lymphocytes in the stroma l (A, B) and tumor (C) regions.

[0083] IV. D. Modifying labels from machine learning models using region priors

[0084] FIG. 7A The use of the mask-recurrent convolutional neural network to generate labels for images after initial training of the network using bootstrap cell-specific labels is shown. The model is trained to select one of the following labels for each cell: tumor, fibroblast, lymphocyte, plasma cell, inflammatory infiltrate, and other nuclei.

[0085] FIG. 7B The region-specific labels for the images are shown. Notably, the region-specific labels can be selected from a larger set of labels. In the depicted case, the yellow regions are associated with necrotic cells, and the gray regions are associated with glandular tissue regions. In the underlying object-specific labels for the neural network, there is no corresponding label available. The necrotic cell label is omitted due to the high homogeneity of the spatial features across the cells, and the glandular tissue label is omitted due to the infrequent occurrence of these regions / cells.

[0086] A set of rules can then be applied to the output of the neural network to identify objects for which a region prior label will be assigned instead of the label identified by the network. The set of rules can include a first rule specifying a confidence measure threshold such that each object-specific label associated with a confidence measure (as by the output of the neural network) will be changed to a region corresponding to the object. The set of rules can include a second rule indicating that an object-specific label will be changed to a label corresponding to the region when a pre-identified spatial feature (e.g., size and shape) of the object does not sufficiently (by at least a pre-identified degree) correspond to a spatial feature of the assigned label. FIG. 7C The object-specific labels after application of the set of rules are shown. Notably, the cells within the necrotic cell and glandular tissue regions are correctly labeled.

[0087] IV. E. Facilitating review of selected object specific labels

[0088] FIG. 8A-8DCertain metrics are shown that are used to determine which object-specific labels to present to a reviewer so that the reviewer can provide input indicating whether the assigned object-specific labels are accurate. Specifically, objects can be filtered and / or ranked based on whether they are associated with one or more special features and / or feature values that indicate that the label can not be accurate. FIG. 8A Objects that are identified as corresponding to segmentation artifacts are shown. FIG. 8B Objects that are associated with areas that are below a predetermined threshold or that have an unusual shape (e.g., as indicated by an aspect ratio that is above a predetermined threshold) are shown. FIG. 8C Objects that are assigned object-specific labels corresponding to a cell type that would not be physiologically observed within the area type indicated by the corresponding area-specific label are shown. FIG. 8D Objects for which the object-specific label is associated with a confidence measure that is below a predetermined threshold are shown.

[0089] In some cases, a score can be assigned to each object based on the confidence that the object corresponds to a segmentation artifact, the extent to which the area of the object is atypical (e.g., below a size threshold), the extent to which the shape of the object is atypical (e.g., has an aspect ratio that is above a threshold), whether the object-specific label is inconsistent with the spatially corresponding area-specific label, and the confidence measure. Objects for which the score is close to an extreme (indicating irregularity) can then be selected for review.

[0090] FIG. 9A Multiple framed regions within a slide are shown. Each framed region corresponds to a different field of view. The blue and green frames correspond to fields of view that are representative of a complete set of fields of view across the slide. The red and purple frames correspond to fields of view that are associated with more problematic labels and / or object-specific labels that are flagged for review. FIG. 9B A presentation of a special field of view is shown as presented to a user. A border is drawn around the selected object that has been flagged for review. Nucleus segmentation and shading are used to facilitate correction and expression classification certainty. Segmentation is also used so that it becomes clear as to which object is being corrected when a point is placed for correction.

[0091] V. Process of labeling objects in images using machine learning

[0092] FIG. 10 A process for assigning labels to objects in an image using machine learning is shown. The process 1000 begins with evaluating a frame of an image 1005. The image can correspond to and / or depict a histology slide. In some cases, the image is defined using a set of patches, which can be processed individually to detect and characterize objects (e.g., in this case, the frame 1010 through frame 1025 can be performed for each patch).

[0093] At block 1010, regions within the image are identified. The regions can be defined based on user input that can identify the boundaries of the regions. The regions can alternatively or additionally be defined based on segmentation analysis.

[0094] Blocks 1015 through 1025 are applied for each of the identified regions. At block 1015, a region-specific label is identified for each region. The region-specific label can be identified based on user input that identifies the label and / or based on a computer vision classification method.

[0095] At block 1020, objects within the region are identified. The objects are identified using computer vision and / or segmentation analysis. The identified objects can include cell clusters.

[0096] At block 1025, an object-specific label is assigned to each object. At this block, it is defined that the label assigned to an object is the same as the label assigned to the region that contains the object. Thus, after this block, the same label is assigned to all objects within a given region.

[0097] At block 1030, a training data set is defined that includes some or all of the object-specific labels. At block 1035, an augmented label is defined for a subset of the objects within the training data set. Block 1035 can include changing a selected object-specific label to an "other" classification that will not be used in training. The selected object-specific label can correspond to an object that has characteristics (e.g., shape and / or size) of a special cell state with a challenging label (e.g., necrotic cell), the selected object-specific label can be non-physiological in a given context (e.g., considering other object-specific labels and / or region-specific labels in the region).

[0098] At block 1040, an image processing model can be trained using the training data. The image processing model can include a deep neural network. The image processing model can be configured to receive an image and an object specification (e.g., indicating where the object is within the image) and output a label for each object.

[0099] At block 1045, an incomplete subset of objects can be selected for review. The subset can include a representative sample and / or objects for which the assigned label is associated with a lower confidence measure.

[0100] At block 1050, an augmented label can be identified for each object in at least some of the subset of objects. The augmented label can include a label identified by a reviewer. The augmented label can be different from the object-specific label initially assigned to the object. In some cases, the augmented label is the same as the object-specific label initially assigned to the object (although the augmented label can have been confirmed by the reviewer to be accurate).

[0101] At block 1055, additional training of the image processing model can be performed using the enhanced labels. New images can be processed to detect individual objects. At block 1060, a new image is processed using the image processing model to generate new object-specific labels. At block 1065, a representation of the new object-specific labels is outputted (e.g., transmitted or presented). The representation can include an overlay with one or more of the detected objects identified and / or indicating the special labels assigned to the objects. The representation can additionally or alternatively include higher level data based on the new object-specific labels. The higher level data can identify a count of objects with each of the plurality of labels, a proportion of objects with each of the plurality of labels, a spatial distribution of the labels, etc. Thus, the process 1000 facilitates generating accurate object-specific image annotations using a bootstrap labeling and machine learning techniques.

[0102] VI. Additional considerations

[0103] Some embodiments of the disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-transitory computer- readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods disclosed herein and / or some or all of one or more processes disclosed herein. Some embodiments of the disclosure include a computer program product tangibly embodied in a non-transitory machine- readable storage medium including instructions configured to cause one or more data processors to perform some or all of one or more methods disclosed herein and / or some or all of one or more processes disclosed herein.

[0104] The terms and expressions used are used as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the claimed invention. Accordingly, it is intended that the claimed invention should not be limited to the particular embodiment described by way of example only and should include any and all equivalents falling within the scope of the claimed invention.

[0105] This description provides preferred exemplary embodiments only, and is not intended to limit the scope, applicability or configuration of the disclosure. Rather, the description of the preferred exemplary embodiments will provide those skilled in the art with an enabling description that can be employed it is understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope as set forth in the appended claims.

[0106] In this description, specific details are set forth to provide a thorough understanding of embodiments. However, embodiments can be practiced without these specific details. For example, circuits, systems, networks, processes, and other components can be shown as components in block diagram form, rather than in detail, in order to avoid obscuring embodiments. In other instances, well-known circuits, processes, algorithms, structures, and techniques have not been shown in detail in order to avoid obscuring embodiments.

Claims

1. A computer-implemented method comprising: accessing a set of histology images; for each image in the set of histology images: identifying a plurality of regions within the image, wherein each region in the plurality of regions corresponds to an area of a patch of the image, wherein cells depicted within the area of the image have similar visual characteristics; for each region in the plurality of regions: identifying, via a computer vision technique, a region-specific label for the region, wherein the region-specific label indicates a particular cell type of cells associated with the respective region in an area of a patch of the image; detecting a set of objects within the region; and assigning, via the computer vision technique, to each object in the set of objects, an object-specific label for the object that is the same as the region-specific label assigned to the region in which the object is located; for each image in the set of histology images, defining a training data set to include: object location data, for each object in a detected set of objects, indicating image- within location data for the object; and label data, for each object in the detected set of objects, indicating the object-specific label assigned to the object; and training an image processing model using the training data, wherein the training includes learning a set of parameter values that define a set of parameters of a computation performed by the image processing model, wherein the computation includes one or more of connected component analysis, edge detection, color channel-specific analysis, smoothing algorithms, or thresholding algorithms.

2. The method of claim 1, further comprising: for an image in the set of histology images, selecting an incomplete subset of the set of objects via at least one of a pseudo-random selection technique and a target selection technique; outputting, on a user device, one or more presentations, each of the one or more presentations including: at least a portion of the image; an indication, as one or more portions of the at least a portion of the image, of one or more objects of the incomplete subset of the set of objects, and for each of the one or more objects, the object-specific label assigned to the object; receiving, from the user device, response data, for each of at least some of the objects of the incomplete subset of the set of objects, the response data identifying a different object-specific label for the object; and performing additional training of the image processing model using the different object-specific labels and the learned set of parameter values, wherein the additional training includes generating a second set of parameter values that includes, for at least some of the set of parameters, new parameter values that are different from respective parameter values in the learned set of parameter values.

3. The method of claim 2, further comprising: accessing a new image; processing the new image using the image processing model configured with the second set of parameter values, wherein a result of the processing includes a set of particular object specific labels corresponding to a set of particular objects detected in the new image; and outputting a representation of the set of particular object specific labels.

4. The method of claim 1, wherein: each image in the set of histology images comprises a histology image depicting at least a portion of a tissue section; and for each object in at least some of the set of detected objects: the object corresponds to a cell; and the object specific label corresponds to a cell type.

5. The method of claim 1, wherein the image processing model comprises a deep convolutional neural network.

6. The method of claim 1, further comprising, for each object in the set of objects: determining one or more features of the object, the one or more features relating to a size, shape, texture, and / or color of the object; determining, based on the one or more features and stored label specific feature data associated with the object specific label assigned to the object, whether a label discrepancy condition is satisfied; and when it is determined that the label discrepancy condition is satisfied, updating the object specific label assigned to the object to a different object specific label.

7. The method of claim 1, further comprising, for each image in the set of images: outputting, on a user device, a presentation of the image; receiving first user input defining a set of boundaries in the image, wherein each region in the plurality of regions is identified as corresponding to an area within a boundary in the set of boundaries; and receiving second user input identifying a set of classifications, wherein each region specific label is identified based on a classification in the set of classifications.

8. A system comprising: one or more data processors; and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform a set of operations comprising: accessing a set of histology images; for each image in the set of histology images: identifying a plurality of regions within the image, wherein each region in the plurality of regions corresponds to a piece of area of the image, wherein cells depicted within the piece of area of the image have similar visual features; for each region in the plurality of regions: identifying, via a computer vision technique, a region specific label for the region, wherein the region specific label indicates a particular cell type of cells associated with the respective region in a piece of area of the image; detecting a set of objects within the region; and assigning, via the computer vision technique, to each object in the set of objects, an object specific label for the object that is the same as the region specific label assigned to the region in which the object is located; for each image in the set of histology images, defining a training data set to include: ​ object location data, for each object in the set of detected objects, the object location data indicating an in-image location data for the object; and label data, for each object in the set of detected objects, the label data indicating the object-specific label assigned to the object; and training an image processing model using training data, wherein the training comprises learning a set of parameter values defining a set of parameters of a computation performed by the image processing model, wherein the computation comprises one or more of connected component analysis, edge detection, color channel-specific analysis, smoothing algorithm, or thresholding algorithm.

9. The system of claim 8, wherein operations further comprise: selecting, for an image in the set of histology images, an incomplete subset of the set of objects via at least one of a pseudo-random selection technique and a target selection technique; outputting, on a user device, one or more presentations, each of the one or more presentations comprising: at least a portion of the image; an indication, as one or more portions of the at least a portion of the image, the one or more portions corresponding to one or more objects of the incomplete subset of the set of objects, and for each of the one or more objects, the object-specific label assigned to the object; receiving, from the user device, response data, for each of at least some objects of the incomplete subset of the set of objects, the response data identifying a different object-specific label for the object; and performing additional training of the image processing model using the different object-specific labels and the learned set of parameter values, wherein the additional training comprises generating a second set of parameter values, the second set of parameter values including, for at least some of the set of parameters, a new parameter value different from a respective parameter value in the learned set of parameter values.

10. The system of claim 9, wherein the operations further comprise: accessing a new image; processing the new image using the image processing model configured with the second set of parameter values, wherein a result of the processing comprises a set of particular object-specific labels corresponding to a set of particular objects detected in the new image; and outputting a representation of the set of particular object-specific labels.

11. The system of claim 8, wherein: each image in the set of histology images comprises a histology image depicting at least a portion of a tissue section; and for each of at least some objects in the set of detected objects: the object corresponds to a cell; and the object-specific label corresponds to a cell type.

12. The system of claim 8, wherein the image processing model comprises a deep convolutional neural network.

13. The system of claim 8, wherein operations further comprise, for each object in the set of objects: determining one or more features of the object, the one or more features relating to a size, shape, texture, and / or color of the object; determining whether a label difference condition is satisfied based on the one or more features and stored label-specific feature data associated with the object-specific label assigned to the object; and updating the object-specific label assigned to the object to a different object-specific label when it is determined that the label difference condition is satisfied.

14. The system of claim 8, wherein operations further comprise, for each image in the set of histology images: outputting, on a user device, a rendering of the image; receiving first user input defining a set of boundaries in the image, wherein each region in the plurality of regions is identified as corresponding to an area within a boundary in the set of boundaries; and receiving second user input identifying a set of classifications, wherein each region-specific label is identified based on a classification in the set of classifications.

15. A computer program product tangibly embodied in a non-transitory machine- readable storage medium, including instructions configured to cause one or more data processors to perform a set of operations comprising: accessing a set of histology images; for each image in the set of histology images: identifying a plurality of regions within the image, wherein each region in the plurality of regions corresponds to a patch of area of the image, wherein cells depicted within the area of the image have similar visual features; for each region in the plurality of regions: identifying, via a computer vision technique, a region-specific label for the region, wherein the region-specific label indicates a cell-specific cell type associated with cells in the respective region in a patch of area of the image; detecting a set of objects within the region; and assigning, via the computer vision technique, to each object in the set of objects an object-specific label for the object that is the same as the region-specific label assigned to the region in which the object is located; for each image in the set of histology images, defining a training data set to include: object location data, for each object in a set of detected objects, indicating image- within location data for the object; and label data, for each object in the set of detected objects, indicating the object-specific label assigned to the object; and training an image processing model using the training data, wherein the training includes learning a set of parameter values defining a set of parameters of a computation performed by the image processing model, wherein the computation includes one or more of connected component analysis, edge detection, color channel-specific analysis, smoothing algorithms, or thresholding algorithms.

16. The computer program product of claim 15, further comprising: for an image in the set of histology images, selecting an incomplete subset of the set of objects via at least one of a pseudo-random selection technique and a target selection technique; outputting, on a user device, one or more renderings, each of the one or more renderings including: at least a portion of the image; indicating, as one or more portions of the at least part of the image, the one or more portions corresponding to one or more objects of the incomplete subset of the set of objects, and for each object of the one or more objects, assigning the object-specific label to the object; receiving, from the user device, response data, for each object of at least some of the incomplete subset of the set of objects, the response data identifying a different object-specific label for the object; and performing additional training of the image processing model using the different object-specific labels and the learned set of parameter values, wherein the additional training comprises generating a second set of parameter values, the second set of parameter values including, for at least some of the parameters in the set of parameters, new parameter values different from the respective parameter values in the learned set of parameter values.

17. The computer program product of claim 16, further comprising: accessing a new image; processing the new image using the image processing model configured with the second set of parameter values, wherein a result of the processing includes a set of particular object-specific labels corresponding to a set of particular objects detected in the new image; and outputting a representation of the set of particular object-specific labels.

18. The computer program product of claim 15, wherein: each image in the set of histology images comprises a histology image depicting at least part of a tissue section; and for each object of at least some of the set of detected objects: the object corresponds to a cell; and the object-specific label corresponds to a cell type.

19. The computer program product of claim 15, wherein the image processing model comprises a deep convolutional neural network.

20. The computer program product of claim 15, further comprising, for each object of the set of objects: determining one or more features of the object, the one or more features relating to a size, shape, texture, and / or color of the object; determining, based on the one or more features and stored label-specific feature data associated with the object-specific label assigned to the object, whether a label difference condition is satisfied; and when it is determined that the label difference condition is satisfied, updating the object-specific label assigned to the object to a different object-specific label.

Images