Image-based object classification using boostrapping of region-level annotations

Abstract

The present technology relates to object classification using bootstrapping of region-level annotations. For each of a plurality of images, a region within the image can be identified. For each region, a region-specific label can be identified, a set of objects within the region can be detected, and an object-specific label can be assigned to each object. The object-specific label can be identical to the region-specific label assigned to the region in which the object is located. A training dataset can be defined for each image of the plurality of images to include object position data (indicating in-image position data for the detected object) and label data (indicating the object-specific label assigned to the object). An image processing model can be trained using the training data. Training can include learning values ​​for a set of parameters that define calculations performed by the image processing model.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of and priority to U.S. Provisional Patent Application No. 62 / 929,100, filed October 31, 2019, which is incorporated by reference in its entirety for all purposes. [Background technology]

[0002] Often, the images are processed to detect different objects within each image and classify each of the different objects. This processing may include manual and / or automated approaches. One exemplary use case is in a biological context, where histological images are processed to identify individual cells and identify the cell type for each cell. Completing this processing manually can require significant time and expense, especially when a large number of images are reviewed, as is often the case. For example, histological images of an entire slide can be very large in size and contain depictions of tens of thousands of cells. Furthermore, this labeling can require significant expertise.

[0003] One approach to facilitate this process is to utilize automated techniques to process the images. For example, deep learning models can be used to assign labels to individual objects (e.g., individual cells) in the image. However, the sensitivity and specificity of this approach can be strongly affected by the size of the training dataset. The training data typically includes annotated images (identifying the location of objects in the image and object-specific classifications), traditionally generated as a result of manual evaluation of images.

[0004] Given the significant time and expense involved in manual evaluation, securing a large training set can be costly and / or difficult, and as a result, training an image processing model (e.g., a deep learning model) to accurately perform these tasks can be difficult. Summary of the Invention

[0005] In some embodiments, a bootstrapping technique is used to process the images for object detection and classification. More specifically, each image of one or more images may be processed to first identify higher-level regions within the image, and each higher-level region may be associated with a region-specific label. For example, a higher-level region may be defined to correspond to a portion of the image defined by a user input outline, and the region-specific label may include a cell type indicated by the user input. Lower-level objects within the region may then be detected (e.g., using edge detection techniques, contrast analysis, intensity analysis, color analysis, etc.). The lower-level objects may include (or be) individual cells or cell nuclei.

[0006] Each lower-level object may be assigned an object-specific label that is the same as the region-specific label assigned to the higher-level region in which the lower-level object is located. An image processing model (e.g., a machine learning model such as a deep neural network and / or a convolutional neural network) may then be trained. Post-processing may include identifying a subset of the automatically detected objects for manual review of the associated object-specific labels. For example, one or more representations including a portion or all of the image may be generated and output along with the identification of the subset of objects and their assigned object-specific labels. Any input received may be used as a representation as to whether the object-specific label is accurate (or whether other labels correspond to the identified object). This feedback may be used to retrain the image processing model. Thus, the image processing model may be trained to have high specificity and sensitivity, while not requiring too many inputs suggesting manual review.

[0007] In some embodiments, a computer-implemented method is provided. For each image of the set of images, a plurality of regions within the image may be identified. For each region of the plurality of regions, a region-specific label may be identified for the region, a set of objects within the region may be detected, and an object-specific label may be assigned to each object of the set of objects. The object-specific label may be the same as the region-specific label assigned to the region in which the object is located. A training data set may be defined to include, for each image of the set of images, object position data (indicative of the object's in-image position data for each object of the detected set of objects) and label data (indicative of the object-specific label assigned to the object for each object of the detected set of objects). The image processing model may be trained using the training data. Training may include learning a set of parameter values ​​for a set of parameters defining the calculations performed by the image processing model.

[0008] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium including instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of the methods and / or some or all of the processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause the one or more data processors to perform some or all of the methods and / or some or all of the processes disclosed herein.

[0009] The terms and expressions used are used as terms of description and not of limitation, and in the use of such terms and expressions there is no intention to exclude equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention as claimed. Thus, although the invention as claimed has been specifically disclosed by embodiments and optional features, it is to be understood that modifications and variations of the concepts disclosed herein may be left to those skilled in the art, and such modifications and variations are considered to be within the scope of the invention.

[0010] The present disclosure will be described with reference to the accompanying drawings. [Brief description of the drawings]

[0011] [Figure 1] 1 illustrates an image processing interaction system including multiple devices / systems configured to support inter-device communication and image processing. [Diagram 2] Shows how individual regions in an image can contain multiple object types. [Diagram 3] 1 illustrates a process 300 for generating and training a machine learning model for detecting and labeling objects in an image. [Figure 4] 4A to 4E show images related to region annotation. [Diagram 5] 5A-5F show the results of applying non-machine learning models and image processing heuristics to generate noisy nuclei annotations that are bootstrapped from region annotations. [Figure 6] 6A-6C show object labels generated by a model trained with bootstrapped annotations for three histological images. [Figure 7] 7A-7C show object-specific and region-specific labeling on histology images. [Figure 8]8A-8D show the intrinsic metrics used to determine which object-specific labels should be available to the reviewer. [Figure 9] 9A-9B show patch selection and representation for review of selected object-specific labels. [Figure 10] 1 illustrates the process of using machine learning to assign labels to objects in an image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] In the accompanying drawings, similar components and / or features may have the same reference numbers. Furthermore, various components of the same type may be distinguished by following the reference number with a dash and a second number that distinguishes the similar components. If only a first reference number is used in this specification, the description is applicable to any of the similar components having the same first reference number, regardless of the second reference number.

[0013] I. Image Processing Dialogue System FIG. 1 illustrates an image processing dialogue system 100 including multiple devices / systems configured to support device-to-device communication and image processing. A user device 105 can send an image processing request to an image processing system 110. The image processing request may be sent via an interface, such as an online interface. The image processing request can identify one or more images to be processed. For example, the image processing request can include an identifier (e.g., an object and / or material identifier) ​​that can be used to retrieve the image from an image source 115. The image processing system 110 can then send a request for the image including the identifier and authorization data to the image source 115, which can return the image. As another example, the image processing request can include an image that may have been uploaded at the client device 105 or otherwise associated with the image.

[0014] The images may include histology images, such as H&E images depicting a sample stained with H&E stain. The images may also include separate immunohistochemistry (IHC) images (which by definition use a type of stain other than H&E). A single sample may be associated with corresponding H&E and IHC images. The H&E and IHC images are to be superimposed (using strict or deformable image registration). The H&E and IHC images may then be used for different evaluations (e.g., the H&E image is used to detect regions and the IHC image is used to identify object information for labeling). The images may include brightfield images, non-brightfield images (e.g., electron microscope images), and / or scanned whole slide histology images.

[0015] The image processing system 110 can include a region-specific labeler that processes each image to identify one or more regions within the image. Each region corresponds to a histological region and may correspond (for example) to a tumor region, a stromal region, or a lymphocytic infiltration region. Each region may correspond (for example) to an area in which cells within the region predominantly and substantially have similar visual characteristics (e.g., with respect to size, shape, intensity, and / or color). The labeling of each region may indicate (for example) the biological structure corresponding to the region and / or the prominent cell type of the region.

[0016] The region-specific marking device 120 can define and / or mark each region automatically, semi-automatically, or based on user input. For example, a graphical user interface can be provided to the user device 105. The graphical user interface can present part or all of the image and can include tools (e.g., a lasso tool, a pencil tool, a box selection tool, an ellipse selection tool, etc.) that can be used to identify boundaries of individual regions. The interface can further accept input received at the user device 105 that identifies a marking of the region. The input can be accepted in a window or pop-up component of the interface, which can include a text box that accepts a freeform marking for the region, or a selection component for accepting a selection of a marking from among a set of potential marks. The selection component can include a drop-down menu or multiple selectable radio buttons.

[0017] As another example, an automated or semi-automated approach can process the image to detect significant edges, and regions can be identified and / or presented accordingly (e.g., via representation in an interface). The automated or semi-automated approach may additionally or alternatively identify image metrics (e.g., brightness, hue, and / or contrast) for each of a set of patches in the image. Individual regions can be defined to correspond to sets of image patches having similar image metrics.

[0018] The object detection device 125 can detect objects in the image. Each of the detected objects can include (or be) a biological structure such as a cell or a cell nucleus. The objects can be automatically detected using image processing. Image processing can include utilizing connected component analysis, edge detection, color (or transformed color) channel singularity analysis, smoothing algorithms, thresholding algorithms, and the like. For example, image processing can include performing stain "unmixing" using established methods such as Macenko's Principal Component Analysis (PCA)-based method, or methods that rely on sparse non-negative matrix factorization (SNMF), among others. For each region label, the region prior (e.g., binary region history) can then be transformed (e.g., multiplied) with a stain channel (e.g., hematoxylin channel). One or more smoothing and / or thresholding processes can be performed (e.g., by iteratively applying Gaussian smoothing and Otsu's thresholding) to obtain nuclear boundaries. A connected component analysis can then be performed to isolate the nuclear objects.

[0019] The object-specific labeler 130 can assign an object-specific label to each detected object. Initially, each object-specific label may be defined to be equal to the label associated with the corresponding region. The object-specific labeler 130 can identify an incomplete subset of the set of objects, in which case each object in the incomplete subset is reassigned or assigned to a non-cellular label. The non-cellular label can indicate that the object is predicted to not be a cell and / or not correspond to a cell. The incomplete subset of objects can include (for example) objects having areas below and / or above a corresponding predefined threshold area, and / or objects having aspect ratios, circularity, and / or areas that do not correspond to the baseline instance.

[0020] The model controller 135 can then use the object-specific labels to train a machine learning model. The machine learning model can include (for example) a deep neural network, a recurrent neural network, and / or a convolutional neural network. The training can use overlapping tiles or patches and / or can restrict model predictions to the center or center of each tile or patch, which can reduce the effect of edge artifacts. In some examples, the object-specific labels can be used to define and / or train post-processing (although not required). The post-processing can use binary erosion, binary dilatation, Gaussian smoothing, or another technique to suppress segmentation artifacts.

[0021] After the machine learning model (and, e.g., post-processing techniques) is trained, the model controller 135 can use the model to process image data corresponding to the same image used to train the model and / or different images. The image data can include part or all of the image and object-specific data that identifies which parts of the image correspond to individual objects. The output of the machine learning model (and / or post-processing) can include object-specific labels for each detected object. Of course, the object-specific labels generated using the trained model can differ from the object-specific labels originally assigned based on the region-specific labels.

[0022] The object-specific labeling device 130 may modify the selected labels. A population-based approach may be used to select objects associated with rare (e.g., having a prevalence below a predefined threshold) size, shape, brightness profile, contrast profile, and / or color profile. For each of these selected objects, the label assignment may be modified and / or new labels may be defined. For example, a rare shape of an object may be typical of cells undergoing necrosis. Thus, the shape itself is not necessarily indicative of a cell type, but perhaps a physiological stage. As another example, a given label in a particular situation (e.g., vascular endothelial cells in an image corresponding to breast cancer) may be rare, thus triggering a reassignment of the label. The newly assigned labels may include a label indicating that the label will not be used while training a model to classify the object.

[0023] The model evaluator 140 may select objects to be reviewed. Objects can be selected based on (for example) a confidence metric associated with the label assignment. The confidence metric can indicate the degree to which one or more characteristics of an object (e.g., size, shape, brightness profile, contrast profile, and / or color profile) correspond to or match one or more characteristics of the object type associated with the assigned label. The trained machine learning model can output a confidence metric in association with each label output (corresponding to an individual output). The objects selected for review can include some or all of the objects associated with a confidence metric below an absolute or relative threshold (e.g., within a minimum of 5% of the confidence metric for the entire set of objects).

[0024] The model evaluator 140 may additionally or alternatively select objects for review by using pseudo-random and / or target selection techniques. For example, the predetermined rules may indicate that a certain number of objects should be selected for review and / or may identify upper and / or lower bounds on the number of objects to be identified for review. Target selection techniques may be configured to bias object selection toward and / or constrain object selection to objects having certain characteristics.

[0025] Selection of an object for review may be related to, affect, and / or depend on the selection of one or more images (e.g., corresponding to different fields of view). In cases where a machine learning model processes multiple images to detect and label objects, the review need not include all of the multiple images. Rather, a subset of images (e.g., a single image) may be selected. Each of the subset of images may be selected to have a representative field of view of the slide that represents at least a threshold number, at least a threshold percentage of object-specific labels and / or region-specific labels. Alternatively, each of the subset of images may be selected based on which objects have been flagged for review. For example, the subset may include images that have a high (e.g., above a predetermined threshold and / or high compared to other images) number or density of objects flagged for review. As another example, the subset may be selected such that each of the multiple object-specific labels is associated with an object represented among the images of the subset. To achieve this representation, for each of the object-specific labels, it can be determined which images have an object with the label, and then a single image can be selected (e.g., using a pseudo-random selection technique). The density and / or diversity of objects may also be utilized to enhance stratified sampling of images for review. As yet another example, subsets may be selected to provide a large-scale review of hot spots, or representative foci of objects flagged for review.

[0026] Upon identifying the selected objects and / or regions for review, the model evaluator 140 may initiate and send a communication to the user device 105 (or another user device). Receiving the communication may initiate the presentation of an interface including the selected image, one or more selected objects, and one or more object-specific indicators associated with the selected objects. The interface may further identify boundaries of the one or more regions and the region-specific indicators. For example, for each selected object and / or region, a border or boundary defining the area of ​​the object or region may be overlaid on the image. The associated indicator may be drawn within the area or presented after a mouseover or click on the area.

[0027] The interface may include an input component capable of detecting input that identifies modifications or replacements to the boundaries and / or indicia. For example, a pencil tool or an oval tool may be used to redefine or modify the boundaries and / or indicia may be used to identify the indicia using a text tool (or a radio button tool or a dropdown tool). The interface may further include tools that may be used to indicate whether the identified boundaries or indicia are accepted as appropriate.

[0028] The user device 105 can send communications representing input received at the interface to the image processing system 110 (which are routed to the model evaluator 140).

[0029] The model evaluator 140 can generate one or more evaluation quality metrics that represent an estimated quality of the review. The metrics can be generated based on which user is associated with the user device 105 (e.g., the number of reviews performed by the user, stored credentials associated with the user, etc.), the number of reviews previously performed on the user device 105, the extent to which the percentage or amount of corrected or replaced indicators or boundaries corresponds to a typical percentage or amount (e.g., for a given view type, slide type, classifier confidence or other heuristic), the amount of time the interface was presented and / or the amount of time input activity was detected, the absolute number of corrections or replacements made, and / or the extent to which the actual number of corrections matches an expected number (e.g., for a given view type, slide type, classifier confidence, etc.).

[0030] The rating quality metrics may include metrics that apply to the entire review and / or to reviewers in general. Alternatively or additionally, rating quality metrics may be generated that apply to more specific portions of a review and / or that correspond to more than one review and / or more than one reviewer. For example, a rating quality metric may be generated for a given view. Metrics may be determined based on (for example): The degree of agreement (e.g., regarding the amount of corrections made or the specific corrections made) between reviews generated by multiple reviewers and / or reviews generated by subject matter experts (e.g., senior pathologists) for a common set of images corresponding to a given field of view. · The degree of agreement between reviews produced by the same reviewer across multiple time periods. The degree of discrepancy between reviews (e.g., on the same field of view) between when the initial bootstrapped cells are displayed and when the cells are displayed with model-generated labels.

[0031] The degree of disparity between reviews for the same field of view when the reviewer provides the label from scratch ("de novo") compared to when the reviewer corrects the model-generated label.

[0032] The model controller 135 can then retrain the machine learning model using any revised or new indicators or boundaries. The retraining can include reinforcement training. If one or more evaluation quality metrics are generated, the retraining can be performed using the metrics. For example, a training dataset can be defined to include a plurality of images, the reviewed annotations, and image-related weights that are defined based on the evaluation quality metrics associated with the individual images. The training can be performed via a single stage of training or multiple stages. For example, multi-step training can begin by performing large-scale training using the review data to provide model initialization. This large-scale training can be particularly advantageous when the machine learning model includes a deep learning model (e.g., a masked recurrent convolutional neural network). The training can then continue using high-quality images that have been rigorously inspected (e.g., during review or by reviewers associated with high-rating quality metrics).

[0033] The model controller 135 can then use the retrained machine learning model to process new images. The new images may be provided by a different or the same entity compared to the entity that provided the initial image and / or compared to the reviewer. For example, the model controller 135 can determine that the client device 145 is authorized to use the retrained machine learning model. Using the machine learning model can include unlocking functionality of software operated locally on the client device 145 and / or responding to requests to process images online using the model (e.g., by performing auto-labeling). The retrained machine learning model can then be used to generate accurate object-level annotations without relying on heuristics and / or subject matter expertise.

[0034] Of course, the image processing system 110 may operate to generate highly accurate cell localization and / or segmentation models. To increase the accuracy of the model, multiple reviews of the model generation metrics can be performed, the quality of the reviews can be evaluated, and the model can be repeatedly retrained based on the reviews and quality evaluation. Performing multiple reviews, evaluating the reviews, and retraining can be very time-consuming. Thus, this process can have a moderate impact on the throughput of the system. Alternatively, the image processing system 110 may operate to generate a high-throughput system by implementing a rapid prototyping framework. The framework can de-emphasize user input and / or review. For example, the framework can include using simple heuristic-based algorithmic segmentation, training machine learning models using bootstrapped object annotations, and possibly facilitating high-level review of notable labeling errors or inconsistencies.

[0035] II. Accuracy and Efficiency Benefits of Bootstrapping Machine Learning Techniques Traditional histological image processing has relied heavily on annotation or manual input from users to identify regions and cells. Depending on the scale of the image, annotating cells can be a very time-consuming task. Furthermore, while a region is typically populated by a single cell type, it often contains a significant number of other types of cells (see Figure 2, which shows different regions and cell classes in an image of a breast cancer slide). Furthermore, due to the variability in the appearance of a given type of cell, the expertise required for cell annotation is high.

[0036] In order of cost, time required, and logistical complexity, annotation types can be ranked (from easiest to most difficult) as follows: 1. Image-level classification (e.g., labeling a slide as "invasive ductal carcinoma") 2. Region-level classification (e.g., labeling a tissue region as "containing primarily lymphocytes") 3. Object Center Localization (e.g., Placing a Seed on a Lymphocyte Cell) 4. Bounding box localization (e.g., placing a box around lymphocyte cells) 5. Object segmentation (e.g., drawing a polygon around a lymphocyte cell)

[0037] Therefore, the bootstrapping technique provided herein, in which a machine learning model is trained based on cell labels initially defined for corresponding regions, can significantly improve annotation efficiency. Annotators do not need to spend a lot of time labeling individual cells. Rather, efficiency can be improved by focusing annotation efforts on higher level regions. Furthermore, selection and strategic review and retraining can promote a high level of accuracy.

[0038] III. Process for Bootstrapping Machine Learning Techniques 3 illustrates a process 300 for generating and training a machine learning model for detecting and labeling objects in images. Process 300 begins at block 305 where region annotation is performed for each of a set of images. Each of the sets of images may correspond to a field of view of a histology slide. The fields of view may overlap each other across the set of images.

[0039] Region annotation may include, for each of the one or more regions, identifying a boundary (or other spatial characteristic) of the region within the image. Region annotation may be performed based on user input, may be performed semi-automatically using computer vision techniques, or may be performed automatically using computer vision techniques. Region annotation may further include annotation of a region-specific label for each region. The region-specific label may correspond to a characterization of a particular cell type and / or biological structure. For example, the region-specific label may include "tumor" or "tumor cell." If region annotation is performed using user input, the region-specific label may be identified by a user by entering free text or by selecting from a set of predefined label options.

[0040] Although not shown in Figure 3, object detection can occur before or after block 305. Object detection can be performed automatically using computer vision techniques. Detected objects can include biological cells and possibly other biological structures.

[0041] At block 310, object annotation bootstrapping is performed for each image in the set of images. Bootstrapping may include assigning, for each of at least some objects, a label that corresponds to the region in which the object resides. In some examples, bootstrapping may be selectively performed for objects that have spatial characteristics that correspond to cellular spatial characteristics (e.g., have areas, contrasts, and / or aspect ratios that correspond to comparable statistical profiles of cells, given a field of view of the image). Furthermore, bootstrapping need not be performed for objects that are not within a defined region.

[0042] At block 315, a machine learning model is trained using the training dataset including the bootstrapped annotations. Input to the machine learning model can include one or more image and object data, such as a center location of each detected object and / or a boundary of each detected object. In some examples, preprocessing can be performed to filter out a subset of the set of objects based on spatial characteristics of the objects (e.g., to filter out objects that have shapes that are not typical for cells and / or are close to the edge of the image and / or the edge of the region). Output of the machine learning model can include a label for each identified object and a confidence metric potentially associated with the label. The machine learning model can include a neural network, a convolutional neural network, a recurrent neural network, and / or a deep neural network. Training the model can include learning parameter values ​​that define how the input data is transformed into output data.

[0043] In block 320, the trained machine learning model is executed to generate object-specific labels. The object-specific labels generated in block 320 may be generated by processing images that were in the training dataset or by processing other images.

[0044] In block 325, the subset of object-specific labels generated in block 320 is modified using the region history. More specifically, the trained machine learning model may predict that multiple objects correspond to object-specific labels that are different from the region-specific labels of the regions in which the objects are located in the image. For some of these examples, the object-specific labels may be defined (in block 325) to be identical to the region-specific labels despite the model's opposite prediction. Region-prior labels may be assigned (for example) when the object has atypical (e.g., non-uniform and / or substantially asymmetric across axes) spatial characteristics and / or when the object type associated with the model-predicted label is unlikely to exist within the context of the slide associated with the image. For example, it is unlikely to observe vascular endothelial cells in a slide associated with a breast cancer sample.

[0045] In block 330, a number of object-specific labels are selected for evaluation. The number of object-specific labels selected for evaluation may be less than the number defined for a given slide, field, image, or data set. Selection may be based on a confidence metric and / or a sample selection technique (e.g., a pseudo-random selection technique). Information about the object-specific labels and / or associated objects may be made available to a reviewer. For example, an interface may be generated that includes the image and identifies the location of the object associated with the number of object-specific labels (e.g., by including a mark on the nucleus of a cell object and / or including an outline of the object).

[0046] In block 335, an augmented labeling is detected for a subset of the objects. The subset of objects may correspond to some or all of the plurality of object-specific labels selected for evaluation. The augmented labeling may include labels that are different from the object-specific labels generated by the machine learning model and / or labels that are the same as the object-specific labels generated by the machine learning model. For example, the interface provided to the reviewer may be configured to receive an input that identifies the object-specific labels of the objects. The object-specific labels may be identified explicitly (e.g., by receiving an explicit label identification that may be different or the same as the label generated by the machine learning model) or implicitly (e.g., by inferring that the presented object-specific label generated by the machine learning model is correct if no label to the contrary is provided). The object-specific labels identified by the reviewer (and / or approved by the reviewer) may be tagged as augmented labels.

[0047] At block 340, a quality metric can be generated for the enhanced labeling. The quality metric can be generated based on the reviewer's tracking experience, the length of time the review interface was presented, the percentage of identified label corrections, the consistency (e.g., across users and / or for the same user) in the identified labels across multiple presentations of the same data, the correlation between the likelihood that the label was corrected and the model output confidence score, etc. The quality metric can be assigned to a given dataset, slide, image, reviewer, image type, and / or field of view.

[0048] At block 345, the machine learning model is retrained using the augmented labeling. In some examples, a quality metric is used to weight the portion of the augmented labeling in the retraining. For example, augmented labeling from one reviewer and / or associated with one view may be weighted more highly compared to augmented labeling from another reviewer and / or associated with another view.

[0049] At block 350, new images are processed using the retrained machine learning model. In some examples, region annotation is not performed before processing the new images. In some examples, region annotation is performed as a pre-processing. Post-processing may also be performed (e.g., using region history to modify a subset of object-specific labels, as described herein). The output may be sent to and / or presented at a user device. The output may include one or more images, where objects are located within the images, and labels for each object. The output may additionally or alternatively include higher level information, such as a list of object-specific labels detected in the dataset, the amount of objects associated with each of the object-specific labels, and / or spatial distribution information corresponding to each of the object-specific labels.

[0050] IV. Working Examples The following examples relate to the use of various image processing techniques (including the bootstrapping technique disclosed herein) to generate object-specific labels. More specifically, 125 images were obtained from the TCGA dataset. Each image represents a field of a diagnostic breast cancer slide stained with H&E.

[0051] IV.A. Region annotation Figures 4A-E show images relevant for region annotation. Figure 4A shows an original RGB image of the TCGA dataset. Trained users then manually identified the boundaries of the various histological regions in the image. The users also identified region-specific labels for each region. Figure 4B shows a processed version of the original RGB image, where pixels are colored to indicate the region-specific label for the corresponding region in which the pixel is located.

[0052] In some examples, smoothing can be applied to the manual region annotations. The leftmost images in Fig. 4C and Fig. 4D also show the original RGB image from the TCGA dataset, and the rightmost images in Fig. 4C and Fig. 4D show the histological regions identified by user input. The middle images in Fig. 4C and Fig. 4D show the results of applying fully connected conditional random field smoothing to enhance the correspondence between the manual annotations and the tissue boundaries.

[0053] Figure 4E shows a processed version of the image from Figure 4B. In this example, the image in Figure 4B was processed by color thresholding the image in HSV space and then applying Gaussian smoothing and Otsu's thresholding.

[0054] Thus, FIG. 4 illustrates various techniques by which the regions may be defined (e.g., based purely on manual input, by applying random field smoothing to the manually annotated input, by applying another type of smoothing (e.g., Gaussian smoothing), and / or by applying one or more thresholding techniques).

[0055] IV.B. Application of non-machine learning models and image processing heuristics to generate bootstrapped nuclear annotations from region annotations Figures 5A-5F show the results of applying non-machine learning models and image processing heuristics to generate bootstrapped noisy nuclei annotations from region annotations. Notably, the thresholding and Gaussian smoothing performed to generate the images in Figure 5 resulted in less continuous regions than those in Figures 4B, 4C, and 4D, which may indicate that this technique better detects heterogeneous cell types within a region. The RGB image in Figure 4A was processed using computer vision techniques to detect individual cells. Each of the detected cells was first assigned an object-specific label that matched the region-specific label for the region of pixels. Figure 5A depicts an image with color shades indicating the object-specific labels assigned based on the region-specific labels depicted in Figure 4E. The red shades represent the tumor label, the blue shades represent the lymphocyte label, the green shades represent the fibroblast label, the orange shades represent the necrotic label, and the gray shades represent the "other" label. As can be seen, many cells within the stromal region are classified as fibroblasts. This represents a discrepancy between the labels. The cells should be labeled with lymphocyte labels.

[0056] A heuristic approach was applied to modify the label assignments. More specifically, shape and size constraints were applied to identify objects for which the initial label assignments should be removed. Figure 5B shows the object-specific label assignments after heuristic-based processing. Notably, the label modification affected many of the objects initially identified as lymphocytes within the stromal region.

[0057] Each of Figures 5C-5F shows how an initial label was assigned to the initial RGB image (leftmost image) based on the bootstrapping technique. In Figures 5C and 5D, the bootstrapping technique results in a correct classification of the majority of nuclei (compare the "intermediate" image, second from the right, with the "final" image, second from the left). On the other hand, in Figures 5E and 5F, the bootstrapping technique results in an incorrect assignment of labels to nuclei. Figure 5E shows a stromal region with lymphoid cells, and Figure 5F shows a stromal region with fibroblasts. Figure 5E shows the case where lymphocytes receive the incorrect label of "fibroblast" before applying a rule of thumb to correct the initial label.

[0058] IV.C. Predictions from trained machine learning models using bootstrapped annotations Figures 6A-C show object labels generated by models trained with bootstrapped annotations for three histological images. In each of Figures 6A-C, the top row shows the original image (right-most image), region annotations (second image from the right), and noisy bootstrapped nuclei (left-most image), while the bottom row shows predictions from a mask rcnn (recurrent convolutional neural network) model trained using a training dataset with bootstrapped and labeled information. Notably, the mask rcnn predictions are more extensive (do not match the inaccurate region annotation boundaries). Furthermore, the mask rcnn predictions are more "round" and match the nuclear contours better. In addition, nuclei that are not dominant in the region are correctly classified in the mask rcnn output, even if the bootstrapped labels (used to train the model) were inaccurate. This is most noticeable for lymphocytes in the stroma (A, B) and tumor (C) regions.

[0059] IV.D. Using Domain History to Correct Signs from Machine Learning Models Figure 7A shows the labels generated for an image by a masked recurrent convolutional neural network after the network was first trained on bootstrapped cell-specific labels. The model was trained to select for each cell a label for either tumor, fibroblast, lymphocyte, plasma cell nuclei, inflammatory infiltrate, and other nuclei.

[0060] FIG. 7B shows region-specific labels of the image. In particular, region-specific labels can be selected from among a larger set of labels. In the illustrated example, yellow regions are associated with necrotic labels and grey regions are assigned to glandular tissue regions. There are no corresponding labels available in the set of potential object-specific labels utilized by the neural network. Necrotic labels were omitted as a result of the high heterogeneity of spatial properties between cells, and glandular tissue labels were omitted as a result of the rarity of these regions / cells.

[0061] The rule set can then be applied to the output of the neural network to identify objects that are assigned a region-prior label instead of the label identified by the network. The rule set can include a first rule that specifies a confidence metric threshold such that each object-specific label associated with a confidence metric (output by the neural network) is changed to a label for the region that corresponds to the object. The rule set can include a second rule that indicates that if the spatial characteristics (e.g., size and shape) of the pre-identified object do not sufficiently correspond (at least to the pre-identified extent) to the spatial characteristics of the assigned label, the object-specific label is changed to a label for the corresponding region. Figure 7C shows the object-specific labels after application of the rule set. Notably, cells that were within the necrotic and glandular tissue regions are correctly labeled.

[0062] IV.E. Facilitate review of selected object-specific signage 8A-8D illustrate certain metrics used to determine which object-specific labels should be utilized by the reviewer, who can provide input indicating whether the assigned object-specific label is accurate. Specifically, objects can be filtered and / or ranked based on whether they are associated with one or more particular characteristics and / or characteristic values ​​that indicate the label may be inaccurate. FIG. 8A illustrates objects identified as corresponding to segmentation artifacts. FIG. 8B illustrates objects associated with areas that are below a predefined threshold or have an abnormal shape (e.g., as indicated by an aspect ratio above a predefined threshold). FIG. 8C illustrates objects that have been assigned object-specific labels corresponding to cell types not physiologically observed within the region type, as indicated by the corresponding region-specific labels. FIG. 8D illustrates objects associated with a confidence metric in which the object-specific labels are below a predefined threshold.

[0063] In some examples, each object may be assigned a score based on the confidence as to whether the object corresponds to a segmentation artifact, the extent to which the area of ​​the object is atypical (e.g., below a size threshold), the extent to which the shape of the object is atypical (e.g., has an aspect ratio above a ratio threshold), whether object-specific landmarks do not match spatially corresponding region-specific landmarks, and a confidence metric. Objects with scores that tend toward the extremes (indicating irregularity) may then be selected for review.

[0064] FIG. 9A shows multiple boxed regions within a slide. Each boxed region corresponds to a different field of view. The blue and green boxes correspond to fields of view that represent the complete set of fields across the slide. The red and purple boxes correspond to fields of view associated with more suspicious and / or object-specific labels that have been flagged for review. FIG. 9B shows a presentation of a particular field of view that may be presented to a user. A border is drawn around selected objects that have been flagged for review. Nuclear segmentation and shading was used to facilitate correction and communicate the classifier's certainty. Segmentation was also used to provide clarity as to which object was being corrected when a dot was placed for correction.

[0065] V. A Process for Using Machine Learning to Label Objects in Images 10 illustrates a process for using machine learning to assign labels to objects in an image. Process 1000 begins at block 1005 where an image is accessed. The image may correspond to and / or depict a histology slide. In some examples, a set of patches may be used to define portions of the image that may be processed independently to detect and characterize objects (e.g., in this case blocks 1010-1025 may be performed for each patch).

[0066] At block 1010, regions within the image are identified. The regions may be defined based on user input, which may identify boundaries of the regions. Alternatively, or additionally, the regions may be defined based on segmentation analysis.

[0067] Blocks 1015-1025 may be applied to each of the identified regions. In block 1015, a region-specific label is identified for each region. The region-specific labels may be identified based on user input identifying the labels and / or based on a computer vision classification approach.

[0068] At block 1020, objects within the region are identified. The objects may be identified using computer vision and / or segmentation analysis. The identified objects may include a set of cells.

[0069] In block 1025, an object-specific label is assigned to each object. In this block, the label assigned to an object is defined to be the same label assigned to the region surrounding the object. Thus, after this block, all objects within a particular region are assigned the same label.

[0070] At block 1030, a training data set is defined to include some or all of the object-specific labels. At block 1035, an extended labeling is defined for a subset of objects in the training data set. Block 1035 may include changing the selected object-specific label to an "other" category that is not used during training. The selected object-specific label may correspond to an object having characteristics (e.g., shape and / or size) indicative of a particular cellular state for which labeling is being attempted (e.g., necrosis), and the selected object-specific label may be non-physiological in a given context (e.g., considering other object-specific labels in the region and / or region-specific labels).

[0071] At block 1040, the image processing model may be trained using the training data. The image processing model may include a deep neural network. The image processing model may be configured to receive an image and an object specification (e.g., indicating where the objects are in the image) and output an indication of each object.

[0072] In block 1045, an incomplete subset of objects may be selected for review. The subset may include representative samples and / or samples whose label assignments are associated with a lower confidence metric.

[0073] At block 1050, an extended indicator may be identified for each of at least some of the subset of objects. The extended indicator may include an indicator identified by the reviewer. The extended indicator may be different from the object-specific indicator originally assigned to the object. In some examples, the extended indicator is the same as the object-specific indicator originally assigned to the object (but may have since been confirmed as accurate by the reviewer).

[0074] At block 1055, the augmented labels may be used to perform additional training of the image processing model. The new image may be processed to detect individual objects. At block 1060, the new image is processed using the image processing model to generate new object-specific labels. At block 1065, a representation of the new object-specific labels is output (e.g., transmitted or presented). The representation may include an image with one or more overlays that identify the detected object and / or indicate a particular label assigned to the object. The representation may additionally or alternatively include higher level data based on the new object-specific labels. The higher level data may identify a count of objects having each of the multiple labels, a percentage of objects having each of the multiple labels, a spatial distribution of the labels, etc. Thus, process 1000 facilitates generating accurate object-specific image annotations by using bootstrapped labeling and machine learning techniques.

[0075] VI. Further Considerations Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium including instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0076] The terms and expressions used are used as terms of description and not of limitation, and in the use of such terms and expressions there is no intention to exclude equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention as claimed. Thus, although the invention as claimed has been specifically disclosed by embodiments and optional features, it will be understood that modifications and variations of the concepts disclosed herein may be left to those skilled in the art, and such modifications and variations are considered to be within the scope of the invention as defined by the appended claims.

[0077] This specification provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, this description of preferred exemplary embodiments provides those skilled in the art with an enabling description for implementing various embodiments. It will be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the appended claims.

[0078] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments can be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other examples, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments.

Claims

1. 1. A computer-implemented method comprising: Accessing a set of images; For each image of the set of images: identifying a plurality of regions within the image; For each region of the plurality of regions, identifying a region-specific label for said region; Detecting a set of objects within the region; assigning to each object of the set of objects an object-specific indicator that is identical to the region-specific indicator assigned to the region within which the object is located; For each image of the set of images: - for each object in the set of detected objects, object position data indicative of position data of said object in the image; for each object of the set of detected objects, label data indicative of the object-specific label assigned to the object; defining a training dataset to include: training an image processing model using the training data, said training including learning a set of parameter values ​​for a set of parameters that define computations performed by the image processing model; The method includes:

2. selecting an incomplete subset of the set of objects for images of the set of images; facilitating one or more presentations, each presentation of the one or more presentations comprising: At least a portion of the image; indicating as one or more portions of the at least one portion of the image corresponding to one or more objects of the incomplete subset of the set of objects; and for each of said one or more objects, the object-specific indicator assigned to said object; facilitating one or more presentations, including: receiving, for each of at least some objects of the incomplete subset of the set of objects, response data identifying a different object-specific indicator for the object; performing further training of the image processing model using the different object-specific labels and the learned set of parameter values, the further training including generating a second set of parameter values ​​including new parameter values ​​for at least some of the parameters of the set of parameters that are different from corresponding parameter values ​​in the learned set of parameter values; The method of claim 1 further comprising:

3. Access to new images and processing the new image using the image processing model configured with the second set of parameter values, a result of the processing including a particular set of object specific indicators corresponding to a particular set of objects detected in the new image; outputting a representation of said particular set of object specific indicators; The method of claim 2 , further comprising:

4. each image of the set of images comprises a histological image depicting at least a portion of a slice of tissue; For each object of at least a portion of the set of detected objects, the object corresponds to a cell; The object-specific label corresponds to a cell type; The method of claim 1.

5. The method of claim 1 , wherein the image processing model comprises a deep convolutional neural network.

6. For each object in the set of objects: determining one or more characteristics of the object relating to a size, shape, texture, and / or color of the object; determining whether a label mismatch condition is met based on the one or more characteristics and stored label specific characteristic data associated with the object specific label assigned to the object; if it is determined that the indicator mismatch condition is satisfied, updating the object-specific indicator assigned to the object to a different object-specific indicator; The method of claim 1 further comprising:

7. For each image of the set of images: facilitating presentation of the image; and receiving a first user input defining a set of boundaries within the image, each region of the plurality of regions being identified as corresponding to an area within a boundary of the set of boundaries; receiving a second user input identifying a set of categories, each region-specific marking being identified based on a category of the set of categories; The method of claim 1 further comprising:

8. one or more data processors; a non-transitory computer-readable storage medium comprising instructions that, when executed on the one or more data processors, cause the one or more data processors to perform a sequence of operations, the sequence of operations comprising: Accessing a set of images; For each image of the set of images: identifying a plurality of regions within the image; For each region of the plurality of regions, identifying a region-specific label for said region; Detecting a set of objects within the region; and assigning to each object of the set of objects an object-specific indicator that is identical to the region-specific indicator assigned to the region within which the object is located; For each image of the set of images: - for each object in the set of detected objects, object position data indicative of position data of said object in the image; for each object of the set of detected objects, indicator data indicative of object specific data assigned to said object; defining a training data set, training an image processing model using the training data, said training including learning a set of parameter values ​​for a set of parameters that define computations performed by the image processing model; Including, the system.

9. The operation includes: selecting an incomplete subset of the set of objects for images of the set of images; facilitating one or more presentations, each presentation of the one or more presentations comprising: At least a portion of the image; an indication as one or more portions of the at least a portion of the image corresponding to the one or more objects of the incomplete subset of the set of objects; and for each of said one or more objects, the object-specific indicator assigned to said object; facilitating one or more presentations, including: receiving, for each of at least some of the objects of the incomplete subset of the set of objects, response data identifying a different object-specific indicator of the object; performing further training of the image processing model using the different object-specific labels and the learned set of parameter values, the further training including generating a second set of parameter values ​​including new parameter values ​​for at least some of the parameters of the set of parameters that are different from corresponding parameter values ​​in the learned set of parameter values; The system of claim 8 further comprising:

10. The operation includes: Access to new images and processing the new image using the image processing model configured with the second set of parameter values, a result of the processing including a particular set of object specific indicators corresponding to a particular set of objects detected in the new image; outputting a representation of said particular set of object specific indicators; The system of claim 9 further comprising:

11. each image of the set of images comprises a histological image depicting at least a portion of a slice of tissue; For each object of at least a portion of the set of detected objects, the object corresponds to a cell; The object-specific label corresponds to a cell type; The system of claim 8.

12. The system of claim 8 , wherein the image processing model includes a deep convolutional neural network.

13. The operation comprises, for each object of the set of objects: determining one or more characteristics of the object relating to a size, shape, texture, and / or color of the object; determining whether a label mismatch condition is met based on the one or more characteristics and stored label specific characteristic data associated with the object specific label assigned to the object; if it is determined that the indicator mismatch condition is satisfied, updating the object-specific indicator assigned to the object to a different object-specific indicator; The system of claim 8 further comprising:

14. The operation includes, for each image of the set of images: facilitating presentation of the image; and receiving a first user input defining a set of boundaries within the image, each region of the plurality of regions being identified as corresponding to an area within a boundary of the set of boundaries; receiving a second user input identifying a set of categories, each region-specific marking being identified based on a category of the set of categories; The system of claim 8 further comprising:

15. A computer program product tangibly embodied in a non-transitory machine-readable storage medium comprising instructions configured to cause one or more data processors to perform a sequence of operations, said sequence of operations comprising: Accessing a set of images; For each image of the set of images: identifying a plurality of regions within the image; For each region of the plurality of regions, identifying a region-specific label for said region; Detecting a set of objects within the region; and assigning to each object of the set of objects an object-specific indicator that is identical to the region-specific indicator assigned to the region within which the object is located; For each image of the set of images: - for each object in the set of detected objects, object position data indicative of position data of said object in the image; for each object of the set of detected objects, label data indicative of the object-specific label assigned to the object; defining a training data set, training an image processing model using the training data, said training including learning a set of parameter values ​​for a set of parameters that define computations performed by the image processing model; 2. A computer program product comprising:

16. selecting an incomplete subset of the set of objects for images of the set of images; facilitating one or more presentations, each presentation of the one or more presentations comprising: At least a portion of the image; an indication as one or more portions of the at least a portion of the image corresponding to one or more objects of the incomplete subset of the set of objects; and for each of said one or more objects, the object-specific indicator assigned to said object; facilitating one or more presentations, including: receiving, for each of at least some of the objects of the incomplete subset of the set of objects, response data identifying a different object-specific indicator of the object; performing further training of the image processing model using the different object-specific labels and the learned set of parameter values, the further training including generating a second set of parameter values ​​including, for at least some parameters of the set of parameters, new parameter values ​​different from corresponding parameter values ​​in the learned set of parameter values; 16. The computer program product of claim 15, further comprising:

17. Access to new images and processing the new image using the image processing model configured with the second set of parameter values, a result of the processing including a particular set of object specific indicators corresponding to a particular set of objects detected in the new image; outputting a representation of said particular set of object specific indicators; 20. The computer program product of claim 16, further comprising:

18. each image of the set of images comprises a histological image depicting at least a portion of a slice of tissue; for each object of at least a portion of the set of detected objects; the object corresponds to a cell; The object-specific label corresponds to a cell type; 16. A computer program product according to claim 15.

19. 16. The computer program product of claim 15, wherein the image processing model comprises a deep convolutional neural network.

20. for each object in said set of objects, determining one or more characteristics of the object relating to a size, shape, texture, and / or color of the object; determining whether a label mismatch condition is met based on the one or more characteristics and stored label specific characteristic data associated with the object specific label assigned to the object; if it is determined that the indicator mismatch condition is satisfied, updating the object-specific indicator assigned to the object to a different object-specific indicator; 16. The computer program product of claim 15, further comprising:

Images