Mitotic identification by selective multi-step deep learning analysis

Abstract

Mitosis may be detected using a two-step detection method. This type of detection method may include obtaining a detection output by applying a detection model to a tissue image. Subsequently, for each core in the set of cores identified as undergoing mitotic in the detection output, a deterministic determination is made for that core. Then, for each core in the subset of cores, a classification model may be applied to the core based on a relationship between the confidence of the core and a first threshold. An output may then be provided that identifies each core in the detection output that is identified as experiencing mitotic and whose confidence is above a first threshold as being experiencing mitotic. Further, the output may identify each of the subset of cores that is identified as being subjected to mitosis in the classification results obtained for that core as being subjected to mitosis.

Description

Technical Field

[0001] The present disclosure relates to processing histological images using deep learning techniques to identify cells undergoing mitosis. Background Art

[0002] One of the most important topics in microscopy imaging is the classification of cells, typically stained with hematoxylin and eosin (H&E) dyes. A particularly challenging task in this field is the detection of mitotic figures—cells undergoing division—in tumor tissue. Mitotic figures are defined histologically by the presence of hair-like projections containing chromosomes (nuclear material) in the absence of a nuclear membrane. A common quantitative method is the mitotic count (MC), which involves counting mitotic figures within a standard-sized area of ​​the tumor where the tumor is assumed to have the highest mitotic density. The number of mitotic figures is generally considered one of the most powerful predictors of the biological behavior of many tumor types in humans and animals. However, there are many difficulties associated with this type of mitotic count. For example, even when histological images are annotated by pathologists, studies have found that there can be significant disagreement between human annotators of approximately 17–34% in distinguishing individual mitotic figures from other cellular structures. Similarly, algorithmic approaches have difficulty distinguishing mitotic figures from other cellular structures and can also be expensive in terms of the time and computational resources required to apply the difficult task of trying to interpret mitotic figures. Therefore, there is a need for improved techniques for automatically identifying mitotic figures in histological images. Summary of the Invention

[0003] The techniques disclosed herein are susceptible to implementation in various ways. For example, a method for detecting mitosis in a tissue image using two-step detection can be implemented. This method can include receiving a tissue image. After receiving the tissue image, the method can proceed by applying a detection model to the tissue image to obtain a detection output. In this type of method, the detection output can include, for each nucleus in a set of nuclei in the tissue image, identifying the nucleus as undergoing mitosis and a confidence level for identifying the nucleus as undergoing mitosis. The described method can also include, for each nucleus in the set of nuclei, performing a deterministic determination, including determining a relationship between the confidence level for the nucleus and a first threshold. Following this determination, the method can include, for each nucleus in a first subset of nuclei, applying a classification model to the nucleus based on the relationship between the confidence level for the nucleus and the first threshold, obtaining a classification result indicating whether the nucleus is undergoing mitosis, wherein, for each nucleus in the first subset of nuclei, the confidence level for the nucleus is no greater than the first threshold. Finally, the method can include providing an output, wherein the output identifies each nucleus in the set of nuclei identified as undergoing mitosis in the detection output and having a confidence level greater than the first threshold as undergoing mitosis. The output may also identify each nucleus in the first subset of nuclei that is indicated as undergoing mitosis in the classification result obtained for the nucleus by applying the classification model as undergoing mitosis.

[0004] Other types of implementations, including systems and computer-readable media for performing the described methods, are also possible and will be apparent to those skilled in the art based on this disclosure. Therefore, the example methods provided in this disclosure should be understood to be illustrative only and should not be considered as limiting. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] While the specification concludes with claims which particularly point out and distinctly claim the invention, it is believed the invention will be better understood from the following description of certain examples taken in conjunction with the accompanying drawings, wherein like reference numerals represent like elements, and wherein:

[0006] Figure 1 Describes the architecture of a detection model that can be used to detect mitosis and mitotic pseudonuclei in images;

[0007] Figure 2 It shows how to conceptualize the transformed image as being covered by multiple sets of anchor boxes;

[0008] Figure 3 An example of a process that can be used to create annotated images is shown;

[0009] Figure 4Provides a schematic diagram showing how two stages can be used to extract features at different resolutions, and how the outputs of these stages are combined in the process of creating an annotated image;

[0010] Figure 5 A method is shown that can improve the identification of mitosis and mitosis-like nuclei, such as can be provided by a detection model; and

[0011] Figure 6 is a block diagram of a computing device that can be used to implement aspects of the disclosed technology.

[0012] The accompanying drawings are not intended to be limiting in any way, and it is contemplated that various embodiments of the invention may be carried out in a variety of other ways, including those not necessarily depicted in the drawings. The accompanying drawings, which are incorporated in and form a part of this specification, illustrate several aspects of the invention and, together with the description, serve to explain the principles of the invention; it should be understood, however, that the invention is not limited to the precise arrangements shown. DETAILED DESCRIPTION

[0013] The present disclosure relates to apparatus, systems, and methods for identifying mitotic figures in histological images. As described in detail herein, this type of mitotic figure identification can be performed in a selective, multi-step manner, using a detection model to detect mitoses and mitotic-pseudo-nuclear blocks in an image, and using a classification model to classify nuclear blocks whose detection as mitoses or pseudo-nuclear blocks is uncertain. By employing this type of selective, multi-step approach, the disclosed techniques can allow for improved accuracy and / or time cost relative to other types of mitotic detection.

[0014] To illustrate how to implement the disclosed technology, Figure 1 Describes a detection model architecture that can be used to detect mitosis and mitotic pseudonuclei in images. Figure 1 In the architecture of

[0045] , an input image 101 (e.g., a portion of an H&E stained tissue image) may initially be provided to a feature extraction network 102. The feature extraction network 102 may then apply a set of filters to the input image 101 to obtain a transformed image 103 having a channel for each filter applied by the feature extraction network 102. For example, the feature extraction network 102 may include a first filter as shown in Table 1 (the first filter may be convolved with the input image 101 to identify vertical edges) and a second filter as shown in Table 2 (the second filter may be convolved with the input image 101 to identify horizontal edges).

[0015] [ -1 0 1 ]

[0016] [ -2 0 2 ]

[0017] [ -1 0 1 ]

[0018] Table 1 [ 1 2 1 ] [ 0 0 0 ]

[0021] [ -1 -2 -1 ]

[0022] Table 2

[0023] The outputs of these filters can then be combined into a transformed image 103, where the image created with the first filter is considered to be the first channel in the transformed image 103, the image created with the second filter is considered to be the second channel in the transformed image 103, and any other images created with other filters are considered to be other channels in the transformed image 103.

[0024] Once the transformed image 103 is created, Figure 1 In a system with a detection model architecture, features in the transformed image 103 can be used to identify mitosis and pseudo-nuclei and their locations. Figure 1 As shown, this can be achieved by using two different networks in parallel, a classification network 104 and a box regression network 105, to process the transformed image 103. The role played by each of these networks and how they can potentially be implemented are discussed below.

[0025] Turning first to the classification network 104, this network can be used to generate, for each pixel in the transformed image 103, an indication of the confidence with which the anchor box at that location can be classified as depicting mitosis or a mitosis-like nucleus. To illustrate, consider Figure 2 , which shows how the transformed image 103 (shown in a flattened manner rather than as a superposition of individual channels for ease of understanding) can be conceptualized as being covered by multiple sets 201, 202, 203 of anchor boxes. Each of these sets 201, 202, 203 of anchor boxes forms a tile on the transformed image 103, where the first set 201 of anchor boxes includes anchor boxes with an aspect ratio of 1.2, the second set 202 of anchor boxes includes anchor boxes with an aspect ratio of 2:1, and the third set 203 of anchor boxes includes anchor boxes with an aspect ratio of 1:1. These sets 201, 202, 203 of anchor boxes can be superimposed on the transformed image 103 to obtain a processed image 204, where each pixel in the transformed image 103 is covered by one anchor box from each of the sets 201, 202, 203 of anchor boxes (i.e., each pixel is covered by one 1:2 anchor box, one 2:1 anchor box, and one 1:1 anchor box).

[0026] In order to provide an indication of the confidence with which the anchor box at each location in the transformed image 103 can be classified as depicting mitosis or a mitosis-like nucleus, the classification network 104 can be composed of one or more preparation layers, wherein each preparation layer includes a convolution filter followed by an activation function, and the output of one of the preparation layers is provided as the input of the next. For example, the first preparation layer can take the transformed image 103 as input, convolve the transformed image with a 3x3 filter (as shown in Table 1 or Table 2), and then apply an activation function (e.g., ReLU activation) to each pixel in the convolved transformed image to obtain an output, which is provided as the input of the second preparation layer, and so on until each of the one or more preparation layers comprising the classification network is applied. The classification network 104 may also include an output layer that applies a filter to each set of anchor boxes and each set of categories (e.g., in the case of a set of three anchor boxes and three potential categories of objects to be identified, a total of nine filters), followed by an activation function (e.g., a sigmoid activation function) to provide a value for each anchor box and each category for each pixel in the transformed image. These values ​​can then be considered as confidence values ​​for each potential identification (e.g., in the case of applying two filters to each anchor box in the output layer, the result of applying the activation function to the value provided by the first filter can be considered as the confidence of mitosis, and the result of applying the activation function to the value provided by the second filter can be considered as the confidence of mitosis-likeness). Alternatively, in some cases, some additional processing can be applied to obtain the confidence values. For example, in the case where the classification network 104 uses the categories of mitosis, mitosis-likeness, and background, a softmax function can be applied to the category value of each anchor box to provide a confidence value that each anchor box can be assigned to each category.

[0027] Turning next to the box regression network 105, this network can operate in a manner similar to the discussion above for the classification network 104. That is, it can use a set of anchor boxes and one or more preparation layers, followed by an output layer, to generate a value for each anchor box in the set of applied anchor boxes. However, while the box regression network 105 can operate in a similar manner to the classification network 104, there may also be some differences. For example, while the discussion of the classification network 104 described how its output layer applies a filter to each class and each anchor box, the output layer of the box regression network can simply apply four filters to each anchor box, thereby providing four values ​​for each pixel for each anchor box covering that pixel. These values ​​can then be viewed as encoding the left, right, top, and bottom offsets of the associated anchor box relative to the actual bounding box of any kernel that may exist (in whole or in part) in that anchor box.

[0028] In use Figure 1 In a system with an architecture of , after applying the classification network 104 and the box regression network 105 , the outputs of these networks can be used to create an annotated image 106 . Figure 3 An example of a process that can be used for this purpose is shown in . Initially, in the process, a category and confidence can be determined 301 for one of the anchor boxes in the processing image 205. This can include initially determining the category of the anchor box, for example by looking at the highest confidence category for each pixel covered by the anchor box, and considering the anchor box to have the category corresponding to the largest number of covered pixels. This category can then be used to determine the confidence of the anchor box, for example by averaging the highest confidence for each pixel whose highest confidence category is the same as the category of the anchor box. This process can then be repeated until a category and confidence are determined 301 for each anchor box in the processing image (i.e., for each anchor box in each of the sets 201, 202, 203 of anchor boxes).

[0029] Once the category and confidence are determined for each anchor box, Figure 3 The process can then proceed to identify 302 the most confident box for the location in the processed image 205. This can be achieved by determining, for each anchor box covering the location in the processed image, which of these boxes has the highest confidence as previously determined 301. It can then be determined 303 whether the most confident anchor box depicts mitosis or a mitotic pseudo-nucleus. In the case where the categories assigned by the classification network 104 include mitotic nuclei, mitotic pseudo-nuclei, and one or more other categories, this determination 303 can be performed by simply checking whether the category of the most confident anchor box is mitotic nuclei or mitotic pseudo-nuclei. Alternatively, in the case where the classification network 104 only assigns the categories of mitotic nuclei and mitotic pseudo-nuclei, this determination 303 can be performed by checking whether the confidence value of the anchor box is below a threshold that separates boxes that are more likely to reasonably depict mitotic nuclei or non-mitotic nuclei from boxes that are less likely to reasonably depict either type of nucleus. Figure 3 The process 304 can then determine 306 whether there are more locations (either directly, or after creating 304 and adding 305 the bounding boxes as described below), for example by checking whether there are any locations not covered by the anchor boxes that were not determined 303. If there are more locations, the process can continue to the next location (e.g., a location not covered by the previously processed anchor boxes) and repeat the above identification 302 and determination 303. Otherwise, if there are no remaining locations, the process can end 307, for example by providing an annotated image 106 that includes identifications and bounding boxes of mitosis and / or mitosis pseudo-nuclei in the original input image 101.

[0030] Returning now to determining 303 whether the anchor box depicts a mitotic nucleus or a mitotic pseudo-nucleus, if the anchor box does depict such a nucleus, a bounding box can be created 304 from the anchor box. For example, this can be achieved by combining (e.g., averaging) the offsets determined by the box regression network 105 for each of the pixels covered by the anchor box in question, and then moving the top, bottom, left, and right sides of the anchor box based on these combined offsets. Once created 304, the anchor box can be added 305 as an annotation to the annotated image. This can be achieved by adding the bounding box and / or associated category as metadata that can be displayed in a separate layer on the annotated image, adding the shape of the bounding box directly on a copy of the input image 101 in a color corresponding to the appropriate category, or in any other appropriate way to show where the appropriate nucleus can be found in the input image, taking into account the constraints of the particular embodiment and the applications that the particular embodiment may support. Once the bounding boxes are added 305, the process can continue at the next location if it is determined 306 that more locations exist, or if all locations have been processed, the method can end 307 where all mitosis and mitosis-pseudo-nuclei in the original input image 101 have been detected and marked on the annotated image 106.

[0031] When implemented as Figure 1 When the system is described in the context of , it is possible to train the various parameters of the various networks using a dataset comprising a large number of images in which mitotic and mitotic pseudo-nuclei and their associated bounding boxes have been identified. For example, such a dataset can be obtained by simply asking a human domain expert to identify mitotic nuclei and mitotic pseudo-nuclei in training images and then applying bounding boxes to these nuclei by either adding bounding boxes by the human domain expert or automatically placing bounding boxes on the nuclei based on the expected size of the nuclei in a given image. Alternatively, in some cases, a panel of human reviewers can be asked to independently identify mitotic nuclei in the training images. In this case, nuclei identified as mitotic nuclei by all panel members (preliminarily or after consensus), or by a majority of panel members (if consensus cannot be reached), can be considered mitotic nuclei, while nuclei initially identified as mitotic nuclei by at least one panel member but ultimately not identified as mitotic nuclei can be considered mitotic pseudo-nuclei. These images can then be used to train the network by providing feedback on the difference between the outputs of the various networks and the actual correct labels and bounding boxes using a loss function that reduces the weights of easy decisions so that they do not dominate the training and overload learning that can be provided by difficult cases. An example of such a loss function is provided below, as shown in Equation 1, however other types of loss functions, such as cross entropy loss or balanced cross entropy loss are also possible and can be used in some cases.

[0032] Loss(pt )=-(1-p t ) γ *log(p t )

[0033] Equation 1

[0034] In equation 1, if the truth value is equal to 1, then p t is equal to the true value, otherwise equal to 1-p, where p is the value provided by the trained network. γ is an adjustable parameter whose value is preferably between 0 and 5, inclusive.

[0035] Although the above discussion describes how to use Figure 1 The architecture shown is used to detect mitotic and non-mitotic nuclei, but it should be understood that this discussion is illustrative only and that many variations of this discussion are possible and can be used in systems based on the present disclosure. For example, while the above discussion describes how the classification network 104 and the box regression network 105 use convolutional filters to provide their outputs (including confidence values), other types of implementations of the classification network 104 and the box regression network 105 are also possible and will be immediately apparent to those skilled in the art based on this disclosure. For example, these networks can be implemented using one or more dense layers, i.e., neural networks that take as input all the values ​​produced by the previous network. Similarly, while the examples given in Tables 1 and 2 are 3x3 filters, filters of other sizes, such as larger filters (e.g., 4x4, 5x5) or smaller filters (e.g., 2x2 or 1x1) are also possible and can be used in addition to or as an alternative to the 3x3 filters shown in Tables 1 and 2.

[0036] exist Figure 1 Variations are also possible in other aspects of

[0045] , such as in potential implementations of the feature extraction network 102. For example, in some cases, the feature extraction network 102 may include multiple stages, such as to facilitate identification of features that may be more pronounced at different scales. The outputs of the various stages may then be used to create a collection of intermediate annotated images, which may ultimately be combined into a

[0046] image. Figure 1 The final annotated image 106 is shown in . For illustration, Figure 4 A schematic diagram is provided showing how two stages are used to extract features of different resolutions and how the outputs of these stages are combined in the process of creating the annotated image 106. Figure 4In the schematic diagram of , a high-resolution input (which would be the input image 101 if there are only two stages, or the output of the previous stage if there are more than two stages) is provided as input to an intermediate stage 401. At the intermediate stage 401, the high-resolution input is convolved with one or more filters 402 to obtain a high-resolution feature map. This high-resolution feature map is downsampled using a pooling operation 403 (e.g., an operation that replaces the values ​​in a 2×2 window with a single value, such as the average or maximum of the values ​​in the original window) to produce a reduced-resolution output 404. This reduced-resolution output 404 is then provided as input to a terminal stage 405, which creates a terminal-resolution feature map 408 using its own one or more filters 406 and pooling operations 407 (each of which can be the same or different from the corresponding item used by the previous stage).

[0037] Once created, the end-resolution feature map 408 can be provided as input to the classification network 104 and the box regression network 105 and is processed by the classification network 104 and the box regression network 105 in a manner similar to Figure 1 The annotated image 106 is generated from the transformed image 103 in the manner described in the context of FIG. The result of this processing will be an annotated image at the final resolution 409, which can then be combined with the output of the next highest resolution stage 401 (e.g., by upsampling the result of processing the output of the final stage and then performing a pixel-by-pixel addition with the output of the next higher resolution stage). The result of this combination can then be processed by the classification network 104 and the box regression network 105 (these networks are depicted as being the same as those used to process the output of the final stage, although in some implementations they may be different), and they can provide an annotated image at the resolution of the intermediate stage 401. This process can then be repeated for however many stages there are in applicable implementations until the final high-resolution annotated image 106 is created.

[0038] Apart from Figure 4 Other variants besides multi-stage processing and Figure 1Variants of the classification and box regression networks described in the context of are also possible. For example, mitosis and mitosis pseudonuclei can use methods such as those described by Haoqing Ren, Kaiming He, Ross Girshick, and Jian Sun in Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, arXiv:1506.01497, by Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi in You Only Look Once: Unified Real-Time Object Detection, arXiv:1506.02640, by Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollar in Focal Loss for Dense Object Detection, arXiv:1708.02002, and by Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath The method described by Hariharan and Serge Belongie in "Feature Pyramid Networks for Object Detection, arXiv:1612.03144" is used for detection, all of which are incorporated herein by reference in their entirety. Therefore, the above discussion on how to detect mitosis and mitosis pseudonuclei should be understood as merely illustrative and should not be regarded as limiting.

[0039] Now go to Figure 5 , which illustrates a method for improving the identification of mitosis and mitosis-like nuclei, as provided by the detection model described above. As shown in the figure, identification can begin by segmenting 501 an image of a slice of H&E stained tissue into a plurality of blocks. For example, in performing the following steps using a detection model configured to detect mitosis and mitosis-like nuclei in 256x256 pixel regions: Figure 5 In the case of the method shown in , this segmentation 501 can be performed by dividing the larger image of the H&E stained tissue section into a set of 256x256 pixel blocks.

[0040] Once the image is segmented 501 into blocks, a detection model can be applied 502 to the blocks. This can be done by extracting one of the blocks from the original image (e.g., top left block, bottom right block, random block, etc.) and applying 502 object detection (as described above in Figure 1-4 Once target detection is complete, each identified mitosis or mitosis-like nucleus can be subjected to further processing to ensure the accuracy of the final output of the method. For example, Figure 5 As shown, it can be determined 503 whether a mitosis or mitosis-pseudo-nucleus is associated with a confidence value that is below a low threshold. If so, the nucleus can be discarded 504. For example, where the detection model identifies each nucleus in a block as a mitosis or mitosis-pseudo-nucleus and provides a confidence score corresponding to the identification, if the confidence score is below a threshold associated with a portion of the image that should be identified as background rather than a mitosis or mitosis-pseudo-nucleus, the nucleus can be discarded 504 (e.g., no longer considered a nucleus for execution). Figure 5 (further processing of the system of the method).

[0041] If an object detected by the detection model is not discarded 504, it can be determined 505 whether it is associated with a high confidence score. This can be achieved by comparing the confidence score associated with the object to a threshold established during training that is sufficient to indicate that the detection model's identification of the object is correct. For example, if an identification with a confidence above a certain level is found to have a 99% or greater chance of being correct, then that level can be used as a threshold for determining 505 whether the object is associated with a high confidence score. If the object is associated with a high confidence score, then the identification of the object obtained by applying 502 the detection model can be used as 506 the final classification of the object. That is, if the detection model identifies the object as a mitotic nucleus, then in the final output of the method it can be treated as a mitotic nucleus, and if the detection model identifies the object as a mitotic pseudo-nucleus, then in the final output of the method it can be treated as a mitotic pseudo-nucleus. Alternatively, if it is determined 505 that the object does not have a high confidence score, then a classification model can be applied 507 to determine how it should be treated in the final output of the method. The classification model can be such as Figure 1-4The detection model shown and described in the context of , except that the training data for the model may be limited to objects whose identification by the initial detection model is associated with a confidence that is not high enough to be classified as high, but not low enough to cause these objects to be discarded (for ease of reference, these objects are referred to as "uncertain cases"). Alternatively, in some cases, the classification model can be a different type of model than the model used in the initial detection. For example, if the initial detection is performed using a RetinaNET detection model trained on large-scale annotated H&E-stained images, the subsequent classification can be performed using a VGG19 classification model (trained using RetinaNET's uncertain cases).

[0042] After determining the appropriate treatment for a particular nucleus (e.g., discarded 504, treated as having a classification provided by a detection model 506, or treated as having a classification provided by a classification model 507), the process can be repeated until it is determined 508 that no additional nuclei are in the particular block being analyzed. The detection model can then be applied 502 to the next block in the H&E stained slide image, and the entire process repeated until it is determined 509 that all blocks have been analyzed. Once all blocks have been analyzed, Figure 5 The process can end by outputting 510 an annotated image. This can be achieved, for example, by taking the processed blocks, adding the appropriate identifications of mitoses and mitotic pseudo-nuclei (i.e., those determined by applying 502 the detection model if they are associated with high confidence scores, and those determined by applying 507 the classification model if they are not associated with high confidence scores but have not been discarded 504), and reassembling them into a complete image for subsequent presentation to a physician or other application. In this way, Figure 5 The process shown can selectively apply a secondary classification model, allowing accurate identification of mitosis and mitosis-pseudonuclei without incurring the computational overhead cost of applying a secondary model, unless the initial detection model fails to identify the object with sufficient (or low enough) confidence to allow it to be classified or discarded based solely on the initial detection model.

[0043] Variations of the above methods and examples may also be used to provide the same or similar benefits as those discussed above. For example, while Figure 5 The discarding 504 of objects associated with confidence values ​​below a low threshold is shown, but in some cases, objects that should be discarded may be identified as such by the detection model itself (e.g., by classifying them in a background category, or in a category that is not relevant to a particular application and can therefore be discarded in that application), in which case the separate low confidence value determination 504 may be omitted. As another example of a potential variation, although with Figure 5The associated discussion describes identifying nuclei as indeterminate (505) and classified (507) on a nucleus-by-nucleus basis, but it is possible to reframe Figure 5 , so that all nuclei in the block (or all nuclei in the initial slice image) are evaluated, and then after all nuclei have been evaluated, all nuclei that are evaluated as uncertain are classified.

[0044] Based on the present disclosure, variations in the types of classification that can be performed are also possible. For example, while the examples discussed potentially classify objects as mitotic and mitotic-pseudo-nuclei, in some cases, objects may be classified only as mitotic nuclei, where a confidence value associated with the classification is used to determine whether any particular nucleus will actually be considered a mitotic nucleus (e.g., if the confidence value from the detection model is above a threshold, the nucleus will be considered a mitotic nucleus, if the confidence value from the detection model is below a second threshold, the nucleus will be discarded, and if the confidence value from the detection model is between these two values, the nucleus can be provided to a secondary classification model, which will provide a confidence value that will be considered to indicate a mitotic nucleus if and only if the confidence value is above the threshold). In addition, while the above examples discuss identifying mitosis and mitotic-pseudo-nuclei in H&E stained images, the same methods can also be applied to identifying mitosis and mitotic-pseudo-nuclei in other types of images, or to distinguishing other types of difficult-to-distinguish objects in H&E stained images or other types of images. Therefore, the above methods and applications should be understood as exemplary only and should not be regarded as limiting.

[0045] There may also be variations on the benefits that may be obtained by using the disclosed techniques, and the benefits discussed above are not the only possible ones. To illustrate, consider the problem of false positives and negatives in a system that classifies mitotic nuclei based on confidence values. Because it is difficult to identify mitotic nuclei, using any particular threshold to separate mitotic nuclei from other types of objects will run the risk that some mitotic nuclei will be below the threshold and therefore not be identified. On the other hand, the lower the threshold, the greater the risk of false positive identification, so the threshold for considering an object as a mitotic nucleus can be set at a level that can be expected to achieve a particular balance of false negatives and false positives suitable for a particular application. However, by using a threshold such as Figure 5The method shown in can lower the threshold for potentially identifying a nucleus as a mitotic nucleus to below a threshold that would be expected to provide an appropriate number of false positives and false negatives for the detection model alone, as the classification model will be better able to distinguish between uncertain cases, thereby allowing for a reduction in false negatives without increasing the number of false positives. Similarly, the confidence level available for accepting a classification of a mitotic nucleus from the detection model can be increased, thereby reducing the number of false positives without increasing the risk of false negatives. Thus, the use of a secondary classification model can improve the sensitivity and specificity of automated object detection, not only by improving the classification of uncertain cases for the detection model, but also by reducing the number of false positives and / or false negatives generated by the detection model.

[0046] Figure 6 is a block diagram illustrating an example computing device 600 that can be used in conjunction with various embodiments described herein. For example, the computing device 600 can be programmed to use a method such as implementing Figure 1 、 2 and 4, and using software such as the architecture shown and discussed in the context of Figure 3 and 5 The classification of mitosis and mitotic pseudonuclei can be performed using the methods shown and discussed in the context of FIG. 1 . The computing device 600 can be a server or any conventional personal computer, or any other processor-enabled device capable of wired or wireless data communication. Other computing devices, systems, and / or architectures can also be used, including devices that are not capable of wired or wireless data communication, as will be apparent to those skilled in the art.

[0047] The computing device 600 preferably includes one or more processors, such as a processor 610. The processor 610 can be, for example, a CPU, a GPU, a TPU, or an array or a combination thereof, such as a CPU and TPU combination or a CPU and GPU combination. Additional processors can be provided, such as auxiliary processors for managing input / output, auxiliary processors for performing floating-point mathematical operations (e.g., TPUs), specialized microprocessors with architectures suitable for rapidly executing signal processing algorithms (e.g., digital signal processors, image processors), slave processors (e.g., back-end processors) subordinate to the main processing system, additional microprocessors or controllers for dual-processor systems or multi-processor systems, or coprocessors. Such auxiliary processors can be discrete processors or can be integrated with the processor 610. Examples of CPUs that can be used with the computing device 600 are Pentium processors, Core i7 processors, and Xeon processors, all of which are available from Intel Corporation of Santa Clara, California. An example GPU that can be used with the computing device 600 is the Tesla K80 GPU from Nvidia Corporation of Santa Clara, California.

[0048] The processor 610 is connected to the communication bus 605. The communication bus 605 may include a data channel for facilitating information transfer between the memory and other peripheral components of the computing device 500. The communication bus 605 may also provide a set of signals for communicating with the processor 610, including a data bus, an address bus, and a control bus (not shown). The communication bus 505 may include any standard or non-standard bus architecture, such as a bus architecture that conforms to the Industry Standard Architecture (ISA), the Extended Industry Standard Architecture (EISA), the Micro Channel Architecture (MCA), the Peripheral Component Interconnect (PCI) local bus, or a bus architecture that conforms to a standard promulgated by the Institute of Electrical and Electronics Engineers (IEEE), including IEEE 488 general-purpose interface bus (GPIB), IEEE 696 / S-100, etc.

[0049] The computing device 600 preferably includes a main memory 615 and may also include a secondary memory 620. The main memory 615 provides storage of instructions and data for programs executed on the processor 610 (e.g., one or more functions and / or modules discussed above). It should be understood that the computer-readable program instructions stored in the memory and executed by the processor 610 can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source code or object code written and / or compiled in any combination of one or more programming languages ​​(including but not limited to Smalltalk, C / C++, Java, JavaScript, Perl, Visual Basic, .NET, etc.). The main memory 615 is typically a semiconductor-based memory, such as dynamic random access memory (DRAM) and / or static random access memory (SRAM). Other semiconductor-based memory types include, for example, synchronous dynamic random access memory (SDRAM), Rambus dynamic random access memory (RDRAM), ferroelectric random access memory (FRAM), etc., including read-only memory (ROM).

[0050] The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0051] The secondary storage 620 may optionally include an internal memory 625 and / or a removable medium 630. The removable medium 630 is read and / or written in any known manner. The removable storage medium 630 may be, for example, a tape drive, a compact disc (CD) drive, a digital versatile disc (DVD) drive, other optical disc drives, a flash drive, etc.

[0052] The removable storage medium 630 is a non-transitory computer-readable medium having computer executable code (ie, software) and / or data stored thereon. The computer software or data stored on the removable storage medium 630 is read into the computing device 600 for execution by the processor 610.

[0053] Secondary memory 620 may include other similar elements for allowing computer programs or other data or instructions to be loaded into computing device 600. Such devices may include, for example, external storage media 645 and a communication interface 640 that allows software and data to be transferred from external storage media 645 to computing device 600. Examples of external storage media 645 may include an external hard drive, an external optical drive, an external magneto-optical drive, etc. Other examples of secondary memory 620 may include semiconductor-based memory, such as programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), or flash memory (a block-oriented memory similar to EEPROM).

[0054] As described above, the computing device 600 may include a communications interface 640. The communications interface 640 allows software and data to be transferred between the computing device 600 and external devices (e.g., a printer), a network, or other information sources. For example, computer software or executable code may be transferred from a network server to the computing device 600 via the communications interface 640. Examples of the communications interface 640 include a built-in network adapter, a network interface card (NIC), a Personal Computer Memory Card International Association (PCMCIA) network card, a cardbus network adapter, a wireless network adapter, a Universal Serial Bus (USB) network adapter, a modem, a network interface card (NIC), a wireless data card, a communications port, an infrared port, an IEEE 1394 FireWire, or any other device capable of interfacing the system 550 with a network or another computing device. The communication interface 640 preferably implements industry-promulgated protocol standards, such as Ethernet IEEE 802 standards, fiber channel, digital subscriber line (DSL), asynchronous digital subscriber line (ADS), frame relay, asynchronous transfer mode (ATM), integrated digital services network (ISDN), personal communications service (PCS), transmission control protocol / Internet protocol (TCP / IP), serial line Internet protocol / point-to-point protocol (SLIP / PPP), etc., but may also implement customized or non-standard interface protocols.

[0055] The software and data transmitted via the communication interface 640 typically take the form of electrical communication signals 655. These signals 555 can be provided to the communication interface 640 via a communication channel 650. In embodiments, the communication channel 650 can be a wired or wireless network, or any other type of communication link. The communication channel 650 carries the signals 655 and can be implemented using a variety of wired or wireless communication means, including wire or cable, fiber optics, a traditional telephone line, a cellular telephone link, a wireless data communication link, a radio frequency (RF) link, or an infrared link, to name a few.

[0056] Computer executable code (e.g., computer programs or software) is stored in the main memory 615 and / or the secondary memory 620. The computer program may also be received via the communication interface 640 and stored in the main memory 615 and / or the secondary memory 620. Such a computer program, when executed, enables the computing device 600 to perform the various functions of the disclosed embodiments described elsewhere herein.

[0057] In this document, the term "computer-readable medium" is used to refer to any non-transitory computer-readable storage medium for providing computer-executable code (e.g., software and computer programs) to the computing device 600. Examples of such media include main memory 615, secondary memory 620 (including internal memory 625, removable media 630, and external storage media 645), and any peripheral devices communicatively coupled to the communication interface 640 (including a network information server or other network device). These non-transitory computer-readable media are means for providing executable code, programming instructions, and software to the computing device 500.

[90] In embodiments implemented using software, the software may be stored on a computer-readable medium and loaded into the computing device 600 via the removable media 630, the I / O interface 635, or the communication interface 640. In such embodiments, the software is loaded into the computing device 600 in the form of an electrical communication signal 655. When executed by the processor 610, the software preferably causes the processor 610 to perform the features and functions described elsewhere herein.

[0058] The I / O interface 635 provides an interface between one or more components of the computing device 600 and one or more input and / or output devices. Example input devices include, but are not limited to, a keyboard, a touch screen or other touch-sensitive device, a biometric sensing device, a computer mouse, a trackball, a pen-based pointing device, and the like. Examples of output devices include, but are not limited to, a cathode ray tube (CRT), a plasma display, a light-emitting diode (LED) display, a liquid crystal display (LCD), a printer, a vacuum fluorescent display (VFD), a surface-conduction electron-emitter display (SED), a field emission display (FED), and the like.

[0059] Computing device 600 also includes optional wireless communication components that facilitate wireless communication over voice and / or data networks. The wireless communication components include antenna system 670, radio system 665, and baseband system 660. In computing device 600, radio frequency (RF) signals are sent and received over the air through antenna system 670 under the management of the radio system.

[0060] The antenna system 670 may include one or more antennas and one or more multiplexers (not shown) that perform switching functions to provide transmit and receive signal paths for the antenna system 670. In the receive path, a received RF signal may be coupled from the multiplexer to a low noise amplifier (not shown) that amplifies the received RF signal and transmits the amplified signal to the radio system.

[0061] The radio system 665 may include one or more radios configured to communicate on various frequencies. In an embodiment, the radio system 665 may combine a demodulator (not shown) and a modulator (not shown) in a single integrated circuit (IC). The demodulator and modulator may also be separate components. In the incoming path, the demodulator strips off the RF carrier signal, leaving a baseband receive audio signal, which is sent from the radio system 665 to the baseband system.

[0062] If the received signal contains audio information, the baseband system 5660 decodes the signal and converts it into an analog signal. The signal is then amplified and sent to the speaker. The baseband system 660 also receives analog audio signals from a microphone. These analog audio signals are converted to digital signals and encoded by the baseband system 660. The baseband system 660 also encodes and decodes the digital signals for transmission and generates a baseband transmit audio signal, which is routed to the modulator section of the radio system 665. The modulator mixes the baseband transmit audio signal with an RF carrier signal to generate an RF transmit signal, which is routed to the antenna system 670 and can pass through a power amplifier (not shown). The power amplifier amplifies the RF transmit signal and routes it to the antenna system 670, where the signal is switched to the antenna port for transmission.

[0063] The baseband system 660 is also communicatively coupled to the processor 610, which may be a central processing unit (CPU). The processor 610 may access data storage areas 615 and 620. The processor 610 is preferably configured to execute instructions (i.e., computer programs or software) that may be stored in the main memory 615 or the secondary memory 620. Computer programs may also be received from the baseband processor 660 and stored in the main memory 610 or the secondary memory 620, or executed upon receipt. Such computer programs, when executed, enable the computing device 600 to perform the various functions of the disclosed embodiments. For example, the data storage areas 615 or 620 may include various software modules.

[0064] The computing device also includes a display 675 attached directly to the communications bus 605 , which may be provided instead of or in addition to any display connected to the I / O interface 635 described above.

[0065] Various embodiments may also be implemented primarily in hardware using components such as application specific integrated circuits (ASICs), programmable logic arrays (PLAs), or field programmable gate arrays (FPGAs). Implementation of a hardware state machine capable of performing the functions described herein will also be readily apparent to those skilled in the relevant art. Various embodiments may also be implemented using a combination of hardware and software.

[0066] In addition, it will be understood by those skilled in the art that the various illustrative logic blocks, modules, circuits and method steps described in conjunction with the above-mentioned figures and the embodiments disclosed herein can generally be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been generally described above with respect to their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. Technicians can implement the described functions in different ways for each specific application, but such implementation decisions should not be interpreted as causing departure from the scope of the present invention. In addition, the grouping of functions within modules, blocks, circuits or steps is for ease of description. Without departing from the present invention, specific functions or steps can be moved from one module, block or circuit to another.

[0067] As further illustration of potential implementations and applications of the disclosed technology, the following examples are provided in a non-exhaustive manner in which the teachings herein may be combined or applied. It should be understood that the following examples are not intended to limit the scope of coverage of any claims that may be filed at any time in this application or in a subsequent filing of this application. No disclaimers are intended. The following examples are provided for illustrative purposes only and for no other purpose. It is contemplated that the various teachings herein may be arranged and applied in a variety of other ways. It is also contemplated that some variations may omit certain features mentioned in the following examples. Therefore, unless expressly indicated otherwise by the inventor or the inventor's successor in interest at a later date, no aspect or feature mentioned below should be considered critical. If any claims are made in this application or in a subsequent filing related to this application that include additional features beyond those mentioned below, it shall not be presumed that these additional features were added for any reason related to patentability.

[0068] Example 1

[0069] A method for detecting mitosis in a tissue image using two-step detection, the method comprising: a) receiving a tissue image, b) obtaining a detection output by applying a detection model to the tissue image, the detection output comprising, for each of a set of nuclei in the tissue image, i) identifying the nucleus as undergoing mitosis, and ii) a confidence level for identifying the nucleus as undergoing mitosis; c) for each nucleus in the set of nuclei, performing a deterministic determination comprising determining a relationship between the confidence level for the nucleus and a first threshold; d) for each nucleus in a first subset of nuclei, determining a relationship between the confidence level for the nucleus and a first threshold; the relationship between the detection output and the first threshold, obtaining a classification result indicating whether the nucleus is undergoing mitosis by applying the classification model to the nucleus, wherein, for each nucleus in the first subset of the nuclei, the confidence of the nucleus is not higher than the first threshold; and e) providing an output, wherein the output will include: i) each nucleus in the set of nuclei that is identified as undergoing mitosis in the detection output and has a confidence higher than the first threshold; and ii) each nucleus in the first subset of the nuclei that is indicated as undergoing mitosis in the classification result obtained for the nucleus by applying the classification model as undergoing mitosis.

[0070] Example 2

[0071] A method according to Example 1, wherein: a) for each core in the set of cores, the deterministic determination includes determining a relationship between a confidence score for the core and a second threshold, wherein the second threshold is less than the first threshold; b) the method includes, for each core in a second subset of the set of cores, discarding the core based on the deterministic determination, wherein, for each core in the second subset of the set of cores, the confidence score for the core is not higher than the second threshold; and c) for each core identified in the output, the confidence score for the core is not lower than the second threshold.

[0072] Example 3

[0073] The method according to example 1, wherein the detection output comprises, for each of the second set of nuclei in the tissue image, the detection output comprises: a) identifying the nucleus as a mitotic pseudo-nucleus; and b) a confidence level of the identification of the nucleus.

[0074] Example 4

[0075] A method according to Example 3, wherein the output comprises: a) each nucleus in the second set of nuclei that is identified in the detection output with a confidence level above the first threshold; and b) each nucleus in the first subset of nuclei that is indicated as not undergoing mitosis in the classification result obtained for the nucleus by applying the classification model to the nucleus, being identified as a mitosis-pseudo-nucleus.

[0076] Example 5

[0077] The method according to Example 1, wherein: a) the method comprises: i) obtaining a slice image; and ii) segmenting the slice image into a plurality of image blocks; and b) the tissue image is an image block of the plurality of image blocks.

[0078] Example 6

[0079] According to the method of Example 1, applying the detection model includes: a) for each of a plurality of resolutions, obtaining a plurality of stage outputs by: i) receiving an input feature set; ii) generating an output feature set of the resolution by applying one or more convolution filters to the input feature set; iii) generating a stage output of the resolution by downsampling the output feature set of the resolution using a pooling operation; b) generating the detection output by performing steps including: i) for each of the plurality of resolutions, generating an identification output by: A) generating a set of object identifications of the resolution based on the stage output of the resolution; B) obtaining an upsampled set of object identifications by upsampling the set of object identifications of the resolution; and C) generating a next stage object identification input by combining the upsampled set of object identifications with the stage output of the next higher resolution among the plurality of resolutions; wherein, for each of the plurality of resolutions except the lowest resolution among the plurality of resolutions, the identification outputs are generated based on the stage output of the resolution. The method comprises the steps of: i) generating a set of object identifications for each of the plurality of resolutions by generating the set of object identifications based on a next-stage object identification input generated by a next-lower resolution among the plurality of resolutions; ii) in parallel with generating the identification output, generating, for each of the plurality of resolutions, a box output by: A) generating a set of bounding boxes for the resolution based on the stage output for the resolution, B) obtaining an upsampled set of bounding boxes by upsampling the set of bounding boxes for the resolution; and C) generating a next-stage bounding box input by combining the upsampled set of bounding boxes with the stage output for the next higher resolution among the plurality of resolutions; wherein, for each of the plurality of resolutions except a lowest resolution among the plurality of resolutions, generating the set of bounding boxes for the resolution based on the stage output for the resolution is performed by generating the set of bounding boxes based on a next-stage bounding box input generated by a next-lower resolution among the plurality of resolutions; and iii) obtaining the detection output based on combining the identification output and the box output.

[0080] Example 7

[0081] The method according to Example 1, wherein the tissue image is a hematoxylin and eosin (H&E) stained image.

[0082] Example 8

[0083] A system for detecting mitosis in a tissue image using two-step detection, the system comprising a processor configured with a set of computer-executable instructions operable, when executed, to perform a method comprising: a) receiving a tissue image; b) obtaining a detection output by applying a detection model to the tissue image, the detection output comprising, for each of a set of nuclei in the tissue image: i) identifying the nucleus as undergoing mitosis, and ii) a confidence level for identifying the nucleus as undergoing mitosis; and c) for each nucleus in the set of nuclei, performing a deterministic determination comprising determining a relationship between the confidence level for the nucleus and a first threshold. ; d) for each nucleus in the first subset of nuclei, based on the relationship between the confidence of the nucleus and the first threshold, obtain a classification result indicating whether the nucleus is undergoing mitosis by applying the classification model to the nucleus, wherein, for each nucleus in the first subset of nuclei, the confidence of the nucleus is not higher than the first threshold; and e) provide an output, wherein the output will: i) each nucleus in the set of nuclei that is identified as undergoing mitosis in the detection output and has a confidence higher than the first threshold; and ii) each nucleus in the first subset of nuclei that is indicated as undergoing mitosis in the classification result obtained for the nucleus by applying the classification model is identified as undergoing mitosis.

[0084] Example 9

[0085] A system according to Example 8, wherein: a) for each core in the set of cores, the deterministic determination includes determining a relationship between a confidence score for the core and a second threshold, wherein the second threshold is less than the first threshold; b) the method includes, for each core in a second subset of the set of cores, discarding the core based on the deterministic determination, wherein, for each core in the second subset of the set of cores, the confidence score for the core is not higher than the second threshold; and c) for each core identified in the output, the confidence score for the core is not lower than the second threshold.

[0086] Example 10

[0087] The system according to Example 8, wherein the detection output comprises, for each of the second set of nuclei in the tissue image, the detection output comprises: a) an identification of the nucleus as a mitotic pseudo-nucleus; b) a confidence level of the identification of the nucleus.

[0088] Example 11

[0089] A system according to Example 10, wherein the output identifies a) each nucleus in the second set of nuclei that is identified in the detection output with a confidence level above the first threshold; and b) each nucleus in the first subset of nuclei that is indicated as not undergoing mitosis in the classification result obtained for the nucleus by applying the classification model to the nucleus as a mitosis-pseudo-nucleus.

[0090] Example 12

[0091] The system according to example 8, wherein: a) the method comprises: i) obtaining a slice image; and ii) segmenting the slice image into a plurality of image blocks; and b) the tissue image is an image block of the plurality of image blocks.

[0092] Example 13

[0093] A system according to Example 8, wherein applying a detection model comprises: a) for each of a plurality of resolutions, obtaining a plurality of stage outputs by: i) receiving an input feature set; ii) generating an output feature set for the resolution by applying one or more convolution filters to the input feature set; iii) generating a stage output for the resolution by downsampling the output feature set for the resolution using a pooling operation; b) generating the detection output by performing steps comprising: i) for each of the plurality of resolutions, generating an identification output by: A) generating a set of object identifications for the resolution based on the stage output for the resolution; B) obtaining an upsampled set of object identifications by upsampling the set of object identifications for the resolution; and C) generating a next stage object identification input by combining the upsampled set of object identifications with the stage output for the next higher resolution in the plurality of resolutions; wherein, for each of the plurality of resolutions except the lowest resolution in the plurality of resolutions, the detection output is generated based on the stage output for the resolution. The method comprises the steps of: i) generating a set of object identifications for a first stage of the plurality of resolutions by generating the set of object identifications based on a next stage object identification input generated by a next lower resolution among the plurality of resolutions; ii) in parallel with generating the identification output, generating, for each of the plurality of resolutions, a box output by: A) generating a set of bounding boxes for the resolution based on the stage output for the resolution, B) obtaining an upsampled set of bounding boxes by upsampling the set of bounding boxes for the resolution; and C) generating a next stage bounding box input by combining the upsampled set of bounding boxes with the stage output for the next higher resolution among the plurality of resolutions; wherein, for each of the plurality of resolutions except a lowest resolution among the plurality of resolutions, generating the set of bounding boxes for the resolution based on the stage output for the resolution is performed by generating the set of bounding boxes based on a next stage bounding box input generated by a next lower resolution among the plurality of resolutions; and iii) obtaining the detection output based on combining the identification output and the box output.

[0094] Example 14

[0095] The system according to Example 8, wherein the tissue image is a hematoxylin and eosin (H&E) stained image.

[0096] Example 15

[0097] A non-transitory computer-readable medium storing computer-executable instructions operable to program a computer including a processor to perform a method comprising: a) receiving a tissue image; b) obtaining a detection output by applying a detection model to the tissue image, wherein for each of a set of nuclei in the tissue image, the detection output comprises: i) identifying the nucleus as undergoing mitosis, and ii) a confidence level for identifying the nucleus as undergoing mitosis; c) for each nucleus in the set of nuclei, performing a deterministic determination comprising determining a relationship between the confidence level for the nucleus and a first threshold; d) for a first subset of the nuclei, determining a probability distribution of the nucleus; for each nucleus in the set, obtaining a classification result indicating whether the nucleus is undergoing mitosis by applying the classification model to the nucleus based on the relationship between the confidence of the nucleus and the first threshold, wherein, for each nucleus in the first subset of the nuclei, the confidence of the nucleus is not higher than the first threshold; and e) providing an output, wherein the output will include: i) each nucleus in the set of nuclei that is identified as undergoing mitosis in the detection output and has a confidence higher than the first threshold; and ii) each nucleus in the first subset of the nuclei that is indicated as undergoing mitosis in the classification result obtained for the nucleus by applying the classification model as undergoing mitosis.

[0098] Example 16

[0099] According to the non-transitory computer-readable medium of Example 15, wherein: a) for each core in the set of cores, the deterministic determination includes determining a relationship between a confidence score for the core and a second threshold, wherein the second threshold is less than the first threshold; b) the method includes, for each core in a second subset of the set of cores, discarding the core based on the deterministic determination, wherein, for each core in the second subset of the set of cores, the confidence score for the core is not higher than the second threshold; and c) for each core identified in the output, the confidence score for the core is not lower than the second threshold.

[0100] Example 17

[0101] The non-transitory computer-readable medium of Example 15, wherein the detection output comprises, for each of the second set of nuclei in the tissue image, a) identifying the nucleus as a mitotic pseudo-nucleus; and b) a confidence level for the identification of the nucleus.

[0102] Example 18

[0103] According to the non-transitory computer-readable medium of Example 17, the output will be a) each nucleus in the second set of nuclei that is identified in the detection output with a confidence level above the first threshold; and b) each nucleus in the first subset of nuclei that is indicated as not undergoing mitosis in the classification result obtained for the nucleus by applying the classification model to the nucleus is identified as a mitosis-pseudo-nucleus.

[0104] Example 19

[0105] A non-transitory computer readable medium according to example 15, wherein: a) the method comprises: i) obtaining a slice image; and ii) segmenting the slice image into a plurality of image blocks; and b) the tissue image is an image block of the plurality of image blocks.

[0106] Example 20

[0107] According to the non-transitory computer-readable medium of example 15, wherein applying the detection model comprises: a) for each of a plurality of resolutions, obtaining a plurality of stage outputs by: i) receiving an input feature set; ii) generating an output feature set for the resolution by applying one or more convolution filters to the input feature set; iii) generating a stage output for the resolution by downsampling the output feature set for the resolution using a pooling operation; b) generating the detection output by performing steps comprising: i) for each of the plurality of resolutions, generating an identification output by: A) generating a set of object identifications for the resolution based on the stage output for the resolution; B) obtaining an upsampled set of object identifications by upsampling the set of object identifications for the resolution; and C) generating a next stage object identification input by combining the upsampled set of object identifications with the stage output for the next higher resolution in the plurality of resolutions; wherein, for each of the plurality of resolutions except the lowest resolution in the plurality of resolutions, the stage output for the resolution based on the The method comprises the steps of: i) generating a set of object identifications for the resolutions by generating the set of object identifications based on a next-stage object identification input generated by a next-lower resolution among the plurality of resolutions; ii) in parallel with generating the identification output, generating, for each of the plurality of resolutions, a box output by: A) generating a set of bounding boxes for the resolution based on the stage output for the resolution; B) obtaining an upsampled set of bounding boxes by upsampling the set of bounding boxes for the resolution; and C) generating a next-stage bounding box input by combining the upsampled set of bounding boxes with the stage output for the next higher resolution among the plurality of resolutions; wherein, for each of the plurality of resolutions except a lowest resolution among the plurality of resolutions, generating the set of bounding boxes for the resolution based on the stage output for the resolution is performed by generating the set of bounding boxes based on a next-stage bounding box input generated by a next-lower resolution among the plurality of resolutions; and iii) obtaining the detection output based on combining the identification output and the box output.

[0108] Each calculation or operation described herein can be performed using a computer or other processor with hardware, software and / or firmware. Various method steps can be performed by modules, and modules can include any of various digital and / or analog data processing hardware and / or software arranged to perform the method steps described herein. Modules optionally include data processing hardware, which is suitable for performing one or more of these steps by having an applicable machine programming code associated therewith, and modules for two or more steps (or a part of two or more steps) are integrated into a single processor board, or are separated into different processor boards in any one of various integrated and / or distributed processing architectures. These methods and systems will generally adopt a tangible medium comprising machine-readable code, with instructions for performing the above-mentioned method steps. Suitable tangible media can include memory (including volatile memory and / or non-volatile memory), storage media (such as magnetic recording on floppy disks, hard disks, magnetic tapes, etc.; optical storage such as CD, CD-R / W, CD-ROM, DVD; or any other digital or analog storage media), etc.

[0109] All patents, patent publications, patent applications, journal articles, books, technical references, etc. discussed in this disclosure are incorporated herein by reference in their entirety for all purposes.

[0110] Different arrangements of components depicted in the drawings or described above, as well as components and steps not shown or described, are possible. Similarly, some features and subcombinations are useful and can be employed without reference to other features and subcombinations. The embodiments of the present invention have been described for illustrative purposes, not limiting, and alternative embodiments will become apparent to readers of this patent. In some cases, method steps or operations may be performed in a different order, or operations may be added, deleted, or modified. It is understood that in certain aspects of the present invention, a single component may be replaced by multiple components, and multiple components may be replaced by a single component to provide an element or structure or perform a given function or functions. Unless such substitution is inappropriate for practicing certain embodiments of the present invention, such substitution is considered within the scope of the present invention. Therefore, the claims should not be construed as limited to the examples, drawings, embodiments, and illustrations provided above, but should be construed as having the scope provided when their terms are given the broadest reasonable interpretation provided by a commonly used dictionary. Except where a term or phrase is indicated under the heading "Defined" as having a specific meaning, it should be understood to have that meaning when used in the claims.

[0111] Clear definition

[0112] It should be understood that in the above examples and claims, the statement that something is "based on" another thing should be understood to mean that the thing is at least partially determined by the thing indicated as being based on. In order to indicate that something must be completely determined based on another thing, it is described as "completely based on" anything that must be completely determined by it.

[0113] It should be understood that in the above examples and claims, the term "mitotic pseudo" should be understood as a nucleus that is not undergoing mitosis, but that has one or more characteristics consistent with a nucleus undergoing mitosis.

[0114] It should be understood that in the above examples and claims, the term "set" should be understood as one or more things grouped together. The terms "subset" and "superset" should be understood as synonyms of "set," where the use of "set," "subset," or "superset" is for ease of reference and not intended to convey a substantive distinction.

Claims

1. A method for detecting mitosis in a tissue image using a two-step detection process, the method comprising: a) receiving tissue images, b) obtaining a detection output by applying a detection model to the tissue image, the detection output comprising, for each of a set of nuclei in the tissue image: i) identifying the nucleus as undergoing mitosis, and ii) the confidence with which the nucleus is identified as undergoing mitosis; c) for each kernel in the set of kernels, performing a deterministic determination, including determining a relationship between a confidence score of the kernel and a first threshold; d) for each nucleus in the first subset of nuclei, obtaining a classification result indicating whether the nucleus is undergoing mitosis by applying the classification model to the nucleus based on a relationship between the confidence score of the nucleus and the first threshold, wherein, for each nucleus in the first subset of nuclei, the confidence score of the nucleus is not higher than the first threshold; and e) providing an output, wherein said output will: i) each nucleus in the set of nuclei that is identified in the detection output as undergoing mitosis with a confidence level above the first threshold; and ii) each nucleus in the first subset of nuclei that is indicated as undergoing mitosis in the classification result obtained for the nucleus by applying the classification model Identified as undergoing mitosis.

2. The method according to claim 1, wherein: a) for each kernel in the set of kernels, the deterministic determination comprises determining a relationship between a confidence score of the kernel and a second threshold, wherein the second threshold is less than the first threshold; b) the method includes, for each core in a second subset of the set of cores, discarding the core based on the deterministic determination, wherein, for each core in the second subset of the set of cores, the confidence score of the core is not above the second threshold; and c) For each nucleus identified in the output, the confidence score of the nucleus is not lower than the second threshold.

3. The method according to claim 1, wherein The detection output includes, for each of a second set of nuclei in the tissue image, the detection output includes: a) identifying the nucleus as a mitotic pseudo-nucleus; b) Confidence of the identification of the nucleus.

4. The method of claim 3, wherein the output is: a) each nucleus in the second set of nuclei that is identified in the detection output with a confidence level above the first threshold; and b) each nucleus in the first subset of nuclei that is indicated as not undergoing mitosis in a classification result obtained for the nucleus by applying the classification model Identified as a mitotic nucleus.

5. The method according to claim 1, wherein: a) the method comprises: i) obtaining slice images; and ii) dividing the slice image into a plurality of image blocks; as well as b) The tissue image is an image block among the plurality of image blocks.

6. The method of claim 1 , wherein applying the detection model comprises: a) For each of the multiple resolutions, multiple stage outputs are obtained by: i) receiving an input feature set; ii) generating an output feature set at the resolution by applying one or more convolutional filters to the input feature set; iii) generating a stage output at said resolution by downsampling the output feature set at said resolution using a pooling operation; b) generating the detection output by performing the steps comprising: i) for each of the plurality of resolutions, generating a logo output by: A) generating a set of object identifiers at the resolution based on the stage output at the resolution; B) obtaining an upsampled set of object identifiers by upsampling the set of object identifiers at the resolution; as well as C) generating a next stage object identification input by combining the upsampled set of object identifications with a stage output of a next higher resolution among the plurality of resolutions; wherein, for each of the plurality of resolutions except a lowest resolution of the plurality of resolutions, generating a set of object identifications for the resolution based on a stage output of the resolution is performed by generating the set of object identifications based on a next stage object identification input generated by a next lower resolution of the plurality of resolutions; ii) in parallel with generating the identification output, for each of the plurality of resolutions, generating an output by the following block: A) generating a set of bounding boxes at the resolution based on the stage output at the resolution; B) obtaining an upsampled set of bounding boxes by upsampling the set of bounding boxes at the resolution; and C) generating a next stage bounding box input by combining the upsampled set of bounding boxes with a stage output of a next higher resolution among the plurality of resolutions; wherein, for each of the plurality of resolutions except a lowest resolution of the plurality of resolutions, generating a set of bounding boxes for the resolution based on a stage output of the resolution is performed by generating the set of bounding boxes based on a next stage bounding box input generated by a next lower resolution of the plurality of resolutions; as well as iii) obtaining the detection output based on combining the identification output and the box output. The method according to claim 1 , wherein the tissue image is a hematoxylin and eosin (H&E) stained image.

8. A system for detecting mitosis in a tissue image using two-step detection, the system comprising a processor configured with a set of computer-executable instructions operable, when executed, to perform a method comprising: a) receiving tissue images, b) obtaining a detection output by applying a detection model to the tissue image, the detection output comprising, for each of a set of nuclei in the tissue image: i) identifying the nucleus as undergoing mitosis, and ii) the confidence with which the nucleus is identified as undergoing mitosis; c) for each kernel in the set of kernels, performing a deterministic determination, including determining a relationship between a confidence score of the kernel and a first threshold; d) for each nucleus in the first subset of nuclei, obtaining a classification result indicating whether the nucleus is undergoing mitosis by applying the classification model to the nucleus based on a relationship between the confidence score of the nucleus and the first threshold, wherein, for each nucleus in the first subset of nuclei, the confidence score of the nucleus is not higher than the first threshold; and e) providing an output, wherein said output will: i) each nucleus in the set of nuclei that is identified in the detection output as undergoing mitosis with a confidence level above the first threshold; and ii) each nucleus in the first subset of nuclei that is indicated as undergoing mitosis in the classification result obtained for the nucleus by applying the classification model Identified as undergoing mitosis.

9. The system of claim 8, wherein: a) for each core in the set of cores, the deterministic determination comprises determining a relationship between a confidence score of the core and a second threshold, wherein the second threshold is less than the first threshold; b) the method includes, for each core in a second subset of the set of cores, discarding the core based on the deterministic determination, wherein, for each core in the second subset of the set of cores, the confidence score of the core is not above the second threshold; and c) For each nucleus identified in the output, the confidence score of the nucleus is not lower than the second threshold.

10. The system according to claim 8, wherein: The detection output includes, for each of a second set of nuclei in the tissue image, the detection output includes: a) identifying the nucleus as a mitotic pseudo-nucleus; b) Confidence of the identification of the nucleus.

11. The system of claim 10, wherein the output will: a) each nucleus in the second set of nuclei that is identified in the detection output with a confidence level above the first threshold; and b) each nucleus in the first subset of nuclei that is indicated as not undergoing mitosis in a classification result obtained for the nucleus by applying the classification model Identified as a mitotic nucleus.

12. The system of claim 8, wherein: a) the method comprises: i) obtaining slice images; and ii) dividing the slice image into a plurality of image blocks; as well as b) The tissue image is an image block among the plurality of image blocks.

13. The system of claim 8, wherein applying the detection model comprises: a) For each of the multiple resolutions, multiple stage outputs are obtained by: i) receiving an input feature set; ii) generating an output feature set at the resolution by applying one or more convolutional filters to the input feature set; iii) generating a stage output at said resolution by downsampling the output feature set at said resolution using a pooling operation; b) generating the detection output by performing the steps comprising: i) for each of the plurality of resolutions, generating a logo output by: A) generating a set of object identifiers at the resolution based on the stage output at the resolution; B) obtaining an upsampled set of object identifiers by upsampling the set of object identifiers at the resolution; as well as C) generating a next stage object identification input by combining the upsampled set of object identifications with a stage output of a next higher resolution among the plurality of resolutions; wherein, for each of the plurality of resolutions except a lowest resolution of the plurality of resolutions, generating a set of object identifications for the resolution based on a stage output of the resolution is performed by generating the set of object identifications based on a next stage object identification input generated by a next lower resolution of the plurality of resolutions; ii) in parallel with generating the identification output, for each of the plurality of resolutions, generating an output by the following block: A) generating a set of bounding boxes at the resolution based on the stage output at the resolution; B) obtaining an upsampled set of bounding boxes by upsampling the set of bounding boxes at the resolution; and C) generating a next stage bounding box input by combining the upsampled set of bounding boxes with a stage output of a next higher resolution among the plurality of resolutions; wherein, for each of the plurality of resolutions except a lowest resolution of the plurality of resolutions, generating a set of bounding boxes for the resolution based on a stage output of the resolution is performed by generating the set of bounding boxes based on a next stage bounding box input generated by a next lower resolution of the plurality of resolutions; as well as iii) obtaining the detection output based on combining the identification output and the box output.

14. The system of claim 8, wherein the tissue image is a hematoxylin and eosin (H&E) stained image.

15. A non-transitory computer-readable medium storing computer-executable instructions operable to program a computer comprising a processor to perform a method comprising: a) receiving tissue images, b) obtaining a detection output by applying a detection model to the tissue image, the detection output comprising, for each of a set of nuclei in the tissue image: i) identifying the nucleus as undergoing mitosis, and ii) the confidence with which the nucleus is identified as undergoing mitosis; c) for each kernel in the set of kernels, performing a deterministic determination, including determining a relationship between a confidence score of the kernel and a first threshold; d) for each nucleus in the first subset of nuclei, obtaining a classification result indicating whether the nucleus is undergoing mitosis by applying the classification model to the nucleus based on a relationship between the confidence score of the nucleus and the first threshold, wherein, for each nucleus in the first subset of nuclei, the confidence score of the nucleus is not higher than the first threshold; and e) providing an output, wherein said output will: i) each nucleus in the set of nuclei that is identified in the detection output as undergoing mitosis with a confidence level above the first threshold; and ii) each nucleus in the first subset of nuclei that is indicated as undergoing mitosis in the classification result obtained for the nucleus by applying the classification model Identified as undergoing mitosis.

16. The non-transitory computer-readable medium of claim 15, wherein: a) for each core in the set of cores, the deterministic determination comprises determining a relationship between a confidence score of the core and a second threshold, wherein the second threshold is less than the first threshold; b) the method includes, for each core in a second subset of the set of cores, discarding the core based on the deterministic determination, wherein, for each core in the second subset of the set of cores, the confidence score of the core is not above the second threshold; and c) For each nucleus identified in the output, the confidence score of the nucleus is not lower than the second threshold.

17. The non-transitory computer readable medium of claim 15, wherein: The detection output includes, for each of a second set of nuclei in the tissue image, the detection output includes: a) identifying the nucleus as a mitotic pseudo-nucleus; b) Confidence of the identification of the nucleus.

18. The non-transitory computer readable medium of claim 17, wherein the output will: a) each nucleus in the second set of nuclei that is identified in the detection output with a confidence level above the first threshold; and b) each nucleus in the first subset of nuclei that is indicated as not undergoing mitosis in a classification result obtained for the nucleus by applying the classification model Identified as a mitotic nucleus.

19. The non-transitory computer-readable medium of claim 15, wherein: a) the method comprises: i) obtaining slice images; and ii) dividing the slice image into a plurality of image blocks; as well as b) The tissue image is an image block among the plurality of image blocks.

20. The non-transitory computer-readable medium of claim 15, wherein applying the detection model comprises: a) For each of the multiple resolutions, multiple stage outputs are obtained by: i) receiving an input feature set; ii) generating an output feature set at the resolution by applying one or more convolutional filters to the input feature set; iii) generating a stage output at said resolution by downsampling the output feature set at said resolution using a pooling operation; b) generating the detection output by performing the steps comprising: i) for each of the plurality of resolutions, generating a logo output by: A) generating a set of object identifiers at the resolution based on the stage output at the resolution; B) obtaining an upsampled set of object identifiers by upsampling the set of object identifiers at the resolution; as well as C) generating a next stage object identification input by combining the upsampled set of object identifications with a stage output of a next higher resolution among the plurality of resolutions; wherein, for each of the plurality of resolutions except a lowest resolution of the plurality of resolutions, generating a set of object identifications for the resolution based on a stage output of the resolution is performed by generating the set of object identifications based on a next stage object identification input generated by a next lower resolution of the plurality of resolutions; ii) in parallel with generating the identification output, for each of the plurality of resolutions, generating an output by the following block: A) generating a set of bounding boxes at the resolution based on the stage output at the resolution; B) obtaining an upsampled set of bounding boxes by upsampling the set of bounding boxes at the resolution; and C) generating a next stage bounding box input by combining the upsampled set of bounding boxes with a stage output of a next higher resolution among the plurality of resolutions; wherein, for each of the plurality of resolutions except a lowest resolution of the plurality of resolutions, generating a set of bounding boxes for the resolution based on a stage output of the resolution is performed by generating the set of bounding boxes based on a next stage bounding box input generated by a next lower resolution of the plurality of resolutions; as well as iii) obtaining the detection output based on combining the identification output and the box output.

Images