Computer-assisted review of tumors in histological images and postoperative tumor margin evaluation

Abstract

Computer devices and methods are provided for identifying and visualizing tumors in histological images and measuring tumor margins. A CNN is used to classify pixels within an image based on whether they are associated with one or more classes of non-tumor tissue or tumor tissue. Segmentation is performed based on the CNN results to generate a mask indicating the area occupied by individual tumors. Summary statistics for each tumor are calculated and provided to a filter that edits the segmentation mask by filtering out tumors considered insignificant. Optionally, tumors passing through the filter may be ranked according to the summary statistics, for example, in order of clinical relevance or in order of reasonable review by a pathologist. A visualization application may display the histological image associated with the segmentation mask, summary statistics, and / or ranking. Tumor masses extracted by resection are painted with ink to highlight surface areas. A CNN is trained to distinguish between tumor and non-tumor tissues, as well as between ink and non-ink tissues. The CNN is applied to the histological image to generate an output image having pixels assigned to tissue classes. The status of the tumor margin of a tissue section is determined by the presence or absence of tumor-ink classified pixels. Tumor margin invasion and tumor margin distance are determined by calculating additional parameters based on the distance parameter between classification-specified pixels.

Description

Technology Field

[0001] The present invention relates to image processing of histological images containing a tumor using a convolutional neural network (CNN) to assist in the review and diagnosis by a pathologist evaluating the margins of a tumor mass removed from a patient by resection. Background Technology

[0002] Digital pathology is continuously changing the way pathologists view slides and make diagnoses. The traditional method for a pathologist to examine a slide is to observe a glass slide under a microscope. The pathologist begins by viewing the slide through a low-magnification objective lens. If an area of ​​potential diagnostic value is observed, the pathologist switches to a high-magnification objective lens to view that area in more detail. Afterward, the pathologist switches back to low magnification to continue examining other areas of the slide. This low-high-low magnification viewing sequence can be repeated multiple times on the slide until a definite and complete diagnosis can be made.

[0003] Over the past 20 years, the introduction of digital scanners has changed this workflow. Digital scanners can acquire so-called entire glass slide (WSI) images, which are images of the entire glass slide, and save them as digital image data files in an automated process that does not require a pathologist. The resulting image data files are typically stored in a slide database provided to pathologists via a clinical network at a viewing workstation equipped with a high-resolution display, and the workstation has visualization applications for this purpose.

[0004] Recent developments in pathology are increasing research interest in CNN methods. There are growing reports that CNN methods are just as effective as pathologists in identifying and diagnosing tumors in histological images.

[0005] Wang et al. 2016 describes a CNN approach for detecting breast cancer metastasis to lymph nodes. US 2015 / 213302 A1 describes how cell mitosis is detected in cancer tissue regions. After training the CNN, classification is performed based on an automated nuclear detection system that performs mitotic counting, and this system is used to grade the tumor.

[0006] Hou et al. 2016 process brain and lung cancer images. Image patches from WSI are used to make patch-level predictions provided by a patch-level CNN.

[0007] Liu et al 2017 detect and localize tumors by processing image patches extracted from gigapixel breast cancer histological images with a CNN to assign tumor probabilities to every pixel of the image.

[0008] Bejnordi et al. (2017) apply two stacked CNNs to classify tumors in image patches extracted from WSIs of breast tissue stained with hermatoxylin and eosin (H&E). The performance was found to be good for object detection and segmentation (segmentation) in these pathological images. Additionally, Bejnordi et al. provide an overview of other CNN-based tumor classification methods applied to breast cancer samples (see References 10-13).

[0009] Esteva et al. (2017) apply a deep CNN to analyze skin lesions and classify them according to a tree-structure classification into various malignant and non-malignant types, including malignant types such as acrolentiginous melanoma, amelanotic melanoma, and lentigo melanoma, and non-malignant types such as blue nevus, halo nevus, and Mongolian spot. Images of skin lesions (e.g., melanoma) are sequentially warped into probability distributions for clinical classes to perform classification.

[0010] Mobadersany et al. (2017) disclose a computational method based on survival CNNs to predict the overall survival of patients diagnosed with brain tumors. Pathological image data from tissue biopsies (histological image data) is provided to the model, along with patient-specific genomic biomarkers, to predict patient outcomes. This method uses adaptive feedback to simultaneously learn visual patterns and molecular biomarkers associated with patient outcomes.

[0011] Tumor resection is generally a surgical procedure to remove tumor tissue by removing part or all of an organ or gland; partial removal is referred to as segmental resection. The purpose of tumor resection is to remove the entire mass of tissue containing the tumor, along with a small amount of normal tissue surrounding the tumor margins.

[0012] To confirm that the tumor has been completely removed by resection, the following pathological examinations are performed after the tissue mass is removed. The surface of the resected tissue mass is coated with colored ink. The tissue mass is prepared in various ways to preserve its fine tissue structure. The tissue mass is sliced ​​into thin sections using a microtome. After slicing, additional preservation measures for the microscopic tissue structure may be performed. The tissue sections are mounted on slides. Tissue sections are typically stained with specialized stains or markers to enhance cell visibility or to identify specific types of cells using markers specific to particular gene expression or related receptors. In the case of breast cancer tumors, the typical receptors targeted are ER, ER-, PR+, PR-, HER2+, and HER2-, where ER represents the estrogen receptor, PR represents the progesterone receptor, and HER2 represents the human epidermal growth factor receptor 2.

[0013] The traditional method of evaluating margins involves a pathologist examining stained slides under a microscope to determine whether tumor cells are present on the ink-treated surface or too close to it. If tumor cells are found in the ink, the pathologist reports the surgical margin as benign (Lester et al. 2009). A benign margin generally indicates that more surgery is required to remove a larger mass of tissue. In the case of a benign margin, the pathologist may also report the degree of margin-relatedness using subcategories such as focal, minimal / moderate, and extensive, which relate to the extent of invasion into the margin. If cancer cells are close to the ink surface but not completely over it (less than about 2 mm), the pathologist may report the tumor margin as 'close'. If cancer cells are not present on the ink-coated surface (not 'benign') or are not close to it (not 'close'), for example, if the margin is 2 mm or more, the pathologist reports the margin as 'clear'. In the case of clear or closed margins, pathologists may report the margin distance, which indicates the distance between the ink-covered surface and the tumor cell closest to the ink-covered surface. In the case of benign margins, additional surgery may be required to remove the remaining cancerous tissue. While accurately reporting the margin status is important for subsequent cancer treatment, there is a significant difference in the quality of margin status reporting (Persing et al 2011).

[0014] Therefore, as mentioned above, a system and method are needed to overcome these significant problems found in conventional systems. The problem to be solved

[0015] The purpose is to provide a system and method capable of overcoming the aforementioned problems. means of solving the problem

[0016] According to one aspect of the present invention, a method for identifying a tumor in a histological image (or set thereof) is provided, and the method comprises the following:

[0017] A step of receiving a histological image including a two-dimensional pixel array;

[0018] A step of applying a convolutional neural network to generate an output image having a two-dimensional pixel array having a mapping of the histological image, wherein the output image is generated by assigning one of a plurality of tissue classes to each pixel, wherein the plurality of tissue classes includes at least one class representing non-tumor tissue and at least one class representing tumor tissue;

[0019] Steps for calculating summary statistics for each tumor; and

[0020] A step of applying filters to the summary statistics of each tumor to edit the segmentation mask by selecting and deselecting tumors according to the filter.

[0021] The proposed approach allows pathologists to be marked with locations within images where potential or potentially significant tumors are present, directing them to move directly to those areas. This approach can improve the reliability of the analysis because it reduces the likelihood of the pathologist missing tumor-containing regions. Additionally, throughput can be increased as pathologists no longer need to scan the entire slide area to locate tumor regions. Furthermore, by allocating limited analysis time based on filtering and ranking derived from CNN analysis results, pathologists can be implicitly encouraged to allocate their time more effectively to studying specific slide images.

[0022] The above method can be extended by scoring each tumor according to a scoring algorithm to assign a score to each tumor. It is possible to rank the tumors based on the scores, and the ranking order may indicate various useful sequences, for example, to demonstrate perceived clinical relevance or to predict the order in which a clinician wishes to review the tumors. The ranking may be limited to only the tumors remaining after applying filters. The ranking may be stored, for example, and added to the record as additional metadata related to the histological images. Histological images may be received from records stored in a virtual slide library or another database, or may be received immediately after being acquired by a digital scanner; in this case, the method may be performed by a digital scanner or a computer device connected to, for example, a digital scanner or another data source.

[0023] A visualization of the histological image can be generated in relation to the edited segmentation mask. The visualization can then be displayed to the user on a display by a visualization application running on a computer device. For example, the visualization may include an overview view window where the segmentation mask is overlaid on the histological image. Another option for the visualization is to include individual overview view windows displayed adjacent to each other (e.g., side-by-side) for a one-to-one comparison of the segmentation mask and the histological image. The overview view window(s) may include rank labels for each tumor based on rank. A user interface tumor selection control is provided so that the user can interact with the visualization to select a tumor from the edited segmentation mask. The visualization may include a zoom view window that zooms in on the currently selected tumor. The tumor selection control may incorporate a scrolling function to browse through tumors in the edited segmentation mask in rank order. The tumor selection control may also incorporate diagnostic functions to allow the user to perform additional computational diagnostic processes on the currently selected tumor.

[0024] In its simplest form, CNNs have only two tissue classes: neoplastic and non-neoplastic tissues. However, more than two classes may be provided for neoplastic tissues. For example, tissue classes for neoplastic tissues can distinguish between invasive tumors and in situ tumors. There may be only one tissue class for non-neoplastic tissues, or there may be multiple classes, for example, to distinguish bone from other tissues. There may also be a non-tissue class for areas of the entire slide image where no sample is present, which can be particularly useful for tissue microarray samples.

[0025] The above method may further include a step of assembling output image patches into a probability map for histological images (or a set thereof) after the allocation step.

[0026] The above method may further include, after the assembly step, a step of storing the probability map, for example, in a record of a data repository so that the probability map is linked to a histological image (or set thereof).

[0027] In the current implementation, since depth increases as dimensionality decreases in each successive convolution step, the convolution layer continuously increases in depth and decreases in dimensionality; similarly, since depth decreases as dimensionality increases in each successive transpose convolution step, the depth of the deconvolution layer continuously decreases and the dimensionality continues to increase. Consequently, the final convolution layer has a maximum depth and a minimum dimensionality. Instead of the approach of successive depth increase and decrease going through convolution and deconvolution steps, an alternative is to design a neural network where all layers except the input and output layers have the same depth.

[0028] The above method may further include the step of displaying histological images (or a set thereof) on a display, for example, along with probability maps placed on top of or side by side with them. The probability maps can be used to determine which immunohistochemistry (IHC) scoring algorithm should be used for scoring. The probability maps can also be used to generate a set of contours around tumor cells that can be displayed on the display. For example, a pathologist can evaluate the results generated by the CNN.

[0029] In a specific embodiment, the convolutional neural network has one or more skip connections. Each skip connection takes an intermediate result from at least one convolutional layer of a larger dimension than the final convolutional layer and passes the result to a transposed convolution as needed to obtain one or more additional recovery layers with a size matching the input image patch. These are then combined with the recovery layers mentioned above prior to the step of assigning a tissue class to each pixel. An additional processing step considers the results obtained from the skip connections by combining the recovery layers with each respective recovery layer to recalculate probabilities.

[0030] In certain embodiments, a softmax operation is used to generate probabilities.

[0031] Image patches extracted from histological image(s) can cover the entire area of ​​the image(s). The patches can be non-overlapping image tiles or image tiles that overlap at the margins to aid in the stitching of the probability map. Since CNNs are designed to accommodate only fixed-size pixel arrays, each image patch must have a fixed number of pixel widths and heights that can be matched with the CNN, but this does not mean that each image patch must correspond to the same physical area on the histological image. This is because pixels in the histological image can be combined into low-resolution patches covering larger areas; for example, each 2×2 array of adjacent pixels can be combined into a single "super" pixel to form a patch with four times the physical area of ​​the patch extracted at the base resolution of the histological image.

[0032] This method can be performed for prediction once the CNN is trained. The purpose of training is to assign appropriate weights to the connections between layers. For training, baseline truth data is provided, in which each pixel of a histological image (or set thereof) is assigned to one of the tissue classes. The baseline truth data is based on the use of expert clinicians to annotate a sufficiently large number of images. Training is performed by iteratively applying the CNN, and each iteration involves adjusting weight values ​​by comparing the baseline truth data with the output image patches. In the current implementation, weights are adjusted during training using gradient descent.

[0033] Although there are various options for establishing tissue classes, most embodiments will have in common that classes between non-tumor tissue and tumor tissue will be distinguished. A non-tumor tissue class may include one, two, or more classes. A tumor tissue class may also include one, two, or more classes. For example, in the current implementation, we have three tissue classes, one for non-tumor tissue and two for tumor tissue, where the two tumor tissue classes are for invasive tumors and in situ tumors.

[0034] In some embodiments, the CNN is applied to one histological image at a time. In other embodiments, the CNN may be applied to a synthetic histological image formed by combining a set of histological images taken from differently stained adjacent sections of a tissue region. In yet another embodiment, the CNN may be applied in parallel to each of the images in the set of images taken from differently stained adjacent sections of a tissue region.

[0035] Using the results of the CNN, the method can be extended to include a scoring process based on tumor and pixel classification defined in the corresponding class by referencing a probability map. For example, the method includes the steps of defining regions of the histological image corresponding to tumors according to the probability map; scoring each tumor according to a scoring algorithm and assigning a score to each tumor; and storing the scores, for example, in records of a data repository. Thus, scoring occurs in the histological image but is limited to regions identified by the probability map as containing tumor tissue.

[0036] The results can be displayed to the clinician. That is, histological images can be displayed, for example, on top of or alongside the associated probability maps. Tumor scores can also be displayed in some convenient way, for example, on the tumor or next to a text label or image pointing to the tumor.

[0037] A convolutional neural network can be a complete convolutional neural network.

[0038] Another aspect of the present invention relates to a computer program product for identifying a tumor in a histological image (or set thereof), wherein the computer program product includes a machine-readable instruction for performing the method described above.

[0039] Another aspect of the present invention relates to a computer device for identifying a tumor in histological images (or a set thereof), the device comprising:

[0040] For example, an input capable of operating to receive a histological image (or a set thereof) comprising a two-dimensional array of pixels from a stored record; and

[0041] It includes a processing module, and the processing module is:

[0042] It is configured to apply a convolutional neural network to generate an output image having a two-dimensional pixel array having a mapping of the above histological image, wherein the output image is generated by assigning one of a plurality of tissue classes to each pixel, the plurality of tissue classes including at least one class representing non-tumor tissue and at least one class representing tumor tissue;

[0043] Configured to generate a segmentation mask from an output image, indicating the area occupied by individual tumors;

[0044] Configured to calculate summary statistics for each tumor; and

[0045] It is configured to apply the filter to the summary statistics of each tumor to edit the segmentation mask by selecting and deselecting tumors according to the filter.

[0046] The computer device may further include a post-processing module configured to assemble output image patches into a probability map for a histological image (or set thereof). Furthermore, the computer device may further include an output operable to store the probability map, for example, in a record of a data repository, so that the probability map is linked to the histological image (or set thereof). The device may further include: a display and a display output operable to transmit the histological image (or set thereof) and the probability map to the display, so that the histological image is displayed with the probability map, for example, on top of or together with the probability map.

[0047] Another aspect of the present invention relates to a system comprising the computer device described above combined with one or more other elements. For example, the system may include an image acquisition device operable to acquire histological images, such as a microscope operable to acquire a histological image or a set thereof and store the image in a record in a data repository. Such a microscope may be integrated into a digital scanner. The system may also include a data repository configured to store a record of patient data including histological images; and a network connection capable of transmitting the patient data record or a portion thereof between the computer device and the data repository.

[0048] In summary, methods, computer devices, computer program products, and systems for identifying and visualizing tumors in histological images may be provided. Convolutional Neural Networks (CNNs) are used to classify pixels in an image based on whether the pixels are associated with non-tumor tissue or with one or more classes representing tumor tissue. Segmentation is performed based on the CNN results to generate a mask that displays the area occupied by individual tumors. Then, summary statistics for each tumor are calculated and provided to a filter that edits the segmentation mask to filter out tumors considered insignificant. Optionally, tumors passing through the filter may be ranked according to their summary statistics, for example, in order of clinical relevance or in a reasonable order of review for pathologists. Visualization applications may display histological images taking into account the segmentation mask, summary statistics, and / or rankings.

[0049] The histological image may include one or more additional two-dimensional pixel arrays, for example, one for each of a plurality of stains, or one for each of different depths in a sample (so-called z-stacking) obtained by stepping the focal plane of the microscope through a sample of transparent or translucent finite depth. The output image generated by the CNN also includes one or more two-dimensional pixel arrays, wherein there is a defined mapping between the (input) histological image and the output image, which may be a one-to-one mapping, a many-to-one mapping, or a one-to-many mapping. It will be understood that in at least some embodiments, the histological image(s) are a digital representation of a two-dimensional image of a tissue sample cut by a microscope, particularly a conventional optical microscope, a confocal microscope, or other type of microscope suitable for obtaining histological images of unstained or stained tissue samples. In the case of a set of histological images, these may be a series of microscopic images of adjacent sections (i.e., slices) of a tissue region, where each section may be stained differently. A single histological image may also be a composite calculated by merging multiple histological images obtained from differently stained adjacent sections of a tissue region using a simple overlay or an overlay supported by a warp transform.

[0050] In one embodiment, CNN can be applied to a computer automation method for automatically determining the tumor margin status after surgical resection.

[0051] According to one aspect of the present disclosure, a method for image processing histological images to determine the margin condition of a tumor mass is provided, said method comprising:

[0052] A step of receiving a histological image of a tissue sample section obtained from a tumor mass extracted by resection, and receiving a histological image including a two-dimensional pixel array painted with ink to highlight the surface;

[0053] A step of applying a convolutional neural network process using at least one neural network trained to distinguish tissue classes to generate an output image in a two-dimensional pixel array mapped to the histological image, wherein the output image is generated by assigning a tissue class to each pixel to distinguish between ink-non-ink and tumor-non-tumor;

[0054] A step of calculating the margin state as positive or negative, respectively, depending on the presence or absence of tumors and ink pixels; and

[0055] Step for outputting margin status.

[0056] The positive or negative margin status can be calculated at least partially based on whether the number of tumor and ink pixels is greater than or less than a threshold value. The positive and negative statuses can be subdivided. The calculation of the margin status can subdivide the negative status between the Close and Clear sub-states, where Close and Clear are distinguished by the distance between tumor and non-ink pixels and between ink and non-tumor pixels being less than or greater than a threshold value. The calculation of the margin status can subdivide the positive status between multiple sub-states collectively referred to as margin invasion. Margin invasion is classified into focal, minimal / moderate, and extensive. Margin invasion can be determined based on the prevalence of tumor and ink pixels, where the prevalence can be quantified at least partially based on calculating the maximum distance between tumor and ink pixels.

[0057] Therefore, we propose using deep learning to detect tumor margin conditions and an image analysis algorithm capable of reporting margin distance and margin invasion. The algorithm can be run unsupervised on a batch of virtual slide images that pathologists can review. Providing this margin information to pathologists through this computer automation method can accelerate the pathologist's workflow and generate higher quality and more consistent tumor margin reports.

[0058] In one group of embodiments, a single convolutional neural network is used to perform the entire classification. That is, the convolutional neural network process involves applying a single neural network trained to distinguish tissue classes for tumor and ink, tumor and non-ink, non-tumor and ink, and non-tumor and non-ink.

[0059] In an embodiment of another group, separate convolutional neural networks are applied, one for classifying tumor types and the other for classifying ink and non-ink, and the results of these two neural networks are logically combined to generate a complete set of classes. That is, the convolutional neural network process includes separately applying a first neural network trained to distinguish tissue classes for tumors and non-tumors and output a first output image, and separately applying a second neural network trained to distinguish tissue classes for ink and non-ink and output a second output image. The output image is generated by combining the first and second output images to assign tissue classes for tumors and ink, tumors and non-ink, non-tumors and ink, and non-tumors and non-ink.

[0060] In a classification system, there may be only one type of tumor tissue or there may be multiple types. In one embodiment, since there are separate tumor tissue types for invasive and in situ tumors and one class for non-tumor tissues, a set of six classes is provided with the understanding that there is a division between ink and non-ink in parallel with the tumor tissue types.

[0061] The histological image received by the processing method may be loaded from a record stored in a virtual slide library or another database, or may be acquired directly after acquiring a histological image with a digital scanner; in this case, the method may be performed by the digital scanner itself or by a computer device connected to, for example, the digital scanner or another data source.

[0062] The above method may further include, after the allocation step, a step of assembling the output image into a probability map for the histological image (or set thereof). After the assembly step, the above method may further include a step of storing the probability map linked to the histological image or set thereof in a common record, for example, along with the probability map constituting metadata for the histological image data.

[0063] In the current implementation, since depth increases as dimensionality decreases in each successive convolution step, the convolution layer continues to increase in depth and decrease in dimensionality; and since depth decreases as dimensionality increases in each successive pre-convolution step, the depth of the deconvolution layer continues to decrease and the dimensionality continues to increase. Then, the final convolution layer has a maximum depth and a minimum dimensionality. Instead of the approach of successive depth increase and decrease going through convolution and deconvolution steps, an alternative is to design a neural network in which all layers except the input and output layers have the same depth.

[0064] The above method may further include the step of displaying a histological image or a set thereof on a display, for example, together with a probability map placed on top of or next to each other.

[0065] In a specific embodiment, the convolutional neural network has one or more skip connections. Each skip connection takes an intermediate result from at least one convolutional layer of a larger dimension than the final convolutional layer and passes the result to a transposed convolution as needed to obtain one or more additional recovery layers with a size matching the input image patch. These are then combined with the recovery layers mentioned above prior to the step of assigning a tissue class to each pixel. An additional processing step considers the results obtained from the skip connections by combining the recovery layers with each respective recovery layer to recalculate probabilities.

[0066] In certain embodiments, a softmax operation is used to generate probabilities.

[0067] Image patches extracted from histological image(s) can cover the entire area of ​​the image(s). The patches can be non-overlapping image tiles or image tiles that overlap at the margins to aid in the stitching of the probability map. Since CNNs are designed to accommodate only fixed-size pixel arrays, each image patch must have a fixed number of pixel widths and heights that can be matched with the CNN, but this does not mean that each image patch must correspond to the same physical area on the histological image. This is because pixels in the histological image can be combined into low-resolution patches covering larger areas; for example, each 2×2 array of adjacent pixels can be combined into a single "super" pixel to form a patch with four times the physical area of ​​the patch extracted at the base resolution of the histological image.

[0068] This method can be performed for prediction once the CNN is trained. The purpose of training is to assign appropriate weights to the connections between layers. For training, baseline truth data is provided, in which each pixel of a histological image (or set thereof) is assigned to one of the tissue classes. The baseline truth data is based on the use of expert clinicians to annotate a sufficiently large number of images. Training is performed by iteratively applying the CNN, and each iteration involves adjusting weight values ​​by comparing the baseline truth data with the output image patches. In the current implementation, weights are adjusted during training using gradient descent.

[0069] In an embodiment using separate convolutional neural networks, one is for classifying tumor tissue classes and the other is for classifying ink and non-ink, and the supply of training data may be simpler because two separate training datasets can be used. One is based only on tumor segmentation and the other is based only on ink boundary segmentation. Therefore, the ink training data may be non-specific to the tumor type or the subclassification type of the tumor tissue type. Additionally, the tumor training data may use appropriate pathology images rather than images of resection slides with ink. On the other hand, if the same convolutional neural network is used for 4-way, 6-way, or higher even classification, the training dataset must be images of painted resection tissue masses that have been appropriately professionally analyzed and augmented with underlying factual data.

[0070] Although there are various options for establishing tumor tissue classes, there will be a commonality in that, in most but not all embodiments, a distinction will be made in the classification between non-tumor tissue and tumor tissue. A non-tumor tissue class may include one, two, or more classes. A tumor tissue class may also include one, two, or more classes. For example, in the current implementation, we have three tissue classes: one for non-tumor tissue and two for tumor tissue, where the two tumor tissue classes are for invasive tumors and in-issue tumors.

[0071] In some embodiments, the CNN is applied to one histological image at a time. In other embodiments, the CNN may be applied to a synthetic histological image formed by combining a set of histological images taken from differently stained adjacent sections of a tissue region. In yet another embodiment, the CNN may be applied in parallel to each image of the set of images taken from differently stained adjacent sections of a tissue region.

[0072] The results can be displayed to the clinician. That is, histological images can be displayed along with associated probability maps, for example, on top of or side by side. Tumor scores can also be displayed in some convenient way, for example, above the tumor or next to a text label or image pointing to the tumor.

[0073] A convolutional neural network can be a complete convolutional neural network.

[0074] A further aspect of the present invention relates to a computer program product for identifying a tumor in a histological image or set thereof, wherein the computer program product comprises a machine-readable instruction for performing the method described above.

[0075] Another aspect of the present invention relates to a computer device for processing histological images to determine the margin status of a tumor mass, the device comprising:

[0076] An input operable to receive a histological image comprising a two-dimensional array of pixels of a tissue sample section obtained from a tumor mass extracted by resection, and a histological image painted with ink to highlight its surface; and

[0077] A convolutional neural network processing module comprising at least one neural network trained to distinguish tissue classes to generate an output image, wherein the neural network has a two-dimensional pixel array having a mapping of histological images and assigns a tissue class to each pixel to distinguish between ink and non-ink, and tumor and non-tumor;

[0078] A margin evaluation module capable of operating to calculate a margin state as positive or negative, respectively, depending on the presence or absence of tumor-ink pixels in an output image; and

[0079] Output capable of outputting margin status.

[0080] The output can be connected to a data store to store margin status and optionally output images in a record. The stored output images may include tissue classes for each pixel and / or probabilities for each pixel, i.e., probability maps. Storing output images can be useful for constructing visualizations, for example, because histological images can be displayed with tissue class maps or probability maps overlaid. Whether stored in a record or not, the output can operate to transmit both the histological images and the output images to a display, so that the histological images are displayed together with the output images.

[0081] Another aspect of the present invention relates to a system such as a clinical network, comprising the computer device described above in combination with one or more of the following elements: an image acquisition device, such as a microscope, operable to acquire a histological image or a set thereof; and a data store configured to store a record of patient data including the histological image (or a set thereof). It will be understood that a suitable network connection may be provided between one of these elements to enable data transmission, such as the transmission of a record of patient data or a part thereof between the computer device and the data store.

[0082] In at least some embodiments, the histological image(s) are digital representations of two-dimensional images of a tissue sample cut by a light microscope, which may be a conventional optical microscope, a confocal microscope, or any other type of microscope suitable for obtaining histological images of unstained or stained tissue samples. In the case of a set of histological images, these may be a series of microscope images of adjacent sections (i.e., slices) of a tissue region, where each section may be stained differently.

[0083] Other features and advantages of the present invention will become more readily apparent to those skilled in the art after reviewing the following detailed description and the accompanying drawings. Effects of the invention

[0084] A method and system for identifying tumors in histological images can be provided. Brief explanation of the drawing

[0085] The structure and operation of the present invention can be understood by reviewing the following detailed description and the attached drawings, and in the attached drawings, the same reference numerals refer to the same parts. FIG. 1a is a schematic diagram of a neural network architecture used in one embodiment of the present invention. Figure 1b illustrates a method for generating a feature map that predicts an individual class for each pixel of an input image patch by combining global and local feature maps within the neural network architecture of Figure 1a. Figure 2a is a drawing showing a raw histological patch image in operation as a color image. Figure 2b is a diagram showing the CNN prediction of Figure 2a in operation on a color image. The CNN prediction shows the predicted tumor region indicated in dark red in the color image. Figure 2c is a diagram showing a raw histological image in operation in color. The labeled raw histological image shows the pathologist's hand-annotation (red) and CNN prediction (pink and yellow). Figure 2d is a diagram showing CNN prediction in operation as a color image. The CNN prediction image shows a non-tumor region (green), an invasive tumor region (red, corresponding to pink in Figure 2c), and a non-invasive tumor (blue, corresponding to yellow in Figure 2c). Figure 3a is a diagram showing an example of an input RGB image patch in operation as a color image. The image patch represents a pathologist's manual overview of an invasive tumor (red) and additionally shows an overlay of neural network predictions (pink and yellow). Figure 3b is a diagram showing the final output tumor probability heat map in operation as a color image. The heat map represents the overlay of the neural network prediction (reddish-brown and blue, respectively). Figure 4 is a flowchart showing the steps involved in CNN training. Figure 5 is a flowchart showing the steps involved in prediction using CNN. Figure 6 is a flowchart illustrating the steps involved in using a trained CNN to evaluate margins. FIG. 7 is a flowchart of a method according to one embodiment of the present invention. Figure 8 is a block diagram of a TPU that can be used to perform calculations related to implementing the neural network architecture of Figures 1a and 2b. FIG. 9 shows an exemplary computer network that can be used with embodiments of the present invention. FIG. 10 is a block diagram of a computing device that can be used as a host computer for, for example, the TPU of FIG. 10. FIG. 11a is a block diagram showing an exemplary processorable device (550) that can be used in connection with various embodiments described in this specification. FIG. 11b is a block diagram showing an exemplary line scan camera having a single linear array. FIG. 11c is a block diagram showing an exemplary line scan camera having three linear arrays. FIG. 11d is a block diagram showing an exemplary line scan camera having a plurality of linear arrays. Specific details for implementing the invention

[0086] The following detailed description is provided for illustrative purposes only and not for limitation, to aid in understanding the invention. It will be apparent to those skilled in the art that the present disclosure may be implemented in other embodiments that deviate from these specific details.

[0087] Cancer is the second leading cause of death for women in North America. Among all types of cancer in women, breast cancer is the most common and second-leading cause of cancer death. Therefore, the accuracy of breast cancer treatment has a significant impact on the lifespan and quality of life for a substantial portion of women who develop breast cancer at some point in their lives.

[0088] Breast cancer can be classified into various molecular subtypes based on the expression of specific genes. The commonly used classification system is as follows.

[0089] 1. Luminal A: ER+, PR+, HER2-

[0090] 2. Luminal B : ER+, PR-, HER2+

[0091] 3. Triple-Negative Breast Cancer (TNBC): ER-, PR-, HER2-

[0092] 4. HER2-enriched: HER2+, ER-, PR-

[0093] 5. Normal (Normal-like).

[0094] ER stands for estrogen receptor. PR stands for progesterone receptor. HER2 represents human epidermal growth factor receptor 2.

[0095] We describe a computer-automated tumor detection method that automatically detects and outlines invasive and in situ breast cancer cell nuclei. This method is applied to a single input image, such as a WSI, or a set of input images, such as a set of WSIs. Each input image is a digitized histological image, such as a WSI. For a set of input images, they may be different stained images of adjacent tissue sections. We use the term staining broadly to include staining with biomarkers as well as staining using conventional contrast-enhancing staining.

[0096] Since computer-automated tumor outlining is much faster than manual outlining, it enables the processing of the entire image rather than merely manually annotating tiles extracted from the image. Therefore, the proposed automated tumor outlining must allow pathologists to calculate the positive (or negative) ratio for all tumor cells in an image, which should yield more accurate and reproducible results.

[0097] The proposed computer automation method for tumor detection, contouring, and classification uses a Convolutional Neural Network (CNN) to locate each nuclear pixel in WSI and then classifies each of these pixels into one of the non-tumor classes and one of multiple tumor classes from our current implementation of breast tumor classes.

[0098] The neural network we implemented is<http: / / www.robots.ox.ac.uk / ~vgg / research / very_deep / > and similar to the VGG-16 architecture and design described in Simonyan and Zisserman 2014, the full contents of which are incorporated herein by reference.

[0099] The input image is a pathological image stained with one of several conventional stains, as discussed in more detail elsewhere in this document. For the CNN, image patches are extracted from specific pixel sizes (e.g., 128×128, 256×256, 512×512, or 1024×1024 pixels). Image patches can be of arbitrary size and do not need to be square, but the number of pixels in the rows and columns of the patch follows 2n, where n is a positive integer. Such numbers are generally more suitable for direct digital processing by a suitable single CPU, GPU, or TPU, or an array thereof.

[0100] 'Patch' is a technical term generally used to refer to a portion of an image derived from a square or rectangular WSI. In this regard, since a WSI can contain over a billion pixels (gigapixel images), image processing is typically applied to patches of a manageable size for CNN processing (e.g., approximately 500×500 pixels). Therefore, the WSI is processed by subdividing it into patches, analyzing the patches with a CNN, and then reassembling the output (image) patches into a probability map of the same size as the WSI. The probability map is overlaid, for example, semi-transparently on the WSI or a portion thereof, allowing the pathology image and the probability map to be viewed together. In this sense, the probability map serves as an overlay image for the pathology image. Since the patches analyzed by the CNN may all have the same magnification or a mixture of different magnifications such as 5×, 20×, or 50×, they correspond to physical regions of different sizes within the sample tissue. Depending on the different magnifications, these can correspond to the physical magnification at which WSI was acquired or the effective magnification obtained by digitally reducing a physical image at a higher magnification (i.e., higher resolution).

[0101] Figure 1a is a schematic diagram of the neural network architecture. Layers Cl, C2 ... C10 are convolution layers. Layers D1, D2, D3, D4, D5, and D6 are transpose convolution (i.e., deconvolution) layers. Lines connecting specific layers represent skip connections between the convolution, C, layers and the deconvolution, D, layers. Skip connections allow for the combination of local features from larger-dimensional, shallow-depth layers ("larger" and "thinner" refer to lower-index convolution layers) with global features from the last (i.e., smallest and deepest) convolution layer. These skip connections provide more accurate outlines. Maxpool layers, used to reduce the width and height of the patch by a factor of two respectively, exist behind the C2, C4, and C7 layers, but are only implicitly indicated by the resulting reduction size of the patch and are not directly shown in the schematic diagram. In some implementations of neural networks, the max-pool layer is replaced with lxl convolution to become a complete convolution network.

[0102] The convolutional part of the neural network has the following lasers in order: an input layer (RGB input image patch); two convolutional layers, Cl, C2; a first max-pool layer (not shown); two convolutional layers C3, C4; a second max-pool layer (not shown); three convolutional layers, C5, C6, C7, and a third max-pool layer (not shown). The outputs of the second and third max-pool layers are directly connected to the deconvolutional layer using skip connections in addition to the normal connections to layers C5 and C8, respectively.

[0103] The outputs of the final convolution layer, C1O, the second max-full layer (i.e., the layer after C4), and the third max-full layer (i.e., the layer after C7) are respectively connected to a separate "deconvolution layer" sequence that upgrades the convolution feature maps to the same size as the input (image) patch; that is, the convolution feature maps are converted into feature maps with the same width and height as the input image patch, and the same number of channels (i.e., the number of feature maps) as the number of tissue classes to be detected (i.e., non-tumor types and one or more tumor types). For the second max-full laser, only one step of deconvolution is required, so a direct link to the D6 laser is shown. For the third max-full laser, two steps of deconvolution are required through the intermediate deconvolution layer D4 to reach the D5 laser. For the deepest convolution laser, C10, three steps of deconvolution are required to reach the D3 layer through D1 and D2. As a result, three arrays D3, D5, and D6 of the same size as the input patch are generated.

[0104] A simplified version of the one described in Fig. 1a, although it may have lower performance, can omit skip connections in this case, layers D4, D5, and D6 do not exist, and output patches are calculated only in layer D3.

[0105] Figure 1b illustrates in more detail how the final step is performed in the neural network architecture of Figure 1a. Specifically, the global feature map layer D3 and local feature map layers D5 and D6 are combined to generate a feature map that predicts the individual class for each pixel of the input image patch. In particular, Figure 1b shows how the final three transposed convolution layers D3, D5, and D6 are processed into the tumor class output patch.

[0106] Now, we discuss how the aforementioned approach differs from known CNNs currently used in digital pathology. Conventional CNNs assign a single class selected from several available classes to each image patch. Examples of this type of CNN can be found in the papers by Wang et al. 2016, Liu et al. 2017, Cruz-Roa et al. 2017, and Vandenberghe et al. 2017. However, what has just been described is that a single class selected from several available classes within a given image patch is assigned to each and every pixel. Therefore, instead of generating a single class label for each image patch, the neural network outputs a class label for each individual pixel of a given patch. Since our output patch has a one-to-one pixel-to-pixel correspondence with the input patch, each pixel in the output patch is assigned one of several available classes (non-tumor, tumor 1, tumor 2, tumor 3).

[0107] In these conventional CNNs, a series of convolutional layers are used to assign a single class to each patch, followed by one or more fully connected layers, and then an output vector containing as many values ​​as there are classes to be detected. The predicted class is determined by the location of the maximum value in the output vector. The trained CNN takes pixels from a digital slide image as input and returns a probability vector for each pixel (Goodfellow, Bengio, and Courville 2016). The length of the vector is N, where N is the number of classes the CNN is trained to detect. For example, if the CNN is trained to distinguish three classes—invasive tumor, in situ tumor, and non-tumor—the vector v will have a length of 3. Each coordinate in the vector represents the probability that the pixel belongs to a specific class. Thus, v[0] could represent the probability that the pixel belongs to the invasive tumor class, v[l] the probability that the pixel belongs to the in situ class, and v[2] the probability that it belongs to the non-tumor class. The class of each pixel is determined from the probability vector. A simple way to assign pixels to classes is to assign them to the class with the highest probability.

[0108] To predict the class of individual pixels, CNNs use a different architecture that follows convolutional layers. Instead of a series of fully connected layers, they follow a convolutional layer structure consisting of a series of transposed convolutional layers. Fully connected layers are removed in this architecture. Each transposed layer doubles the width and height of the feature map while simultaneously halving the number of channels. In this way, the feature map is upscaled back to the size of the input patch.

[0109] In addition, to improve predictions, skip linking is used as described in Long et al. 2015, the full content of which is incorporated here by reference.

[0110] Skip connections use shallower features to improve the approximate predictions made by upscaling from the final convolution layer C1O. Local features from the skip connections included in layers D5 and D6 of Fig. 1a are connected to features generated by upscaling the global features included in layer D3 of Fig. 1a from the final convolution layer. The global and local feature layers D3, D5, and D6 are connected as combined layers as shown in Fig. 1b.

[0111] From the combined layer of Fig. 1b (or directly from the final deconvolution layer D3 if skip connections are not used), the number of channels is reduced by the lxl convolution of the combined layer to match the number of classes. Then, a softmax operation on this classification layer converts the values ​​of the combined layer into probabilities. The size of the output patch layer is N×N×K, where N is the width and height of the input patch pixels and K is the number of classes detected. Thus, for every pixel P of the image patch, there is an output vector V of size K. Then, a unique class can be assigned to each pixel P based on the location of the maximum value in the corresponding vector V. Thus, the CNN marks each pixel as non-cancerous or belonging to one or more of several different cancer (tumor) types. While breast cancer is of particular interest, it can also be applied to histological images of other cancers such as bladder cancer, colon cancer, rectal cancer, kidney cancer, blood cancer (leukemia), endometrial cancer, lung cancer, liver cancer, skin cancer, pancreatic cancer, prostate cancer, brain cancer, spinal cancer, and thyroid cancer.

[0112] Our specific neural network implementation is configured to operate on input images with specific fixed pixel dimensions. Therefore, as a preprocessing step for both training and prediction, patches can be extracted from a WSI with desired pixel dimensions, e.g., N×N×h pixels, where n=3 if there are three pixels with three primary colors at each physical location—typically associated with RGB when the WSI is a color image acquired by a conventional visible light microscope. (As mentioned further below, 'h' can be three times the number of composite WSIs when two or more color WSIs are combined.) Additionally, 'h' will have a value of 1 for a single monochromatic WSI. To accelerate training, the input patches are also centered and normalized at this stage.

[0113] Our preferred approach is to process the entire WSI or at least the entire area of ​​the WSI containing tissue; therefore, in our case, the patch is a tile covering at least the entire tissue area of ​​the WSI. Since the tiles can be non-overlapping and adjacent, or have overlapping edge margin areas of, for example, 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10 pixels in width, the output patches of the CNN can be stitched together, taking into account the discrepancies. However, our approach can be applied to random samples of patches of the same or different scales through the WSI if desired, as in existing techniques, or performed by a pathologist.

[0114] Our neural network has a design similar to the VGG-16 architecture of Simonyan and Zisserman 2014. Very small 3×3 kernels are used in all convolution filters. Max pooling is performed with a small 2×2 window and a stride of 2. Unlike the VGG-16 architecture, which has a series of fully connected layers following the convolution layer, we follow the convolution layer with a series of "deconvolutions" (more precisely, transposed convolutions) to generate a segmentation mask. This type of upsampling for semantic segmentation was previously used in natural image processing by Long et al. 2015, the full details of which are included here for reference.

[0115] Each deconvolution layer scales the input feature map by a factor of two in the width and height dimensions. This prevents the scaling effect of the max-pool layer, generating class feature maps of the same size as the input image. The outputs of each convolution and deconvolution layer are transformed by non-linear activation layers. Currently, non-linear activation lasers use the ReLU(x)=max(0, x) rectifier function. Other activation functions, such as ReLU, Leaky ReLU, eLU, etc., can be used as desired.

[0116] The proposed method can be applied to any number of tissue classes without modification. The only constraint is the availability of suitable training data classified in the manner desired for replication in the neural network. Additional examples of breast pathologies include invasive lobular carcinoma or invasive ductal carcinoma. In other words, the single invasive tumor class in the previous examples can be replaced by multiple invasive tumor classes. The accuracy of the neural network is largely determined by the number of images available for each class, the similarity between the classes, and how deep the network can be made before reaching memory limits. Generally, network accuracy increases with a larger number of images per class, a deeper network, and greater class diversity.

[0117] A softmax regression layer (i.e., a multinomial logistic regression layer) is applied to each channel patch to convert the values ​​of the feature map into probabilities.

[0118] After this final transformation by softmax regression, the value at position (x, y) of channel C in the final feature map contains the probability P(x, y) of the pixel at position (x, y) in the input image patch belonging to the tumor type detected by channel C in the input image patch.

[0119] You will notice that the number of convolution and deconvolution layers can be increased or decreased depending on the memory limitations of the hardware running the neural network and your preferences.

[0120] We train the neural network using mini-batch gradient descent. The learning rate decreases from an initial rate of 0.1 using exponential decay. We prevent neural network overfitting by using the "dropout" procedure described by Srivastava et al. 2014

[2017] , the full details of which are included here for reference. Network training can be performed on a GPU, CPU, or FPGA using one of several available deep learning frameworks. While we are currently using Google TensorFlow for the implementation, the same neural network may have been implemented in other deep learning frameworks such as Microsoft CNTK.

[0121] The neural network outputs an N×N×K probability map, where N is the width and height of the input patch in pixels and K is the number of detected classes. These output patches are reconnected to a W×H×K probability map, where W and H are the width and height of the original WSI before being subdivided into patches.

[0122] The probability map can be reduced to a W×H labeled image by recording the class index with the maximum probability at each position (x, y) of the labeled image.

[0123] In the current implementation, our neural network assigns every pixel to one of three classes: non-tumor, invasive tumor, and in situ tumor.

[0124] When multiple tumor classes are used, the output images can be post-processed into a simpler binary classification of non-tumor and tumor. In other words, the multiple tumor classes are combined. Binary classification can be used optionally when creating images from the base data, while the multi-class tumor classification is retained in the stored data.

[0125] While the above description of a specific implementation of the present invention focuses on a specific approach using CNNs, it will be understood that our approach can be implemented in a wide variety of convolutional neural networks. In general, any neural network that uses convolution to detect increasingly complex features and then uses transposition convolution ("deconvolution") to upscale the feature map back to the width and height of the input image should be suitable.

[0126] Example 1

[0127] Figures 2a to 2d are color images in operation and show the difference between patch-level predictions generated by Google’s CNN solution for Camelyon competition (Liu et al. 2017) and pixel-level predictions for tumor classes generated by the CNN described above.

[0128] Figures 2a and 2b are copied from Figure 7 of Liu et al 2017, while Figures 2c and 2d are comparable tiles for examples according to the currently described CNN applied to classify tumor tissue classes (including ink / non-ink).

[0129] Figure 2a is a patch of H&E-stained WSI, where the larger, dark purple cell cluster in the lower right quadrant is a tumor and the smaller dark purple cells are lymphocytes.

[0130] Figure 2b is a tumor probability heatmap generated by CNN by Liu et al. 2017, in which the authors note that it accurately identifies tumor cells while ignoring connective tissue and lymphocytes.

[0131] FIG. 2c is a raw image patch of an exemplary WSI to which the CNN method embodying the present invention is applied. In addition to the raw image, FIG. 2c shows contours manually drawn by a pathologist (red boundary lines in solid). Furthermore, referring to FIG. 2d, FIG. 2c also shows the results of the CNN method described above (a first region shaded in pink with pink boundary lines, i.e., the tumor type indicated in red in FIG. 2d, corresponds to the first tumor type; a second region shaded in yellow with pink boundary lines, i.e., the tumor type shaded in blue in FIG. 2d, corresponds to the second tumor).

[0132] Figure 2d is a tumor probability heatmap generated by our CNN. It shows how our approach to pixel-level prediction generates regions with smooth boundary contours. In the heatmap, different (arbitrarily selected) colors represent different classes. Specifically, green for non-tumor, red for tumor type 1, and blue for tumor type 2.

[0133] Example 2

[0134] Figures 3a and 3b are color images in operation and show examples of an input RGB image patch (Figure 3a) and a final output tumor probability heat map (Figure 3b).

[0135] Figure 3a additionally shows the pathologist's manual contour setting of an invasive tumor (red outline) and an additional overlay of our neural network's prediction (pink shade and yellow area) shown separately in Figure 3b.

[0136] Figure 3b is a tumor probability heatmap generated by the CNN. For our heatmap, different (arbitrarily selected) colors represent different classes. Specifically, green represents non-tumor, reddish-brown (indicated as pink in Figure 3a) represents invasive tumors, and blue (indicated as yellow in Figure 3a) represents in situ tumors. Once again, we can see how our approach to pixel-level prediction generates regions with smooth boundary contours. Furthermore, we can verify how the CNN predictions are compatible with a pathologist's manual markings. Additionally, the CNN can better distinguish between invasive and non-invasive (in situ) tissues, which a pathologist has not performed, and is essentially part of a multi-channel CNN design that can be programmed and trained to classify tissues into a random number of different clinically relevant types as desired.

[0137] Acquisition and Image Processing

[0138] The starting point of the method is to section the tissue sample, that is, to slice it and stain adjacent sections with different stains. Adjacent sections will have very similar tissue structures because the sections are thin, but they will not be identical because they are in different layers.

[0139] For example, there may be five adjacent sections with different stains, such as ER, PR, p53, HER2, H&E, and Ki-67. Then, microscopic images are acquired from each section. Although adjacent sections have very similar tissue morphologies, the stains highlight different features, such as the nucleus, cytoplasm, and all features by general comparative enhancement. Then, the different images are aligned, warped, or preprocessed to match the coordinates of any given feature in one image to the same feature in another image. Mapping handles differences between images caused by factors such as slight differences in magnification and orientation resulting from differences in microscope slide alignment or mounting of tissue slices on slides.

[0140] It should be noted that WSIs can be merged into a single composite WSI through coordinate mapping between different WSIs of a set composed of different adjacent sections, and such composite patches can be extracted for processing by a CNN, where the composite patch has dimensions of N×N×3m and |m| is the number of composite WSIs forming the set.

[0141] Then, some standard processing of the images is performed. These image processing steps can be performed at the WSI level or at the individual image patch level. If the CNN is configured to operate on grayscale rather than color images, the images can be converted from color to grayscale. The images can be modified by applying contrast enhancement filters. Then, some segmentation may be performed to identify common tissue regions within the image set or simply to remove backgrounds unrelated to the tissue. Segmentation may include some or all of the following image processing techniques.

[0142] 1. Variance-based analysis to identify seed organization regions

[0143] 2. Setting adaptive thresholds

[0144] 3. Morphological work (e.g., stain analysis)

[0145] 4. Contour Identification

[0146] 5. Contour merging based on proximity heuristic rules

[0147] 6. Calculation of Invariant Image Moments

[0148] 7. Edge extraction (e.g., Sobel edge detection)

[0149] 8. Curvature Flow Filtering

[0150] 9. Histogram Matching to Eliminate Intensity Variations Between Serial Sections

[0151] 10. Multi-resolution rigid / affine image registration (Gradient Descent Optimization Tool)

[0152] 11. Non-rigid deformation / transformation

[0153] 12. Superpixel Clustering

[0154] Furthermore, it will be understood that the aforementioned type of image processing step can be performed on the WSI or individual patches after patch extraction. In some cases, it may be useful to perform the same type of image processing before and after patch extraction. That is, it is equivalent to CNN preprocessing and CNN postprocessing, respectively. In other words, some image processing can be performed on the WSI before patch extraction, and other image processing can be performed on the patches after extraction from the WSI.

[0155] These image processing steps are described by way of example and should not be interpreted as limiting the scope of the invention in any way. For example, if sufficient processing power is available, the CNN can work directly with color images.

[0156] Learning and prediction

[0157] Figure 4 is a flowchart showing the steps involved in CNN training.

[0158] In step S40, training data containing WSIs for clinician-annotated processing is retrieved to locate, contour, and classify tumors. The clinician's annotations represent ground truth data.

[0159] In step S41, the WSI is subdivided into image patches, which are input image patches for the CNN. In other words, image patches are extracted from the WSI.

[0160] In step S42, the image patch is preprocessed as described above. (Alternatively or additionally, the WSI may be preprocessed as described above prior to step S41.)

[0161] In step S43, initial values ​​are set for the CNN weights, i.e., the weights between layers.

[0162] In step S44, each batch of input image patches is input into and processed by a CNN to locate, outline, and classify the patches pixel by pixel, as described further with reference to Figures 1a and 1b. Since our method identifies each tumor (or tumor type) pixel, the term "outline" is not strictly technically correct to use here, so it would be more accurate to say that the CNN determines the tumor region for each tumor type.

[0163] In step S45, the CNN output image patches are compared with the underlying factual data. This can be performed on a patch-by-patch basis. Alternatively, if patches covering the entire WSI are extracted, this can be performed at the WSI level or in sub-regions of the WSI composed of a series of patches (e.g., one quadrant of the WSI). In this variation, the output image patches can be reassembled into a probability map for the entire WSI or its adjacent parts, and the probability map can be visually compared with the underlying factual data by the computer and the user, for example, if the probability map is displayed as a translucent overlay on the WSI.

[0164] In step S46, the CNN learns from these comparisons and updates the CNN weights using, for example, a gradient descent approach. Thus, the learning is fed back to the iterative processing of the training data by a return loop in the processing flow, as shown in Fig. 4, so that the CNN weights can be optimized.

[0165] After training, CNN can be applied to WSI for real-time use for prediction, regardless of the underlying factual data.

[0166] Figure 5 is a flowchart showing the steps involved in prediction using CNN.

[0167] In step S50, one or more WSIs are retrieved for processing, for example from a laboratory information system (LIS) or another histological data repository. The WSIs are pre-processed, for example, as described above.

[0168] In step S51, image patches are extracted from each WSI. The patches may include the entire WSI, or be random or non-random selections.

[0169] In step S52, the image patch is preprocessed, for example, as described above.

[0170] In step S53, each batch of input image patches is input into a CNN and processed to locate, contour, and classify patches pixel by pixel as described above with reference to FIGS. 1a and 1b. The output patches can be reassembled into a probability map for the WSI from which the input image patches were extracted. For example, if the probability map is displayed on a display as a translucent overlay of the WSI or together with the WSI, the probability map can be visually compared with the WSI by a computer device for digital processing and a user.

[0171] In step S54, tumor sites are filtered to exclude tumors that may be falsely positive, such as areas that are too small or areas that could be margin defects.

[0172] In step S55, the scoring algorithm is executed. The scoring is cell-specific, and the scores can be aggregated for each tumor and / or additionally aggregated for WSI (or sub-regions of WSI).

[0173] In step S56, the results are presented to a pathologist or other relevant skilled clinician for diagnosis. The annotated WSI is displayed on an appropriate high-resolution monitor.

[0174] In step S57, the CNN results—namely, probability map data and optionally arbitrary additional diagnostic information added by the pathologist—along with metadata related to the CNN parameters, are stored in a manner associated with a patient data file containing the WSI or WSI set processed by the CNN. Thus, the patient data file in the LIS or other histological data repository is supplemented by the CNN results.

[0175] Margin evaluation

[0176] Figure 6 is a flowchart illustrating the steps involved in prediction using a CNN. In step S1, one or more WSIs of ink and stained tissue samples for processing are retrieved using a laboratory information system (LIS) to retrieve data, for example, from a virtual slide library or other histological data repository.

[0177] In step S2, WSI is preprocessed as, for example, as described above.

[0178] In step S3, the first CNN is applied to perform 3-way tissue type classification (non-tumor, invasive tumor, in situ tumor). Image patches are extracted from one or more WSIs. The patches may include the entire WSI or be random or non-random selections. The image patches may be preprocessed, for example, as described above. Each group of input image patches is input into the CNN and processed to classify the patches pixel by pixel, as described above with reference to Figures 1a and 1b. The output patches are reassembled into a probability map of tumor locations.

[0179] In step S4, a second CNN is applied to perform binary classification to recognize ink pixels (ink, non-ink). A probability map is generated as described for step S3, and this time the probability map recognizes the ink locations, i.e., the boundaries of the tissue masses.

[0180] In step S5, some post-processing is performed on the two probability maps to filter the probability values ​​according to a simple threshold or more complex numerical processing. For example, segmentation may be applied to the probability values ​​to identify ink boundaries and adjust the probability values ​​of pixels of the ink category accordingly. Post-processing may also use segmentation to exclude tumor areas with a potential for false positives, such as areas that are too small or areas that may be edge defects. If desired, the two probability maps output in steps S3 and S4 may be merged or combined into a single probability map, and post-processing may be performed on the merged probability map.

[0181] As described above, steps S3 and S4 are effectively merged in another embodiment where a single CNN performs 6-way classification and generates a single probability map, in which case the post-processing of step S5 will be performed on the single probability map. (Note that the probability map can also be used to generate visualizations in a viewing application as an overlay on a histological image that can be presented to a pathologist, for example, along with the margin evaluation data output from the process flow of FIG. 6.)

[0182] In step S6, a mask is generated by assigning each pixel to one of six classes. (It should also be noted that the mask can be used to generate visualizations in a viewing application as one or more overlays on the histological image, such as an ink boundary overlay or a tumor pixel overlay. These overlays can be presented to a pathologist along with the margin evaluation data output from the process flow of Fig. 6.)

[0183] In step S7, it is determined whether there is more than a threshold number of pixels of the tumor and ink classes. If 'no', in step S8 the margin status is set to 'negative' and the process flow proceeds with the margin distance calculation; otherwise, if 'yes', in step S12 the margin status is set to 'positive' and the process flow proceeds with the calculation of margin invasion, the degree to which the tumor tissue invades the boundary of the tissue mass resected by the resection surgery. The threshold text may apply a single threshold to the sum of all pixels of the tumor and ink classes or apply separate thresholds to the individual sums of each tumor and ink class. For example, in the latter case, if the "Invasive Tumor and Ink" or "In Situ Tumor on Ink" class is detected at a level higher than the respective threshold (which may be the same or different), the threshold test result is "Yes" and the margin status is "Positive," whereas if the level is the same or lower in the comparison of the two thresholds, the test result is "No" and the margin status is "Negative." The threshold(s) may be set to empirically defined value(s) or any other appropriate value(s).

[0184] Following the negative branch, in step S9, the margin distance is calculated and tested against an appropriate threshold in step S10. If pixels of the tumor-ink class are detected at a level lower than the threshold, a margin exists, so the margin status must be "Clear" or "Close" depending on the margin distance. In step S11, if the margin distance is less than the threshold, the status is set to Negative / Close. In step S14, if the margin distance is less than the threshold, the status is set to Negative / Clear. The margin distance is defined as the shortest distance between the tumor tissue and the ink surface. The coordinates (xt, yt) ∈ R of all tumor pixels 2 Set S1 including, coordinates (xm, ym) of all margin pixels ∈ R 2It can be defined as including a set S2 containing. Therefore, the problem of calculating the margin distance is the problem of calculating the minimum distance between two sets of planar points. Since this measure can be calculated in O(nLogn) time, it can be calculated quickly even on large sets of points (Toussaint and Bhattacharya 1983). If the margin distance Dm is less than a specified threshold θ that can be set by the user, the margin state returns to "closed". If Dm ≥ θ, the margin state returns to "clear". Of course, the actual calculated margin distance can be output. This can be a simple number representing the smallest distance, or a more sophisticated output providing additional data such as individual distances for the minimum and maximum margin distances, and statistical values ​​(e.g., standard deviation) such as the average margin distance and probability distribution.

[0185] Following the positive branch, margin invasion is calculated in step S13. Since pixels of the tumor-ink class were detected at one or more of the threshold ε or threshold ε, there are no tumor-free margins, i.e., no positive margins. For positive margins, margin invasion can be calculated as the total length of tumor cells on the ink surface. This is equivalent to finding the maximum distance between two pixels belonging to each tumor and summing the results. Since this problem is identical to the minimum distance problem, it can be solved in O(nLogn) time (Vaidya 1989). Current College of American Pathologists guidelines require only approximate reporting ranges. One method for classifying the degree of margin invasion based on total length is as follows.

[0186] · Single focal point: A single focal point area of ​​a tumor where the maximum distance is, for example, at a margin less than a threshold of < 4 mm.

[0187] · Multifocal: Two or more carotid focal points in the margin, and the sum of the maximum distances is less than the monofocal threshold.

[0188] · Minimum / Normal: For monofocal or multifocal, the sum of the maximum distances is greater than the second threshold used for monofocal and multifocal classification, e.g., < 5 mm.

[0189] · Extensive: Cancer present on the margins spanning a wide frontal area (sum of maximum distances > 5mm)

[0190] If desired, single-focus and multi-focus can be merged into a single class of displayed focus.

[0191] Finally, the process flow ends at step S15 by the margin state, i.e., the output, being reported, for example, to a display and / or saved to a file.

[0192] In summary, we describe a computer-automated method for processing histological images of tissue sections obtained from tumor masses extracted by resection, and then painting the surface areas with ink to highlight them. One or two convolutional neural networks are trained to distinguish tissue classes, differentiating between ink and non-ink, and between tumor and non-tumor. The trained neural networks are applied to the histological images to generate output images in which pixels are assigned to tissue classes. The benign or negative margin status of the tissue section can be determined based on the presence or absence of tumor and ink classification pixels, respectively. Additionally, benign and negative margin statuses can be further examined to generate substates for margin invasion (positive substate) and margin distance (negative substate) by calculating additional parameters based on the distance between pixels by classification.

[0193] Tumor region filtering and ranking

[0194] FIG. 7 is a flowchart according to one embodiment of the present invention.

[0195] Step S71 provides an image data file containing image data of WSI that can be generated by a slide scanner. The image data file may include multiple images, for example, one for each of a plurality of blotches, or one for each of different depths of a sample (so-called z-stack) obtained by stepping the focal plane of a microscope through a transparent or translucent sample of finite depth.

[0196] Step S72 is an optional step in which some CNN preprocessing may be performed, such as variance-based analysis, adaptive threshold setting, and morphological operations, for example as described above.

[0197] Step S73 executes the CNN described above, as described specifically with reference to steps S51 through S54 of FIG. 5. Pixel-wise classification of tissue types is performed to identify tumor pixels, followed by segmentation to outline the tumor (i.e., tumor region). The tissue type is a classification based on the type of carcinoma. Segmentation is typically performed when adjacent tumor pixels—that is, pixels that are in contact with or close to each other—belong to a common tumor. However, more complex segmentation criteria are generally included to improve reliability, for example, to identify two contacting tumors of different pixel classifications associated with two different cancer cell classifications. The CNN assigns a probability to each pixel, which is a probability vector representing the probability of a pixel belonging to each of the N classes that the CNN has been trained to detect. For example, for a CNN trained to distinguish invasive, in situ, and non-tumor regions, a vector of length 3 is assigned to each pixel. A pixel at location k may have a probability vector [0.1, 0.2, 0.7] indicating that the pixel has a 10% probability of being in an invasive area, a 20% probability of being in an in situ area, and a 70% probability of being in a non-tumor area.

[0198] In step S74, the data generated by the tumor discovery CNN in step S73—that is, tumor-specific data—is used to calculate a set of summary statistics for each tumor. For example, for each tumor, the score can be calculated as the mathematical mean of the aforementioned probability values ​​for all pixels included in that tumor (region). Other summary statistics, such as the median and weighted average, can also be used to calculate the score. For example, other measurements such as the dimensions or morphological attributes of the tumor may be used, such as the tumor area measured by the number of pixels in the tumor or the shape of the tumor region, or the tumor area measured by the prevalence of specific pixel classifications like invasive tumors and in situ tumors. Generally, for each tumor, the mean and standard deviation of the tumor probability, tumor area, and maximum dimension length of the tumor are included. The tumor regions are not necessarily taken from a single slide; they may belong to separate slides. For example, tissue samples from two slides are stained with different colors to highlight different classes of tumor cells, so that some tumors are identified on the first slide and others on the second. Optionally, more sophisticated scores can be calculated. For example, patient risk can be predicted using a CNN trained on a combination of histological image data (e.g., tumors identified in image data) and patient-specific genetic data (e.g., genome). This type of CNN is described in Mobadersany et al. 2018. In other implementations, scores can be calculated using traditional image processing techniques applied to tumors identified by the CNN (e.g., as listed above regarding segmentation). For instance, shape and texture measures can be combined with genetic data to create a series of statistical measures to be included in summary statistics. Support Vector Machines or Random Forest algorithms can use these features to predict metastasis risk.We define the risk of metastasis as the probability that tumor cells will spread to other parts of the body. In either case, a metastasis risk score is calculated and associated with each tumor area.

[0199] Step S75 performs filtering of tumors based on summary statistics calculated and compiled in Step S74. The filter may operate by a simple pass through a list of tumors to remove or retain tumor entities based on the comparison result, after comparing a value of one of the parameters in the summary statistics, or a composite value derived from a logical combination and / or formula containing multiple parameters present in the summary statistics, with a threshold. For example, a filter that may be configured or preset by a pathologist may select to pass only tumors that have a maximum dimension exceeding a threshold (e.g., 100 micrometers) and an average probability higher than a threshold (e.g., 50%). Another exemplary filter may pass only tumors classified as invasive, and among invasive tumors, only those with an average probability exceeding a threshold (e.g., 80%).

[0200] Step S76 takes the set of tumors filtered in Step S75 and ranks them in order. The tumors can then be ranked based on summary statistics and by applying a series of criteria. Presets of standard ranking approaches are provided to the user, allowing the user to select one of the presets. Furthermore, the user may be provided with a user interface to define which criteria will be applied. For example, in the case of scalar value criteria based on length dimensions, area, or integer counts (e.g., cell count), the user may set thresholds or value ranges for these criteria. The ranking order may be based on a composite score indicating importance for patient survival, such as, for example, the 5-year survival probability, or a simple single-parameter ranking based on tumor size parameters, such as area or maximum dimensions, or morphological parameters, such as roundness.

[0201] Step S77 generates a visualization of the slide image and displays it to the user in a GUI window on the display device. The visualization also takes into account filtering and ranking. In particular, tumors considered to be the most important in the WSI are displayed in a clinically relevant manner, along with summary statistics that are available in all cases as part of the overall display or optionally respond to user GUI commands; for example, clicking on a tumor of interest in the WSI with a cursor generates a pop-up window that displays statistics for that tumor in a table or other appropriate form. This approach can be presented to the user by highlighting potentially important tumors and providing the user with ranking information among potentially important tumors and a statistical summary of these tumors. The statistical summary may display not only individual parameter values ​​used as filtering and / or ranking criteria, but also composite parameters such as rank numbers or significance scores calculated from formulas and / or Boolean logic combinations of multiple filtering and / or ranking criteria.

[0202] Generally, the displayed image will be in the form of a combined overlay view or a multi-tile view. In the overlay view, raw data (processable) is displayed at the top along with segmentation data, and the segmentation data is converted into shades and / or outlines of each tumor for visualization. The shades or outlines may be color-coded, for example, by tumor classification. Non-tumor regions of the tissue may not be displayed at all or may be shaded with a high-transparency color wash (e.g., a blue wash). In the multi-tile view, different layers of the overlay view are displayed side-by-side as tiles, so there will be tiles showing raw image data (processable) and segmentation data of filtered tumor regions. If desired, separate segmentation data tiles may be displayed for each tumor classification type. The display of tumor regions in the display takes into account the filtering performed in step S75 and, preferably, the ranking performed in step S76. Factors such as tumor scores, classified tumor types, and other parameters of summary statistics related to tumor regions may be used alone or in combination to configure the display.

[0203] There are several options for displaying detected, fillerated, and ranked tumors to the user.

[0204] In the WSI view, one method of displaying tumor information is to overlay a series of markers, which may include thumbnail images and / or text information, on the low-resolution image of the WSI. The markers may be sorted by rank or explicitly marked. Pathologists are provided with GUI tools suitable for selecting tumors by selecting markers, which may also be used to navigate tumors from high-ranking to low-ranking tumors and in other ways. Suitable GUI tools include keyboard shortcuts; keyboard up / down or left / right, arrow keys, keyboard page up / page down keys, mouse navigation (e.g., scrolling up and down using a scroll wheel), or other input devices (voice navigation, multi-touch gestures on touch sensors, etc.). When a ranked marker is selected in the GUI, summary statistics of the corresponding tumor or a subset thereof may be displayed and / or a high-resolution view of the tumor may be displayed in a pop-up or a separate high-resolution viewing window linked to the display of the tumor in the low-resolution image. The user can display a high-resolution, e.g., full native resolution image of the tumor by appropriate GUI commands, such as keyboard commands, mouse commands (e.g., double-click), or other suitable inputs. A corresponding GUI command is provided to return to a low-resolution WSI view. The visualization application preferably provides GUI controls that allow the user to move the tumor up and down by ranking it in both the low-resolution and high-resolution views.

[0205] One exemplary WSI view displays all tumors that have passed the filter in step S76 along with rank marker labels 1, 2, 3, etc. (i.e., segmentation data is displayed). Clicking a rank marker label (or tumor) may generate a popup listing the selected set of summary statistics, specifically the statistics used by the filter, and / or thumbnail views of the tumor at a higher resolution than the WSI view. Alternatively, the view may be a split-screen view where an overlay image or image tile is displayed on one part of the screen and a table of filtered tumors is displayed on another part of the screen. The table may initially be displayed sorted by rank, but the GUI may have a feature that allows the user to reorder by other columns or a combination of multiple columns. The other columns may be any criteria, such as summary statistics or filter criteria like tumor area, tumor classification, metastasis probability, etc. For example, sorting may be based on tumor classification followed by tumor area. For multi-slide image files, sorting may be based on slide number and several other parameters.

[0206] Generally, a multi-resolution view includes one viewing window of lower resolution (e.g., 10x magnification) that reproduces the WSI and another viewing window of higher resolution (e.g., 60x magnification, i.e., 6x or 600% zoom compared to the 10x view). For example, the presented initial view may consist of a low-resolution WSI image window and a high-resolution image window centered on the highest-ranked tumor region. Using other appropriate GUI buttons or button combinations, such as a cascading down arrow (or a pair of down and up arrows) or a physical or virtual scroll wheel, allows the user to step through the filtered tumor regions by ranking them one by one. The GUI can allow the user to adjust the resolution in the high-resolution image window via user input. The GUI can also select an initial resolution for the high-resolution image window to resize it so that the tumor substantially fills the high-resolution viewing window.

[0207] Therefore, the visualization application can determine the items displayed to the user, the tumors highlighted by segmented data and summary statistics, and the chronological order for optional views—that is, the order of tumors displayed to the user.

[0208] Another method for displaying tumor regions that is particularly suitable for a slide set is to create a montage in which low-resolution images of the tumor and summary statistics are displayed as aligned tiles. The alignment can be a method of displaying a one-dimensional (ID) list or a two-dimensional (2D) grid. It is also possible to arrange the tumor in three dimensions using virtual reality goggles. Users can navigate the ID list or 2D tile array using keyboard input, mouse scrolling, or other input methods (voice, touch, etc.). Once a tile is selected, the pathologist can quickly move to a high-resolution version of the tumor on the corresponding slide and selectively validate the tumor more deeply by comparing it with the CNN analysis results presented in the summary statistics and segmentation data, or perform further processing on the tumor by selectively applying one or more additional analysis algorithms that may aid in diagnosis.

[0209] At step S78, the pathologist has an additional option to select any one of these tumors (or a subset of the graded tumors, or actually the entire set of filtered tumors) for further study after viewing the filtered and graded tumors.

[0210] In step S79, these additional studies are performed. Additional studies refer to the application of one or more additional algorithms to provide additional information about the selected tumor to aid in diagnosis. This can help confirm a provisional diagnosis or provide an initial diagnosis that a pathologist may consider as a starting point.

[0211] For example, in the case of breast cancer, at step S78, the pathologist selects a tumor containing an invasive breast cancer tumor, and then at step S79, applies an algorithm to calculate the number of mitotic divisions of the tumor. That is, the visualization application includes such an algorithm. Another breast cancer example involving image data from IHC stained slides is to apply an algorithm to calculate a score for the expression of one or more diagnosis-related genes, such as ER, PR, or HER2, in the selected tumor.

[0212] In summary, the visualization is configured to guide users to inspect image areas where potentially serious tumors may be present. In particular, compared to existing visualization applications, users do not need to perform manual visual scans of the entire slide at low resolution or zoom in and out of high-resolution views whenever a potential tumor area is identified. As mentioned above, the automated preprocessing of the visualization presented to pathologists based on CNN-assisted filtering and ranking not only reduces the time required for pathologists to review slides and make appropriate diagnoses but also reduces the likelihood of important tumors being missed due to human error.

[0213] Examples

[0214] In one embodiment, a system for identifying a tumor in a histological image is configured to perform a method for identifying a tumor in a histological image. The method includes receiving a histological image comprising a two-dimensional array of pixels, applying a convolutional neural network to generate an output image having a two-dimensional array of pixels, and mapping pixels of the output image to pixels of the histological image. The output image is generated by assigning one of a plurality of tissue classes to each pixel, wherein the plurality of tissue classes includes at least one class representing non-tumor tissue and at least one class representing tumor tissue.

[0215] Next, the system generates a segmentation mask from the output image. The segmentation mask identifies the area occupied by individual tumors. The system also calculates summary statistics for each tumor, and is configured to filter the segmentation mask to select and deselect individual tumors based on these summary statistics. The system is also configured to generate visualizations of the histological image based on the edited segmentation mask, receive the selection of individual tumors from the filtered segmentation mask, and then coordinate the execution of additional computational diagnostic processes for the currently selected tumors.

[0216] In one embodiment, the additional computational diagnostic process includes the step of transmitting a processing task to the additional computational diagnostic process and receiving the result of the processing task from the additional computational diagnostic. The additional computational diagnostic process may be executed locally or on a remote processor-supported device that is communicated via a data communication network. Part of the processing task may include a histological image. The processing task may also include metadata related to the histological image, and in some cases, the metadata related to the histological image may include a segmentation mask. Additionally, the metadata related to the histological image may include one or more of patient information, staining information, staining protocol information, and scanning protocol information.

[0217] CNN computing platform

[0218] The proposed image processing can be performed on various computing architectures, particularly architectures optimized for neural networks that may be based on CPUs, GPUs, TPUs, FPGAs, and / or ASICs. In some embodiments, the neural network is implemented using Google's TensorFlow software library running on Nvidia GPUs from Nvidia Corporation in Santa Clara, California, such as the Tesla K80 GPU. In other embodiments, the neural network may run on a general-purpose CPU. Faster processing may be achieved by processors designed for the purpose of performing CNN computations, for example, TPUs disclosed in Jouppi et al. 2017, the full contents of which are incorporated herein by reference.

[0219] FIG. 8 shows the TPU of Jouppi et al. 2017, which is a simplified version of FIG. 1 of Jouppi. The TPU (100) has a matrix multiplication unit (MMU) (102) containing 256×256 MACs capable of performing 8-bit multiplication and addition on signed or unsigned integers. Weights for the MMU are supplied through a weight FIFO buffer (104) that reads weights from memory (106) in the form of off-chip 8GB DRAM via an appropriate memory interface (108). An integration buffer (UB) (110) is provided to store intermediate results. The MMU (102) is connected to receive inputs from the weight FIFO interface (104) and the UB (110) (via the shrink data setting unit (112)) and outputs the 16-bit product of the MMU processing to the accumulator unit (114). The activation unit (116) performs a non-linear function on the data held in the accumulator unit (114). After further processing by the normalization unit (118) and the pooling unit (120), the intermediate result is sent to the UB (110) to be fed back to the MMU (102) via the data setting unit (112). The pooling unit (120) can perform max pooling (i.e., max pooling) or average pooling as desired. The programmable DMA controller (122) transfers data to or from the host computer of the TPU and the UB (110). TPU commands are transferred from the host computer to the controller (122) via the host interface (124) and the command buffer (126).

[0220] It will be understood that the computing power used to run neural networks, whether CPU, GPU, or TPU-based or not, can be hosted locally in the clinical network described below, for example, or remotely in a data center.

[0221] Network, computing, and scanning environments

[0222] The proposed computer automation method generally operates in the context of a Laboratory Information System (LIS), which is part of a large-scale clinical network environment such as a Hospital Information System (HIS) or a Picture Archiving and Communications System (PACS). In the LIS, WSIs will typically be stored in a database, which is a patient information database containing individual patients' electronic medical records. Since the microscope acquiring the WSI is equipped with a barcode reader, the WSIs will be retrieved from stained tissue samples mounted on slides with printed barcode labels that tag the WSIs with metadata appropriate for them. From a hardware perspective, the LIS will be a general computer network, such as a Local Area Network (LAN), using wired and wireless connections as desired.

[0223] FIG. 9 illustrates an exemplary computer network that can be used with embodiments of the present invention. The network (150) includes a LAN of a hospital (152). The hospital (152) has a plurality of workstations (154) each having access to a hospital computer server (156) that has an associated storage device (158) via a local area network. An LIS, HIS, or PACS archive is stored in the storage device (158) so that data within the archive can be accessed from any workstation (154). One or more workstations (154) may have access to a graphics card and software for a computer implementation of the method of generating images as described above. The software may be stored locally at each workstation (154) or remotely and may be downloaded to the workstation (154) via the network (150) when necessary. In another example, the method of implementing the present invention may be executed on a computer server with a workstation (154) acting as a terminal. For example, a workstation may be configured to receive user input defining a desired histological image dataset and to display the resulting images while CNN analysis is performed elsewhere in the system. Additionally, multiple histological and other medical imaging devices (160, 162, 164, 166) are connected to a hospital computer server (156). Image data collected by the devices (160, 162, 164, 166) may be stored directly in an LIS, HIS, or PACS archive on the storage device (156). Thus, histological images can be viewed and processed immediately after the corresponding histological image data is recorded. A local area network is connected to the Internet (168) by a hospital internet server (170), which allows remote access to the LIS, HIS, or PACS archive. This is used for remote data access and data transfer between hospitals, for example, to allow patients to move or perform external research.

[0224] FIG. 10 is a block diagram illustrating an exemplary computing device (500) that may be used in connection with various embodiments described herein. For example, the computing device (500) may be used as a computing node in the aforementioned LIS or PACS system, for example, a host computer where CNN processing is performed with a suitable GPU, or a TPU shown in FIG. 8.

[0225] The computing device (500) may be a server or any conventional personal computer, or any other processorable device capable of wired or wireless data communication. As will be obvious to those skilled in the art, other computing devices, systems, and / or architectures including devices that cannot perform wired or wireless data communication may also be used.

[0226] The computing device (500) preferably includes one or more processors, such as a processor (510). The processor (510) may be, for example, an array such as a CPU, GPU, TPU, a combination of CPU and TPU or a combination of CPU and GPU, or a combination thereof. Additional processors may be provided, such as an auxiliary processor for managing input / output, an auxiliary processor for performing floating-point mathematical operations (e.g., TPU), a special-purpose microprocessor with an architecture suitable for the rapid execution of signal processing algorithms (e.g., digital signal processor, image processor), a slave processor dependent on the main processing system (e.g., backend processor), or an additional microprocessor or controller or coprocessor for a dual or multi-processor system. These auxiliary processors may be individual processors or may be integrated with the processor (510). Examples of CPUs that may be used with the computing device (500) include Pentium processors, Core i7 processors, and Xeon processors, all of which are available from Intel Inc. in Santa Clara, California. An exemplary GPU that can be used with the computing device (500) is the Tesla K80 GPU from Nvidia Corporation in Santa Clara, California. The processor (510) is connected to a communication bus (505). The communication bus (505) may include a data channel to facilitate the transfer of information between the storage device and other peripheral components of the computing device (500). The communication bus (505) may further provide a set of signals used for communication with the processor (510), including a data bus, an address bus, and a control bus (not shown).The communication bus (505) may include any standard or non-standard bus architecture, such as the Industrial Standard Architecture (ISA), the Extended Industrial Standard Architecture (EISA), the Micro Channel Architecture (MCA), the Peripheral Components Interconnect (PCI) local bus compatible bus architecture, or standards published by the IEEE (Institute of Electrical and Electronics Engineers), including the IEEE 488 General Purpose Interface Bus (GPIB), IEEE 696 / S-100, etc.

[0227] The computing device (500) preferably includes main memory (515) and may also include secondary memory (520). The main memory (515) provides storage of instructions and data for programs executed on a processor (510), such as one or more of the functions and / or modules discussed above. Computer-readable program instructions stored in memory and executed by the processor (510) may be assembler instructions, ISA (Instruction Set Architecture) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, configuration data for an integrated circuit, or source code or object code written and / or compiled in a combination of Smalltalk, C / C++, Java, JavaScript, Perl, Visual Basic, NET, and other programming languages. The main memory (515) is generally a semiconductor-based memory such as dynamic random access memory (DRAM) and / or static random access memory (SRAM). Other semiconductor-based memory types include, for example, synchronous dynamic random access memory (SDRAM), Rambus dynamic random access memory (RDRAM), and ferroelectric random access memory (FRAM), and also include read-only memory (ROM).

[0228] Computer-readable program instructions can be executed wholly on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or connected to an external computer (e.g., via the Internet using an Internet Service Provider (ISP)).

[0229] The auxiliary memory (520) may optionally include internal memory (525) and / or removable media (530). The removable media (530) is read and / or written in any known manner. The removable storage media (530) may be, for example, a magnetic tape drive, a CD (compact disc) drive, a DVD (Digital Versatile Disc) drive, other optical drives, a flash memory drive, etc.

[0230] A removable storage medium (530) is a non-transient computer-readable medium in which computer-executable code (i.e., software) and / or data is stored. Computer software or data stored on the removable storage medium (530) is read by a computing device (500) for execution by a processor (510).

[0231] The auxiliary memory (520) may include other similar elements for enabling computer programs or other data or instructions to be loaded into the computing device (500). Such means may include, for example, an external storage medium (545) and a communication interface (540), which enables software and data to be transferred from the external storage medium (545) to the computing device (500). Examples of the external storage medium (545) may include an external hard disk drive, an external optical drive, an external optical-magnetic drive, etc. Other examples of the auxiliary memory (520) may include semiconductor-based memory such as programmable read-only memory (PROM), eraseable programmable read-only memory (EPROM), electrically eraseable read-only memory (EEPROM), or flash memory (block-oriented memory similar to EEPROM).

[0232] As described above, the computing device (500) may include a communication interface (540). The communication interface (540) enables software and data to be transmitted between the computing device (500) and an external device (e.g., a printer), a network, or other information source. For example, computer software or executable code may be transmitted from a network server to the computing device (500) through the communication interface (540). Examples of the communication interface (540) include an embedded network adapter, a network interface card (NIC), a PCMCIA (Personal Computer Memory Card International Association) network card, a card bus network adapter, a wireless network adapter, a USB (Universal Serial Bus) network adapter, a modem, a network interface card (NIC), a wireless data card, a communication port, an infrared interface, an IEEE 1394 FireWire, or any other device capable of interfacing the system (550) with a network or other computing device. The communication interface (540) is preferably implemented with industry-published protocol standards such as Ethernet IEEE 802 standard, Fibre Channel, Digital Subscriber Line (DSL), Asynchronous Digital Subscriber Line (ADSL), Frame Relay, Asynchronous Transfer Mode (ATM), Integrated Digital Service Network (ISDN), Personal Communication Service (PCS), Transmission Control Protocol / Internet Protocol (TCP / IP), Serial Line Internet Protocol / Point-to-Point Protocol (SLIP / PPP), but may also be implemented with a custom or non-standard interface protocol.

[0233] Software and data transmitted through the communication interface (540) are generally in the form of telecommunication signals (555). These signals (555) may be provided to the communication interface (540) through a communication channel (550). In one embodiment, the communication channel (550) may be a wired or wireless network, or any various other communication link. The communication channel (550) may be implemented using various wired or wireless communication means, including wired or cable, optical fiber, conventional telephone line, mobile phone link, wireless data communication link, radio frequency ("RF") link, or infrared link, etc., to transmit the signals (555).

[0234] Computer executable code (i.e., computer program or software) is stored in main memory (515) and / or auxiliary memory (520). Computer programs may also be received via a communication interface (540) and stored in main memory (515) and / or auxiliary memory (520). When such computer programs are executed, they enable the computing device (500) to perform various functions of the disclosed embodiments as described elsewhere in this specification.

[0235] In this document, the term “computer-readable medium” is used to refer to any non-transient computer-readable storage medium used to provide computer-executable code (e.g., software and computer programs) to a computing device (500). Examples of such media include main memory (515), auxiliary memory (520) (including internal memory (525), removable media (530), and external storage media (545)), and any peripheral device coupled to communicate with a communication interface (540) (including a network information server or other network device). Such non-transient computer-readable media are means for providing executable code, programming instructions, and software to the computing device (500). In an embodiment implemented using software, the software may be stored on a computer-readable medium and loaded into the computing device (500) via a removable medium (530), an I / O interface (535), or a communication interface (540). In such an embodiment, the software is loaded into the computing device (500) in the form of an electrical communication signal (555). When the software is executed by the processor (510), it preferably causes the processor (510) to perform the features and functions described elsewhere in this specification.

[0236] The I / O interface (535) provides an interface between one or more components of the computing device (500) and one or more input and / or output devices. Exemplary input devices include, but are not limited to, a keyboard, a touch screen or other touch sensing device, a biometric device, a computer mouse, a trackball, a pen-based pointing device, etc. Examples of output devices include, but are not limited to, a cathode ray tube (CRT), a plasma display, a light-emitting diode (LED) display, a liquid crystal display (LCD), a printer, a vacuum fluorescent display (VFD), a surface conduction electron emitter display (SED), a field emission display (FED), etc.

[0237] The computing device (500) also includes optional wireless communication components that facilitate wireless communication over a voice network and / or a data network. The wireless communication components include an antenna system (570), a wireless system (565), and a baseband system (560). In the computing device (500), radio frequency (RF) signals are transmitted and received wirelessly by the antenna system (570) under the management of the wireless system (565).

[0238] The antenna system (570) may include one or more antennas and one or more multiplexers (not shown) that perform switching functions to provide transmission and reception signal paths to the antenna system (570). In the reception path, the received RF signal may be combined with a low-noise amplifier (not shown) that amplifies the received RF signal from the multiplexer and transmits the amplified signal to the wireless system (565).

[0239] The wireless system (565) may include one or more wireless devices configured to communicate via various frequencies. In one embodiment, the wireless system (565) may combine a demodulator (not shown) and a modulator (not shown) in a single integrated circuit (IC). The demodulator and the modulator may be separate components. In the receiving path, the demodulator removes the RF carrier signal while leaving the baseband received audio signal transmitted from the wireless system (565) to the baseband system (560).

[0240] If the received signal contains audio information, the baseband system (560) decodes the signal and converts it into an analog signal. The signal is then amplified and transmitted to a speaker. The baseband system (560) also receives an analog audio signal from a microphone. This analog audio signal is converted into a digital signal and encoded by the baseband system (560). The baseband system (560) also codes the digital signal for transmission and generates a baseband transmission audio signal that is routed to the modulator portion of the wireless system (565). The modulator mixes the baseband transmission audio signal with an RF carrier signal to generate an RF transmission signal that can be routed to the antenna system (570) and passed through a power amplifier (not shown). The power amplifier amplifies the RF transmission signal and routes it to the antenna system (570), where the signal is switched to an antenna port for transmission. The baseband system (560) is also telecommunicationally coupled with a processor (510), which may be a central processing unit (CPU). The processor (510) has access to a data storage area (515 or 520). The processor (510) is preferably configured to execute instructions (i.e., computer programs or software) that may be stored in main memory (515) or auxiliary memory (520). Computer programs may also be received from a baseband processor (560) and stored in main memory (510) or auxiliary memory (520), or executed upon receipt. Such computer programs enable the computing device (500) to perform various functions of the disclosed embodiment when executed. For example, the data storage area (515 or 520) may include various software modules.

[0241] The computing device further includes a display (575) directly attached to a communication bus (505) that may be provided or added instead of any display connected to the I / O interface (535) mentioned above.

[0242] Various embodiments may also be implemented primarily in hardware using components such as, for example, Application-Specific Integrated Circuits (ASICs), Programmable Logic Arrays (PLAs), or Field Programmable Gate Arrays (FPGAs). Implementations of hardware state machines capable of performing the functions described herein will also be apparent to those skilled in the art. Various embodiments may also be implemented using a combination of hardware and software.

[0243] Furthermore, those skilled in the art will recognize that the various exemplary logic blocks, modules, circuits, and method steps described in connection with the drawings and embodiments disclosed herein may often be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate this interoperability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have generally been described in terms of function. Whether these functions are implemented in hardware or software depends on the specific application and design constraints imposed on the overall system. Those skilled in the art may implement the functions described in various ways for each specific application, but such implementation decisions should not be interpreted as being outside the scope of the invention. Additionally, groups of functions within a module, block, circuit, or step are intended for ease of explanation. Specific functions or steps may be moved from one module, block, or circuit to another without departing from the invention.

[0244] Furthermore, the various exemplary logic blocks, modules, functions, and methods described in connection with the embodiments disclosed herein may be implemented or executed by a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic device, individual gate or transistor logic, individual hardware components, or a combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in other examples, the processor may be any processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors associated with a DSP core, or any other such configuration.

[0245] Additionally, steps of the method or algorithm described in connection with the embodiments disclosed herein may be directly implemented in hardware, software modules executed by a processor, or a combination of both. Software modules may exist in other forms of storage media, including RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or network storage media. Exemplary storage media may be coupled to a processor so that the processor can read information from the storage media and write information to the storage media. As another example, the storage media may be integrated into the processor. The processor and the storage media may also exist in an ASIC.

[0246] The computer-readable storage media mentioned in this specification should not be interpreted as being limited to radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses passing through fiber optic cables), or transient signals such as electrical signals transmitted through wires.

[0247] All software components described herein may take various forms. For example, a component may be a standalone software package or a software package integrated as a "tool" in a larger software product. It may be downloaded from a network, such as as an add-on package for installation in a website, a standalone product, or an existing software application. It may also be used as a client-server software application, a web-enabled software application, and / or a mobile application.

[0248] Embodiments of the present invention are described herein with reference to flowchart examples and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block of the flowchart examples and / or block diagrams, and combinations of blocks of the flowchart examples and / or block diagrams, may be implemented by computer-readable program instructions.

[0249] Computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to generate a machine such as an instruction that is executed through the processor of the computer or other programmable data processing device, and can generate means for implementing a specified function / operation in blocks or blocks of a flowchart and / or block diagram.

[0250] Computer-readable program instructions may also be loaded into a computer, other programmable data processing device, or other device to perform a series of work steps performed on the computer, other programmable device, or other device, and instructions executed on the computer, other programmable device, or other device may implement functions / actions specified in flowchart and / or block diagram blocks or blocks.

[0251] The illustrated flowcharts and block diagrams illustrate the architecture, function, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block of a flowchart or block diagram may represent a module, segment, or part of an instruction, which includes one or more executable instructions for implementing a specified logical function(s). In some alternative implementations, the functions mentioned in the blocks may occur differently from the order mentioned in the drawings. For example, two blocks depicted consecutively may actually be executed substantially simultaneously, or blocks may be executed in reverse order depending on the related functions. Additionally, each block of the block diagram and / or flowchart, and combinations of blocks of the block diagram and / or flowchart, may be implemented by a special-purpose hardware-based system that performs a specified function or operation, or by a combination of special-purpose hardware and computer instructions. Devices and methods implementing the present invention may be hosted in a cloud computing environment and delivered thereby. Cloud computing is a service delivery model that enables convenient on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services), which can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0252] The characteristics are as follows.

[0253] On-demand self-service: Cloud consumers can unilaterally provision computing functions, such as server time and network storage, as needed without human interaction with service providers.

[0254] Extensive network access: This feature is provided over a network and accessed through standard mechanisms that facilitate the use of heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0255] Resource Pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, and various physical and virtual resources are dynamically allocated and reallocated based on demand. While consumers generally lack control or knowledge regarding the exact location of the provided resources, there is location independence in that locations can be specified at a higher level of conceptualization (e.g., country, state, or data center).

[0256] Rapid Elasticity: In some cases, features can be automatically provisioned quickly and elastically to scale out rapidly and release rapidly to scale in. To consumers, the features available for provisioning appear unlimited, and they can be purchased at any time regardless of quantity.

[0257] Measured Services: Cloud systems automatically control and optimize resource usage by leveraging measurement capabilities at a conceptual level appropriate for service types (e.g., storage, processing, bandwidth, and active user accounts). Since resource usage can be monitored, controlled, and reported, it can provide transparency to both providers and consumers of the services utilized.

[0258] The service model is as follows.

[0259] SaaS (Software as a Service): The functionality provided to consumers is the use of a provider's applications running on cloud infrastructure. Applications can be accessed from various client devices through thin client interfaces, such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or individual application functions, except for limited user-specific application configuration settings.

[0260] PaaS (Platform as a Service): The function provided to consumers is the deployment of applications created or acquired by the consumer, generated using programming languages ​​and tools supported by the provider, onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they control the deployed applications and, where possible, the configuration of the application hosting environment.

[0261] IaaS (Infrastructure as a Service): The function provided to consumers is the provisioning of processing, storage, network, and other underlying computing resources that enable consumers to deploy and run any software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they control the operating system, storage, and deployed applications, and can have limited control over some networking components (e.g., host firewalls).

[0262] The deployment model is as follows.

[0263] Private Cloud: Cloud infrastructure operates exclusively for an organization. It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0264] Community Cloud: Cloud infrastructure is shared across multiple organizations and supports specific communities that share common goals (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by organizations or third parties and can exist on-premises or off-premises.

[0265] Public Cloud: Cloud infrastructure is available for use by the general public or large industry groups and is owned by the organization that sells cloud services.

[0266] Hybrid Cloud: A configuration of two or more clouds (private, community, or public) combined by standardized or proprietary technologies (e.g., cloud bursting for load balancing between clouds) that enable data and application portability, while the cloud infrastructure remains a distinct entity.

[0267] Cloud computing environments are service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. The core of cloud computing is an infrastructure comprising a network of interconnected nodes. It will be apparent to those skilled in the art that many improvements and modifications can be made to the exemplary embodiments described above without departing from the scope of the invention.

[0268] FIG. 11a is a block diagram showing an exemplary processorable device (551) that may be used in connection with the various embodiments described herein. As understood by those skilled in the art, alternative forms of the device (551) may also be used. In an exemplary embodiment, the device (551) is provided as a digital imaging device (also referred to herein as a scanner system or scanning system) comprising one or more processors (556), one or more memories (566), one or more motion controllers (571), one or more interface systems (576), one or more movable stages (580) each supporting one or more glass slides (585) having one or more samples (590), one or more lighting systems (595) illuminating the samples, one or more objective lenses (600) each defining an optical path (605) moving along an optical axis, one or more objective lens positioners (630), one or more optional epi-illumination systems (635) (e.g., included in a fluorescence scanner system), one or more focusing optical devices (610), one or more line scan cameras (615) and / or one or more area scan cameras (620) each defining a separate angle of view (625) on the sample (590) and / or glass slide (585). Various elements of the scanner system (551) are communically combined through one or more communication buses (560). Although each of the various elements of the scanner system (551) may be one or more, for the sake of simplification of the following description, these elements will be described in the singular, except where it is necessary to describe them in the plural to convey appropriate information.

[0269] One or more processors (556) may include, for example, a central processing unit ("CPU") and a separate graphics processing unit ("GPU") capable of processing instructions in parallel, or one or more processors (556) may include a multi-core processor capable of processing instructions in parallel. Separate additional processors may be provided to control specific components or to perform specific functions such as image processing. For example, additional processors may include an auxiliary processor for managing data input, an auxiliary processor for performing floating-point mathematical operations, a special-purpose processor having an architecture suitable for the rapid execution of signal processing algorithms (e.g., a digital signal processor), a slave processor dependent on the main processor (e.g., a backend processor), and an additional processor for controlling a line scan camera (615), a stage (580), an objective lens (225), and / or a display (not shown). These additional processors may be separate individual processors or may be integrated with the processor (556).

[0270] Memory (566) provides storage of instructions and data for a program that can be executed by the processor (556). Memory (566) may include one or more volatile and permanent computer-readable storage media for storing data and instructions, such as random access memory, read-only memory, hard disk drives, removable storage drives, etc. The processor (556) is configured to communicate with various elements of the scanner system (551) via a communication bus (560) to execute instructions stored in memory (566) and to perform the full functions of the scanner system (551).

[0271] One or more communication buses (560) may include a communication bus (560) configured to transmit analog electrical signals and a communication bus (560) configured to transmit digital data. Accordingly, communication from the processor (556), motion controller (571), and / or interface system (576) through one or more communication buses (560) may include both electrical signals and digital data. The processor (556), motion controller (571), and / or interface system (576) may also be configured to communicate with one or more of the various elements of the scanning system (551) via a wireless communication link.

[0272] The motion control system (571) is configured to precisely control and adjust the XYZ movement of the stage (580) and the objective lens (600) (e.g., via the objective lens positioner (630)). The motion control system (571) is also configured to control the movement of any other moving part in the scanner system (551). For example, in a fluorescent scanner embodiment, the motion control system (571) is configured to adjust the movement of an optical filter, etc., in the epi-illumination system (635).

[0273] The interface system (576) enables the scanner system (551) to interface with other systems and human operators. For example, the interface system (576) may include a user interface for providing information directly to the operator and / or allowing direct input from the operator. The interface system (576) is also configured to facilitate communication and data transmission between the scanning system (551) and one or more external devices directly connected (e.g., a printer, a removable storage medium) or external devices such as an image server system, an operator station, a user station, and a management server system connected to the scanner system (551) via a network (not shown).

[0274] The lighting system (595) is configured to illuminate a portion of the sample (590). The lighting system may include, for example, a light source and an illumination optical system. The light source may be a variable-intensity halogen light source with a concave reflective mirror to maximize light output and a KG-1 filter to suppress heat. The light source may also be any type of arc lamp, laser, or other light source. In one embodiment, the lighting system (595) illuminates the sample (590) in a transmission mode so that a line scan camera (615) and / or an area scan camera (620) detect optical energy transmitted through the sample (590). Alternatively, or in combination, the lighting system (595) may also be configured to illuminate the sample (590) in a reflection mode so that a line scan camera (615) and / or an area scan camera (620) detect optical energy reflected from the sample (590). Overall, the lighting system (595) is configured to be suitable for irradiation of the microscope sample (590) in any known optical microscope mode.

[0275] In one embodiment, the scanner system (551) optionally includes an epi-illumination system (635) to optimize the scanner system (551) for fluorescence scanning. Fluorescence scanning is the scanning of a sample (590) containing a fluorescent molecule, which is a photon-sensitive molecule capable of absorbing light at a specific wavelength (excitation). This photon-sensitive molecule also emits light at a higher wavelength (emission). Because the efficiency of this photoluminescence phenomenon is very low, the amount of emitted light is often very low. This small amount of emitted light generally interferes with conventional techniques (e.g., transmission mode microscopes) for scanning and digitizing the sample (590). Advantageously, in an embodiment of the optional fluorescence scanner system of the scanner system (551), the use of a line scan camera (615) comprising a multi-linear sensor array (e.g., a time delay integrated ("TDI") line scan camera) increases the light sensitivity of the line scan camera by exposing the same area of ​​the sample (590) to each of the multi-linear sensor arrays of the line scan camera (615). This is particularly useful when scanning faint fluorescence samples with low emission light.

[0276] Accordingly, in the embodiment of the fluorescence scanner system, the line scan camera (615) is preferably a monochrome TDI line scan camera. Advantageously, the monochrome image is ideal for fluorescence microscopy because it more accurately represents the actual signals of the various channels present in the sample. As understood by those skilled in the art, the fluorescence sample (590) may be labeled with multiple fluorescent dyes that emit light at different wavelengths, also referred to as "channels."

[0277] Additionally, since the low-end and high-end signal levels of various fluorescent samples represent a wide spectrum of wavelengths that the line scan camera (615) can detect, it is desirable that the low-end and high-end signal levels that the line scan camera (615) can detect are similarly wide. Accordingly, in the fluorescent scanner embodiment, the line scan camera (615) used in the fluorescent scanning system (551) is a monochrome 10-bit 64 linear array TDI line scan camera. It should be noted that various bit depths for the line scan camera (615) may be used for use with the fluorescent scanner embodiment of the scanning system (551).

[0278] The movable stage (580) is configured for precise XY movement under the control of the processor (556) or the motion controller (571). The movable stage may also be configured for movement in Z under the control of the processor (556) or the motion controller (571). The movable stage is configured to place a sample at a desired location during image data capture by the line scan camera (615) and / or the area scan camera. The movable stage is also configured to accelerate the sample (590) in the scanning direction to a substantially constant speed and then maintain a substantially constant speed during image data capture by the line scan camera (615). In one embodiment, the scanner system (551) may use a high-precision and tightly coordinated XY grid to assist in positioning the sample (590) on the movable stage (580). In one embodiment, the movable stage (580) is a linear motor-based XY stage having a high-precision encoder used for both the X and Y axes. For example, a very precise nanometer encoder can be used on the axis of the scanning direction, the direction perpendicular to the scanning direction, and the axis in the same plane as the scanning direction. The stage is also configured to support a glass slide (585) on which the sample (590) is placed.

[0279] The sample (590) can be anything that can be examined with an optical microscope. For example, glass microscope slides (585) are frequently used as observation substrates for specimens containing tissues and cells, chromosomes, DNA, proteins, blood, bone marrow, urine, bacteria, beads, biopsy material, or other types of biological material, or materials that are dead or living, stained or unstained, or labeled or unlabeled. The sample (590) can also be an array of any type of DNA or DNA-related material, such as cDNA, RNA, or protein, deposited on any type of slide or other substrate, including any and all samples, generally known as a microarray. The sample (590) can be a microtitier plate, for example, a 96-well plate. Other examples of the sample (590) include integrated circuit boards, electrophoretic recordings, Petri dishes, films, semiconductor materials, forensic materials, or mechanical parts.

[0280] In one embodiment, the objective lens (600) is mounted on an objective positioner (630) capable of using a very precise linear motor to move the objective lens (600) along the optical axis defined by the objective lens (600). For example, the linear motor of the objective lens positioner (630) may include a 50 nanometer encoder. The relative positions of the stage (580) and the objective lens (600) in the XYZ axes are adjusted and controlled in a closed-loop manner using a motion controller (571) under the control of a processor (556) that uses memory (566) to store information and commands, including computer-executable program steps for operating the entire scanning system (551).

[0281] In one embodiment, the objective lens (600) is a planar achromatic ("APO") infinite correction objective lens having a numerical aperture corresponding to the highest desired spatial resolution, wherein the objective lens (600) is suitable for a transmission mode illumination microscope, a reflection mode illumination microscope, and / or an epi-illumination mode fluorescence microscope (e.g., Olympus 40×, 0.75 NA or 20×, 0.75 NA). Advantageously, the objective lens (600) can correct chromatic aberration and spherical aberration. Since the objective lens (600) is corrected to infinity, a focusing optical device (610) may be placed in an optical path (605) on the objective lens (600) such that the light beam passing through the objective lens becomes a collimated light beam. The focusing optical device (610) focuses the optical signal captured by the objective lens (600) onto the light-responding element of the line scan camera (615) and / or area scan camera (620) and may include optical components such as filters, magnification changing lenses, etc. The objective lens (600) combined with the focusing optical device (610) provides total magnification for the scanning system (551). In one embodiment, the focusing optical device (610) may include a tube lens and an optional 2× magnification changing device. Advantageously, the 2× magnification changing device enables the native 20X objective lens (600) to scan the sample (590) at a 40× magnification.

[0282] A line scan camera (615) comprises at least one linear array of image elements ("pixels"). The line scan camera may be monochrome or color. While color line scan cameras generally have three or more linear arrays, monochrome line scan cameras may have a single linear array or multiple linear arrays. Any type of single or multiple linear arrays packaged as part of the camera or custom-integrated into an imaging electronic module may also be used. For example, a color line scan camera with three linear arrays ("Red-Green-Blue" or "RGB") or a monochrome TDI with 96 linear arrays may be used. TDI line scan cameras generally sum intensity data from previous image regions of a sample to provide a substantially better signal-to-noise ratio ("SNR") in the output signal, increasing the SNR in proportion to the square root of the number of integrated stages. TDI line scan cameras are composed of multiple linear arrays. For example, TDI line scan cameras may be used with 24, 32, 48, 64, 96, or more linear arrays. The scanner system (551) also supports linear arrays manufactured in various formats, some of which are 512 pixels, some of which are 1024 pixels, and others of which are 4096 pixels. Similarly, linear arrays with various pixel sizes can also be used in the scanner system (551). The most important requirement for selecting any type of line scan camera (615) is that the movement of the stage (580) can be synchronized with the line speed of the line scan camera (615) so that the stage (580) can move relative to the line scan camera (615) during digital image capture of the sample (590).

[0283] Image data generated by the line scan camera (615) is stored in a portion of memory (566) and processed by the processor (556) to generate a continuous digital image of at least a portion of the sample (590). The continuous digital image can be further processed by the processor (556), and the modified continuous digital image can also be stored in memory (566).

[0284] In an embodiment having two or more line scan cameras (615), at least one of the line scan cameras (615) may be configured to function as a focusing sensor in combination with at least one of the line scan cameras (615) configured to function as an image sensor. The focusing sensor may be logically positioned on the same optical axis as the imaging sensor, or the focusing sensor may be logically positioned before and after the imaging sensor with respect to the scanning direction of the scanner system (551). In this embodiment having at least one line scan camera (615) functioning as a focusing sensor, image data generated by the focusing sensor is stored in a portion of memory (566) and processed by one or more processors (556) to generate focus information so that the scanner system (551) adjusts the relative distance between the sample (590) and the objective lens (600) to maintain focus on the sample during scanning. Additionally, in one embodiment, at least one line scan camera (615) functioning as a focusing sensor may be oriented such that each of a plurality of individual pixels of the focusing sensor is located at a different logical height along the optical path (605).

[0285] When in operation, various components of the scanner system (551) and programmed modules stored in memory (566) enable automatic scanning and digitization of a sample (590) placed on a glass slide (585). The glass slide (585) is securely placed on a movable stage (580) of the scanner system (551) to scan the sample (590). Under the control of the processor (556), the movable stage (580) accelerates the sample (590) to a substantially constant speed for detection by the line scan camera (615), wherein the speed of the stage is synchronized with the line speed of the line scan camera (615). After scanning a stripe of image data, the movable stage (580) decelerates and brings the sample (590) to a substantially complete stop. The movable stage (580) then moves perpendicular to the scanning direction to position the sample (590) to scan a subsequent stripe of image data, for example, an adjacent stripe. Additional stripes are scanned continuously until the entire portion of the sample (590) or the entire sample (590) is scanned.

[0286] For example, while digitally scanning the sample (590), a continuous digital image of the sample (590) is acquired in a plurality of consecutive viewing angles that are combined together to form an image strip. A plurality of adjacent image strips are similarly combined together to form a continuous digital image of part or all of the sample (590). Scanning of the sample (590) may involve acquiring a vertical image strip or a horizontal image strip. Scanning of the sample (590) may be from top to bottom, from bottom to top, or both (bidirectional) and may start at any point on the sample. Alternatively, scanning of the sample (590) may be from left to right, right to left, or both (bidirectional) and may start at any point on the sample. Additionally, it is not necessary to acquire image strips in an adjacent or continuous manner. Furthermore, the resulting image of the sample (590) may be an image of the entire sample (590) or a part of the sample (590).

[0287] In one embodiment, computer-executable instructions (e.g., programmed modules and software) are stored in memory (566) and, when executed, enable the scanning system (551) to perform the various functions described herein. In this description, the term “computer-readable storage medium” is used to refer to any medium used to store and provide computer-executable instructions to the scanning system (551) for execution by the processor (556). Examples of such media include memory (566) and any removable or external storage medium (not shown) that is communicationally coupled to the scanning system (551) directly or indirectly, for example, through a network (not shown).

[0288] FIG. 11b illustrates a line scan camera having a single linear array (640) that can be implemented as a charge coupling device ("CCD") array. The single linear array (640) includes a plurality of individual pixels (645). In the illustrated embodiment, the single linear array (640) has 4096 pixels. In alternative embodiments, the linear array (640) may have more or fewer pixels. For example, common forms of the linear array include 512, 1024, and 4096 pixels. The pixels (645) are arranged linearly to define the field of view (625) for the linear array (640). The size of the field of view depends on the magnification of the scanner system (551).

[0289] FIG. 11c illustrates a line scan camera having three linear arrays, each of which can be implemented as a CCD array. The three linear arrays are combined to form a color array (650). In one embodiment, each individual linear array of the color array (650) detects different color intensities, such as red, green, or blue. Color image data from each individual linear array of the color array (650) are combined to form a single field of view (625) of color image data.

[0290] FIG. 11d illustrates a line scan camera having multiple linear arrays, each of which can be implemented as a CCD array. The multiple linear arrays are combined to form a TDI array (655). Advantageously, the TDI line scan camera can increase the SNR (also called the integration step) by summing the intensity data of previously imaged sample areas to provide a substantially better SNR in the output signal, which is proportional to the square root of the number of linear arrays. The TDI line scan camera may include a larger number of linear arrays, for example, common forms of TDI line scan cameras include 24, 32, 48, 64, 96, 120, and many more linear arrays.

[0291] The above description of the disclosed embodiments is provided to enable those skilled in the art to manufacture or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles described herein may be applied to other embodiments without departing from the spirit or scope of the present invention. Accordingly, the description and drawings presented herein represent currently preferred embodiments of the present invention and should be understood as representing the subject matter broadly considered by the present invention. Furthermore, it is understood that the scope of the present invention fully encompasses other embodiments that may be apparent to those skilled in the art, and that the scope of the present invention is not limited accordingly.

Claims

Claim 1 A computer device for identifying tumors in histological images comprises: a memory configured to store computer-executable instructions; and a hardware processor communicating with said memory, wherein when said computer-executable instructions are executed by said processor, said processor generates, from an output image, said segmentation mask having regions occupied by individual tumors in said output image marked on said segmentation mask—wherein said output image is generated by a convolutional neural network based on said histological image, said output image has one of a plurality of tissue classes assigned to each pixel of said output image that is mapped to said histological image and represents a classified tissue of said histological image; calculates statistics for each tumor marked on said segmentation mask; applies a filter to said statistics of said tumor to edit said segmentation mask—wherein said application of the filter selects and deselects tumors according to said filter to edit said segmentation mask to remove unimportant tumors; and scores each tumor in said edited segmentation mask according to said scoring algorithm to assign a score to each tumor. and a computer device configured to rank the tumors according to the above score. Claim 2 A computer device according to claim 1, wherein when the computer-executable instructions are executed by the processor, the processor is configured to receive a histological image comprising a two-dimensional pixel array; and to generate the output image using the convolutional neural network, wherein the convolutional neural network is trained based on a training data set comprising (a) a training histological image and (b) annotations assigning one of a plurality of tissue classes to pixels of the training histological image, the plurality of tissue classes comprising at least one class representing non-tumor tissue and at least one class representing tumor tissue. Claim 3 A computer device, wherein the ranked tumor in paragraph 1 is a tumor remaining after applying the filter. Claim 4 A computer device according to claim 1, wherein, when the computer-executable instructions are executed by the processor, the processor is additionally configured to score the ranking of the tumor. Claim 5 A computer device according to claim 1, wherein, when the computer-executable instructions are executed by the processor, the processor is further configured to generate a visualization of the histological image based on the edited segmentation mask. Claim 6 In paragraph 5, the visualization comprises a computer device including respective overview view windows in which the segmentation mask and the histological image are displayed adjacent to each other. Claim 7 In paragraph 6, the overview view window of the segmentation mask includes a ranking label for each tumor, a computer device. Claim 8 A computer device according to claim 5, wherein, when the computer-executable commands are executed by the processor, the processor is further configured to display an operable user interface tumor selection control that allows the user to interact with the visualization to select a tumor from the edited segmentation mask. Claim 9 In paragraph 5, the above visualization is a computer device comprising an enlarged view window that magnifies the currently selected tumor. Claim 10 In paragraph 5, the above visualization is a computer device comprising an overview view window in which the segmentation mask is overlaid on the histological image. Claim 11 In paragraph 6, the above overview view window is a computer device that includes a ranking label for each tumor. Claim 12 A computer device according to claim 1, wherein the histological image is a complex comprising a plurality of histological images obtained from adjacent sections of tissue regions that are stained differently from each other. Claim 13 In a non-transient computer-readable medium for identifying tumors in histological images, the hardware processor has program instructions to perform the following method, the following method comprises: generating a segmentation mask from an output image having regions occupied by individual tumors in the output image marked on the segmentation mask, wherein the output image is generated by a convolutional neural network based on the histological image, and the output image has one of a plurality of tissue classes assigned to each pixel of the output image that is mapped to the histological image and represents a classified tissue of the histological image; calculating statistics for each tumor marked on the segmentation mask; applying a filter to the statistics of the tumor to edit the segmentation mask, wherein the application of the filter selects and deselects tumors according to the filter to edit the segmentation mask to remove unimportant tumors; scoring each tumor of the edited segmentation mask according to a scoring algorithm to assign a score to each tumor; A non-transient computer-readable medium comprising the step of ranking the tumors according to the above score. Claim 14 In claim 13, the method comprises: receiving a histological image comprising a two-dimensional pixel array; and generating the output image using the convolutional neural network, wherein the convolutional neural network is trained based on a training dataset comprising (a) a training histological image and (b) annotations assigning one of a plurality of tissue classes to pixels of the training histological image, and wherein the plurality of tissue classes include at least one class representing non-tumor tissue and at least one class representing tumor tissue, a non-transient computer-readable medium. Claim 15 In paragraph 13, the above method further comprises the step of scoring the ranking of the tumor, a non-transient computer-readable medium. Claim 16 In paragraph 13, the method further comprises the step of generating a visualization of the histological image based on the edited segmentation mask, wherein the visualization comprises an overview view window in which the segmentation mask is overlaid on the histological image, and the overview view window comprises a ranking label for each tumor, in a non-transient computer-readable medium. Claim 17 In paragraph 16, the visualization comprises a non-transient computer-readable medium including an overview view window in which the segmentation mask and the histological image are displayed adjacent to each other. Claim 18 A method for identifying tumors in a histological image, the method comprises: generating a segmentation mask from an output image having regions occupied by individual tumors in the output image marked on the segmentation mask, wherein the output image is generated by a convolutional neural network based on the histological image, and the output image has one of a plurality of tissue classes assigned to each pixel of the output image that is mapped to the histological image and represents a classified tissue of the histological image; calculating statistics for each tumor marked on the segmentation mask; applying a filter to the statistics of the tumor to edit the segmentation mask, wherein the application of the filter selects and deselects tumors according to the filter to edit the segmentation mask to remove unimportant tumors; scoring each tumor of the edited segmentation mask according to a scoring algorithm to assign a score to each tumor; and ranking the tumors according to the score. Claim 19 In claim 18, the method comprises: receiving a histological image comprising a two-dimensional pixel array; generating the output image using the convolutional neural network, wherein the convolutional neural network is trained based on a training dataset comprising (a) a training histological image and (b) annotations assigning one of a plurality of tissue classes to pixels of the training histological image, wherein the plurality of tissue classes include at least one class representing non-tumor tissue and at least one class representing tumor tissue; and scoring the rank of the tumor. Claim 20 In claim 18, the method further comprises the step of generating a visualization of the histological image based on the edited segmentation mask, wherein the visualization comprises an overview view window in which the segmentation mask is overlaid on the histological image. Claim 21 In claim 18, the method further comprises the step of generating a visualization of the histological image based on the edited segmentation mask, wherein the visualization comprises an overview view window in which the segmentation mask and the histological image are displayed adjacent to each other. Claim 22 delete Claim 23 delete Claim 24 delete Claim 25 delete Claim 26 delete Claim 27 delete Claim 28 delete Claim 29 delete Claim 30 delete Claim 31 delete Claim 32 delete Claim 33 delete Claim 34 delete Claim 35 delete Claim 36 delete Claim 37 delete Claim 38 delete Claim 39 delete Claim 40 delete Claim 41 delete Claim 42 delete Claim 43 delete Claim 44 delete Claim 45 delete Claim 46 delete Claim 47 delete Claim 48 delete Claim 49 delete Claim 50 delete Claim 51 delete Claim 52 delete Claim 53 delete Claim 54 delete Claim 55 delete Claim 56 delete Claim 57 delete Claim 58 delete Claim 59 delete Claim 60 delete Claim 61 delete Claim 62 delete Claim 63 delete Claim 64 delete Claim 65 delete Claim 66 delete Claim 67 delete Claim 68 delete Claim 69 delete Claim 70 delete Claim 71 delete

Images