Focus-weighted machine learning classifier error prediction for microscope slide images

By training an OOF classifier to calculate the loss of focus of microscope slice images and generate focus-weighted classifier error prediction, the problem of loss of focus in digital microscope slice images is solved, the diagnostic accuracy is improved and the consistent focus quality evaluation is provided, suitable for Different scanning devices.

CN112106107BActive Publication Date: 2024-11-01GOOGLE LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201880093264.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-05-07
Filing Date
2018-12-17
Publication Date
2024-11-01
Estimated Expiration
2038-12-17

AI Technical Summary

Technical Problem

The out-of-focus of digital microscope section images leads to a decrease in diagnostic accuracy. The existing automatic focusing algorithms are difficult to distinguish between tissue blur and out-of-focus blur, and the focus evaluation methods of different scanners are inconsistent, which affects the spatial distribution and clinical correlation of image quality.

Method used

The loss of focus of each tile is calculated by training a convolutional neural network (OOF classifier), and a focus-weighted classifier error prediction is generated based on the expected classifier error statistics. Deep convolutional neural network is used to identify disease or tumor cells , combined with data enhancement steps such as data compression, noise artifacts, translational jitter and brightness perturbation, a synthetic out-of-focus image is generated.

Benefits of technology

The focus quality evaluation and weighting of digital microscope slice images is achieved, the accuracy of machine learning diagnosis is improved, and the focus quality measurement is provided consistent and universally applicable. It is suitable for different scanning devices and enhances support for clinical diagnosis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112106107B_ABST
    Figure CN112106107B_ABST
Patent Text Reader

Abstract

A method for generating a prediction of classification error for a magnified digital microscope slide image, such as an image of a tissue sample, is described. The image includes a plurality of tiles or mosaics of pixel image data. A degree of defocus for each tile is calculated using a machine learning defocus classifier. Data representing expected classifier error statistics for the machine learning classifier for a plurality of degrees of defocus are retrieved. Based on the calculated degree of defocus for each tile, a mapping of the expected classifier error statistics to each tile of the digital microscope slide image is calculated, thereby generating a classifier error prediction for each tile. The classifier error predictions so generated are aggregated over all tiles.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] The present disclosure relates to a method for generating predictions of classification errors generated by a machine learning classifier for microscope slide images. The predicted errors are weighted by the degree to which portions of the image are out-of-focus ("OOF").

[0002] In the medical field of histopathology, microscopic images of human tissue samples (which are prepared on glass slides) are used to make cancer diagnoses. In classical histopathology, tissue samples are diagnosed visually by experts using a microscope. In contrast, in the newer subfield of digital pathology, high-resolution digital images of the sample are first acquired by a full-slice scanner, and then diagnosed on a computer screen in a subsequent step. Alternatively, the identification of cancerous cells in tissue images can be assisted by a machine learning algorithm (usually embodied as a deep convolutional neural network), which is trained to find cancer cells in magnified tissue images. Such an algorithm can generate a so-called "heat map" image in which areas of the slice are shown in a contrasting color (e.g., red) to indicate areas predicted to contain cancer cells.

[0003] Tissue images from whole-slide scanners are typically gigapixel in size (e.g., 100,000 x 100,000 pixels at 40X magnification). However, one of the main technical issues is that areas of the digitized image can often be blurred and out of focus, making the corresponding image area unusable for accurate diagnosis by human pathologists and machine learning algorithms. Achieving accurate focus is particularly challenging for whole-slide scanners because (1) the depth of field is very shallow due to the high objective power used to digitize the image, and (2) tissues are typically not flat and not in the same focal plane.

[0004] The depth of field is inversely proportional to the magnification, so at high magnifications the depth of field is only extremely shallow. The depth of field (also called "focus range"), especially at high magnifications, is usually close to or even less than the thickness of the tissue to be captured. Moreover, tissue samples are usually not perfectly planar, but rather uneven, and their thickness often varies. Therefore, slide scanners usually use local autofocus methods when capturing images in smaller strips or tiles, which are then digitally stitched together to form a full slide image. The autofocus solutions used by different scanner manufacturers are not perfect, but may not be able to keep most of the tissue in focus in certain image areas, resulting in varying degrees of out-of-focus blur.

[0005] Therefore, the main challenge for the autofocus algorithm is to distinguish between (a) blur in the in-focus image area caused by tissue with a smooth appearance and (b) blur of any tissue pattern caused by varying degrees of defocus. A secondary challenge is to prevent focusing on foreign particles (such as dust or debris) atop the "cover slip" (the plastic or glass slide covering the tissue sample), which typically causes the tissue to be far out of focus.

[0006] References to the problem of quantifying the degree of out-of-focus in tissue images and related topics include: G. Campanella et al., Towards machine learned quality control: A benchmark for sharpness quantification in digital pathology, Computerized Medical Imaging and Graphics (2017) https: / / doi.org / 10.1016 / j.compmedimag, 2017.09.001; K. Kayser et al., How to measure image quality in tissue-based diagnosis (diagnostic surgical pathology), from the 9th European Telepathology Congress and the 3rd International Congress of Virtual Microscopy, Toledo, Spain, Diagnostic Pathology 2008 3 (Suppl 1); J. Liao et al., Rapid focus map surveying for whole-slide imaging with continues[sic]sample motion), arXiv:1707.03039[cs.CV], June 2017; S. Shakeri et al., Optical quality assessment of whole-slide imaging systems for digital pathology, Optics Express, Vol. 23, No. 2, pp. 1319-1336 (2015); X. Lopex et al., Automated Blur Detection Method for Histological Whole-slide Imaging, PLOS One, (Dec. 13, 2013), https: / / doi.org / 10.1371 / journal.pone.0082710; Samuel Yang et al., “Assessing microscope image focus quality with deeplearning,” BMC Bioinformatics (2018) 19:77; and M. Gurcan et al., Histopathological Image Analysis: A Review, IEEE Biomedical Engineering Reviews, 2009; 2:147-171. .

[0007] The inventors have recognized that the degree to which a slide is out of focus can affect the accuracy of machine-learned diagnosis or cancer cell identification, and that there is a need to quantify the error in a focus-weighted machine-learned disease classifier (i.e., the error specifically attributable to the degree to which portions of a microscope slide image are out of focus). The present disclosure addresses this need. Summary of the invention

[0008] In one aspect, a method for generating a prediction of classification error for a magnified digital microscope slide image of a sample is disclosed. The sample can take the form of a tissue sample (e.g., a prostate or lymph node sample), but other types of samples can be used. The method includes the following steps:

[0009] (a) scanning a microscope slide containing a sample with a slide scanner and generating a digital microscope slide image, wherein the digital microscope slide image includes a plurality of blocks of pixel image data;

[0010] (b) Calculate the defocus of each tile of the digital microscope slice image;

[0011] (c) retrieving data representing expected classifier error statistics of the machine learning classifier for multiple degrees of defocus;

[0012] (d) calculating a mapping of expected classifier error statistics to each tile of the digital microscope slide image based on the calculated defocus for each tile calculated in step (b) and the data retrieved in step (c), thereby generating a classifier error prediction for each tile; and

[0013] (e) The classifier error predictions generated in step (d) are summarized over all tiles.

[0014] In one configuration, the calculated out-of-focus degree of each tile is performed by an out-of-focus (OOF) classifier. The OOF classifier is trained from a synthetically blurred out-of-focus image. The out-of-focus image is generated by simulating one or more of (a) data compression and noise artifacts, (b) translational jitter, and (c) brightness perturbations, and (d) applying a computational bokeh filter. In one configuration, the OOF image is generated by simulating all of (a), (b), (c), and (d).

[0015] In another aspect, a pathology system is disclosed, the combination comprising:

[0016] a) a slide scanner suitable for generating digital slide images of microscope slides;

[0017] b) a memory storing:

[0018] 1) Parameters of a convolutional neural network (OOF classifier) ​​trained to calculate the degree of out-of-focus for each tile of a digital microscopy slide image generated by a slide scanner;

[0019] 2) data representing expected classifier error statistics of the machine learning classifier for multiple degrees of defocus; and

[0020] c) a computer configured to (1) calculate the defocus of each tile of the digital microscope slide image using a deep convolutional neural network, (2) calculate a mapping of expected classifier error statistics to each tile of the digital microscope slide image based on the calculated defocus of each tile, thereby generating a classifier error prediction for each tile; and (3) calculate a summary of the classifier error predictions over all tiles.

[0021] The OOF classifier may take the form of a shallow neural network, e.g., a neural network with three convolutional layers, or a deep neural network, e.g., a neural network with 10 or 18 convolutional layers. As described above, the OOF classifier is trained from the OOF image, and the OOF image may be generated by simulating one or more of: (a) data compression and noise artifacts, (b) translational jitter, and (c) brightness perturbations, and (d) applying a computational bokeh filter. In one configuration, the OOF image is generated by simulating all of (a), (b), (c), and (d).

[0022] In one configuration, the microscope slides contain prostate tissue samples. A machine learning classifier is trained to assign Gleason scores to various portions of the prostate tissue image. In another configuration, the tissue sample contains lymph node samples obtained from breast cancer patients. In this case, a machine learning classifier is trained to assign cancer / no cancer labels to various portions of the lymph node tissue image.

[0023] In one configuration, the memory and computer are local to the slide scanner. Alternatively, the memory and computer are remote from the slide scanner, such as in a computing resource on a local area network connected to the slide scanner on a service provider computing resource in the cloud.

[0024] In another aspect, a method for generating a prediction of classification error for a magnified digital microscope slide image of a sample is described. The digital microscope slide image includes a plurality of tiles of pixel image data. The method includes the following steps:

[0025] (a) Calculate the defocus of each tile of the digital microscope slice image;

[0026] (b) retrieving data representing expected classifier error statistics of the machine learning classifier for multiple degrees of defocus;

[0027] (c) calculating a mapping of expected classifier error statistics to each tile of the digital microscope slide image based on the calculated defocus for each tile calculated in step (a), thereby generating a classifier error prediction for each tile; and

[0028] (d) Summarize the classifier error predictions generated in step (c) over all tiles.

[0029] The calculation of the out-of-focus degree of each tile can be performed by a convolutional neural network (e.g., a shallow or deep CNN) that is trained to classify tiles of tissue images by their degree of out-of-focus. As explained above and in detail in the specific embodiments, the CNN (here, an OOF classifier) ​​is trained from synthetically blurred images.

[0030] In yet another aspect, a method for characterizing a classifier configured to generate a classification label for a digital microscopic slide of a tissue sample or a portion thereof is disclosed. The method comprises the following steps:

[0031] a) obtaining (e.g., receiving via an API call) a set of slice images ( Figure 2 , each slice image comprising a patch of pixel image data that is in focus and associated with a ground truth label for each image patch;

[0032] b) Define a set of defocus degrees, and for each degree:

[0033] 1) applying a corresponding amount of synthetic defocus to each tile of the image in the set of slices;

[0034] 2) Calculate the classification error for each patch in the image; and

[0035] 3) Calculate the error metric on all patches in the image;

[0036] c) storing the error measure calculated in step b)3) for all degrees defined in step b) as the expected error of the classifier for the defocus degrees defined in step b); and

[0037] d) Repeating steps b1), b2), b3) and c) for each slice image in the group.

[0038] The error metric calculated in step b) 2) may take the form of a mean error. It may also be calculated as a receiver operating characteristic (ROC) area under the curve (AUC) metric. This latter approach requires first calculating the ROC curve over all tiles and then calculating the area under it ("area under the curve"). It is therefore not a metric that is calculated per individual tile and then aggregated.

[0039] Therefore, in a variant, in step b) of the above method, a set of defocus degrees is defined, and for each degree, the method comprises performing steps 1)-4):

[0040] 1) applying a corresponding amount of synthetic defocus to each tile of the image in the set of slices;

[0041] 2) Compute classification predictions and probabilities.

[0042] 3) Compute the receiver operating characteristic (ROC) curve for the ground truth labels on all patches, and

[0043] 4) Calculate the area under the ROC curve (=AUC).

[0044] In one embodiment, the method may further include the steps of repeating steps b), c) and d) for the slice images in the group at different magnifications. This results in generating classifier error statistics for the slice images at different magnifications. In one embodiment, the method may further include the steps of repeating steps a), b), c) and d) at least once for each of a plurality of different slice scanners from different manufacturers. Different slice scanners from different manufacturers may produce images of different quality, so the classifier error statistics may vary from machine to machine. In this embodiment, classifier error statistics are obtained for each of a group of different slice scanners, so that the method of the present disclosure can be generally used for a variety of different machines from different manufacturers.

[0045] In one embodiment, a computational bokeh filter is used to obtain a synthetic defocus applied to the primary image. Other methods for artificially blurring the primary image may be used. Our experiments show that computational or digital bokeh (Bokeh) filtering is a preferred method. For example, the synthetic defocus may be generated by simulating the OOF method used in the development of the OOF classifier, as explained below (simulating JPG compression artifacts, brightness perturbations, and translational jitter).

[0046] In another aspect, an out-of-focus (OOF) classifier and training method are described. The OOF classifier is in the form of a convolutional neural network (CNN) and can be configured as a deep or shallow CNN. The method includes the step of synthetically blurring a set of training images. The synthetically blurred images are generated by simulating one or more of the following: (a) data compression (e.g., JPG compression) and noise artifacts, (b) translational jitter, (c) brightness perturbations, and (d) computational bokeh filtering; and the OOF classifier is trained with the synthetically blurred images to classify the synthetically blurred images by degree of out-of-focus. In one embodiment, all of (a), (b), (c), and (d) are used to generate the synthetically blurred images. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart illustrating the process of predicting focus-weighted disease classifier errors for digital microscope images.

[0048] Figure 2 is shown for generating and calculating Figure 1 Flowchart of a process for generating expected disease classifier error statistics for a method of . The flowchart is performed on a set of in-focus, ground truth annotated images (herein referred to as "master images") obtained from a specific full-slide scanner of a specific manufacturer. The images consist of a multitude of pixel image data patches. The process can be repeated several times to generate expected disease classifier error statistics for different manufacturers or different makes and models of full-slide scanners, as the expected disease classifier error statistics may differ between different makes and models of slide scanners.

[0049] Figure 3 is a graph of the average disease classifier error computed per tile for different degrees of defocus for the two primary images.

[0050] Figure 4 is a graphical representation of data representing expected disease classifier error statistics for different degrees of defocus, averaged over all reference images. Figure 4The data shown in the figure is stored in the memory and used according to Figure 1 and Figure 6 The process of computing the focused weighted disease classifier error for the new slice.

[0051] Figure 5 is a graph of the disease classifier error against the degree of defocus for all primary images.

[0052] Figure 6 is used Figure 4 The stored error statistics are based on Figure 2 Generated, based on Figure 1 A more detailed illustration of the process of predicting the focally weighted disease classifier error for new slices.

[0053] Figure 7 is based on Figure 6 Illustration of the expected disease classifier error computed for each image patch.

[0054] Figure 8 yes Figure 1 Illustration of an application of the method in a computing environment with a whole slide scanner and a networked pathology workstation.

[0055] Fig. 9 yes Figure 1 An illustration of a second application of the method in a computing environment with a whole slide scanner and a networked pathology workstation, wherein generation of a focused weighted disease classifier error occurs in a computing system in the remote network and the results are transmitted to the pathology workstation. DETAILED DESCRIPTION

[0056] Digital pathology is advancing into the clinical workflow, driven primarily by the potential of powerful image analysis tools using new machine learning techniques and made possible by the recent regulatory approval of the first whole slide image (WSI) scanner for initial diagnosis in the United States and the wider availability of cloud storage and large IT infrastructure for processing gigapixel image files. However, the digitization process introduces sources of artifacts into the imaging process, including color or contrast issues and out-of-focus areas. These artifacts, especially out-of-focus areas, can negatively impact the suitability of digital slide images for accurate diagnosis by pathologists or the accuracy of automated image analysis. Human pathologists often flag such images as low quality and order a rescan, which can cause long delays in cases. Worse, in automated image analysis, such image artifacts translate directly into detection and classification errors. For example, some studies have found that system false positives can be traced back to poor focus quality, such as out-of-focus germinal centers being misclassified as tumor metastases by the algorithm.

[0057] One option to mitigate these problems is to have a technician pre-screen all digital slides produced by the scanner. However, this type of manual quality control is impractical from a time and cost perspective and is not feasible even for thorough screening of small artifacts. Another option is to perform automated quality control and assessment of focus quality, which does not have any of the above limitations. Although each WSI scanner has a built-in focus assessment that can be used to automatically rescan affected areas or make quality reports, existing methods still have several shortcomings: (1) despite this built-in focus assessment, many slides scanned by WSI scanners still have out-of-focus areas, (2) focus assessment methods vary between scanners, hindering comparisons between devices, (3) focus metrics are often not exported to users with sufficient detail, such as the spatial distribution of focus quality, and (4) the assessment does not take into account the clinical relevance of focus quality. For example, diagnoses based on cellular features (e.g., detection of breast cancer metastases) often require even higher focus quality than diagnoses based primarily on larger tissue morphology (e.g., Gleason grading of prostate adenocarcinoma).

[0058] This paper introduces a focus quality assessment that addresses all these shortcomings - it provides a generally applicable metric that is highly consistent with human-assessed focus quality, can be applied to any WSI regardless of the scanning device, and provides spatial information of focus quality across the WSI. Furthermore, we study the focus sensitivity of two diagnostic models for tumor detection and introduce a quality metric that takes into account the impact of focus quality on the specific clinical diagnostic task at hand, providing a relevance-weighted quality score for slice images.

[0059] At a higher level, the method Figure 1 The method utilizes a deep convolutional neural network that is trained to identify disease or tumor cells in digital microscope images (herein a "classifier" or "disease classifier"). The deep convolutional neural network or machine learning model is not shown in Figure 1, but may be configured as described in PCT application entitled “Method and System for Assisting Pathologist Identification of Tumor Cells in Magnified Tissue Images,” serial number PCT / US17 / 019051, filed on February 23, 2017, publication number WO2018 / 156133, the contents of which are incorporated herein by reference. Deep convolutional neural network pattern recognizers are well known in the field of pattern recognition and machine vision, and therefore a detailed description thereof is omitted for brevity. The Google Inception-v3 deep convolutional neural network architecture is described in the scientific literature, which is a suitable architecture. See the following references, the contents of which are incorporated herein by reference: C. Szegedy et al., Going Deeper with Convolutions, arXiv:1409.4842 [cs.CV] (September 2014); C. Szegedy et al., Rethinking the Inception Architecture for Computer Vision, arXiv:1512.00567 [cs.CV] (December 2015); see also U.S. Patent Application Serial No. 14 / 839,452 filed on August 28, 2015 by C. Szegedy et al., “Processing Images Using Deep Neural Networks.” The fourth generation, referred to as Inception-v4, is considered an alternative architecture. See C. Szegedy et al., Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning, arXiv:1602.0761 [cs.CV] (February 2016). See also C. Vanhoucke's U.S. patent application Ser. No. 15 / 395,530, filed Dec. 30, 2016, for "Image Classification Neural Networks". The descriptions of convolutional neural networks in these papers and patent applications are incorporated herein by reference.

[0060] Basically, in Figure 1 In the method, in step 100, expected disease classifier error statistics of the disease classifier neural network described in the previous paragraph are obtained for different degrees of defocus. Figure 2 The way these statistics are obtained from a large set of master images in the form of focused, ground truth images that are artificially blurred to varying degrees of out-of-focus is shown in and described in detail below. An alternative approach is to use partially out-of-focus scans as master images, determine the in-focus patches by running an OOF classifier (described below), and run the OOF classifier on these image patches. Figure 2 The remainder of the "calibration algorithm" of . Once these statistics are obtained by either method, they are stored as shown in 102 and then used later as shown in step 104. In a sense, step 100 can be considered a quasi-calibration step. Basically, we take a measure of the tile-based error (through AUC for binary disease classifiers and through Cohen's Kappa for multi-class classifiers) and assume that these measures are similar when the disease model is applied to other digitized pathology images of the same tissue and stain type and with similar focus quality. Therefore, it can be said that in step 100, we calibrate the tile-level error metrics (AUC and Cohen's kappa, respectively) to the OOF level.

[0061] At step 104, we use the stored error statistics (step 102) to predict the focus-weighted disease classifier error on previously unseen (i.e., “new”) tissue images. Step 104 is performed when in use, i.e., when a new tissue image is captured by a whole slide scanner and the user wishes to understand the disease classifier error before assigning a diagnosis or label to the image or a portion thereof. Step 104 utilizes a second convolutional neural network (“OOF classifier”) that is trained to identify the degree to which patches in new images are out of focus. The convolutional neural network can be deep (e.g., 18 convolutional layers) or shallow (e.g., 3 convolutional layers), and our studies have shown that shallow neural networks with 3 convolutional layers perform well. The architecture of the OOF classifier can also be as previously described. In one possible configuration, it can be configured and trained in accordance with the following reference by Samuel Yang et al.: “Assessing microscope image focus quality with deep learning,” BMC Bioinformatics (2018) 19:77. Step 104 is Figure 6A notable difference between our approach (similar to Yang et al.) and the approach of Campanella et al. cited previously in the background is that the latter does not apply synthetic noise after synthetic blur (to simulate the original image sensor pixel noise that is almost completely eliminated by the synthetic blur). In our experiments, we found that synthetic noise is very important for training a proper OOF classifier based on synthetic training data. Without it, we found that the classifier failed to detect particularly strong actual OOF where the noise was very noticeable.

[0062] Our studies have shown that it is preferable to perform additional data augmentation steps when training the OOF classifier mentioned in the previous paragraph. These steps include simulating (a) data compression and noise artifacts, (b) translational jitter, and (c) brightness perturbations. It is possible and advantageous to combine all of these in training in addition to or instead of computational Bokeh filtering (described below). These will now be described in more detail.

[0063] (a) Data compression and noise artifacts

[0064] During the development of the OOF classifier, hyperparameters of all models were manually tuned by qualitatively evaluating the predicted OOF heatmaps on the validation test set. In these experiments, CNNs trained on Bokeh-style or Gaussian blurred synthetic examples produced poor prediction accuracy on real OOF images; incorrectly predicting almost all OOF test patches as in focus. We hypothesize that this is caused by artificial smoothing that removes real artifacts. For example, grid-like artifacts at the edges of scan lanes and JPEG blocks can be smoothed out in artificially blurred images, but appear in real OOF images. Therefore, several categories of additional artifact types were added back after synthetic blur to explore their importance.

[0065] Visual comparison of synthetically blurred actual in-focus images with actual OOF at high magnification reveals other artifacts besides OOF: pixel noise, which most likely originates from the image sensor; and JPEG compression artifacts, which likely originate from the lossy JPEG compression applied by the scanner post digitization. We found that the latter was applied to all our training and test images, with compression levels ranging from 40% to 90%. Most scanner models employ post digitization. However, both of these other artifacts are faint or almost non-existent in synthetically blurred images, depending on the synthetic blur size, even though they are present in the actual in-focus input image. This is because both pixel noise and JPEG artifacts typically include high spatial frequencies, which are attenuated by blur.

[0066] Therefore, simulated JPEG compression artifacts were added back to the synthetically blurred images, which was achieved through JPEG encoding and decoding and image quality parameters selected between 40% and 90%, which are the typical compression settings used by the scanner from which our images were obtained.

[0067] Regarding analog pixel noise, it is well known that image sensors generate Poisson-distributed noise when digitizing. c =P x c *s each color channel value x c ∈[0,1],c∈{R,G,B} maps to the noisy version x c ′, to simulate pixel-wise Poisson noise, where P denotes the Poisson distribution and s inversely controls the signal-to-noise ratio. Since the latter was observed to vary significantly between different scanners and objective magnifications, the noise component during training can be varied by randomly sampling s from the interval [0.01, 64.0] for each training tile, which additionally showed better experimental results compared to using a fixed signal-to-noise ratio. Although both data augmentations, JPEG artifacts and Poisson noise, significantly reduce false in-focus positives, the overall accuracy can still be improved, so simulations (b) and (c) were explored.

[0068] (b) Translational jitter

[0069] More pronounced prediction gains are observed by adding translation perturbations in the range of + / -8 pixels (both horizontally and vertically) to each 139×139 training tile. This is achieved by first taking 155×155 image tiles and then cropping randomly translated 139×139 regions from them.

[0070] (c) Brightness disturbance

[0071] A substantial improvement in the overall accuracy of the OOF classifier is finally achieved by implementing another more general data augmentation when perturbing the brightness of the training patches. In this step, random negative / positive intensity offsets are added to all pixels in the training patches (i.e., fixed positive / negative increment values ​​for all three RGB color channels). More specifically, for each training example, RGB values ​​in the range [0,1] are offset using an intensity shift randomly selected from the interval [-0.8,+0.8] (followed by clipping to [0,1]). A similar approach is proposed in Liu et al., Detecting Cancer Metastases on Gigapixel Pathology Images, arXiv [cs.CV] (2017), available at http: / / arxiv.org / abs / 1703.02442, which supports the assumption that OOF images can vary significantly in brightness, and therefore the classifier should be relatively invariant to brightness changes.

[0072] Additionally, random perturbations to the contrast, hue, and saturation of the training patches were added, but only the contrast perturbation resulted in a small performance gain.

[0073] Figure 2 is shown in more detail Figure 1 Flow chart of steps 100. Figure 2The process shown in is to calculate the expected classification error statistics of the classifier. As shown in 200, we obtain multiple (e.g., about 100, possibly more, such as 1000 or tens of thousands, and optionally with different magnification levels) fully focused, digital microscope slice images with reference truth annotations of a given tissue type (e.g., lymph node or prostate tissue of a breast cancer patient). These images 200 are referred to as "primary images" in this article. These images can be obtained from scanners of different manufacturers. There is a reference truth label for each image tile, i.e., (i) cancer / no cancer of lymph node breast metastasis, or (ii) Gleason score of prostate tissue. These gigapixel images include multiple rectangular pixel tiles or mosaics, and tiles are shown in 210 in box 208. Tile annotation can be performed manually by a trained pathologist or a team of pathologists. The size of the tile can vary according to the tissue type and the specific disease classifier used. In the case of using a team of pathologists to assign reference truth (tile annotation), the reference truth is determined from multiple annotations of the same tile. If there is no consensus, the following approaches can be used: (a) majority voting, (b) forcing the pathologist / annotator to adjudicate for each tile where the number of disagreement votes is above a threshold or greater than zero; (c) applying (a) and for tiles where there is no majority vote (e.g., for multi-class labels such as Gleason), then doing (b).

[0074] At step 202, we add synthetic (artificial) out-of-focus blur to image 200 along with horizontally fixed pixel noise, both to varying degrees. The blur is added to varying degrees or amounts of blur, indicated by parameter Y, where Y can take integer values ​​from 0 (no blur) to N, where N can be 4, 10, or some other number, such as 29. In contrast, the degree of synthetic noise (multiplicative Poisson noise, to be precise) does not depend on Y, but is randomly selected from a range that was previously determined based on measuring the noise level of the (non-blurred) original image. As shown in loop 204, we repeat the process of generating synthetic OOF blur and perform steps 206 and 212 for all values ​​of Y. OOF blur is preferably performed using a computational Bokeh filter, the details of which are explained in the following paper by Tim McGraw: Fast Bokeh Effects Using Low-Rank Linear Filters, Visual Computers, Vol. 31, No. 5 (May 2014), the contents of which are incorporated herein by reference. The summed area table method described on page 3 of McGraw's paper is a suitable method for this application. We found that the model trained using Gaussian blurred examples overestimated weak OOF regions and underestimated strong OOF regions. Therefore, we next experimented with bokeh blur (filtering), which is a more physically accurate representation of the underlying optics. Bokeh blur is achieved by convolving the image with a circular step-function kernel.

[0075] Then, for each degree of out-of-focus (Y=0...N), steps 206 and 212 are performed. In step 206, we perform inference using the disease classifier and assign a label (e.g., cancer / no cancer or Gleason score) to each of the tiles in one of the images for that degree of OOF. The result of step 206 can be represented as a "heat map" that shows the predicted disease probability for each tile in the full slice 200. The heat map is shown at 208, where the tiles 210 are not shown to scale. For example, a small circle in a tissue blob can be red to indicate a high probability of containing cancer cells.

[0076] At step 212, we first compute the disease classifier error for each tile using the predictions generated at step 206 and the ground truth annotations for each tile. We then aggregate these results to compute the average error = 1-AUC over all tile predictions for that particular defocus level Y. We then store this result as the expected error for that synthetic blur level and that particular disease classifier. Please note: the 1-AUC error metric is applicable to 2-class disease classifiers. For multi-class classifiers, other metrics are applicable. Such metrics are described in further detail below.

[0077] As shown in step 214 , we repeat loop 204 for each in-focus image 200 that has a ground truth.

[0078] By repeating steps 206 and 212 for each OOF degree, we obtain Figure 3 For each OOF degree (0, 1, 2, ...), we calculated the average error in step 212 for each master image 200 (labeled as case 1, case 2, ...).

[0079] Then, at step 216, we average the error for each OOF degree across all case / master images. As shown in loop 218, we repeat this process for different scanning magnifications to obtain the expected disease error statistics for all images at different magnifications. The different magnifications can be obtained from upsampling or downsampling the image 200 or by obtaining physical tissue sections instead of the image 200 and scanning them at different magnifications. The results are then stored in Figure 4 As shown in Table 220. Figure 4 Expected disease classifier error statistics for different degrees of defocus at different magnification levels are shown.

[0080] will understand, Figure 2 The process may be performed multiple times for different types of primary images 200 and disease classifiers, for example, once for a set of focused lymph node primary images and lymph node metastasis classifiers, and once for a set of focused prostate tissue primary images and prostate Gleason score classifiers. Additionally, the process may be performed repeatedly. Figure 2 to generate disease classifier error statistics for different makes and models of whole slide scanners. For example, one can execute Figure 2 The process is performed once to generate disease classifier error statistics for Hamamatsu whole slide scanners and then a second time for Aperio line scan whole slide scanners or other makes and models currently commercially available. This is because disease classifier error statistics may vary from scanner to scanner due to differences in scanner performance in terms of image and focus quality.

[0081] Figure 5 The performance (AUC metric) of a disease classifier at a given magnification power is shown in terms of a graph of a specific disease classifier (in this example, lymph node metastasis classification for a pixel patch). Figure 2 The X-axis indicates the degree of defocus (0 is completely in focus) and examples of images 500, 502, 504 and 506 at each degree of defocus. Figure 5 In this example, only OOF degrees up to "3" are shown, but it is possible to plot AUC performance for other OOF degrees ( Figure 5 (not shown in the figure). Figure 5 As can be seen in , as expected, the disease classifier has the highest classification accuracy for OOF degree 0 (completely in focus), while the classification accuracy drops significantly for OOF degrees 1 and 2, and is even lower for OOF degree 3. Figure 4 It will also be understood that while disease error statistics indicate that disease classifiers perform better at 40X magnification than at 20X and 10X magnifications, this is not necessarily true for all types of disease classifiers, and for some disease classifiers, the classifier's performance may actually be higher at lower magnification (10X) than at higher magnifications, for example, when larger scale structures and tissue morphology are important in generating disease classifications that are most easily understood at lower magnifications.

[0082] Once based on Figure 2 The process obtains and stores disease classifier error statistics, which can then be used on new (previously unseen) digitally enlarged tissue images to generate disease classifier error predictions for all or part of the image. The process is shown in Figure 1 104 of , and in more detail at Figure 6 The "new" tissue image is shown at 600. At step 602, we use the previously described OOF classifier (deep or shallow CNN) to perform inference on image 600. As explained above, the classifier is trained from synthetically blurred training images to use Figure 2 The out-of-focus scale used in 600 (in this example, Y=0...29, where 0 is completely in focus and 29 is completely out of focus) is used to generate a prediction of the degree of out-of-focus of each tile in the image 600. The result of the inference step 206 can be represented as a heat map of the image 600, such as shown at 604, where, according to the scale on the right side of the heat map 604, the tiles in the image that are more out of focus are lightly colored, while the areas that are more in focus are given darker colors.

[0083] Then, in step 606, for each tile, we use Figure 2 The calculated error statistics (see Figure 4) maps the focus to the expected classifier error. This mapping can be conceptualized as Figure 7 , where the heat map 604 ( Figure 6 ) is converted or mapped to Figure 6 The classifier error is shown at 606 in . Note that Figure 6 The most out-of-focus tiles in heatmap 604 have the highest expected classifier error (higher classifier errors are in Figure 7 is also light in color, while the region with the smallest classifier error is Figure 7 ). Therefore, this mapping simply assigns the classifier error to a tile based on its OOF degree using the stored error statistics and repeats this algorithm for all tiles.

[0084] Then in step 608, we predict the patch error ( Figure 7 ) to the full slice image. In one embodiment, this summary can take the form of calculating the classifier error score for the full slice. There are several possible methods for calculating this summary. One method is as follows:

[0085] a. Apply a disease / tissue type specific threshold to the predicted classifier error predictions for each tile, e.g. 1.3%, essentially producing a binary mask of those tiles where the prediction error is above or below the threshold.

[0086] b. Apply Connected Component Analysis to the binary mask of a.

[0087] c. discarding each “connected component” region whose area is smaller than a disease / tissue type specific minimum area A (e.g. 200 square microns),

[0088] d. Sum the areas of the remaining regions to obtain a final metric, in this example an area metric, which indicates how much of the slice has a disease classifier error below a threshold, and excludes areas that are too small according to c. This metric can be expressed as an absolute area metric, or as a percentage metric, for example, the percentage of tissue area in the image for which the disease classifier metric is below a threshold.

[0089] Another simple way to aggregate the predictions of the patch errors to the full slice would be to just count the pixels above / below the error threshold without doing any connected component analysis and report the results as a percentage of the tissue portion of the image.

[0090] At step 608, we then report the focused weighted disease classifier error prediction for the full slice. Figure 6On a computer workstation used by a pathologist with the image 600, the error prediction can be reported in a pop-up window on the workstation display next to the image. Alternatively, the disease classifier error prediction can be reported in a screen used by the pathologist to enter findings or diagnoses associated with the tissue image.

[0091] The example above was done in the context of a lymph node metastasis detector that predicts the probability of breast cancer metastasis in an image of lymph node tissue, and the classifier would typically operate at 40X or 20X magnification, for example, generating a cancer / no cancer class label for each tile in the image and using as an error metric 1.0–AUC, where AUC is a receiver operating characteristic graph of the classifier's performance. Figure 3 and Figure 4 Such an error metric is shown in . For a prostate cancer classifier, the classifier predicts a Gleason score for each pixel tile in a prostate tissue image and is typically operated at 10X magnification. The class label assigned by the classifier to each tile is in the set {None, 1, 2, 3, 4, 5, and 6}, where 1...6 are Gleason scores known in the art. The error metric for such a classifier may take different formats, such as weighted Cohen's Kappa (κ), a known statistical metric, or an average over the AUC of other classes. The expected error in a disease classifier can be expressed as 1.0 minus the area under the receiver operating characteristic curve (AUC) for a binary classifier, or 1.0-Cohen's Kappa when the classifier is a multi-class classifier.

[0092] Figure 6 The calculations shown in can be implemented locally in the computing resources of the whole slide scanner, so that the scanner scans the slide and then generates disease classifier error statistics and reports them all to a connected pathology workstation. In this embodiment, the scanner includes a processing unit and a deep CNN focused classifier and a storage Figure 4 Data store for statistical information, and Figure 6 The process explained in the software.

[0093] Several other alternative configurations are possible, one of which is Figure 8 In this embodiment, the whole slide scanner 802 is connected to a local area network 806 to which the pathology workstation 804 is also connected. The network 806 may, for example, take the form of a network in a hospital or medical clinic equipped with scanners and pathology workstations. The network also has a computing resource 808, which may also take the form of a general purpose computer connected to a data store or memory that stores the parameters of the deep CNN focused classifier and Figure 4The disease classifier error statistics and the Figure 6 In use, when the scanner 802 scans a slice (not shown) and generates Figure 6 When the image 600 is transmitted to the computing resource 808, the computing resource 808 then implements Figure 6 The process of forwarding the disease classifier error metric to the workstation 804. Report of disease classifier error prediction ( Figure 6 Step 610) may be performed using, for example, Figure 7 The heat map shown in FIG. 6 and the form of a summary prediction of the tile errors calculated in step 608 .

[0094] Fig. 9 is another example of a system implementing the methods of the present disclosure. In this example, a full slide scanner 802 and a workstation 804 are on a network 806 (the Internet) connected to a cloud 820. A service provider in the cloud 820 operates a computing resource 808 with a connected data store 810. The service provider receives an image (600, Figure 6 ), and by storing the OOF classifier and disease classifier error statistics in the data store 810, a disease classifier error prediction for the image is generated and reported back to the pathology workstation 804. Image 600 ( Figure 6 ) to the service provider is accompanied by metadata for the image that indicates the make / model and magnification level of the whole slide scanner so that the appropriate disease classifier error statistics can be retrieved and used appropriately for the particular image. The service provider performs this service for multiple distributed pathology labs, hospitals, clinics, universities, etc. (one of which is indicated by a separate whole slide scanner and pathology workstation 904 that is also connected to the cloud 820).

[0095] Other considerations

[0096] Figure 2 The number of OOF degrees of the scale used in loop 204 may vary, and a floating point scale may also be used.

[0097] Figure 6 The summary of the predictions of the tile errors for the new image in step 608 can take other forms, such as a) the absolute tissue area and the percentage of the total tissue area for which the prediction error is above the classifier-related threshold, or b) the total area of ​​different tissue regions that are larger than the disease-specific size and have prediction errors larger (or smaller) than the disease-specific prediction threshold, or c) the average expected error (i.e., the average of all tissue tiles) or minimum / maximum error statistics or some other format.

[0098] The primary use of a summary error metric is to report it along with a slice-level prediction (e.g., a slice-level Gleason score). For example. an automated system may report to a clinician / pathologist a Gleason score of 4+3, but a focus-related prediction error of 3%. The predicted error may then trigger (a) additional microscopy-based manual review by a pathologist, or (b) if the error is particularly high, it may trigger a rescan of the slide. Furthermore, in scenario (a), the pathologist may use a heat map to prioritize areas with high error predictions when performing additional manual microscopy review.

[0099] From the foregoing, it will be appreciated that we have described a pathology system comprising in combination: a) a slide scanner ( Figure 8 , 802); b) memory ( Figure 8 , 810), the memory stores:

[0100] 1) Parameters of a convolutional neural network (OOF classifier) ​​trained to compute the degree of out-of-focus for each tile of a digital microscope slide image generated by a slide scanner; preferably, the OOF classifier is trained with synthetically blurred images using the techniques described in detail previously;

[0101] 2) data representing expected classifier error statistics of a machine learning disease classifier for multiple degrees of defocus; and c) a computer ( Figure 8 , 808), which is configured to (1) calculate the defocus of each tile of the digital microscope slide image using a deep convolutional neural network ( Figure 6 , 602), (2) based on the calculated defocus of each tile, calculate the expected classifier error statistics to each tile of the digital microscope slide image, thereby generating a classifier error prediction for each tile ( Figure 6 , 606); and (3) computing the summary of classifier error predictions over all tiles ( Figure 6 , 608).

[0102] It will also be appreciated that we have described a method for generating predictions of classification errors for magnified digital microscope slide images of tissue samples ( Figure 6 , 600), the digital microscope slice image includes a plurality of blocks of pixel image data. The method comprises steps, wherein the steps include the following steps:

[0103] (a) Calculate the defocus of each image block of the digital microscope slice image; Figure 6 , 602). This step is performed by the OOF classifier trained as explained above.

[0104] (b) retrieving data representing expected classifier error statistics of a machine learning disease classifier for multiple degrees of defocus; Figure 2 , 220)

[0105] (c) calculating a mapping of expected classifier error statistics to each tile of the digital microscope slide image based on the calculated defocus for each tile calculated in step (a), thereby generating a classifier error prediction for each tile; and ( Figure 6 , 606)

[0106] (d) Summarize the classifier error predictions generated in step (c) over all tiles. Figure 6 , 608).

[0107] It will also be appreciated that we have described a method for characterizing a classifier configured to generate classification labels for digital microscopic sections of tissue samples or portions thereof, see generally Figure 2 The method comprises the following steps: obtaining a set of slice images ( Figure 2 , 200) (e.g., via an application programming interface (API) call to a data store storing such images, or from physically scanning a set of tissue slices with a whole-slice scanner), each slice image comprising a tile of pixel image data that is in focus and associated with a ground truth label for each image tile;

[0108] b) define a set of defocus levels (e.g., defocus level Y on a scale from 0 in focus to N, where N is some integer in the range of 1, 2, 3, ... N), and for each level:

[0109] 1) applying a corresponding amount of synthetic defocus to each tile of the image in the set of slices; Figure 2 , 202)

[0110] 2) Calculate the classification error of each block in the image; Figure 2 , 206)

[0111] 3) Calculate the average error of all patches in the image; Figure 2 , 212)

[0112] c) storing the average error calculated in step b)3) for all defocus levels defined in step b) as the expected error of the classifier for the defocus levels defined in step b); Figure 2 , for a loop 204 for all synthetic defocus levels)

[0113] d) Repeat steps b1), b2), b3) and c) for each slice image in the group (loop Figure 2 , loop 214)

[0114] In the above method, steps a), b), c) and d) may be repeated at least once for each of a plurality of different slide scanners from different manufacturers.

[0115] The error metric calculated in step b) 2) may take the form of a mean error. It may also be calculated as a receiver operating characteristic (ROC) area under the curve (AUC) metric. This latter approach requires first calculating the ROC curve over all tiles and then calculating the area under it ("area under the curve"). It is therefore not a metric that is calculated per individual tile and then aggregated.

[0116] Therefore, in a variant, in step b) of the above method, a set of defocus levels is defined, and for each defocus level, the method comprises performing steps 1)-4):

[0117] 1) applying a corresponding amount of synthetic defocus to each tile of the image in the set of slices;

[0118] 2) Calculate disease classification predictions and probabilities.

[0119] 3) Compute the receiver operating characteristic (ROC) curve for the ground truth labels on all patches, and

[0120] 4) Calculate the area under the ROC curve (AUC). In this variant, the method continues with step c) storing the AUC calculated in step b)4) for all degrees defined in step b) as the expected error of the disease classifier for the defocus degree defined in step b); d) repeating steps b1), b2), b3), b4) and c) for each slice image in the group.

[0121] In one embodiment, the tissue sample is in the form of a prostate tissue sample. Alternatively, the tissue sample may be a lymph node sample. In a preferred embodiment, the synthetic defocus applied in step b)1) is applied using a calculated bokeh filter. In one embodiment, the expected error may be represented by 1.0 minus the area under the receiver operating characteristic curve (AUC) of the disease classifier. Figure 3 and Figure 4 As shown in , the average errors stored in step c) are stored in a tabular format of the degree of defocus and the associated expected disease classifier errors.

[0122] In yet another aspect, we have described an OOF classifier and a method of training an OOF classifier. The OOF classifier is in the form of a convolutional neural network that can be trained from synthetically blurred images. The synthetically blurred images are generated by simulating one or more of the following: (a) data compression and noise artifacts, (b) translational jitter, and (c) brightness perturbations. In addition to or as an alternative to computing bokeh filtering, it is possible and advantageous to combine all of these in training.

[0123] Although the above description has been provided in the context of pathology microscopy images (tissue samples), the method is generally applicable to other types of microscopy images and classifiers built to recognize or classify objects within microscopy images.

Claims

1. A method for characterizing a classifier configured to generate classification labels for digital microscope slide images, the method The following steps are involved: a) acquiring a set of slice images, each slice image comprising a patch of pixel image data, the patches of pixel image data being in focus and associated with a ground truth label for each image patch; b) Define a set of defocus degrees, and for each degree: 1) applying a corresponding amount of synthetic defocus to each tile of the image in the set of slices; 2) Calculate the classification error of each block in the image; 3) Calculate the average error of all patches in the image; c) storing the average error calculated in step b)3) for all degrees defined in step b) as the expected error of the classifier for the degree of defocus defined in step b); d) Repeating steps b)1), b)2), b)3) and c) for each slice image in the group.

2. The method according to claim 1, further comprising: The following steps are involved: Repeat steps b), c) and d) at different magnifications for the digital microscope slide images in the set.

3. The method according to claim 1, further comprising: The following steps are involved: Steps a), b), c) and d) are repeated at least once for each of a plurality of different slide scanners from different manufacturers.

4. The method according to claim 1, in, The digital microscope image is an image of a tissue sample.

5. The method according to claim 1, in, Tissue samples include a lymph node sample or a prostate tissue sample.

6. The method according to any one of claims 1 to 5, in, The synthetic defocus applied in step b)1) is applied using a computational bokeh filter.

7. The method according to any one of claims 1 to 5, in, The expected error is expressed as 1.0 minus the area under the receiver operating characteristic curve (AUC) of the classifier, or 1.0-Cohen's Kappa if the classifier is a multi-class classifier.

8. The method according to any one of claims 1 to 5, in, The average errors stored in step c) are stored in a tabular format of the degree of defocus and the associated expected classifier error.

9. The method according to claim 1, in, The defocus degree of each tile is calculated by a defocus classifier, and wherein the defocus classifier is trained from a synthetically blurred defocus image, and wherein the synthetically blurred defocus image is generated by simulating (a) data compression and noise artifacts, (b) translation jitter, (c) brightness perturbations, and (d) computational bokeh filtering.

10. A method of characterizing a classifier configured to generate classification labels for digital microscope slides, the method The following steps are involved: a) acquiring a set of slice images, each slice image comprising a patch of pixel image data, the patches of pixel image data being in focus and associated with a ground truth label for each image patch; b) Define a set of defocus degrees, and for each degree: 1) applying a corresponding amount of synthetic defocus to each tile of the image in the set of slices; 2) Calculate classification error predictions and probabilities; 3) Compute the receiver operating characteristic (ROC) curve for the ground truth labels on all patches, and 4) Calculate the area under the ROC curve (AUC), c) storing the AUC calculated in step b)4) for all the degrees defined in step b) as the expected error of the classifier for the defocus degrees defined in step b); d) Repeating steps b)1), b)2), b)3), b)4) and c) for each slice image in the group.