Mechanical learning technique for detecting artifact pixel in image

Abstract

To provide a method for detecting a predicted artifact in target image resolution by using a machine learning model, storage media, and a system.SOLUTION: A method accesses a machine learning model trained to detect artifact pixels in an image of target image resolution, transforms an image representing at least a part of a biological sample at initial image resolution to target image resolution, and applies the machine learning model to a transformed image to identify one or more artifact pixels from the transformed image. A system trains the machine learning model so as to detect expected artifacts at the target image resolution.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a part of the "Machine-Learning" This application claims priority to U.S. Provisional Patent Application No. 63 / 256,328, entitled "Techniques For Detecting Artifact Pixels In Images," the contents of which are incorporated herein by reference in their entirety for all purposes. [Background technology]

[0002] Background of the Invention Immunohistochemistry (IHC) assays allow visualization and quantification of biomarker locations, which plays an important role in both cancer diagnosis and oncology research. In addition to the "gold-standard" DAB (3,3'-diaminobenzidine)-based IHC assays, recent advances have been made in both bright-field multiplexed IHC assays and multi-fluorescence IHC assays. These multiplexed IHC assays can be used, among other things, to identify multiple biomarkers in the same slide image. Such assays not only improve the efficiency of biomarker identification in a single slide, but also facilitate the identification of additional features associated with such biomarkers (e.g., colocalized biomarkers).

[0003] Quality control of slide images can be performed to improve performance and reduce errors in digital pathology analysis. In particular, quality control enables accurate detection of diagnostic or prognostic biomarkers from slide images by digital pathology analysis. Quality control may include, among other things, detecting and excluding pixels in the slide image that are predicted to depict one or more image artifacts. Artifacts can include tissue folds, foreign objects, blurred image areas, and any other distortions that prevent accurate display of corresponding regions of a biological sample. For example, tissue folds present in a biological sample can blur one or more portions of an image. These artifacts can cause errors or inaccurate results in subsequent digital pathology analysis. For example, artifacts detected in a slide image can cause an error in the count of detected cells or erroneously identify a group of tumor cells as normal in digital pathology analysis. In fact, artifacts can cause an inaccurate diagnosis of a subject associated with a slide image. Summary of the Invention

[0004] overview In some embodiments, a method for generating training data for training a machine learning model to detect expected artifacts in an image is provided. The method can include accessing an image displaying at least a portion of a biological sample. The method can further include applying an image preprocessing algorithm to the image to generate a preprocessed image. In some cases, the preprocessed image includes a plurality of labeled pixels. Each labeled pixel of the plurality of labeled pixels can be associated with a label that predicts whether the pixel accurately displays a corresponding point or region of at least a portion of the biological sample.

[0005] Additionally, the method may include applying a machine learning model to the preprocessed image to identify one or more labeled pixels from the plurality of labeled pixels. In some cases, one or more labeled pixels are predicted to be incorrectly labeled by the image preprocessing algorithm. Further, the method may include correcting the label for each of the one or more labeled pixels. Further, the method may further include generating training images including at least the one or more labeled pixels with the corrected labels. Further, the method may include outputting the training images.

[0006] In some embodiments, a method for training a machine learning model to detect expected artifacts in an image at a target image resolution is provided. The method can include accessing training images displaying at least a portion of a biological sample. In some cases, the training images include a plurality of labeled pixels, each labeled pixel of the plurality of labeled pixels having an associated label that predicts whether the pixel accurately displays a corresponding point or region of the at least a portion of the biological sample.

[0007] The method may further include accessing a machine learning model including a set of convolutional layers. In some cases, the machine learning model is configured to apply each convolutional layer of the set of convolutional layers to a feature map representing the input image. The method may further include training the machine learning model to detect one or more artifact pixels in the image at the target image resolution. In some cases, the artifact pixel of the one or more artifact pixels is predicted to not accurately represent a point or region of at least a portion of the biological sample.

[0008] In some cases, the training includes, for each labeled pixel among the plurality of labeled pixels of the training image, (i) determining a first loss for the labeled pixel at the first image resolution by applying a first convolutional layer of the set of convolutional layers to a first feature map representing the training image at the first image resolution, (ii) determining a second loss for the labeled pixel at the second image resolution by applying a second convolutional layer of the set of convolutional layers to a second feature map representing the training image at the second image resolution, where the second resolution has a higher image resolution than the first image resolution, (iii) determining a total loss for the labeled pixel based on the first loss and the second loss, and (iv) determining that the machine learning model is trained to detect one or more artifact pixels at the target image resolution based on the total loss. Further, the method may include outputting the trained machine learning model.

[0009] In some embodiments, a method is provided for using a machine learning model to detect predicted artifacts at a target image resolution. The method can include accessing an image displaying at least a portion of a biological sample, the image having a first image resolution. Further, the method can include accessing a machine learning model trained to detect artifact pixels in the image having a second image resolution. In some cases, the first image resolution has a higher image resolution than the second image resolution.

[0010] The method may further include transforming the image to generate a transformed image that displays at least a portion of the biological sample at a second image resolution. The method may further include applying a machine learning model to the transformed image to identify one or more artifact pixels from the transformed image. In some cases, the artifact pixels of the one or more artifact pixels are predicted to not accurately represent a point or region of at least a portion of the biological sample. The method may further include detecting an output image that includes one or more artifact pixels. generating a force.

[0011] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product containing instructions configured to cause one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0012] The terms and expressions which have been employed are used as terms of description rather than of limitation, and there is no intention in the use of such terms and expressions to exclude any equivalents of the features shown and described, or portions thereof, but it is understood that various modifications are possible within the scope of the invention as claimed. Thus, although the claimed invention has been specifically disclosed by embodiments and optional features, it will be understood that modifications and variations of the concepts disclosed herein may be made by those skilled in the art, and that such modifications and variations are considered to be within the scope of the invention as defined by the appended claims.

[0013] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0014] The features, embodiments, and advantages of the present disclosure will be better understood upon consideration of the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0015] [Figure 1] 1 shows an example set of images containing artifact pixels. [Figure 2] 1 shows a flowchart illustrating an example process for generating training data according to some embodiments. [Figure 3] 1 shows a flowchart illustrating an example process for using a machine learning model to generate training data according to some embodiments. [Figure 4] 1 illustrates an example set of images for which labels have been generated according to some embodiments. [Figure 5] 1 shows an example image with image portions each exhibiting different levels of blur. [Figure 6] 1 shows an exemplary set of images including one or more tissue folds in corresponding biological samples. [Figure 7] 1 shows an exemplary schematic diagram for generating training data for tissue fold regions according to some embodiments. [Figure 8] 1 includes a set of exemplary three-class tissue fold masks according to some embodiments. [Figure 9] An exemplary image (eg, FOV) is shown that includes a tissue fold region, showing both unblurred and blurred regions of the tissue fold region. [Figure 10] 1 shows a set of exemplary images displaying various artifact regions. [Figure 11] 1 illustrates an exemplary artifact mask generated using a composite artifact classification technique according to some embodiments. [Figure 12] 10 illustrates a further artifact mask using a complex artifact classification technique according to some embodiments. [Figure 13] 1 illustrates a slide image with two or more classification labels associated with pixels according to some embodiments. [Figure 14] 10 illustrates a process for identifying classification labels for pixels associated with both blurred and tissue fold regions according to some embodiments. [Figure 15] 10 shows a comparison between predicted classifications with ground truth masks according to some embodiments. [Figure 16] 1 shows predicted regions in slide images from different types of IHC assays according to some embodiments. [Figure 17] An exemplary set of images is shown, where each image is stained using a staining protocol corresponding to a particular type of IHC assay. [Figure 18] FIG. 1 shows a schematic diagram illustrating an example architecture used to train a machine learning model to detect artifacts in images, according to some embodiments. [Figure 19] 1 shows a flowchart illustrating an example process for training a machine learning model to accurately detect artifact pixels according to some embodiments. [Figure 20] 1 shows a flowchart illustrating a process for training a machine learning model for detecting artifact regions in slide images according to some embodiments. [Figure 21] 1 shows a flowchart illustrating an example process for using a trained machine learning model to accurately detect artifact pixels according to some embodiments. [Figure 22]1 shows an example set of graphs identifying precision and recall scores for a machine learning model trained to detect artifact pixels. [Figure 23] 1 shows a set of exemplary image masks generated for a set of blind images from the same type of assay and the same type of tissue as the training images. [Figure 24] 1 shows a set of exemplary image masks generated from images displaying non-visible assay patterns or tissue types. [Figure 25] 1 illustrates an example computer system for implementing some embodiments disclosed herein. DETAILED DESCRIPTION OF THE INVENTION

[0016] Detailed Description I. Overview The following examples are provided to introduce particular embodiments. In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of the examples of the present disclosure. However, it will be apparent that various examples may be practiced without these specific details. For example, devices, systems, structures, assemblies, methods, and other components may be shown as components in block diagram form so as not to obscure the examples in unnecessary detail. In other instances, well-known devices, processes, systems, structures, and techniques may be shown without necessary detail so as not to obscure the examples. The drawings and description are not intended to be limiting. The terms and expressions used in this disclosure are used in terms of description rather than limitation, and the use of such terms and expressions is not intended to exclude equivalents of the features shown and described, or portions thereof. The term "example" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as an "example" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0017] Several techniques for detecting artifacts have been used for quality control of slide images. An exemplary technique may involve examining a given image and manually identifying sets of pixels within the image that are predicted to display one or more artifacts. However, manual identification of artifacts can be time-consuming. The manual process relies heavily on experts to inspect each image and accurately determine whether the image contains artifacts. Furthermore, the classification of a particular type of artifact can be subjective and vary from expert to expert. For example, a first expert may label a set of pixels in a slide image as representing a blurred tissue region, while a second expert may label the same set of pixels in the same slide image as a non-blurred tissue region. Such potential discrepancies in artifact identification can reduce the accuracy of subsequent digital pathology analysis.

[0018] As an alternative to manual identification, machine learning models can predict which pixels display artifacts. While these machine learning models have been successful in detecting certain types of artifacts, their accuracy is limited by several factors. For example, these factors may be due to existing training techniques: (1) the inability to efficiently generate accurate training data, (2) the inability to efficiently train machine learning models to accurately detect artifacts in images with various staining patterns, and (3) the inability to incorporate artifact detection machine learning models into subsequent digital pathology analysis (e.g., cell classification models, image segmentation techniques) while minimizing increases in processing time and consumption of computing resources. In fact, existing machine learning models typically require significant computing resources and processing time for training and testing. Furthermore, using existing machine learning models to detect artifacts can significantly increase processing time and consume large amounts of computing resources for subsequent digital pathology analysis (e.g., cell classification). As described in detail below, embodiments of the present application can address each of the three factors to optimize the performance and increase the efficiency of artifact detection.

[0019] A. Generating training data A first factor that can impair accurate artifact detection by machine learning models involves existing training techniques that are unable to efficiently generate accurate training data. Existing techniques can involve manually annotating sets of pixels in images that may not accurately represent the corresponding portions of the biological specimen. However, manually annotating artifacts in slide images can be time-consuming. This problem can be exacerbated when machine learning models require large amounts of training data to achieve acceptable performance levels.

[0020] In addition to the above, manual annotation can result in inconsistent training data. Typically, several experts are involved in manually annotating images to generate training data. As mentioned above, each expert may have a different perspective on how blurred a given set of pixels in an image is considered to be, especially when the image includes pixels with various levels of blur. For a particular set of pixels in an image, an annotation from a first expert (e.g., non-blurred pixels) may be opposite to an annotation from a second expert (e.g., blurred pixels). Such differences in perspective can cause inconsistencies in the training data. The inconsistencies can result in machine learning models being trained to perform to less than optimal accuracy levels. Therefore, there is a need to generate consistent and accurate training data while reducing the time required for generation.

[0021] To address the above challenges, some embodiments determine, for each pixel in an image, whether the pixel accurately represents a corresponding point or region of a (e.g., stained) biological sample. A technique is used to generate labels that identify the blurred image. The labels can then be used as training data for training a machine learning model. In some cases, the label identifies whether the pixel represents at least a portion of a blurred image portion of the image. A pixel associated with a "blurred" label can be determined by estimating the amount of blur of the pixel and comparing the estimated amount to a blur threshold. As used herein, the term "blur threshold" corresponds to a blur level that is predicted to result in a performance degradation of the classification model that exceeds an acceptable threshold. If the estimated amount exceeds the blur threshold, the label can indicate that the corresponding pixel does not accurately represent the corresponding point or region of the biological sample. In some cases, the blur threshold is determined by performing digital pathology analysis of other images at a specific blur level, determining that the output of the digital pathology analysis produces results below an acceptable threshold (e.g., the amount of incorrectly classified pixels), and setting this specific blur level as the blur threshold.

[0022] The technique can include using an image preprocessing algorithm to generate an initial set of labels and a machine learning model to modify the initial set of labels. For example, image blur detection can be applied to the image to generate a preprocessed image. The preprocessed image can identify an initial label for each pixel in the image. A machine learning model can be applied to the preprocessed image to modify the label for each of the mislabeled pixel sets. The image with the pixel sets with the modified labels can be used as a training image to train a model for detecting artifacts in images.

[0023] B. Training a machine learning model to accurately detect artifact pixels A second factor that can impair accurate artifact detection by machine learning models is the inability of existing training techniques to efficiently train machine learning models to detect artifacts in images with various staining patterns. In particular, multiple stains corresponding to a particular type of IHC assay (e.g., Ki67 IHC assay) can be applied to a tissue sample to determine a specific diagnosis or prognosis for a subject. Images displaying such tissue samples may exhibit distinct staining patterns.

[0024] Recent developments in IHC assay technology have made it easier to detect multiple biomarkers in a single image. For example, fluorescence-based IHC assays can use multispectral imaging to separate several different fluorescence spectra, which can enable accurate identification of multiple antigens on the same tissue section. However, these multiplex IHC assays can produce more complex staining patterns than single IHC assays (e.g., IHC assays targeting a single type of antigen). However, training a single machine learning model to detect artifacts across images with complex staining patterns can be challenging, especially when various types of IHC assays are considered. Existing techniques can include training a machine learning model with a first set of training images corresponding to a first type of assay, and then training the machine learning model with a second set of training images corresponding to a second type of assay. In some cases, the machine learning model is trained with a set of training images collected from several IHC assays under study. These techniques can lead to time-consuming labeling and training processes. Therefore, there is a need to efficiently train a machine learning model to detect artifacts in images with various staining patterns.

[0025] To address the above challenges, some embodiments include techniques for training a machine learning model to detect artifacts in images having various staining patterns. The techniques can include accessing training images displaying at least a portion of a biological sample. The training images include a plurality of labeled pixels, each pixel having an associated label. The labels indicate where the pixel corresponds to a portion of the at least a portion of the biological sample. Predicting whether a point or region is accurately represented. For example, pixels representing out-of-focus regions of a biological specimen can be labeled as not accurately representing the corresponding region.

[0026] In some cases, the training images are converted into grayscale images. The grayscale images are used to train machine learning to detect artifact pixels. As used herein, "artifact pixel" refers to a pixel that is predicted to not accurately represent a corresponding point or region of at least a portion of a biological sample. In some cases, the artifact pixel is predicted to represent at least a portion of an artifact. For example, the artifact pixel can be predicted to represent a portion of a blurred portion of a given image or a portion of a foreign object (e.g., a hair, a dust particle, a fingerprint) shown in the image. Additionally or alternatively, the training images are converted into preprocessed images by converting the pixels from a first color space (e.g., RGB) to a second color space (e.g., L*a*b). A first color channel (e.g., L channel) of the second color space can be extracted and used to train a machine learning model to detect artifact pixels.

[0027] In some cases, to train a machine learning model, a set of image features can be added to the training images. For example, the set of image features can include a matrix of image gradient values. The matrix of image gradient values ​​can identify, for each pixel in the training images, the image gradient value of the pixel. The image gradient value indicates whether the corresponding pixel corresponds to an edge of an image object. In some cases, the matrix of image gradient values ​​is determined by applying a Laplacian of Gaussian (LoG) filter to the training images.

[0028] A machine learning model can include a set of convolutional layers. Each convolutional layer can be configured to include one or more filters (alternatively called "kernels"). For each pixel, a loss based on a comparison between the output of the set of convolutional layers and a value representing the pixel's label can be backpropagated to modify the parameters of each filter in the set of convolutional layers.

[0029] In some cases, the machine learning model includes or corresponds to a machine learning model that includes a reduction pass and an expansion pass. For example, the machine learning model may include or be a U-Net machine learning model. The reduction pass may include a first set of processing blocks, each corresponding to processing training images at a corresponding image resolution. For example, a processing block may include applying two 3x3 convolutions (convolutions without padding) to the input (e.g., training images), each followed by a rectified linear unit (ReLU). Thus, the output of the processing block may include a feature map of the training images at the corresponding image resolution. Furthermore, the processing block may include a 2x2 max-pooling operation with a stride of 2 to downsample the feature map of the processing block to a subsequent processing block that can repeat the above steps at a lower image resolution. At each downsampling step, the number of feature channels may be doubled.

[0030] Following the reduction pass, the expansion pass includes a second set of processing blocks, each corresponding to processing of a feature map output from the reduction pass at a corresponding image resolution. For example, a processing block in the second set of processing blocks may receive a feature map from the preceding processing block, apply a 2x2 convolution ("deconvolution") that halves the number of feature channels, and concatenate the feature map with the cropped feature map from the corresponding processing block in the reduction pass. The processing block may then apply two 3x3 convolutions to the concatenated feature map, each followed by a ReLU. The output of the processing block is a corresponding image resolution. The processing block may include a feature map of image resolution that can be used as input for subsequent processing blocks of higher image resolution. The processing blocks may be applied until a final output is generated. The final output may include an image mask. The image mask may identify a set of artifact pixels, each of which is predicted to not accurately represent a point or region of at least a portion of the biological sample.

[0031] In some cases, the loss for each processing block in the second set of processing blocks is calculated and used to determine a total loss for the U-Net machine learning model. For example, the total loss may correspond to the sum of the losses generated from each of the second set of processing blocks. The total loss for the U-Net machine learning model may then be used to learn parameters of the U-Net machine learning model (e.g., parameters of one or more filters in a convolutional layer). In some cases, the loss for each processing block in the second set of processing blocks may be determined by applying a 1x1 convolutional layer to the feature maps output by the processing blocks to generate modified feature maps and determining the loss from the modified feature maps.

[0032] Additionally or alternatively, a set of machine learning models can be trained by training each model in the set at a specific image resolution. The set of machine learning models can be used to determine a target image resolution for detecting artifact pixels in slide images. In some cases, the output from a trained machine learning model at a lower image resolution is compared to a set of labels for training images at a higher image resolution, and a minimized loss is determined. If the minimized loss can indicate that the output can detect artifact pixels within an acceptable level of accuracy, the machine learning model can be deployed to detect artifact pixels in images at a lower image resolution. For example, if a machine learning model can be trained to detect artifact pixels within an acceptable level of accuracy at 5x, there is no need to deploy a machine learning model for image resolutions higher than 5x (e.g., 10x, 20x, 40x). In this way, the inference time for artifact detection can be reduced by 16 times compared to another machine learning model processing images at 20x the original image resolution.

[0033] C. Implementation of a machine learning model to detect artifact pixels in images A third factor that can impair accurate artifact detection of machine learning models corresponds to the inability of existing training techniques to incorporate machine learning models for artifact detection into subsequent digital pathology analysis while minimizing increases in processing time and computing resource consumption. In particular, existing digital pathology analyses for detecting objects of interest (e.g., tissue, tumors, lymphocytes) in a given whole slide image can include dividing the whole slide image into a set of smaller image tiles. For each image tile in the set of image tiles, an analysis can be performed on the image tile to determine the classification of each image object appearing in the image tile. Thus, incorporating artifact detection into such digital pathology analysis can include, for each image tile in the set of image tiles of an image, (i) applying a machine learning model to detect artifact pixels in the image tile, (ii) excluding the detected artifact pixels from the image tile, and (iii) performing digital pathology analysis (e.g., an image segmentation algorithm) to classify the image objects displayed in the image tile excluding the artifact pixels. By applying multiple algorithms to each image tile, digital pathology of images with artifact pixel detection can suffer from increased processing time and consume additional computer resources, which can make the digital pathology analysis overall inefficient.

[0034] Furthermore, digital pathology analysis requires high image resolution for scanning to achieve accurate results. For example, to detect tumor biomarkers in an image, a machine learning model used in digital pathology analysis may require scanning the image at 20 or 40 times the original image resolution. Thus, tumor biomarker detection can already be resource-intensive and time-consuming. If a machine learning model for detecting artifact pixels requires the same image resolution, the processing time for detecting tumor biomarkers may further increase. Therefore, it is necessary to incorporate a machine learning model for artifact detection into digital pathology analysis in a way that keeps the increase in processing time and computing resource consumption to an acceptable level.

[0035] To address the above challenges, some embodiments include techniques for using different image resolutions to detect artifact pixels in an image. In some cases, the artifact pixels are predicted to represent a portion of an artifact. The artifact pixels can be detected during slide scanning and / or after a digital image of the slide is generated. In some embodiments, a machine learning model is trained to generate an image mask including a set of pixels of the image. The set of pixels in the image mask indicates artifact pixels, which are predicted to not accurately represent points or regions of at least a portion of the biological sample. The machine learning model is further trained to process images at a particular image resolution to generate the image mask. Thus, in some cases, an image having a higher image resolution is converted to a lower image resolution, and the machine learning model is applied to the converted image to generate the image mask. Additionally or alternatively, the machine learning model can be further trained to identify the amount of artifact pixels (e.g., the percentage of artifact pixels relative to the total number of pixels in the image). For example, the estimated amount may include a count of predicted artifact pixels, a cumulative area corresponding to multiple or all artifact pixels, a percentage of slide or tissue area corresponding to the predicted artifact pixels, etc.

[0036] In some cases, an image is divided into a set of image tiles. A machine learning model can be applied to each image tile in the set of image tiles to generate an image mask. The image mask can identify a subset of the image tiles, each image tile in the subset of image tiles can display one or more artifact pixels. The image mask can then be applied to the image to allow a user to deselect one or more image tiles in the subset of image tiles, with the deselected image tiles being excluded from further digital pathology analysis. Additionally or alternatively, the image mask can be applied to the image to select a subset of the image tiles of the image without user input and subsequently exclude them from further digital pathology analysis.

[0037] The trained machine learning model can be applied to images at a particular time point to generate an image mask. For example, the machine learning model can be applied to an existing scanned image to generate an image mask. In another example, the machine learning model can be applied while the image is being captured by the scanning device. Additionally or alternatively, a preview image (e.g., a thumbnail image) can be initially captured by the scanning device. An image pre-processing algorithm, such as a blur detection algorithm, can be applied to the preview image. If a tissue region is detected in the preview image, an initial image displaying the biological sample can be scanned. The initial image can display the biological sample at a target image resolution.

[0038] A machine learning model can be applied to the initial image to generate an image mask that identifies predicted artifact pixels and identifies the amount of artifact pixels present in the image. If the amount of artifact pixels exceeds an image area threshold, a warning can be generated indicating that the image is unlikely to produce accurate results when a subsequent digital pathology analysis is performed. In some examples, the artifact region threshold corresponds to a value (e.g., 40%, 50%, 60%, 70%, 80%, 90%) that represents the relative size of an image portion within the image. If the amount of artifact pixels exceeds the artifact region threshold, it can be predicted that one or more artifacts will occupy a large portion of the image and therefore likely cause poor performance of subsequent digital pathology analysis (e.g., cell classification). In such a case, the image can be rejected (e.g., automatically or in response to receiving user input corresponding to an instruction to reject the image) and / or the biological sample can be rescanned to capture another image. In some cases, an image mask is overlaid on the image to show the image with the predicted artifact pixels on a user interface. Additionally or alternatively, the application of the machine learning model and the generation of the warning can be performed for each image tile in the set of image tiles that form the image. In this way, a decision to rescan (for example) the biological sample can be made before the entire image is scanned, thus saving additional processing time and reducing the use of computing resources.

[0039] II. Generating Training Data for Training a Machine Learning Model to Detect Artifact Pixels To improve digital pathology for accurately detecting diagnostic or prognostic biomarkers from slide images, quality control can be performed to detect and remove artifacts from slide images. Artifacts can include tissue folds, foreign objects, blurred image areas, and any other image distortions. FIG. 1 shows an example set of images 100 containing artifact pixels. As shown in FIG. 1, example image 102 shows a biological sample with acceptable focus quality, with cell phenotype classification results overlaid as dots. Red dots correspond to positively stained cells. Black dots correspond to negatively stained cells. In contrast, example image 104 shows the same image as image 102 but with a set of artifact pixels on the left side. The small number of red dots in example image 104 indicates that the cell phenotype classification model was unable to identify all of the positively stained cells present in the biological sample. Due to the artifact pixels, the cell phenotype classification model was unable to perform its corresponding digital pathology analysis. Although the example image shown in FIG. 1 clearly shows pixels that display blurred image portions, other images may include pixels that display blurred image portions with varying levels of blur.

[0040] To increase the efficiency of training data generation from slide images (e.g., image 104 of FIG. 1 ), some embodiments include accelerating label collection for whole slide image quality control. In some cases, the proposed framework is applicable to label collection for other types of digital pathology analysis.

[0041] For artifact identification, two options exist: (1) pixel-by-pixel classification using image segmentation techniques, in which a classification label is assigned to each image pixel, and (2) tile-by-tile classification using image classification techniques, in which a classification label is assigned to each image tile. As used herein, an image tile refers to a portion of an image (e.g., a rectangular portion, a triangular portion) that includes a set of pixels. An image tile may represent a corresponding point or region of a biological sample, such as a cell and / or a biomarker. In some cases, a given slide image includes multiple image tiles, and the number of image tiles may range from tens or hundreds or even thousands. The image tiles can be distributed so that the entire image or a region of interest within the image is covered by the image tiles.

[0042] To generate training data, pixel-by-pixel classification is used to identify each image pixel as either passing or failing quality control (e.g., the pixel is blurred). This allows for greater flexibility in downstream analysis compared to tile-by-tile classification. Flexibility can be attributed to the pixel-level accuracy that image segmentation algorithms provide. Additionally or alternatively, tile-wise classifications can be used to generate pixel-wise classifications.

[0043] A. A framework for generating training data 2 shows a flowchart illustrating an exemplary process 200 for generating training data according to some embodiments. Images for generating the training data may be accessed. In some cases, the images represent at least a portion of a biological sample. The images may be slide images representing tissue sections of a particular organ. In some instances, the biological sample has been stained using one or more types of assays (e.g., IHC, H&E).

[0044] In step 202, a specific quality control problem is determined. The specific quality control problem may include detecting artifact pixels. Additionally or alternatively, the quality control problem may include detecting other types of artifacts, such as foreign bodies, tissue folds, or any other image objects or distortions that result in an inaccurate representation of a portion of the biological sample. In step 204, it is determined whether an existing deep learning model or an existing labeled dataset for a similar purpose with the same image characteristics exists. If such resources are available, process 200 proceeds to step 206, where initial labels are generated by (1) performing inference on the target dataset using an existing model designed for a similar purpose, and (2) training a related model on the existing labeled dataset and then applying such model to the target dataset. If such resources (i.e., models or labeled datasets) are from different image characteristics or image distributions, unsupervised domain adaptation can be utilized to adapt the existing model to the unlabeled target dataset.

[0045] If none of the aforementioned resources are available or valid, process 200 proceeds to step 208, where it is determined whether the quality control problem can be reduced to an image processing problem. If so (the "Yes" path from step 208), an image preprocessing algorithm can be applied to predict labels (step 210). As a result, a set of initial labels can be generated. Each label can predict whether a corresponding pixel in the image accurately represents a corresponding point or region of a portion of the biological sample. In some cases, the image preprocessing algorithm includes image segmentation, morphological processing, image thresholding, image filtering, image contrast enhancement, blur detection, other image preprocessing algorithms, or a combination thereof. Additionally or alternatively, the image preprocessing algorithm can include using one or more other machine learning models to preprocess the image so that the set of initial labels can be generated.

[0046] For example, the image preprocessing algorithm can include blur detection to predict artifact pixels, which involves image filtering by image gradient calculation followed by thresholding to identify low-gradient pixels. A set of pixels with low image gradient can be defined as a group of adjacent image pixels with relatively small intensity variations. In particular, pixels with low image gradient are considered to have more uniform pixel intensity than pixels with relatively high image gradient. In another example, the image preprocessing algorithm can include tissue fold detection to predict tissue folds (i.e., where one portion of tissue folds over another, creating a darker tissue region). Tissue fold detection can include identifying a set of pixels with low image intensity that is significantly darker than other tissue regions. The set of pixels can be identified by first applying an image filter (in this case, using a smoothing kernel such as a Gaussian filter) followed by intensity thresholding.

[0047] If image preprocessing algorithms are unavailable or invalid (the "No" path from step 208), one or more weakly supervised image processing models can be used to generate initial labels (step 212). For example, a learning-based interactive segmentation model can be used with a graphic user interface, which allows a user to provide weak annotations such as mouse clicks to generate an object segmentation map.

[0048] Once the initial labels are generated in the presence of existing resources, they can be modified to correct errors (step 214). Although not shown, correction of the initial labels can also be performed after step 212. In some cases, a machine learning model is applied to the initial labels to determine that a subset of the initial labels are mislabeled. For example, the initial labels may indicate that corresponding pixels accurately represent corresponding points or regions of the biological specimen, but the corresponding pixels contain one or more artifacts. By applying a machine learning model, this error can be addressed by modifying the initial labels.

[0049] Once the set of labels (including the modified labels) is obtained, training images including the set of labels can be generated. In step 216, the training images with the set of labels can be used to iteratively generate additional labels to generate additional training data (step 216). The additional training data can include additional training images, each including a corresponding set of labels. For example, if there is an available pre-trained model, preferably from a similar or same image domain, transfer learning or few-shot learning can be applied to the training images to generate an initial model. Additional training data can then be generated by using the initial model to make predictions on other unlabeled images and generate labels for the other unlabeled images. In another example, active learning can be applied to the training images to select a set of images from multiple images, and a subset of the images can be used to generate a corresponding set of labels. In yet another example, semi-supervised or fully supervised domain adaptation can be performed based on the training images to generate additional training data. Process 2 then ends.

[0050] Using the above framework, various types of artifacts that affect the accurate representation of a biological sample can be considered as labels. In some cases, additional types of artifacts are added to existing label types associated with the training data. For example, new types of artifacts can be merged with existing labels so that all artifacts have the same classification label, "artifact tissue." In some embodiments, new types of artifacts are associated with a new label that is separate from any of the existing labels, allowing for an expansion of the number of label types. For example, a new classification label for "tissue fold" can be generated.

[0051] In some cases, multiple labels are assigned to the same pixel to generate training data. In this case, each label of the multiple labels can predict whether the corresponding pixel displays at least a portion of an artifact associated with a particular artifact type (e.g., blur, foreign body, tissue fold). For example, tissue folds may be interwoven or otherwise correlated with blur artifacts. A pixel labeled as "tissue fold" may also display a blurred portion of the image. Thus, the pixel can be associated with two labels: (i) "tissue fold" and (ii) "blur artifact." Machine learning techniques, such as multi-label classification techniques, can be used to predict each image pixel as being associated with one or more types of artifacts.

[0052] B. The process for generating labels for training images 3 shows a flowchart illustrating an example process 300 for using a machine learning model to generate training data according to some embodiments. The example process 300 for generating training data can include generating labels that predict whether a corresponding pixel accurately represents a point or region of a portion of a biological sample. The example process can be incorporated into the example process presented in FIG.

[0053] In step 302, an image displaying at least a portion of a biological sample can be accessed. The image can be a slide image displaying a tissue section of a particular organ. In some instances, the biological sample has been stained using a staining protocol corresponding to a particular type of assay (e.g., IHC, H&E). For example, the image can display Ki67 The biological sample stained using the corresponding staining protocol for the IHC assay can be displayed.

[0054] In step 304, image preprocessing can be applied to the image to generate a preprocessed image. The preprocessed image can include a plurality of labeled pixels. Each labeled pixel of the plurality of labeled pixels can be associated with a label that predicts whether the pixel accurately represents a corresponding point or region of at least a portion of the biological sample. Thus, the label can indicate whether the corresponding pixel originates from an artifact, non-artifact tissue, or another type of region.

[0055] In some cases, the image preprocessing algorithm includes image segmentation, morphological processing, image thresholding, image filtering, image contrast enhancement, blur detection, other image preprocessing algorithms, or combinations thereof. Image preprocessing can include analyzing image gradients of pixels across the image. For example, image preprocessing can be used to identify a set of smooth pixels (i.e., no or very little local image intensity variation). Smooth pixels can be identified by calculating image gradients and applying a segmentation threshold. The segmentation threshold can represent a value that predicts whether a given pixel displays at least a portion of an edge shown in the image. The segmentation threshold can be a predetermined value. In some cases, the segmentation threshold is determined by performing Otsu's method or a balanced histogram thresholding method. Smooth pixels with image gradients lower than the segmentation threshold can be identified as either blurred tissue or non-tissue regions with uniform image intensity. Additionally or alternatively, the image preprocessing algorithm can include using one or more other machine learning models to preprocess the image so that multiple labeled labels can be generated.

[0056] In step 306, a machine learning model can be applied to the preprocessed image to identify one or more labeled pixels from the plurality of labeled pixels. The label of each of the one or more labeled pixels may be predicted as incorrectly labeled by the image preprocessing algorithm. The error may result from the image preprocessing algorithm not being effective enough to identify the correct label for all pixels. For example, a segmentation threshold applied as part of the image preprocessing algorithm may accurately identify artifacts in some images, but the same segmentation threshold may be too low for the remainder of the image. In another example, a segmentation threshold that can correctly identify artifacts in some portions of an image may be too low for the remainder of the same image. In both examples, some of the artifact pixels may be improperly labeled as tissue regions.

[0057] In step 308, for each of the one or more labeled pixels, the label is In some cases, the modification is performed by a user via a graphical user interface. Additionally or alternatively, the labels can be modified automatically using one or more executable instructions (e.g., if-else conditional statements).

[0058] In step 310, training images can be generated. The training images can include labeled pixels, such as labeled pixels with modified labels. In some cases, additional image features (e.g., image gradient values) can be associated with each labeled pixel to further facilitate training the machine learning model to identify artifact pixels.

[0059] In step 312, the training images are output. The training images can be used to generate additional training data. The additional training data can include additional training images, each of which includes a corresponding set of labels. Various types of machine learning techniques can be used to generate the additional training data. For example, machine learning techniques can include, but are not limited to, using a machine learning model trained via active learning, transfer learning, few-shot learning, or domain adaptation. Thereafter, process 300 ends.

[0060] C. Example training images with labels FIG. 4 illustrates an exemplary set of images 400 for which labels have been generated according to some embodiments. Each label indicates whether the corresponding pixel accurately represents at least a portion of a biological sample. Image 402 illustrates a thumbnail image corresponding to a whole slide image displaying a tissue section. Image 404 illustrates a gradient-based map of the image generated by applying Laplacian filtering followed by Gaussian smoothing to the thumbnail image or an image corresponding to a whole slide image at another resolution. As shown in image 404, pixels with image gradients below a segmentation threshold can be identified as either blurred tissue or non-tissue regions with uniform image intensity.

[0061] Image 406 shows a tissue mask generated by applying a uniform filter and then thresholding the thumbnail image (or a corresponding image at another resolution). For example, a tissue detector can be applied to the image by smoothing the image with a uniform filter and applying a segmentation threshold for the intensities of the R, G, and B channels. As previously described, the segmentation threshold can represent a value that predicts whether a given pixel displays at least a portion of an edge shown in the given image. The segmentation threshold can be a predetermined value. In some cases, the segmentation threshold is determined by performing Otsu's method or a balanced histogram thresholding method. Pixels with intensity values ​​higher than the edge detection threshold across all three channels can be identified as tissue pixels. The tissue mask can be used to generate a global sliding blur mask (e.g., image 408) with three classifications, including non-tissue, blurred tissue, and unblurred tissue.

[0062] Image 408 shows a preprocessed image (e.g., a blur map) generated by merging images 406 and 404. For example, the preprocessed image shown in image 408 shows predicted artifact pixels in a thumbnail image, with dark red colors identifying the predicted artifact pixels.

[0063] Image 410 shows a pre-processed image in which a set of image tiles can be identified. In some cases, image tiles with some artifact pixels are automatically selected. Pre-processed image 410 can correspond to image 408 and display labels with varying amounts of blur. Image tiles 412 and 414 represent image tiles selected from pre-processed image 410. In particular, image tile 412 displays a region of a biological sample stained using an ER-Dabsyl IHC assay (ER: estrogen receptor). Image tile 414 identifies an initial label within the region. The initial label can include multiple classifications, such as blurred tissue, non-tissue, and non-blurred tissue.

[0064] Image 416 shows a screenshot of an interactive graphical user interface showing image tiles, with which a user can interact (e.g., mouse click) to modify the initial labels. In some cases, the modification of the initial labels is performed by applying a machine learning model to the preprocessed image 410 or selected image tiles from 410. The machine learning model can modify the entire blur mask with high accuracy using a limited number of annotations. The machine learning model can be a separate process (not shown) or can be integrated into the graphical user interface. The application of the machine learning model can be performed using a CPU or by leveraging highly parallelized computation using a GPU, thus ensuring efficient label correction.

[0065] D. Determining the blur level for accurate label determination Subjectively determining a specific threshold for detecting artifact pixels can inevitably lead to discrepancies between experts' perceptions of blur and blur levels. Such discrepancies can significantly degrade the performance of digital pathology algorithms. FIG. 5 illustrates an example image including image portions each exhibiting various levels of blur. In particular, a pathologist may identify image 500 as analyzable for over 70% of the image. However, in practice, a large portion of image 500 may be blurred, which can pose a problem for classification models. For example, image tile 502 may not be considered blurry by a particular expert, but is not sufficiently focused for a classification model to accurately detect biomarkers.

[0066] To improve the consistency of identifying artifact pixels in an image, the change in performance of a classification model (e.g., cell classification) can be quantitatively evaluated at various blur levels. A blur threshold can be selected that corresponds to a blur level predicted to result in a performance degradation of the classification model above an acceptable threshold. The blur threshold can be used to indicate any tiles in the image (e.g., image tiles 504) that are considered to be more blurred. In some cases, any pixels in an image tile are labeled as blurred tissue if they are located in an image portion corresponding to a tissue region (e.g., within a tissue region in tissue mask 406) and their respective image gradients are lower than the blur threshold.

[0067] In some cases, the blur threshold can be determined by generating a set of sample images, each of which can be generated by applying Gaussian filtering at a particular sigma value to display one or more regions of the sample at various levels of blur.

[0068] Additionally or alternatively, the volume scan feature of the digital pathology scanner can be used to set the blur threshold. For example, a z-stack on the digital pathology scanner and / or microscope can be used to scan the z-axis of the slide, obtaining sets of scans at increasing distances from the nominal focal plane. The sets of scans can correspond to increasing levels of blur. An exemplary process for using the volume scan feature to determine the blur threshold can be as follows: First, for a fixed assay and a fixed downstream digital pathology analysis (e.g., a cell classification model), rescan the training images with labels in the "volume scan" mode of the scanning device. A volume scan image can be generated by performing a volume scan. In some cases, the scan settings for the volume scan include scanning training images using non-nominal focus scan planes at regular intervals (e.g., 1 micron). Based on the volume scan image, a set of pixels that will result in insufficient accuracy for digital pathology analysis can be detected. In some cases, a range of image gradients for the identified set of pixels can be calculated. The maximum image gradient within the range of image gradients can be set as a blur threshold. Pixels with image gradients that exceed the blur threshold can be predicted as pixels that will cause accuracy degradation beyond an acceptable level for subsequent digital pathology analysis.

[0069] E. Identifying tissue fold artifacts to generate training data Tissue folds typically occur during tissue processing (e.g., tissue slide preparation), where one or more portions of a tissue section do not adhere firmly to the glass slide and flip over onto another portion of the tissue section. FIG. 6 shows an exemplary image set 600 including one or more tissue folds in a corresponding biological sample. As shown in FIG. 6, tissue folds can have a variety of appearances. For example, a first image 602 shows a tissue fold that is much darker in intensity than the surrounding non-tissue fold area. A second image 604 shows a tissue fold that is brighter in intensity, while still allowing significant visibility of cells in the underlying tissue layer. A third image 606 shows a tissue fold area with blurred areas. The blurred areas may be caused by the thickness of the tissue fold exceeding the scanner's depth of field.

[0070] Different processes may be used to generate ground truth images including tissue fold regions. FIG. 7 shows an exemplary schematic diagram 700 for generating training data for tissue fold regions according to some embodiments. A binary tissue fold mask can then be generated using the training data. In FIG. 7 , there was no existing machine learning model, no existing ground truth, and no effective image processing technique for generating the initial ground truth. Therefore, referring again to FIG. 2 , the answers to steps 204 and 208 were both identified as “No.” As a result, step 212 was performed to generate training data for tissue fold regions, in which an interactive GUI was utilized to generate ground truth images. Generating training data for tissue fold regions may include using an interactive segmentation GUI to generate a binary tissue fold mask with two classifications: tissue fold and non-tissue fold.

[0071] In some cases, the tissue fold mask 702 is generated based on one of three techniques: (1) blurred ground truth (in the case of an FOV 704 selected from the blurred ground truth) (block 706), (2) regions identified by an image processing algorithm (e.g., identifying an FOV with tissue folds from an additional Mosaic WSI (block 708)), or (3) regions selected by an interactive GUI (block 710). In some cases, each of the three techniques is performed in turn to generate the tissue fold mask. For example, it can be determined (e.g., based on visual inspection) whether the tissue fold mask generated based on the blurred ground truth is accurate. If it is not accurate, the regions generated by the image processing algorithm can be used. If the regions generated by the image processing algorithm do not result in an accurate tissue fold mask, regions manually selected by the interactive GUI can be used to generate the tissue fold mask (block 710).

[0072] In some cases, the interactive GUI includes one or more machine learning models to facilitate selection of tissue fold regions. For example, the interactive GUI can include: (i) a first GUI component that enables manual description to select image regions and visualizes the selected regions for iterative manual correction, and (ii) a second GUI component that enables user input, such as writing and mouse clicks, to guide automatic identification of target regions. For an interactive GUI, an image processing method can be designed or a machine learning model can be trained to generate a segmented mask in response to user input. For example, a machine learning model can be trained with simulated user clicks in a target image region as well as the original image as model input and output a segmentation mask. In practice, a deep learning interactive GUI can learn to identify a target region with user input that is typically only a few pixels or a portion of the target image region. Furthermore, a user can iteratively modify existing inputs or add new inputs to refine the segmentation mask until the mask is accurate and can be used as ground truth for training a machine learning model.

[0073] A binary tissue fold mask can be combined with a corresponding tissue mask to generate a three-class tissue fold mask. Figure 8 includes a set of exemplary three-class tissue fold masks 800 according to some embodiments. A first three-class tissue fold mask 804 corresponds to a first slide image 802, and a second three-class tissue fold mask 808 corresponds to a second slide image 806. Each of the three-class tissue fold masks 804 and 808 can include, for each pixel, a first classification for non-tissue regions, a second classification for non-tissue fold tissue regions, and a third classification for tissue fold regions.

[0074] F. Integrating different types of artifacts into classification labels To train a machine learning model to detect more than one type of artifact, the artifact regions detected in the training images can be differentiated between blurred ground training labels and tissue fold ground labels. For example, four types of classifications can be integrated into the segmentation mask, including: (1) non-tissue regions, (2) blurred non-tissue fold regions, (3) unblurred tissue fold regions, and (4) analyzable tissue regions.

[0075] However, while the four-class labeling scheme assumes that blurred tissue regions and tissue fold regions are mutually exclusive, this is not necessarily the case. For example, as shown in images 602 and 606 of FIG. 6, tissue fold regions are often accompanied by blurred regions. For example, FIG. 9 shows an example image 900 (e.g., FOV) including a tissue fold region, where the tissue fold region displays both unblurred and blurred regions. Thus, assigning the blurred pixels in the tissue fold region in FIG. 9 as either the “blurred” class or the “tissue fold” class can cause confusion when training a corresponding machine learning model.

[0076] In another example, FIG. 10 shows an exemplary set of images 1000 displaying various artifact regions. For example, FOV 1002 displays blurred regions interwoven with tissue fold regions. A tissue fold binary mask 1004 can distinguish tissue folds from non-tissue fold regions. In contrast to the tissue fold binary mask 1004, a blurred ground truth mask 1006, which segments the image into four classifications, displays a confusing pattern of classified regions. For example, the presence of interwoven blurred regions can divide a tissue fold region into multiple small tissue fold subregions, which can be semantically unclear and significantly increase the difficulty for a machine learning model to learn meaningful features for tissue fold classes.

[0077] To address inaccurate classification of tissue fold regions, two types of classification strategies can be implemented. The first strategy can involve combining tissue fold regions and blurred tissue regions into a single class (e.g., a non-analyzed region class). The blurred tissue regions and tissue fold regions are classified as a "non-analyzed tissue" class. In practice, a three-class segmentation can be output that classifies each pixel into one of the following three classes: non-tissue, analyzable tissue, and non-analyzed tissue. For example, Figure 11 shows the results of some implementations. 11 shows an example artifact mask 1100 generated using a morphological composite artifact classification technique. FIG. 12 shows a further artifact mask 1200 using a composite artifact classification technique according to some embodiments. In FIG. 12, three-class ground truth masks 1204 and 1208 are generated using the TAMRA-ER Figures 11 and 12 segment the regions of the biological sample shown in the FOV (raw RGB) stained with QM-Dabsyl-ER / TAMRA-PR 1202 and QM-Dabsyl-ER / TAMRA-PR 1206. For FOV 1202, the non-analyzed region was mostly unblurred. For FOV 1206, the non-analyzed region included unblurred tissue fold regions and blurred regions. Thus, Figures 11 and 12 classify image regions into one of three classes: (i) non-tissue, (ii) analyzable tissue, and (iii) non-analyzed tissue.

[0078] A second classification strategy can include associating pixels with two or more classification labels. Multi-label segmentation can facilitate classifying each pixel as one or more of the following four classes: non-tissue, analyzable tissue, non-analyzable tissue, and tissue fold. To generate multiple classifications, a binary value (either positive or negative for that classification) can be assigned to each classification at each pixel location. For example, FIG. 13 illustrates a slide image 1300 in which pixels are associated with two or more classification labels according to some embodiments. In FIG. 13, slide image 1300 illustrates pixels 1302 and 1306 associated with a single label. Furthermore, slide image 1300 illustrates pixel 1304 associated with two labels (e.g., a blurred tissue label and a tissue fold label). As illustrated in FIG. 13, regions of a biological sample illustrated in slide image 1300 can be associated with multiple different labels. Thus, the 4x1 array shown in Figure 13 identifies a label for each pixel, where 0 indicates a negative presence for the corresponding region (e.g., this pixel does not belong to this class) and 1 indicates a positive presence for the corresponding region (e.g., this pixel belongs to this class).

[0079] FIG. 14 illustrates a process 1400 for identifying classification labels for pixels associated with both blurred and tissue fold regions according to some embodiments. In step 1402, pixels associated with multiple classifications (e.g., non-analyzed and tissue fold) can be identified. In step 1404, the pixel can be associated with a binary value for each classification in a set of classifications. The set of classifications can include: (a) non-tissue, (b) analyzable tissue, (c) blurred tissue, and (d) tissue fold. The binary value can indicate the presence of an object associated with the corresponding classification (e.g., tissue fold). For example, pixel 1304 includes a binary value of "1" for both blurred tissue and tissue fold, and a binary value of "0" for non-tissue and analyzable tissue. A binary value of "1" can indicate that pixel 1304 displays both blurred tissue and tissue fold. In some cases, the "blurred tissue" class can be replaced with a non-analyzed class (either blurred or tissue fold). In some cases, one of the multiple classifications is selected to represent the pixel. In optional step 1406, the set of classifications can be ranked based on their respective predicted probability values ​​at the pixel location. In particular, a probability of how likely the pixel is to belong to each classification in the set of classifications can be generated (e.g., using a machine learning model). For example, a three-class segmentation model generates three numbers for each pixel, such as [0.1, 0.2, 0.7], which are the probabilities that this pixel belongs to each class. In this example, the set of classifications can be ranked according to probability, with class 3 being considered to be ranked with the highest value. In practice, the highest probability corresponds to the class to which the pixel is most likely to belong. In optional step 1408, the classification with the highest probability value can be selected as the final predicted label for the pixel.

[0080] To generate labeled images for training a machine learning model, no additional processing for the ground truth masks is required. Rather, two sets of ground truth masks can be used, including a first set corresponding to three-class blur masks (e.g., non-tissue, analyzable tissue, and blurred tissue) and a second set corresponding to tissue fold regions within the tissue region (e.g., a binary tissue fold mask). In some cases, labeling of each pixel can be performed using a 4x1 array during model training.

[0081] The above-described machine learning techniques can facilitate accurate classification of regions within slide images. For example, FIG. 15 illustrates a comparison 1500 between predicted classifications with ground truth masks according to some embodiments. For example, a first set of images 1502 illustrates a comparison between predicted masks and corresponding ground truth masks for images stained using a single IHC containing estrogen receptor (ER). A second set of images 1504 illustrates a comparison between predicted masks and corresponding ground truth masks for images stained using a single IHC containing cytokeratin 7 (CK7). A third set of images 1506 illustrates a comparison between predicted masks and corresponding ground truth masks for images stained using a dual IHC containing estrogen receptor and progesterone receptor (ER / PR). Based on the comparison, it can be seen that the predicted segmented masks are qualitatively similar to the corresponding ground truth masks.

[0082] 16 further illustrates predicted regions in a slide image 1600 from different types of IHC assays, according to some embodiments. For example, a first image 1602 illustrates an image tile stained using a dual IHC assay including LIV / HER2 and a corresponding predicted segmentation mask including three classifications (e.g., unblurred tissue, blurred tissue, and non-tissue). A second image 1604 illustrates a second image tile stained using a triple IHC assay including ER / Ki67 / PR and a corresponding predicted segmentation mask including three classifications. A third image 1606 illustrates a third image tile stained using a triple IHC assay including CD8 / BCL2 / CD3 and a corresponding predicted segmentation mask including three classifications. Qualitative evaluation of the predicted segmentation masks demonstrates accurate classification of unblurred tissue, blurred tissue, and non-tissue.

[0083] III. Training a Machine Learning Model to Accurately Detect Artifact Pixels As mentioned above, since images can be stained using various types of IHC assays, training a machine learning model to accurately detect artifact pixels can be complicated. For example, fluorescence-based IHC assays can use multispectral imaging to separate several different fluorescence spectra, which can enable accurate identification of multiple antigens on the same tissue section. However, these multiplex IHC assays can produce more complex staining patterns than single IHC assays (e.g., IHC assays targeting a single type of antigen).

[0084] 17 shows an exemplary set of images 1700, each stained using a staining protocol corresponding to a particular type of IHC assay. Image 1702 shows a biological sample stained with hematoxylin only. Image 1704 shows a biological sample stained using a single IHC assay. In particular, image 1704 shows the nuclear staining pattern of a biological sample with Dabsyl-stained estrogen receptors, in which Dabsyl was used as the chromogen, resulting in yellow staining. Identifying artifacts (e.g., artifact pixels) in images 1702 and 1704 can be relatively straightforward.

[0085] The artifact detection process becomes significantly more difficult in images stained using staining protocols corresponding to multiple IHC assays. For example, image 1706 may be a dual IHC 17 shows a biological sample stained using an assay. In particular, image 1706 shows the nuclear staining pattern of a biological sample stained with Tamra to identify estrogen receptors (i.e., ER) and Dabsyl to identify progesterone receptors (i.e., PR). In image 1706, Tamra can represent purple staining, and Dabsyl can represent yellow staining. However, image 1706 also shows a blend of both stains, exhibiting various hues that may result from various factors, including the staining protocol, chromogen interference, and relative expression levels of the biomarkers. In another example, image 1708 shows a biological sample stained using another type of dual IHC assay. In particular, image 1708 shows a biological sample stained using Tamra-PDL1 (programmed death ligand 1) and Daybsyl-CK7 (cytokeratin 7), where the tissue area stained with PDL1 exhibits primarily membranous staining, and the tissue area stained with CK7 exhibits primarily cytoplasmic staining. However, image 1708 also shows tissue areas where both stains overlap, so detecting artifacts from these types of images can be difficult.

[0086] Thus, a machine learning model can be trained to detect artifact pixels in images having various staining patterns. The technique can include accessing training images displaying at least a portion of a biological sample. The training images can include a plurality of labeled pixels, each pixel having an associated label. The label predicts whether the pixel is an artifact pixel. The training images can be used to train the machine learning model. The machine learning model can include a set of convolutional layers, and a first loss calculated for a first convolutional layer and a second loss calculated for a second convolutional layer can be used to train the machine learning model to detect artifact pixels at a target image resolution.

[0087] A. Architecture for training a machine learning model for artifact pixel detection To improve the ability of a machine learning model to effectively detect artifact pixels across various image resolutions, a teacher can be added during the training phase of the machine learning model, which is trained at each of a set of image resolutions. Figure 18 shows a schematic diagram illustrating an example architecture 1800 used to train a machine learning model to detect artifacts in images according to some embodiments. Figure 18 shows an encoder-decoder model architecture for image segmentation, where features from each of multiple image resolutions in the decoder path can be utilized for pixel-by-pixel classification.

[0088] In some cases, the encoder-decoder model architecture includes U-Net. The machine learning model includes a U-Net machine learning model trained to detect artifact pixels in an image. The U-Net machine learning model can include a reduction pass and an expansion pass. The reduction pass can include a first set of processing blocks, each corresponding to processing a training image at a corresponding image resolution. For example, a processing block can include applying two 3x3 convolutions (convolutions without padding) to the input (e.g., the training image), each followed by a rectified linear unit (ReLU). Thus, the output of the processing block can include a feature map of the training image at the corresponding image resolution. Furthermore, the processing block can include a 2x2 max-pooling operation with a stride of 2 to downsample the feature map of the processing block to a subsequent processing block that can repeat the above steps at a lower image resolution. At each downsampling step, the number of feature channels can be doubled.

[0089] Following the reduction pass, the expansion pass includes a second set of processing blocks, each of which corresponds to processing the feature map output from the reduction path at the corresponding image resolution. For example, a processing block in the second set of processing blocks receives the feature map from the previous processing block, applies a 2x2 convolution ("deconvolution") that halves the number of feature channels, and concatenates the feature map with the cropped feature map from the corresponding processing block in the reduction path. The processing block can then apply two 3x3 convolutions to the concatenated feature map, each followed by an (optional) batch normalization layer and ReLU. The output of the processing block includes a feature map at the corresponding image resolution, which can be used as input for a subsequent processing block at a higher image resolution. Processing blocks can be applied until a final output is generated. The final output can include an image mask. The image mask can identify a set of artifact pixels, each of which is predicted to not accurately represent a point or region of at least a portion of the biological sample.

[0090] In some cases, the loss for each processing block or for some processing blocks of the second set of processing blocks is calculated and can be used to determine the total loss of the U-Net machine learning model. The total loss can be calculated based on the losses calculated from selected processing blocks. For example, the total loss can be determined based on the sum or weighted sum of the losses generated from each of the second set of processing blocks. In a second example, the total loss can be determined based on the average or weighted average between the losses calculated for each processing block. The total loss of the U-Net machine learning model can then be used to learn parameters of the U-Net machine learning model (e.g., parameters of one or more filters in a convolutional layer). Using the total loss of the U-Net machine learning model enables detection of artifact pixels across a variety of image resolutions.

[0091] In some cases, the loss of each processing block in the second set of processing blocks can be determined by applying a 1x1 convolutional layer to the feature maps output by the processing block to generate one or more modified feature maps and determining the loss from the one or more modified feature maps. In particular, 1x1 convolutions can be applied such that the number of modified feature maps corresponds to the number of class labels (e.g., three modified feature maps for three label types). In some cases, the modified feature maps are upsampled to the same resolution as the output of the machine learning model (e.g., image masks). Additionally or alternatively, the image masks (having the same size as the training images) can be downsampled to the same resolution as the modified feature maps.

[0092] B. Integrating global information to detect large-scale artifacts in images In some cases, the second machine learning model is trained to detect artifact pixels in the image that are predicted to correspond to larger artifacts in the image. The use of an additional machine learning model can circumvent limitations on input tile size due to limited computing resources (e.g., hardware memory). To this end, to incorporate information from neighboring image regions of each image tile, parameters of the additional machine learning model can be learned based not only on features of a particular image tile in an image, but also on features of neighboring tiles in the same image. Thus, the additional machine learning model can be trained using information corresponding to dependencies between the target image tile and its neighboring image tiles. In some cases, the additional machine learning model includes a recurrent neural network (e.g., a gated recurrent neural network) and a long-short-term memory.

[0093] The second machine learning model can be trained to (i) replace a machine learning model with a set of convolutional layers (e.g., a convolutional neural network), (ii) be used before or after the execution of the machine learning model, and / or (iii) be integrated into the machine learning model.

[0094] A recurrent neural network comprises a chain of repeating neural network modules ("cells"). Specifically, the operation of a recurrent neural network involves iterating through a single cell indexed by the location of the image tile (t) of interest. To provide its recurrent behavior, a recurrent neural network generates a hidden state s, which is provided as input to the next iteration of the network. t The hidden state may be a vector or a matrix representing information from neighboring image tiles. As referred to herein, the variable s t and h t are used interchangeably to represent the hidden state of a recurrent neural network. A recurrent neural network generates a feature representation x of a target image tile. t and the hidden state value s determined using the set of input features of the adjacent image tiles. t-1 In some cases, we receive a feature representation x of the target image tile. t is generated using a machine learning model with a set of convolutional layers. The following equation expresses the hidden state s t This provides how is determined. s t =φ(Ux t +Ws t-1 ) where U and W are the x t and s t-1 where φ is a nonlinear function such as tanh or ReLU.

[0095] As shown, Ux t and Ws t-1 generated based on the application of t The values ​​can be used as hidden state values ​​for the next iteration of the recurrent neural network that processes features corresponding to subsequent image tiles.

[0096] The output of the recurrent neural network is expressed as follows: o t= softmax(Vs t ) where V is the hidden state value s t is the weight value applied to

[0097] Therefore, the hidden state s t can be called the memory of the network. In other words, the hidden state s t Step o depends on information related to the input and / or output that is used or otherwise derived from one or more previous image tiles. t The output at is a set of values ​​used to identify artifact pixels that are calculated based at least in part on the memory at the image tile location t of interest.

[0098] C. Process for training a machine learning model to accurately detect artifact pixels 19 shows a flowchart describing an example process 1900 for training a machine learning model to accurately detect artifact pixels according to some embodiments. In step 1902, training images displaying at least a portion of a biological sample may be accessed. The training images include a plurality of labeled pixels, each of which has a label associated with it. The label predicts whether a corresponding pixel accurately represents a corresponding point or region of at least a portion of the biological sample. For example, pixels displaying an out-of-focus region of the biological sample may be labeled as not accurately representing a corresponding region (e.g., a portion of a tissue section).

[0099] In some cases, the training images are converted to grayscale images. The grayscale images are used to train machine learning to detect artifact pixels. Additionally or alternatively, the training images are converted to preprocessed images by converting their pixels from a first color space (e.g., RGB) to a second color space (e.g., L*a*b). A first color channel (e.g., L channel) of the second color space is extracted and used to train a machine learning model to detect artifact pixels. By converting to a different color space, uninformative color information is removed from artifact detection modeling. Machine learning models are implemented to learn discriminatory image features that can be filtered out and are independent of complex color variations and non-uniform staining patterns, which are largely unrelated to artifacts.

[0100] In step 1904, a machine learning model including a set of convolutional layers may be accessed. For example, the machine learning model is a U-Net architecture. In some cases, the machine learning model is configured to apply each convolutional layer of the set of convolutional layers to a feature map representing an input image.

[0101] In step 1906, the machine learning model is trained to detect one or more artifact pixels in the image at the target image resolution. The artifact pixels of the one or more artifact pixels are predicted to not accurately represent at least a portion of a point or region of the biological specimen. For example, the artifact label can predict the presence of an artifact (e.g., blur, tissue fold, foreign object) that may result in a pixel not accurately representing a corresponding region of the biological specimen.

[0102] In some cases, a set of image features is used along with the training images to train the machine learning model. For example, the set of image features can include a matrix of image gradient values. The matrix of image gradient values ​​can identify, for each pixel in the training images, the image gradient value of the pixel. The image gradient value indicates whether the corresponding pixel corresponds to an edge of an image object. In some cases, the matrix of image gradient values ​​is determined by applying a Laplacian of Gaussian (LoG) filter to the training images.

[0103] Training the machine learning model can include learning parameters of the machine learning model based on a loss value calculated for each pixel. For each labeled pixel of the plurality of labeled pixels of the training image, the training can include determining a first loss for the labeled pixel at the first image resolution by applying a first convolutional layer of the set of convolutional layers to a first feature map representing the training image at the first image resolution. Then, a second loss for the labeled pixel at a second image resolution can be determined by applying a second convolutional layer of the set of convolutional layers to a second feature map representing the training image at the second image resolution. In some cases, the second image resolution has a higher image resolution than the first image resolution.

[0104] The training can further include determining a total loss for the labeled pixels based on the first loss and the second loss. The total loss can be used to determine that the machine learning model has been trained to detect one or more artifact pixels at the target image resolution.

[0105] In step 1908, the trained machine learning model is output. The trained machine learning model can be used by another system to detect artifacts in other images with different staining patterns. Thereafter, process 1900 ends.

[0106] D. Training a Machine Learning Model to Predict Two or More Types of Artifacts In some cases, the machine learning model is trained using a three-class tissue fold mask (e.g., three-class tissue fold mask 804 of FIG. 8 ) to identify two or more types of artifacts for at least a portion of a given slide image (e.g., FOV). For example, the machine learning model can be trained to detect a first set of pixels corresponding to blurred regions and a second set of pixels corresponding to tissue folds. To train a multi-classification model, the pixels of each training image are classified into "tissue regions," "non-analysis regions" corresponding to blurred regions, and "non-analysis regions" corresponding to non-analysis regions. One or more of the tissue fold regions within the tissue region may be labeled as "non-analyzed tissue."

[0107] A multi-classification model can be first trained to detect and segment artifact regions. FIG. 20 shows a flowchart illustrating a process 2000 for training a machine learning model to detect artifact regions in slide images according to some embodiments. In FIG. 20, a first training stage can be performed to train the machine learning model to detect artifact regions displayed in slide images, and a second training stage can be performed to evaluate the performance of the machine learning model. For the first training stage, a set of slides can be selected to generate a ground truth mask (step 2002). In step 2004, a first set of training images can be generated. The first set of training images can include segmentation masks that identify two or more types of artifacts. The process for generating the first set of training images can include process 700 of FIG. 700. In step 2006, the first set of training images can be divided into a model training dataset, a validation dataset, and a test dataset. In step 2008, the machine learning model can be trained using the first set of training images. Training the machine learning model using the first set of training images is further described in process 1900 of Figure 19. In step 2010, the segmentation performance of the machine learning model can be tested based on performance scores such as precision and recall.

[0108] For the second training stage, an additional set of slides can be selected (step 2012). In some cases, the additional set of slides includes slides from unseen tissue types, biomarkers, and chromogens. The additional set of slides can be different from those used in the first training stage because the goal is to have a separate set of slides to pressure-test model performance to assess generalizability to unseen images, chromogens (combinations), biomarkers, and tissue types. In some cases, an FOV is selected from the additional set of slides. A second set of training images can then be generated based on the additional set of slides. In step 2014, labels can be assigned to pixels of each training image in the second set of training images. The labels can be assigned by receiving two types of readouts from an annotator. The first type of readout can include whether tissue folds are present within the FOV, and the second type of readout can include the percentage of non-analyzed tissue within the tissue region within each selected FOV. In step 2016, the machine learning model can be trained and tested using a second set of training images for an independent test of model generalizability. In some cases, the annotation percentages are compared to the model predictions as a proxy for model generalizability. As a result, the machine learning model can be trained and tested to detect artifact regions in other slide images.

[0109] IV. Implementing a Machine Learning Model to Detect Artifact Pixels in Images Due to the large size of whole slide images, automated digital pathology analysis needs to be performed as efficiently as possible without sacrificing accuracy. Typically, digital pathology analysis (e.g., cell classification model) can involve generating a set of image tiles from the whole slide image, where an image tile can represent a portion of the image having a particular size and dimensions (e.g., 20x20 pixels). Cell classification (for example) can then be performed on each image tile to generate corresponding predicted results, which can then be reassembled to the whole slide image resolution.

[0110] Therefore, applying quality control to digital pathology analysis can double the processing time. However, the main digital pathology analysis, which is typically 20x or 40x higher resolution, It is not necessary to perform slide quality control at the same resolution as the analysis, because many types of artifacts can be identified at lower image resolutions. Furthermore, large artifacts, such as large tissue folds, cannot fit into an image tile at high resolutions. Therefore, in some cases, performing quality control at high image resolutions will produce inconsistent results.

[0111] To improve the efficiency of implementing artifact detection in digital pathology analysis, some embodiments include using different image resolutions to detect artifact pixels in images. A set of training images can be acquired. For each training image, labels corresponding to each pixel can be collected at higher image resolutions (e.g., 40x, 20x, 10x). An artifact detection machine learning model (e.g., the U-Net machine learning model of FIG. 18) is trained using the set of training images. The machine learning model is further trained and tested using images with lower image resolutions to determine a target image resolution. At the target image resolution, the machine learning model can maintain artifact pixel detection accuracy while improving efficiency. For example, if a machine learning model can be trained to detect artifact pixels within an acceptable level of accuracy at 5x, the target resolution can be determined to be 5x, and there is no need to apply the machine learning model to detect artifact pixels at higher image resolutions (e.g., 10x, 20x). In this way, inference time can be reduced by 16 times compared to another machine learning model that detects artifact pixels at (for example) 20x image resolution.

[0112] A. Process for integrating artifact detection into digital pathology analysis 21 shows a flowchart describing an exemplary process 2100 for using a trained machine learning model to accurately detect artifact pixels according to some embodiments. In step 2102, an image displaying at least a portion of a biological sample is accessed. For example, the image may be a slide image displaying a tissue section of a particular organ. The biological sample may include a tissue section stained using a staining protocol for a particular type of IHC assay. In some cases, the image is at a first image resolution (e.g., 40x).

[0113] In step 2104, a machine learning model trained to detect artifact pixels in an image at a second image resolution is accessed. The machine learning model may be a machine learning model having a set of convolutional layers (e.g., U-Net). In some cases, the first image resolution of the image (e.g., 40x) has a higher image resolution relative to the second image resolution (e.g., 5x).

[0114] In step 2106, the image is transformed to generate a transformed image that displays at least a portion of the biological sample at a second image resolution. For example, one or more image resolution transformation algorithms including mipmapping, nearest neighbor interpolation, and Fourier transform can be used to change the image resolution and generate the transformed image.

[0115] In step 2106, the machine learning model is applied to the transformed image to identify one or more artifact pixels from the transformed image, where the artifact pixels of the one or more artifact pixels are predicted to not accurately represent at least a portion of a point or region of the biological sample. For example, the artifact pixel may be predicted to represent a portion of a blurred portion of a given image or a portion of a foreign object shown in the image.

[0116] In step 2108, an output is generated that includes one or more artifact pixels. In some cases, the output is an artifact pixel that identifies the artifact pixel with pixel-level accuracy. The artifact mask may include an artifact mask. The artifact mask may be used to identify portions of the image corresponding to various classes (e.g., unblurred tissue, blurred tissue, non-tissue). Additionally or alternatively, the output may indicate the quantity of artifact pixels (e.g., the percentage of artifact pixels relative to the total number of pixels in the image). For example, the estimated quantity may include a count of predicted artifact pixels, a cumulative area corresponding to some or all artifact pixels, a percentage of slide area or tissue area corresponding to the predicted artifact pixels, etc. Process 2100 then terminates.

[0117] In some cases, predicted artifacts in the image (e.g., artifacts represented by one or more artifact pixels) are classified into one of the following categories: (a) a first artifact category, in which the artifact occurs only during slide scanning, and (b) a second artifact category, in which the artifact occurs at all times (e.g., experiment, staining). If the predicted artifact corresponds to the first artifact category, the digital pathology analysis can proceed without further quality control operations. If the predicted artifact corresponds to the second artifact category, a warning is generated to the user prompting them to reject the image and / or rescan the biological sample to generate another image representing the biological sample. In some cases, the graphical user interface is configured to allow the user to reject the image. Additionally or alternatively, a quality control algorithm can be designed for each type of predicted artifact. The quality control algorithm for the predicted type of artifact can output a result that triggers rejection of the image and / or rescanning of the biological sample.

[0118] B. Configuration for quality control It is not uncommon for some artifacts (e.g., blurred image areas) to be present in slide images. From a user experience perspective, scanned slides with a large number of artifacts introduced during scanning are undesirable. Furthermore, given the large size of histology slides, digitizing every slide with obvious quality issues can increase storage space and scanning time. This issue can become more pronounced in large projects when scanning speeds are not optimal. Therefore, artifact detection during the scanning phase can be considered as an alternative to performing artifact detection after slide digitization.

[0119] 1. Slide Image Preprocessing In some cases, image preprocessing is applied to the image before the machine learning model is applied to the image to detect artifact pixels. For example, a preview image (e.g., a thumbnail image) can be first captured by scanning a slide displaying the biological sample with a scanning device. An image preprocessing algorithm, such as a blur detection algorithm, can be applied to the preview image. If a tissue region is detected in the preview image, an initial image displaying the biological sample can be scanned. The initial image can display the biological sample at a target image resolution.

[0120] As an illustrative example, a slide of a biological specimen can be scanned at thumbnail resolution (e.g., 1.25x) or another lower resolution to generate a preview image. The lower resolution of the preview image allows the scan time to be within a predetermined time threshold. The predetermined time threshold can be selected from various time values, such as 10 seconds, 15 seconds, 20 seconds, or any larger value. Image preprocessing can be applied to the preview image to identify portions of the image that are predicted to display one or more tissue regions of the biological specimen. If no tissue regions are identified, the quality control process ends. If one or more tissue regions are identified, a machine learning model can be applied to an image captured at a relatively higher resolution (e.g., 4x). It can be applied to statues.

[0121] 2. Artifact detection during image scanning Scanning systems for digital pathology typically include line scanners and tile-based area scanners. In a line scanner system, a line sensor can perform image acquisition one line / stripe at a time, where the line may be one pixel wide and have a length specified by the design of the sensor in the system. After scanning is completed for the entire slide, the image data acquired from the line scan can be reorganized into image tiles according to the pixel locations corresponding to the image tiles on the slide. These image tiles can then be stitched together into an image of the entire slide. In a tile-based scanner system, an area sensor performs image acquisition one tile at a time, where a tile corresponds to a rectangular field of view.

[0122] In both types of scanner systems, image tiles can be generated during scanning, at which time machine learning models can be applied to detect artifact pixels. For line scanners, the image data acquired from the line sensor is not an image tile. Thus, scan data can be generated every few line sweeps. The scan data can then be reorganized into image tiles, at which time machine learning models can be applied to the image tiles to detect artifact pixels. In some cases, processing can be performed using hardware components (e.g., FPGAs) and / or software components.

[0123] Artifact detection during scanning allows the scanner or scanner-related software to alert the user to slide quality issues (e.g., artifact type, artifact location, artifact size) during scanning so that the user can decide whether to save or delete a particular scan. Additionally or alternatively, artifact detection during scanning can be used by the scanner to intelligently and automatically adjust settings in response to the detection of a predicted artifact. For example, autofocus parameters can be adjusted by the scanner in response to determining that artifact pixels are present in a portion of an image displaying a tissue region of a biological specimen, or that the amount of artifact pixels exceeds an artifact area threshold.

[0124] (a) Artifact detection from an initial scan at low image resolution For slides with detectable tissue regions, a machine learning model can be applied to the image to generate an image mask that identifies artifact pixels and identifies the amount of artifact pixels present in the image. In some cases, artifact detection can be performed at a low image resolution. Low-resolution artifact detection can be used to detect artifact pixels that are predicted to represent artifacts that occupy a large portion of the image, including large tissue folds, large blurred areas caused by tissue folds, etc.

[0125] For example, a machine learning model can be trained to detect artifact pixels at a target image resolution. During scanning, a first scan of a slide displaying a biological sample can be performed at the target image resolution. The machine learning model can be applied to the first scan to identify one or more artifact pixels. The amount of artifact pixels can be determined. A value representing the amount of artifact pixels can be compared to an artifact region threshold. In some examples, the artifact region threshold corresponds to a value representing the relative size of an image portion within an image (e.g., 40%, 50%, 60%, 70%, 80%, 90%). The artifact region threshold can be selected by a user. If the amount of artifact pixels exceeds the artifact region threshold, it can be predicted that one or more artifacts will likely occupy a large portion of the image and therefore cause poor performance in subsequent digital pathology analysis (e.g., cell classification). If it is determined that the value representing the amount of artifact pixels exceeds the artifact region threshold, it can be determined that a quality control failure is possible. In some cases, a warning is generated in response to the determination of a quality control failure.

[0126] Additionally or alternatively, an image mask (sometimes referred to as an "artifact mask") containing one or more artifact pixels can also be generated. The artifact mask can be used by a graphical user interface to identify portions of the image that are overlaid on the image and are therefore predicted to contain artifacts. This allows the user to decide whether to rescan the slide or reject the image (e.g., the user may repeat the experiment and generate another image with better image quality).

[0127] If the value representing the amount of artifact pixels is determined to be below the artifact region threshold, the biological sample can be scanned at a higher image resolution for digital pathology analysis. In some cases, scanning at a higher image resolution involves switching magnification, such as using a different objective lens or changing the scanner's tube lens. Both actions may involve moving optical elements.

[0128] Switching resolutions can involve scanning two passes through the slide, which requires additional scanning time. By scanning initially at a lower resolution, the additional scanning time can be minimized because the initial scan can be faster than the time required to scan the slide at the target image resolution. For example, scanning at 5x resolution can produce 1 / 16 the number of pixels compared to scanning at 20x resolution. Such a difference can mean that only a small portion of the time is required to scan at 5x resolution. In another example, with a line scanner, if the length of the stripes / lines is large enough to cover the width (or height) of a given slide at low resolution, the scan can be completed in a single sweep through the slide, thus minimizing the increase in total scanning time.

[0129] (b) Artifact detection by converting scanned images to a lower image resolution. In some cases, machine learning models are applied to slide images after scanning biological samples at a high image resolution and then converting the slide images to a lower image resolution. Artifact detection can be performed on slide images before they are further processed (e.g., stored in a separate database for further digital pathology analysis). Such a design can facilitate early elimination of low-quality scans before other time-consuming processes (e.g., data transfer, long-term data storage) occur. Recent advances in computing hardware and software algorithms make such implementations feasible, as processing a whole slide image at (for example) 20x resolution can be completed within tens of seconds.

[0130] For example, a slide displaying a biological specimen can be scanned at a higher image resolution to generate an initial image. A machine learning model for detecting artifacts can be applied to the transformed image to identify one or more artifact pixels. The transformed image can be generated by transforming the initial image into an image with a lower image resolution. The amount of artifact pixels can be determined. A value representing the amount of artifact pixels can be compared to an artifact region threshold. If the value is determined to exceed the artifact region threshold, a potential quality control failure can be determined. In some cases, a warning is generated in response to a quality control failure determination. Additionally or alternatively, an artifact mask containing one or more artifact pixels may be generated to allow the user to rescan the slide or reject the image (e.g., the user may redo the experiment and generate another image with better image quality).

[0131] If the value is determined to be below the artifact region threshold, the initial image scanned at the higher image resolution can be accepted and saved directly in DICOM format and / or another file format. In some cases, information corresponding to the artifact pixels (e.g., the location of the artifact pixels in the initial image, an artifact mask at the same or lower image resolution as the initial image, etc.) is saved with the initial image and / or in a separate file format that is distinct from the initial image. Further, subsequent digital pathology analysis can be performed on the initial image.

[0132] (c) Artifact detection per image tile. In some cases, a machine learning model can be applied to a slide image on an image tile-by-image tile basis. The slide image can be divided into a set of image tiles. The machine learning model can be applied to each image tile in the set of image tiles to generate an image mask. The image mask can identify a subset of the image tiles, each image tile in the subset of image tiles can display one or more artifact pixels. The image mask can then be applied to the image to allow a user to deselect one or more image tiles in the subset of image tiles, the deselected image tiles being excluded from further digital pathology analysis. Additionally or alternatively, the image mask can be applied to the image to select a subset of the image tiles of the image without user input and subsequently exclude them from further digital pathology analysis.

[0133] As an illustrative example, a portion of a slide of a biological specimen can be scanned to obtain a corresponding portion of an image (e.g., an image tile). The image tile can be scanned at a target image resolution. After the image tile is obtained, a machine learning model is applied to the image tile to identify one or more artifact pixels (e.g., batch size=1). In some cases, the machine learning model is applied to multiple image tiles to identify artifact pixels in each image tile (e.g., batch size≧1). Processing of multiple image tiles can be performed based on multi-processing by a GPU or CPU.

[0134] For each image tile having identified artifact pixels, additional processing can be performed. The additional processing of the image tile having artifact pixels can include: (i) determining the amount of identified artifact pixels in the image tile (e.g., the percentage of artifact pixels relative to the total number of pixels), and (ii) determining the amount of pixels representing tissue regions (e.g., the percentage of pixels representing tissue regions relative to the total number of pixels). The additional processing can be performed while additional image tiles of the image are being scanned and processed by the machine learning model. In some cases, the image tile is first downsampled to represent the biological sample at a lower image resolution, and a machine learning model is applied to identify artifact pixels.

[0135] If the amount of artifact pixels determined from the image tile exceeds an artifact region threshold, a warning can be generated to alert a user that the image tile is predicted not to accurately represent the corresponding point or region of the biological specimen. In some cases, an artifact mask is generated in response to the determination.

[0136] Artifact region: the amount of artifact pixels that represent tissue areas below the threshold The entire slide can then be scanned at a target resolution for subsequent digital pathology analysis. Additionally or alternatively, the scanning system (e.g., tile-based scanner, line-based scanner) used to generate the image tiles can be configured to modify its settings based on the detection of artifact pixels. In some cases, modifying settings, with respect to artifacts corresponding to blurred image portions, involves: (i) comparing the focus quality of scanned / assembled image tiles in multiple z-planes; and (ii) excluding image tiles in z-planes where artifact pixels were identified and / or adjusting the z-planes to reduce the artifacts. Such a configuration can integrate with or replace an existing autofocus system in the scanner.

[0137] 3. Additional Artifact Detection In some cases, in addition to artifact detection of images during scanning, post-scan artifact detection can be performed. Post-scan artifact detection can further improve the accuracy of artifact detection in images. For example, algorithms for artifact detection during scanning can be designed specifically for downstream digital pathology analysis or can be designed to be applied to other general algorithms. When integrating a customized artifact detection algorithm into the scanner is impractical, post-scan artifact detection can be used to maintain quality control of whole slide images for downstream digital pathology analysis.

[0138] In another example, with respect to artifact detection at a low image resolution, new or different artifacts may occur in subsequent scans at a higher image resolution. In particular, different objective lenses or tube lenses may be used, and the scans may result from separate scanning operations, so the out-of-focus image portions may differ between these two scans. Therefore, if post-scan artifact detection is not performed, new artifacts may exist that may reduce the accuracy of downstream digital pathology analysis.

[0139] In some cases, post-scan artifact detection is more effective with certain types of machine learning models. For example, post-scan artifact detection may be more effective when certain machine learning models (e.g., recurrent neural networks) integrate features from adjacent image tiles. While image data generated during scanning can be organized during scanning to evaluate adjacent image tiles, such an approach can slow down scanning speed and / or significantly increase the computational load on hardware integrated into or associated with the scanner.

[0140] V. Experimental Results An evaluation was performed to identify the performance level of machine learning models for detecting artifacts in slide images.

[0141] A. Dataset A set of labels for identifying pixels was collected from each of the 50 whole slide images. The corresponding pixel labels related to one of three types of classes: non-tissue, blurred tissue, and unblurred tissue. The whole slide images displayed at least a portion of biological samples obtained from two cohorts (breast cancer and lung cancer). Each biological sample was stained with one of the following: (1) hematoxylin, (2) single staining for ER, PR, PDL1, or CK7, and (3) double staining for ER / PR or PDL1 / CK7. The chromogens for the assay were Dabsyl (yellow), Tamra (purple), SRB (red), or DAB (single IHC only). All samples for independent testing were analyzed. Body slide images were from a variety of tissue types (breast, lung, liver, kidney, and colon) and from single, duplex, and triplex assays (additional chromogen: Teal, additional biomarkers: LIV1, HER2, CD8, and BCL1).

[0142] From 50 whole slide images, 978 image tiles were selected, each with a size of 512x512 pixels and scanned at 5x image resolution. From the selected image tiles, 462 image tiles were used for training, 246 image tiles were used for validation, and 270 image tiles were used for testing. An additional 100 whole slide images were selected for independent testing.

[0143] B. Model Selection and Configuration Two modified U-Net machine learning models were selected for evaluation. For the first machine learning model, the number of channels in the intermediate convolutional layer was reduced by a factor of two, resulting in Model 1 (7.76 million parameters). For the second machine learning model, the number of channels in the intermediate convolutional layer was reduced by a factor of four, resulting in Model 2 (1.94 million parameters).

[0144] C. Image Preprocessing and Training Each selected image tile was converted to grayscale and augmented with random rescaling, flipping, contrast jittering, and intensity jittering. Each grayscale augmented image tile was concatenated with its corresponding image gradient map (Laplacian filtering with a kernel size of 3, followed by Gaussian filtering with a kernel size of 25 and a sigma of 3). Each grayscale augmented image tile with its corresponding gradient features was used to train two U-Net models. Training of the two U-Net models was performed using the multi-resolution training technique described in Section III. In particular, the loss calculated from each of the last two processing blocks in the augmentation pass was utilized for pixel-level classification.

[0145] D. Results 22 shows an example set of graphs 2200 identifying precision and recall scores for machine learning models trained to detect artifact pixels. Graph 2202 shows the precision scores corresponding to Model 1 and Model 2, and graph 2204 shows the accuracy scores corresponding to Model 1 and Model 2. The precision and recall scores for each of graphs 2202 and 2204 are calculated based on provided test images with corresponding labels. Note that the performance of Model 1 and Model 2 is similar in detecting artifacts. This may mean that a machine learning model with relatively few parameters (e.g., 1.94 million parameters) may be sufficiently effective in detecting artifacts.

[0146] Figure 23 shows an example set of image masks 2300 generated for a set of unseen images from the same type of assay and the same type of tissue as the training images. Each image shows a tissue section stained using a staining protocol corresponding to a particular type of IHC assay. In Figure 23, a predicted image mask 2302 generated from Model 2 is presented along with a ground truth image mask 2304. A comparison between the predicted image mask 2302 and the ground truth image mask 2304 shows that Model 2 can accurately identify artifact pixels.

[0147] Additionally, the trained machine learning model was applied to independent test images (i.e., the set of 100 whole slide images identified in Section V-A) to identify artifact pixels. For example, Figure 24 shows an exemplary set of image masks 2400 generated from images displaying unseen assay patterns or tissue types. Figure 24 shows an example set of image masks 2400 generated from images displaying unseen assay patterns or tissue types. It can be used for qualitative evaluation of machine learning model generalizability of defocus artifact detection to both sparse and unseen tissue types.

[0148] 24, the exemplary image masks show accurate artifact detection from unseen assays 2402 and unseen tissue types 2404. The image masks further demonstrate that the trained machine learning model is capable of performing accurate artifact detection for images displaying various types of biological samples and / or biological samples stained using staining protocols corresponding to different types of assays.

[0149] VI. Computing Environment 25 illustrates an example of a computer system 2500 for implementing some embodiments disclosed herein. The computer system 2500 may include a distributed architecture in which some of the components (e.g., memory and processor) are part of an end-user device and some other similar components (e.g., memory and processor) are part of a computer server. In some cases, the computer system 2500 is a computer system for determining a genetic signature of interest based on the size distribution of nucleic acid molecules in a biological sample and includes at least a processor 2502, a memory 2504, a storage device 2506, input / output (I / O) peripherals 2508, communication peripherals 2510, and an interface bus 2512. The interface bus 2512 is configured to communicate, transmit, and transfer data, control, and commands between the various components of the computer system 2500. The processor 2502 may include one or more processing units, such as a CPU, a GPU, a TPU, a systolic array, or a SIMD processor. The memory 2504 and storage 2506 include computer-readable storage media such as RAM, ROM, electrically erasable programmable read-only memory (EEPROM), hard drives, CD-ROMs, optical storage devices, magnetic storage devices, electronic non-volatile computer storage devices such as flash memory, and other tangible storage media. Any such computer-readable storage media can be configured to store instructions or program code embodying aspects of the present disclosure. Additionally, the memory 2504 and storage 2506 include computer-readable signal media.

[0150] A computer-readable signal medium includes a propagated data signal in which computer-readable program code is embodied. Such a propagated signal may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any combination thereof. A computer-readable signal medium is not a computer-readable storage medium, but includes any computer-readable medium that can communicate, propagate, or transmit a program for use in connection with computer system 2500.

[0151] Additionally, memory 2504 includes an operating system, programs, and applications. Processor 2502 is configured to execute stored instructions and includes, for example, logic processing units, microprocessors, digital signal processors, and other processors. For example, computing system 2500 can execute instructions (e.g., program code) that configure processor 2502 to perform one or more of the operations described herein. Program code includes, for example, code for performing analysis of sequence data and / or any other suitable application that performs one or more of the operations described herein. Instructions may be written in, for example, C, C++, C#, Visual C++, or any other suitable programming language. It may include processor-specific instructions generated by a compiler or interpreter from code written in any suitable computer programming language, including Basic, Java, Python, Perl, JavaScript, and ActionScript.

[0152] The program code can be stored in memory 2504 or any suitable computer-readable medium and executed by processor 2502 or any other suitable processor. In some embodiments, all of the modules in the computer system for predicting loss of heterozygosity in HLA alleles are stored in memory 2504. In additional or alternative embodiments, one or more of these modules from the above computer systems are stored in different memory devices of different computing systems.

[0153] The memory 2504 and / or the processor 2502 may be virtualized and / or hosted within another computing system, for example, in a cloud network or data center. The I / O peripherals 2508 include user interfaces such as keyboards, screens (e.g., touchscreens), microphones, speakers, and other input / output devices, as well as computing components such as graphical processing units, serial ports, parallel ports, universal serial buses, and other input / output peripherals. The I / O peripherals 2508 are connected to the processor 2502 via any of the ports coupled to the interface bus 2512. The communications peripherals 2510 are configured to facilitate communications between the computer system 2500 and other computing devices over a communications network and include, for example, network interface controllers, modems, wireless and wired interface cards, antennas, and other communications peripherals. For example, computing system 2500 can communicate with one or more other computing devices (e.g., a computing device that determines genetic characteristics of a subject based on the size distribution of nucleic acid molecules in a biological sample, another computing device that generates sequence data for the subject's biological sample) over a data network using the network interface device of communication peripheral 2510.

[0154] While the present subject matter has been described in detail with reference to specific embodiments thereof, it will be understood that, upon achieving the above understanding, those skilled in the art may readily make modifications, variations, and equivalents to such embodiments. Accordingly, it should be understood that the present disclosure is presented for purposes of illustration and not limitation, and does not exclude the inclusion of such modifications, variations, and / or additions to the present subject matter as would be readily apparent to one skilled in the art. Indeed, the methods and systems described herein may be embodied in a variety of other forms. Furthermore, various omissions, substitutions, and changes may be made in the form of the methods and systems described herein without departing from the spirit of the present disclosure. The appended claims and their equivalents are intended to cover such forms or modifications as are within the scope and spirit of the present disclosure.

[0155] Unless otherwise indicated, throughout this specification, discussions utilizing terms such as "processing," "computing," "calculating," "determining," and "identifying" are understood to refer to operations or processes of a computing device, such as one or more computers or similar electronic computing devices or devices, that manipulate or transform data represented as physical electronic or magnetic quantities in the memory, registers, or other information storage, transmission, or display devices of the computing platform.

[0156] The one or more systems described herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide a result conditioned on one or more inputs. Suitable computing devices range from general-purpose computing devices to dedicated computing devices that implement one or more embodiments of the present subject matter, including general-purpose microprocessors that access stored software that programs or configures the computing system. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software used to program or configure a computing device.

[0157] Certain embodiments of the methods disclosed herein may be performed in operation on such a computing device. The order of the blocks shown in the above examples may be changed, e.g., the blocks may be rearranged, combined, and / or divided into sub-blocks. Certain blocks or processes may be performed in parallel.

[0158] Conditional language used herein, such as, among others, "can," "could," "might," "may," "eg," and the like, unless otherwise stated or understood otherwise within the context of use, is generally intended to convey that certain examples include particular features, elements, and / or steps, but not other examples. Thus, such conditional language is not generally intended to imply that features, elements, and / or steps are somehow required in one or more examples, or that one or more examples necessarily include logic for determining whether those features, elements, and / or steps are included in or performed in any particular example, with or without authorial input or prompting.

[0159] Terms such as "comprising," "including," and "having" are synonymous and are used in an inclusive, open-ended manner and do not exclude additional elements, features, acts, operations, etc. Also, the term "or" is used in an inclusive (rather than exclusive) sense; for example, when used to connect a list of elements, the term "or" may mean one, some, or all of the elements in the list. The use of "adapted to" or "configured to" herein means open, inclusive language that does not exclude devices adapted or configured to perform additional tasks or steps. Furthermore, the use of "based on" means open, inclusive, in that a process, step, calculation, or other action "based on" one or more enumerated conditions or values ​​may, in fact, be based on additional conditions or values ​​beyond those enumerated. Similarly, the use of "based at least in part on" is meant to be open and inclusive in that a process, step, calculation, or other action "based at least in part on" one or more recited conditions or values ​​may in fact be based on additional conditions or values ​​beyond those recited. Headings, lists, and numbering contained herein are for ease of description only and are not meant to be limiting.

[0160] The various features and processes described above may be used independently of one another or may be combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of the present disclosure. Furthermore, in some implementations, certain method or process blocks may be omitted. The methods and processes described herein are also not limited to any particular sequence, and the associated blocks or states may be performed in other sequences as appropriate. For example, described blocks or states may be performed in an order other than the order specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be performed serially, in parallel, or in some other manner. Blocks or states may be added to or deleted from the disclosed examples. Similarly, The example systems and components described herein may be configured differently than described, for example, elements may be added, removed, or rearranged compared to the disclosed examples.

Claims

1. accessing an image representing at least a portion of the biological sample; applying an image pre-processing algorithm to the image to generate a pre-processed image, the pre-processed image including a plurality of labeled pixels, each labeled pixel of the plurality of labeled pixels having associated therewith a label that predicts whether the pixel accurately represents a corresponding point or region of the at least a portion of the biological sample; applying a machine learning model to the preprocessed image to identify one or more labeled pixels from the plurality of labeled pixels, the one or more labeled pixels being predicted to be mislabeled by the image preprocessing algorithm; and modifying a label for each of the one or more labeled pixels; generating training images including at least the one or more labeled pixels with the modified labels; outputting the training images; A method comprising:

2. The method of claim 1 , wherein the label further identifies a type of artifact, and the pixel is further predicted to display at least a portion of an artifact corresponding to the type of artifact.

3. applying a blur threshold to each labeled pixel of the plurality of labeled pixels; determining that a further labeled pixel of the plurality of labeled pixels is mislabeled based on application of the blur threshold; and modifying the labels corresponding to the further labeled pixels; 3. The method of claim 1 or 2, further comprising:

4. The method of claim 3 , wherein the blur threshold is determined based on the performance of a downstream algorithm on a set of z-axis images displaying at least a portion of the biological sample across a depth dimension.

5. The method of any one of claims 1 to 4, wherein the image pre-processing algorithms include image segmentation, morphological processing, image thresholding, image filtering, image contrast enhancement, blur detection, or a combination thereof.

6. The method of any one of claims 1 to 5, wherein the label further predicts whether the pixel displays at least a portion of an artifact associated with a particular artifact type.

7. The method of claim 6 , wherein the specific artifact types include blurred areas, tissue folds, and foreign objects.

8. accessing training images representing at least a portion of a biological specimen, the training images including a plurality of labeled pixels, each labeled pixel of the plurality of labeled pixels having an associated label that predicts whether the pixel accurately represents a corresponding point or region of the at least a portion of the biological specimen; access to training images, accessing a machine learning model including a set of convolutional layers, the machine learning model configured to apply each convolutional layer of the set of convolutional layers to a feature map representing an input image; training the machine learning model to detect one or more artifact pixels in an image at a target image resolution, wherein an artifact pixel of the one or more artifact pixels is predicted to not accurately represent a point or region of the at least portion of the biological sample; For each labeled pixel of the plurality of labeled pixels of the training images, determining a first loss for the labeled pixels at a first image resolution by applying a first convolutional layer of the set of convolutional layers to a first feature map representing the training images at the first image resolution; determining a second loss for the labeled pixels at a second image resolution by applying a second convolutional layer of the set of convolutional layers to a second feature map representing the training images at the second image resolution, the second resolution having a higher image resolution relative to the first image resolution; and determining a total loss for the labeled pixel based on the first loss and the second loss; determining, based on the total loss, that the machine learning model is trained to detect the one or more artifact pixels at the target image resolution; and training the machine learning model to detect one or more artifact pixels in an image at a target image resolution; outputting the trained machine learning model; A method comprising:

9. The method of claim 8 , further comprising converting the training images to grayscale training images, wherein the machine learning model is trained using the grayscale training images.

10. 9. The method of claim 8, further comprising: converting the plurality of labeled pixels of the training images from a first color space to a second color space to generate modified training images, wherein the machine learning model is trained using the modified training images.

11. The method of claim 8 , wherein the total loss is determined based on a sum of the first loss and the second loss.

12. The method of claim 8 , wherein the total loss is determined based on an average of the first loss and the second loss.

13. The method of claim 8 , wherein the target image resolution is the first image resolution.

14. accessing an image displaying at least a portion of a biological sample, the image being at a first image resolution; accessing a machine learning model trained to detect artifact pixels in images at a second image resolution, wherein the first image resolution has a higher image resolution relative to the second image resolution; Transforming the image to capture the at least a portion of the biological sample at the second image resolution generating a transformed image representing applying the machine learning model to the transformed image to identify one or more artifact pixels from the transformed image, wherein an artifact pixel of the one or more artifact pixels is predicted to not accurately represent a point or region of the at least a portion of the biological sample; and generating an output including the one or more artifact pixels; A method comprising:

15. The output is an image mask including the one or more artifact pixels, and the method comprises: overlaying the image mask on the image to distinguish a set of pixels in the image from the one or more artifact pixels; applying a cell classification model to the set of pixels; 15. The method of claim 14, further comprising:

16. The method of claim 14 or 15, wherein the output specifies an amount of the one or more artifact pixels.

17. The method of any one of claims 14 to 16, further comprising using the output to adjust one or more scanning parameters of a scanning device.

18. one or more data processors; a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more of the methods disclosed herein; A system comprising:

19. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product comprising instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein.

Images