Intelligent quality control for histological images

Abstract

A method for automated intelligent quality control may be provided. Such a method may comprise applying a quality control process to a plurality of histological images. Such a quality control process may comprise, for each of a set of artifacts, determining whether a histological image to which the quality control process is applied features that artifact. Such a quality control process may also comprise, for each artifact from the set of artifacts, based on determining that that histological image to which the quality control process is applied features that artifact, determining whether to generate a flag indicating that the histological image to which the quality control process is applied features that artifact.

Description

INTELLIGENT QUALITY CONTROL FOR HISTOLOGICAL IMAGESBACKGROUNDFIELD

[0001] The embodiments described herein are generally directed to generation of histological images and, more particularly, to automatically identifying artifacts in such images.RELATED ART

[0002] There are a variety of artifacts which can degrade the utility of histological images, such as the images being out of focus, being made of slides which have pen marks, and / or of not having fully imaged the tissue the underlying slide. However, in many cases, practices for addressing these artifacts have significant drawbacks. For example in some cases histological images may be spot checked for artifacts, leaving the vast majority of images created by a lab unexamined until they are actually applied for diagnosis. Accordingly, there is a need for automated quality control technology which can identify and / or respond to these types of artifacts in all images generated by a lab.SUMMARY

[0003] The disclosed technology may be implemented in a variety of manners. For example, in some aspects there may be provided a method for performing automatic quality control on histological images. Such a method may comprise applying a quality control process to a plurality of histological images. Such a quality control process may comprise, for each of a set of artifacts, determining whether a histological image to which the quality control process is applied features that artifact. Such a quality control process may also comprise, for each artifact from the set of artifacts, based on determining that that histological image to which the quality control process is applied features that artifact, determining whether to generate a flag indicating that the histological image to whichthe quality control process is applied features that artifact. In such a method, for a first image from the plurality of histological images, applying the quality control process to that image may comprise determining that that image features a first artifact from the set of artifacts, determining whether to generate the flag indicating that that image features the first artifact, and generating the flag indicating that the first image features the first artifact.

[0004] Other types of implementations, including in the form of systems and computer readable media for performing such methods, are also possible and will be immediately apparent to those of skill in the art in light of this disclosure. Accordingly, the example method provided in this summary should be understood as being illustrative only, and should not be treated as limiting.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The details of aspects of the disclosed technology, both as to its structure and operation, may be gleaned in part by study of the accompanying drawings, in which like reference numerals refer to like parts, and in which:

[0006] FIG. 1 A illustrates an example processor-enabled device that may be used in connection with the various embodiments described herein;

[0007] FIG. IB illustrates an example line-scan camera having a single linear array, according to an embodiment;

[0008] FIG. 1C illustrates an example line-scan camera having three linear arrays, according to an embodiment;

[0009] FIG. ID illustrates an example line-scan camera having a plurality of linear arrays, according to an embodiment;

[0010] FIG. 2 provides a high level illustration of a method which may be used for automatically applying a quality control process to histological images;

[0011] FIG. 3 shows an interface which may be provided on the display of a scanner;

[0012] FIG. 4 shows a schematic view of a laboratory which includes a plurality of scanners;

[0013] FIG. 5 shows a method which may be performed in providing a dashboard interface; and

[0014] FIG. 6 illustrates a method which may be used in determining whether to generate an artifact flag.DETAILED DESCRIPTION

[0015] Disclosed herein is technology which can be used to intelligently perform quality control processes on histological images. After reading this description, it will become apparent to one skilled in the art how to implement the disclosed technology in various alternative embodiments and alternative applications. However, although various embodiments of the disclosed technology are described herein, it is understood that these embodiments are presented by way of example and illustration only, and not limitation. As such, this detailed description of various embodiments should not be construed to limit the scope or breadth of the protection provided by this document or any related document.

[0016] 1. Example Scanning System

[0017] FIG. 1A is a block diagram illustrating an example processor-enabled slide-scanning system 100 that may be used in connection with various embodiments described herein. Alternative forms of scanning system 100 may also be used as will be understood by the skilled artisan. In the illustrated embodiment, scanning system 100 is presented as a digital imaging device that comprises one or more processors 104, one or morememories 106, one or more motion controllers 108, one or more interface systems 110, one or more movable stages 112 that each support one or more glass slides 114 with one or more samples 116, one or more illumination systems 118 that illuminate sample 116, an imaging system 101 that comprises imaging optics 103 such as one or more objective lenses 120 that each define an optical path 122 that travels along an optical axis, one or more objective lens positioners 124, one or more optional epi-illumination systems 126 (e.g., included in a fluorescence-scanning embodiment), one or more focusing optics 128, and one or more line-scan cameras 130, and / or one or more area-scan cameras 132, each of which define a separate field of view 134 on sample 116 and / or glass slide 114. The various elements of scanning system 100 are communicatively coupled via one or more communication busses 102. Although there may be a plurality of each of the various elements of scanning system 100, for simplicity in the description that follows, these elements will be described in the singular, except when needed to be described in the plural to convey the appropriate information.

[0018] Processor 104 may include, for example, a central processing unit (CPU) and a separate graphics processing unit (GPU) capable of processing instructions in parallel, or a multi core processor capable of processing instructions in parallel. Additional separate processors may also be provided to control particular components or perform particular functions, such as image processing. For example, additional processors may include an auxiliary processor to manage data input, an auxiliary processor to perform floatingpoint mathematical operations, a special -purpose processor having an architecture suitable for fast execution of signal-processing algorithms (e.g., digital-signal processor), a slave processor subordinate to the main processor (e.g., back-end processor), an additional processor for controlling line-scan camera 130, stage 112, objective lens 120, and / or a display (e.g., a console comprising a touch panel display integral to scanning system 100). Such additional processors may be separate discrete processors or may be integrated into a single processor.

[0019] Memory 106 provides storage of data and instructions for programs that can be executed by processor 104. Memory 106 may include one or more volatile and / or non-volatile computer-readable storage mediums that store the data and instructions. These mediums may include, for example, random-access memory (RAM), read-only memory (ROM), a hard disk drive, a removable storage drive (e.g., comprising flash memory), and / or the like. Processor 104 is configured to execute instructions that are stored in memory 106, and communicate via communication bus 102 with the various elements of scanning system 100 to carry out the overall function of scanning system 100.

[0020] Communication bus 102 may be configured to convey analog electrical signals and / or digital data. Accordingly, communications from processor 104, motion controller 108, and / or interface system 110, via communication bus 102, may include both electrical signals and digital data. Processor 104, motion controller 108, and / or interface system 110 may also be configured to communicate with one or more of the various elements of scanning system 100 via a wireless communication link.

[0021] Motion control system 108 is configured to precisely control and coordinate X, Y, and / or Z movement of stage 112 (e.g., within an X-Y plane), X, Y, and / or Z movement of objective lens 120 (e.g., along a Z axis orthogonal to the X-Y plane, via objective lens positioner 124), rotational movement of a carousel described elsewhere herein, lateral movement of a push / pull assembly described elsewhere herein, and / or any other moving component of scanning system 100. For example, in a fluorescence-scanning embodiment comprising epi-illumination system 126, motion control system 108 may be configured to coordinate movement of optical filters and / or the like in epi- illumination system 126.

[0022] Interface system 110 allows scanning system 100 to interface with other systems and human operators. For example, interface system 110 may include a console (e.g., atouch panel display) to provide information directly to an operator via a graphical userinterface and / or allow direct input from an operator via a touch sensor. Interface system 110 may also be configured to facilitate communication and data transfer between scanning system 100 and one or more external devices that are directly connected to scanning system 100 (e.g., a printer, removable storage medium, etc.), and / or one or more external devices that are indirectly connected to scanning system 100, for example, via one or more networks (e.g., an image storage system, a Scanner Administration Manager (SAM) server and / or other administrative server, an operator station, a user station, etc.).

[0023] Illumination system 118 is configured to illuminate at least a portion of sample 116. Illumination system 118 may include, for example, one or more light sources and illumination optics. The light source(s) could comprise a variable intensity halogen light source with a concave reflective mirror to maximize light output and a KG-1 filter to suppress heat. The light source(s) could comprise any type of arc-lamp, laser, or other source of light. In an embodiment, illumination system 118 illuminates sample 116 in transmission mode, such that line-scan camera 130 and / or area-scan camera 132 sense optical energy that is transmitted through sample 116. Alternatively or additionally, illumination system 118 may be configured to illuminate sample 116 in reflection mode, such that line-scan camera 130 and / or area-scan camera 132 sense optical energy that is reflected from sample 1 16. Illumination system 118 may be configured to be suitable for interrogation of sample 116 in any known mode of optical microscopy.

[0024] In an embodiment, scanning system 100 includes an epi-illumination system 126 to optimize scanning system 100 for fluorescence scanning. It should be understood that, if fluorescence scanning is not supported by scanning system 100, epi-illumination system 126 may be omitted. Fluorescence scanning is the scanning of samples 116 that include fluorescence molecules, which are photon-sensitive molecules that can absorb light at a specific wavelength (i.e., excitation). These photon-sensitive molecules also emit light at a higher wavelength (i.e., emission). Because the efficiency of thisphotoluminescence phenomenon is very low, the amount of emitted light is often very low. This low amount of emitted light typically frustrates conventional techniques for scanning and digitizing sample 116 (e.g., transmission-mode microscopy).

[0025] Advantageously, in an embodiment of scanning system 100 that utilizes fluorescence scanning, use of a line-scan camera 130 that includes multiple linear sensor arrays (e.g., a time-delay-integration (TDI) line-scan camera) increases the sensitivity to light of line-scan camera 130 by exposing the same area of sample 116 to each of the plurality of linear sensor arrays of line-scan camera 130. This is particularly useful when scanning faint fluorescence samples with low levels of emitted light. Accordingly, in a fluorescence-scanning embodiment, line-scan camera 130 is preferably a monochrome TDI line-scan camera. Monochrome images are ideal in fluorescence microscopy because they provide a more accurate representation of the actual signals from the various channels present on sample 116. As will be understood by those skilled in the art, a fluorescence sample can be labeled with multiple florescence dyes that emit light at different wavelengths, which are also referred to as “channels.”

[0026] Furthermore, because the low-end and high-end signal levels of various fluorescence samples present a wide spectrum of wavelengths for line-scan camera 130 to sense, it is desirable for the low-end and high-end signal levels that line-scan camera 130 can sense to be similarly wide. Accordingly, in a fluorescence-scanning embodiment, line-scan camera 130 may comprise a monochrome 10-bit 64-linear-array TDI line-scan camera. It should be noted that a variety of bit depths for line-scan camera 130 can be employed for use with such an embodiment.

[0027] Movable stage 112 is configured for precise X-Y movement under control of processor 104 or motion controller 108. Movable stage 112 may also be configured for Z movement under control of processor 104 or motion controller 108. Movable stage 112 is configured to position sample 116 in a desired location during image data capture byline-scan camera 130 and / or area-scan camera 132. Movable stage 112 is also configured to accelerate sample 116 in a scanning direction to a substantially constant velocity, and then maintain the substantially constant velocity during image data capture by line-scan camera 130. In an embodiment, scanning system 100 may employ a high- precision and tightly coordinated X-Y grid to aid in the location of sample 116 on movable stage 112. In an embodiment, movable stage 112 is a linear-motor-based X-Y stage with high-precision encoders employed on both the X and the Y axes. For example, very precise nanometer encoders can be used on the axis in the scanning direction and on the axis that is in the direction perpendicular to the scanning direction and on the same plane as the scanning direction. Stage 112 is also configured to support glass slide 114 upon which sample 116 is disposed.

[0028] Sample 116 can be anything that may be interrogated by optical microscopy. For example, glass microscope slide 114 is frequently used as a viewing substrate for specimens that include tissues and cells, chromosomes, deoxyribonucleic acid (DNA), protein, blood, bone marrow, urine, bacteria, beads, biopsy materials, or any other type of biological material or substance that is either dead or alive, stained or unstained, labeled or unlabeled. Sample 116 may also be an array of any type of DNA or DNA- related material, such as complementary DNA (cDNA) or ribonucleic acid (RNA), or protein that is deposited on any type of slide or other substrate, including any and all samples commonly known as microarrays. Sample 116 may be a microtiter plate (e.g., a 96-well plate). Other examples of sample 116 include integrated circuit boards, electrophoresis records, petri dishes, fdm, semiconductor materials, forensic materials, and machined parts.

[0029] Objective lens 120 is mounted on objective positioner 124, which, in an embodiment, employs a very precise linear motor to move objective lens 120 along the optical axis defined by objective lens 120. For example, the linear motor of objective lens positioner 124 may include a fifty-nanometer encoder. The relative positions of stage 112 andobjective lens 120 in X, Y, and / or Z axes are coordinated and controlled in a closed- loop manner using motion controller 108 under the control of processor 104 that employs memory 106 for storing information and instructions, including the computerexecutable programmed steps for overall operation of scanning system 100.

[0030] In an embodiment, objective lens 120 is a plan apochromatic (“APO”) infinity-corrected objective lens which is suitable for transmission-mode illumination microscopy, reflection-mode illumination microscopy, and / or epi-illumination-mode fluorescence microscopy (e.g., an Olympus 40X, 0.75NA or 20X, 0.75 NA). Advantageously, objective lens 120 is capable of correcting for chromatic and spherical aberrations. Because objective lens 120 is infinity-corrected, focusing optics 128 can be placed in optical path 122 above objective lens 120 where the light beam passing through objective lens 120 becomes a collimated light beam. Focusing optics 128 focus the optical signal captured by objective lens 120 onto the light-responsive elements of linescan camera 130 and / or area-scan camera 132, and may include optical components such as filters, magnification changer lenses, and / or the like. Objective lens 120, combined with focusing optics 128, provides the total magnification for scanning system 100. In an embodiment, focusing optics 128 may contain a tube lens and an optional 2X magnification changer. Advantageously, the 2X magnification changer allows a native 20X objective lens 120 to scan sample 116 at 40X magnification.

[0031] Line-scan camera 130 comprises at least one linear array of picture elements 142 (“pixels”). Line-scan camera 130 may be monochrome or color. Color line-scan cameras typically have at least three linear arrays, while monochrome line-scan cameras may have a single linear array or plural linear arrays. Any type of singular or plural linear array, whether packaged as part of a camera or custom-integrated into an imaging electronic module, can also be used. For example, a three linear array (“red-green-blue” or “RGB”) color line-scan camera or a ninety-six linear array monochrome TDI may also be used. TDI line-scan cameras typically provide a substantially better signal-to-noise ratio (“SNR”) in the output signal by summing intensity data from previously imaged regions of a specimen, yielding an increase in the SNR that is in proportion to the square-root of the number of integration stages. TDI line-scan cameras comprise multiple linear arrays. For example, TDI line-scan cameras are available with 24, 32, 48, 64, 96, or even more linear arrays. Scanning system 100 also supports linear arrays that are manufactured in a variety of formats including some with 512 pixels, some with 1,024 pixels, and others having as many as 4,096 pixels. Similarly, linear arrays with a variety of pixel sizes can also be used in scanning system 100. The salient requirement for the selection of any type of line-scan camera 130 is that the motion of stage 112 can be synchronized with the line rate of line-scan camera 130, so that stage 112 can be in motion with respect to line-scan camera 130 during the digital image capture of sample 116.

[0032] In an embodiment, the image data generated by line-scan camera 130 is stored in a portion of memory 106 and processed by processor 104 to generate a contiguous digital image of at least a portion of sample 116. The contiguous digital image can be further processed by processor 104, and the processed contiguous digital image can also be stored in memory 106.

[0033] In an embodiment with two or more line-scan cameras 130, at least one of the line-scan cameras 130 can be configured to function as a focusing sensor that operates in combination with at least one of the other line-scan cameras 130 that is configured to function as an imaging sensor. The focusing sensor can be logically positioned on the same optical axis as the imaging sensor or the focusing sensor may be logically positioned before or after the imaging sensor with respect to the scanning direction of scanning system 100. In such an embodiment with at least one line-scan camera 130 functioning as a focusing sensor, the image data generated by the focusing sensor may be stored in a portion of memory 106 and processed by processor 104 to generate focus information, to allow scanning system 100 to adjust the relative distance betweensample 116 and objective lens 120 to maintain focus on sample 116 during scanning. Additionally, in an embodiment, the at least one line-scan camera 130 functioning as a focusing sensor may be oriented such that each of a plurality of individual pixels 142 of the focusing sensor is positioned at a different logical height along the optical path 122.

[0034] In operation, the various components of scanning system 100 and the programmed modules stored in memory 106 enable automatic scanning and digitizing of sample 116, which is disposed on glass slide 114. Glass slide 114 is securely placed on movable stage 112 of scanning system 100 for scanning sample 116. Under control of processor 104, movable stage 112 accelerates sample 116 to a substantially constant velocity for sensing by line-scan camera 130. After scanning a segment of image data, movable stage 112 decelerates and brings sample 116 to a substantially complete stop. Movable stage 112 then moves orthogonal to the scanning direction to position sample 116 for scanning of a subsequent segment of image data (e.g., an adjacent segment). Additional segments are subsequently scanned until an entire portion of sample 116 or the entire sample 116 is scanned.

[0035] In an embodiment, computer-executable instructions (e.g., programmed modules and software) are stored in memory 106 and, when executed, enable scanning system 100 to perform the various functions (e.g., display the graphical user interface, execute the disclosed processes, control the components of scanning system 100, etc.) described herein. In this description, the term “computer-readable storage medium” is used to refer to any media used to store and provide computer-executable instructions to scanning system 100 for execution by processor 104. Examples of these media include memory 106 and any removable or external storage medium (not shown) communicatively coupled with scanning system 100 either directly (e.g., via a universal serial bus (USB), a wireless communication protocol, etc.) or indirectly (e.g., via a wired and / or wireless network).

[0036] FIG. IB illustrates a line-scan camera 130 having a single linear array 140, which may be implemented as a charge- coupled device (“CCD”) or complimentary metal-oxide semiconductor (“CMOS”) array. Single linear array 140 comprises a plurality of individual pixels 142. In the illustrated embodiment, the single linear array 140 has 4,096 pixels 142. In alternative embodiments, linear array 140 may have more or fewer pixels. For example, common formats of linear arrays include 512, 1,024, and 4,096 pixels. Pixels 142 are arranged in a linear fashion to define a field of view 134 for linear array 140. The size of field of view 134 varies in accordance with the magnification of scanning system 100.

[0037] FIG. 1C illustrates a line-scan camera 130 having three linear arrays 140, each of which may be implemented as a CCD array. The three linear arrays 140 combine to form a color array 150. In an embodiment, each individual linear array in color array 150 detects a different color intensity, including, for example, red, green, or blue. The color image data from each individual linear array 140 in color array 150 is combined to form a single field of view 134 of color image data. Other types of color line scan cameras, such as 4-line RGB / mono or RGB / NIR, Bayer mask, prism-based 3 R / G / B sensors may also be used in embodiments in which the disclosed technology is used in a color imaging context.

[0038] FIG. ID illustrates a line-scan camera 130 having a plurality of linear arrays 140, each of which may be implemented as a CCD array. The plurality of linear arrays 140 combine to form a TDI array 160. Advantageously, a TDI line-scan camera may provide a substantially better SNR in its output signal by summing intensity data from previously imaged regions of a specimen, yielding an increase in the SNR that is in proportion to the square-root of the number of linear arrays 140 (also referred to as integration stages). A TDI line-scan camera may comprise a larger variety of numbers of linear arrays 140. For example, common formats of TDI line-scan cameras include 24, 32, 48, 64, 96, 120, and even more linear arrays 140.

[0039] 2. Quality Control Process and Application

[0040] A high level illustration of a method which may be used for automatically applying a quality control process to histological images such as could be created by a scanner as described in the context of FIGS. 1 A-1D is provided in FIG. 2. As shown in that image, after the process starts 201, a determination 202 could be made of whether the histological image to which the process is applied features an artifact. As discussed in more detail below, this may be done, for example, by using one or more machine learning models which are trained for a particular type of artifact to process a histological image to determine whether that image should be treated as featuring the artifact that they are trained for. If this determination 202 is that the image features the artifact, then a second determination 203 could be made as to whether to generate a flag indicating that the histological image being processed features the artifact. If it does, then the flag for that image may be generated 204, for example, by providing an indication in an interface, or storing a flag in a computer readable memory, indicating that the image features that artifact. Alternatively, if the image did not feature the artifact, or if it was determined 203 not to generate the flag despite the image featuring the artifact, then a check 205 could be made of whether there were further artifacts that the image should be analyzed for. For example, in a case where the quality control process was performed using a set of machine learning models, each of which was trained for a particular type of artifact, the check 205 could be made by determining if there were any models which had not yet been used to process the histological image. If there were additional artifacts that needed to be processed then the method of FIG. 2 could proceed 206 to the next artifact (e.g., by incrementing a counter indicating the artifact the histological image was currently being processed for) and repeat. Otherwise, the method may be treated 207 as complete (e.g., the quality control process may then move on to the next image in a batch of histological images created by a scanner such as the scanning device of FIG. 1A).

[0041] In embodiments which include a method such as shown in FIG. 2, the processing performed in that method may be applied in a variety of ways to address artifacts identified in histological images. To illustrate, consider FIG. 3, which shows an interface which may be provided on the display of a scanner (e.g., a display integrated into the scanner itself, or a separate display which is connected to the scanner) based on information generated using a method such as discussed above in the context of FIG. 2. As shown in FIG. 3, such an interface may provide flags which identify the particular artifacts identified for particular histological images, such as the “F” flag shown in FIG. 3 to indicate that slide 15 was out of focus. Additionally, an interface which includes information generated using a method such as FIG. 2 may also provide other information, such as the status of slides with respect to scanning and / or quality control. For example, as shown in FIG. 3, a slide may have a status indicating that an image has been created for that slide, but that the slide has not yet been through quality control (e.g., the “A” status illustrated in FIG. 3). A slide may also be shown with a status indicating that a histological image corresponding that slide is currently being created (e.g., the “S” status illustrated in FIG. 3), or that it has not yet begun being scanned (e.g., the “W” status illustrated in FIG. 3). It is also possible that status information may be provided beyond information for particular slides. For instance, when an instrument is configured to be loaded with a carousel of slides for scanning, if the carousel had one or more slots without slides, those slots may be displayed with their own status, such as the “E” for “Empty” status shown in FIG. 3. Other statuses (e.g., different flags for different artifacts which may be flagged as affecting particular slide images) and other functionality beyond simply displaying statuses (e.g., displaying trends for a scanner, such as numbers of artifacts overall, or particular artifacts, detected over time; allowing a user to view an image generated from a particular slide, etc.) are also possible, and may be provided in different embodiments. Accordingly, the interface of FIG. 3 should be understood as being illustrative only, and should not be treated as limiting.

[0042] Information such as could be generated using a method such as shown in FIG. 2 may also be applied in other ways, such as by allowing a user to obtain a broader view of operations of an overall laboratory. To illustrate, consider FIG. 4, which shows a schematic view of a laboratory which includes a plurality of scanners 401-404 which could be used to create histology images, as well as an information technology manager computer 405 which can be used to provide a dashboard interface 406 with information regarding all of the scanners. In some cases, such a dashboard interface 406 may be provided using a method such as shown in FIG. 5, which includes aggregating 501 the flags generated based on the application of a quality control process to images created on the scanners 401-404, and then using those aggregated flags to generate 502 the dashboard interface. As shown in FIG. 5, the aggregated flags may be matched against stored remediation data to display 503 potential remediations the user may apply to address the issues causing the flagged artifacts. An example of this type of remediation data is provided below in table 1, which identifies illustrative artifacts which may be flagged and steps which could be taken to address those artifacts.Table 1 : example artifacts and potential remediations

[0043] As another example of a potential feature which could be provided by a dashboard interface 406, in some cases a dashboard interface may provide trend information, suchas notifying 504 a user of an increase in artifacts which has exceeded a user-specified trend threshold. This may be, for example, an increase in the overall number (or percentage) of artifacts flagged in images generated by the lab, in the number (or percentage) of a particular type of artifact, and / or the number (or percentage) of artifacts (or artifacts of a particular type) flagged in images generated by a particular scanner. It is also possible that this trend information may be used to provide remediation steps in a manner similar to that described above in the context of table 1. For example, a trend of increasing digital artifacts (e.g., out of focus images, or missing or clipped tissue) in images generated by a particular scanner may cause a notification to be provided to a user that he or she should have the impacted scanner serviced or replaced.

[0044] It should be understood that the applications described above in the context of FIGS. 3- 5 are not intended to exhaustively describe all potential applications of data such as could be generated using a method such as that illustrated in FIG. 2. For example, in some cases, rather than notifying a user of a potential remediation, a remediation may be automatically applied (e.g., if an image is flagged as featuring image striping, then a system implemented using the disclosed technology may automatically rescan that image while using a new calibration point to measure slide opacity for white balance). It is also possible that hybrid manual / automated remediations may be supported in some cases (e g., if a scanner was identified as generating an increasing number of flagged images, a system implemented based on this disclosure may automatically generate a service call for that scanner, while also providing a notification that the flagged images should be regenerated with a different scanner). As another example of a type of variation which may be supported in some cases, it is also possible that aggregated flagging information may be stored and / or provided to data repository for use in downstream analytics. It is also possible that, in implementations which provide potential remediations, more complicated logic may be used for providing those potential remediations than that described above. For example, in the case of a slideflagged for having trapped air, remediation identification logic may consider whether the slide was associated with a retrospective case. If it was, then a potential remediation of modifying the lab’s storage procedure may be provided. Otherwise, then a potential remediation of modifying a mounting step in the lab’s histological workflow may be provided. Other variations are also possible, and could be implemented without undue experimentation by those of skill in the art in light of this disclosure. Accordingly, the above exemplary variations, like the examples given in the context of FIGS. 3-5, should be treated as illustrative only, and should not be treated as limiting the protection provided by this document or any related document.

[0045] 3. Additional Variations

[0046] It should be understood that, while the above disclosure provided various examples of alternative approaches which may be taken in implementing the disclosed technology, those alternative approaches are intended to be illustrative only, and are not intended to be an exhaustive description of all potential variations. To illustrate, consider the determinations which may be made of whether to flag artifacts other than out of focus. While in some cases approaches such as described above in the context of out of focus (e.g., identifying an affected area, comparing that affected area with a threshold to determine whether to flag the image) could be applied to other artifacts (e.g., image striping, chatter, Venetian blinds, stain intensity, presence of dirt & debris on a slide, excessive mounting media, etc.), other approaches are also possible. For example, in some cases, rather than relying on machine learning models, image analysis may be used, such as using edge detection and filtering of horizontal lines that cross an entire image to identify digital striping, and similar line detection could also be applied to identify chatter, though in that case it would preferably be generalized to consider lines of any orientation or locality. As another example of a potential variation, in the case of identifying missing or clipped tissue, an initial image may be captured by a scanner prior to scanning a slide, and the bounding box of the tissue in that initial image may becompared with the bounding box of the tissue in the scanned image to determine whether to flag the scanned image as having missing or clipped tissue (e.g., if more tissue was shown in the initial image than the scanned image, then the scanned image may be flagged for this artifact).

[0047] Variations are also possible even in cases where machine learning is used to implement the affected area identification followed by evaluating whether that area was sufficient for the image to be flagged. For example, it is possible that semantic segmentation (e.g., using a deep learning model such as DeepLabV3 (described in Chen et al., Rethinking Atrous Convolution for Sematic Image Segmentation, available at https: / / arxiv.org / abs / I706.05587, the disclosure of which is hereby incorporated by reference in its entirety) with a ResNet-101 (as described in He, et. al., Deep Residual Learning for Image Recognition, available at htps: / / arxiv.org / abs / 1512.O3385, the disclosure of which is hereby incorporated by reference in its entirety) backbone could be used to identify areas on an image featuring an artifact (e.g., pen marks or other occlusions) on an individual pixel level, rather relying on the tiling approach described above in the context of FIG. 6.

[0048] Variations are also possible in aspects of how the disclosed technology can be implemented beyond the determination of how to flag a particular artifact. For example, in some cases, in addition to flagging an artifact, in some cases the disclosed technology may include functionality to capture the artifact for subsequent recordkeeping or analytics purposes (e.g., a portion of an image identified as having a pen mark occluding tissue may be stored so that any information conveyed by the pen mark would be preserved, even if the slide was cleaned and rescanned to obtain a clear image of the tissue itself). Similarly, in some cases additional functionality may be provided in a dashboard and / or scanner interface such as those discussed above in the context of FIGS. 3 and 4. For example, in an implementation which uses thresholds to determine reporting (e.g., severity thresholds for determining whether an area is affected, areathresholds for determining whether an affected area should trigger a flag, trend thresholds to determine if a user should be notified of a change in artifact detection) the ability for the user to modify one or more of those thresholds may be provided by the scanner interface, the dashboard interface, or both. In cases where this type of threshold modification is supported, there may also be functionality provided for applying threshold changes selectively, such as to a particular scanner, to a particular batch of slides, or even to individual slides themselves.

[0049] In some cases, there may also be variations on implementing a high level method such as shown in FIG. 2. For example, while it is possible that some implementations following a method such as shown in FIG. 2 may determine if each artifact is featured in each image, it is also possible that the disclosed technology may be implemented such that once certain types of artifacts are identified (e.g., missing tissue), the analysis of an image may cease so that the identified artifact can be addressed (e.g., by rescanning a slide, which may render other artifacts identified in the original image moot). Similarly, while some implementations may identify and generate flags for artifacts on an artifact by artifact basis, it is also possible that some implementations of the disclosed technology may identify all artifacts for an image and then generate the flags for those artifacts at the same time, rather than both identifying and generating flags for each artifact before moving to the next.

[0050] Variations are also possible in the physical components which may be used in implementing the disclosed technology. For example, while in some cases the identification and generation of flags for artifacts in an image may be performed on the scanner on which the image was captured, it is also possible that the images may be sent to a separate processor (e.g., a centralized computer for a lab, or a cloud server accessible over a wide area network) for the identification of artifacts and generation of flags. It is also possible that different portions of various methods may be performed by different physical devices. For example, in some cases an image may be sent to a remotesystem (e.g., a cloud server) for application of machine learning models (e g., for semantic segmentation or determining severity scores) and the results may then be returned to a lab (or scanner) for comparison with thresholds to determine whether to generate a flag. Other architectures and approaches to performing the processing tasks described herein are also possible, and may be implemented without undue experimentation by those skilled in the art, and so the example variations described above should be understood as being illustrative only, and should not be treated as limiting.

[0051] 5. Additional Non-Limiting Examples

[0052] The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles described herein can be applied to other embodiments without departing from the spirit or scope of the invention. Thus, it is to be understood that the description and drawings presented herein represent a presently preferred embodiment of the invention and are therefore representative of the subject matter which is broadly contemplated by the present invention. It is further understood that the scope of the present invention fully encompasses other embodiments that may become obvious to those skilled in the art and that the scope of the present invention is accordingly not limited. Other variations are also possible, and will be immediately apparent to those of skill in the art in light of this disclosure. For instance, the following examples are provided as concrete (though non-limiting) illustrations of various approaches which could be taken when implementing the disclosed technology.

[0053] Example 1

[0054] A method comprising automatically, through execution of computer executable instructions, apply a quality control process to a plurality of histological images,wherein: the quality control process comprises, for each artifact from a set of artifacts: determining whether a histological image to which the quality control process is applied features that artifact; and based on determining that the histological image to which the quality control process is applied features that artifact, determining whether to generate a flag indicating that the histological image to which the quality control process is applied features that artifact; and for a first image from the plurality of histological images, applying the quality control process to that image comprises: determining that that image features a first artifact from the set of artifacts; determining whether to generate the flag indicating that that image features the first artifact; and generating the flag indicating that the first image features the first artifact

[0055] Example 2

[0056] The method of example 1, wherein: determining that the first image features the first artifact comprises determining an area on the first image that features the first artifact; and determining whether to generate the flag indicating that the first image features the first artifact comprises comparing the area on the first image that features the first artifact with an affected area threshold for the first artifact.

[0057] Example 3

[0058] The method of example 2, wherein determining the area on the first image that features the first artifact comprises, for each of a plurality of portions of the first image: determining a severity of the first artifact in that portion of the first image; and comparing the severity of the first artifact in that portion of the first image with a severity threshold for the first artifact.

[0059] Example 4

[0060] The method of example 3, wherein the method comprises providing an interface operable by a user to specify: the affected area threshold for the first artifact; and the severity threshold for the first artifact.

[0061] Example 5

[0062] The method of any of examples 1-4, wherein: the plurality of histological images comprises a plurality of sets of histological images; for each set of histological images from the plurality of sets of histological images; that set of histological images corresponds to a scanner from a set of scanners; no scanner from the set of scanners corresponds to more than one set of histological images from the plurality of sets of histological images; the histological images in that set of histological images are generated by, for each slide from a set of slides corresponding to that set of images, scanning that slide using the scanner corresponding to that set of histological images; and the method comprises generating a dashboard interface based on aggregating each flag generated based on applying the quality control process to each histological image from the plurality of sets of histological images.

[0063] Example 6

[0064] The method of example 5, wherein the method comprises displaying, via the dashboard interface, a potential remediation for at least one artifact from the set of artifacts.

[0065] Example 7

[0066] The method of any of examples 5-6, wherein the method comprises notifying the user, via the dashboard interface, of an increase in artifacts based on the increase in artifacts exceeding a user-specified trend threshold.

[0067] Example 8

[0068] The method of any of examples 1-7, wherein the method comprises displaying, on a display of a first scanner, a scanner interface comprising the flag indicating that the first image features the first artifact, wherein the first image is a histological image created by scanning a slide using the first scanner.

[0069] Example 9

[0070] The method of example 8, wherein the method comprises displaying, via the scanner interface, that, for at least one slide from the plurality of slides: a histological image corresponding to that slide has been created; and the histological image corresponding to that slide has not yet had the quality control process applied to it.

[0071] Example 10

[0072] The method of any of examples 1-9, wherein the set of artifacts comprises: image striping, trapped air, marks occluding tissue, missing tissue, and out of focus.

[0073] Example 11

[0074] A system comprising: a first scanner, wherein the first scanner is operable to generate a plurality of histological images based on, for each slide from a plurality of slides, scanning tissue on that slide; one or more non-transitory computer readable media storing instructions to, when executed, apply a quality control process to one or more histological images, wherein the quality control process is operable to, for each histological image to which it is applied, determine, for each artifact from a set of artifacts: whether the histological image to which the quality control process is applied features that artifact; and based on determining that the histological image to which the quality control process is applied features that artifact, whether to generate a flag indicating that the histological image to which the quality control process is applied features that artifact.

[0075] Example 12

[0076] The system of example 11, wherein, for a first artifact from the set of artifacts: the quality control process determining whether to generate the flag indicating that the histological image to which the quality control process is applied features the first artifact comprises: determining an area on the histological image to which the quality control process is applied that features the first artifact; and comparing the area on the image to which the quality control process is applied that features the first artifact with an affected area threshold for the first artifact.

[0077] Example 13

[0078] The system of example 12, wherein determining the area on the image to which the quality control process is applied that features the first artifact comprises, for each a plurality of portions of the image to which the quality control process is applied: determining a severity of the first artifact in that portion of the image to which the quality control process is applied; and comparing the severity of the first artifact in that portion of the image to which the quality control process is applied with a severity threshold for the first artifact.

[0079] Example 14

[0080] The system of example 13, wherein the one or more non-transitory computer readable media store instructions to, when executed, provide an interface operable by a user to specify: the affected area threshold for the first artifact; and the severity threshold for the first artifact.

[0081] Example 15

[0082] The system of any of examples 11-14, wherein: the system comprises a set of scanners, which set of scanners comprises the first scanner; each scanner from the set of scannersis operable to generate a set of histological images; the one or more non-transitory computer readable media comprise instructions to, when executed: apply the quality control process to each histological image generated by the set of scanners; generate a dashboard interface based on aggregating each flag generated based on applying the quality control process to each histological image generated by the set of scanners.

[0083] Example 16

[0084] The system of example 15, wherein the dashboard interface is configured to present at a potential remediation for at least one artifact from the set of artifacts.

[0085] Example 17

[0086] The system of any of examples 15-16, wherein the one or more non-transitory computer readable media comprise instructions operable to, when executed, notify a user, via the dashboard interface, of an increase in artifacts based on the increase in artifacts exceeding a user-specified trend threshold.

[0087] Example 18

[0088] The system of any of examples 11-17, wherein the one or more non-transitory computer readable media comprise instructions to, when executed, display, on a display of the first scanner, the scanner interface presenting each flag generated by applying the quality control process to the plurality of histological images.

[0089] Example 19

[0090] The system of example 18, wherein the scanner interface is configured to display, for each slide from the plurality of slides a status from a set of statuses comprising: a status indicating that a histological image corresponding to that slide has not yet been generated; a status indicating that the histological image corresponding to that slide hasbeen generated but not yet had the quality control process applied to it; and a status indicating that the histological image corresponding to that slide has had the quality control process applied to it.

[0091] Example 20

[0092] The system of any of examples 11-19, wherein the set of artifacts comprises: image striping; trapped air; marks occluding tissue; missing tissue; and out of focus.

[0093] 6. Interpretation

[0094] None of the examples or illustrations set forth herein should be understood as implying limitations on the scope of any claims included in this document or any related document. Instead, the protection provided by this document or any related document, should be understood as being defined by the relevant document’s claims, when the terms in those claims which are explicitly defined herein are given their explicit definitions, and the terms which are not explicitly defined are given their broadest reasonable interpretation as provided by a general purpose dictionary.

[0095] Combinations, described herein, such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof’ include any combination of A, B, and / or C, and may include multiples of A, multiples of B, or multiples of C. Specifically, combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof’ may be A only, B only, C only, A and B, A and C, B and C, or A and B and C, and any such combination may contain one or more members of its constituents A, B, and / or C. For example, a combination of A and B may comprise one A and multiple B’s, multiple A’s and one B, or multiple A’s and multiple B’s.

[0096] A statement that something is “based on” something else should be understood as meaning that that thing is determined at least in part by that which it is “based on.” While the “based on” relationship includes scenarios in which one thing is completely determined by another, the “based on” relationship should not be understood as being limited to only scenarios in which one thing is completely determined by another unless the phrase used is “based exclusively on.”

Claims

CLAIMSWhat is claimed is:

1. A method comprising automatically, through execution of computer executable instructions, apply a quality control process to a plurality of histological images, wherein: the quality control process comprises, for each artifact from a set of artifacts:• determining whether a histological image to which the quality control process is applied features that artifact; and• based on determining that the histological image to which the quality control process is applied features that artifact, determining whether to generate a flag indicating that the histological image to which the quality control process is applied features that artifact; and for a first image from the plurality of histological images, applying the quality control process to that image comprises:• determining that that image features a first artifact from the set of artifacts;• determining whether to generate the flag indicating that that image features the first artifact; and• generating the flag indicating that the first image features the first artifact.

2. The method of claim 1, wherein: determining that the first image features the first artifact comprises determining an area on the first image that features the first artifact; and determining whether to generate the flag indicating that the first image features the first artifact comprises comparing the area on the first image that features the first artifact with an affected area threshold for the first artifact.

3. The method of claim 2, wherein determining the area on the first image that features the first artifact comprises, for each of a plurality of portions of the first image: determining a severity of the first artifact in that portion of the first image; and comparing the severity of the first artifact in that portion of the first image with a severity threshold for the first artifact.

4. The method of claim 3, wherein the method comprises providing an interface operable by a user to specify: the affected area threshold for the first artifact; and the severity threshold for the first artifact.

5. The method of claim 1, wherein: the plurality of histological images comprises a plurality of sets of histological images; for each set of histological images from the plurality of sets of histological images:• that set of histological images corresponds to a scanner from a set of scanners;• no scanner from the set of scanners corresponds to more than one set of histological images from the plurality of sets of histological images;• the histological images in that set of histological images are generated by, for each slide from a set of slides corresponding to that set of images, scanning that slide using the scanner corresponding to that set of histological images; and the method comprises generating a dashboard interface based on aggregating each flag generated based on applying the quality control process to each histological image from the plurality of sets of histological images.

6. The method of claim 5, wherein the method comprises displaying, via the dashboard interface, a potential remediation for at least one artifact from the set of artifacts.

7. The method of claim 5, wherein the method comprises notifying the user, via the dashboard interface, of an increase in artifacts based on the increase in artifacts exceeding a user-specified trend threshold.

8. The method of claim 1, wherein the method comprises displaying, on a display of a first scanner, a scanner interface comprising the flag indicating that the first image features the first artifact, wherein the first image is a histological image created by scanning a slide using the first scanner.

9. The method of claim 8, wherein the method comprises displaying, via the scanner interface, that, for at least one slide from the plurality of slides: a histological image corresponding to that slide has been created; and the histological image corresponding to that slide has not yet had the quality control process applied to it.

10. The method of claim 1, wherein the set of artifacts comprises: image striping; trapped air; marks occluding tissue; missing tissue; and out of focus.

11. A system comprising:a first scanner, wherein the first scanner is operable to generate a plurality of histological images based on, for each slide from a plurality of slides, scanning tissue on that slide; one or more non-transitory computer readable media storing instructions to, when executed, apply a quality control process to one or more histological images, wherein the quality control process is operable to, for each histological image to which it is applied, determine, for each artifact from a set of artifacts:• whether the histological image to which the quality control process is applied features that artifact; and• based on determining that the histological image to which the quality control process is applied features that artifact, whether to generate a flag indicating that the histological image to which the quality control process is applied features that artifact.

12. The system of claim 11, wherein, for a first artifact from the set of artifacts: the quality control process determining whether to generate the flag indicating that the histological image to which the quality control process is applied features the first artifact comprises:• determining an area on the histological image to which the quality control process is applied that features the first artifact; and• comparing the area on the image to which the quality control process is applied that features the first artifact with an affected area threshold for the first artifact.

13. The system of claim 12, wherein determining the area on the image to which the quality control process is applied that features the first artifact comprises, for each a plurality of portions of the image to which the quality control process is applied:determining a severity of the first artifact in that portion of the image to which the quality control process is applied; and comparing the severity of the first artifact in that portion of the image to which the quality control process is applied with a severity threshold for the first artifact.

14. The system of claim 13, wherein the one or more non-transitory computer readable media store instructions to, when executed, provide an interface operable by a user to specify: the affected area threshold for the first artifact; and the severity threshold for the first artifact.

15. The system of claim 11, wherein: the system comprises a set of scanners, which set of scanners comprises the first scanner; each scanner from the set of scanners is operable to generate a set of histological images; the one or more non-transitory computer readable media comprise instructions to, when executed:• apply the quality control process to each histological image generated by the set of scanners;• generate a dashboard interface based on aggregating each flag generated based on applying the quality control process to each histological image generated by the set of scanners.

16. The system of claim 15, wherein the dashboard interface is configured to present at a potential remediation for at least one artifact from the set of artifacts.

17. The system of claim 15, wherein the one or more non-transitory computer readable media comprise instructions operable to, when executed, notify a user, via the dashboardinterface, of an increase in artifacts based on the increase in artifacts exceeding a user- specified trend threshold.

18. The system of claim 11, wherein the one or more non-transitory computer readable media comprise instructions to, when executed, display, on a display of the first scanner, the scanner interface presenting each flag generated by applying the quality control process to the plurality of histological images.

19. The system of claim 18, wherein the scanner interface is configured to display, for each slide from the plurality of slides a status from a set of statuses comprising: a status indicating that a histological image corresponding to that slide has not yet been generated; a status indicating that the histological image corresponding to that slide has been generated but not yet had the quality control process applied to it; and a status indicating that the histological image corresponding to that slide has had the quality control process applied to it.

20. The system of claim 11, wherein the set of artifacts comprises: image striping; trapped air; marks occluding tissue; missing tissue; and out of focus.

Images