Artificial intelligence supported adaptive generation of histological images

Abstract

A histological image can be generated using an adaptive method supported by artificial intelligence. Such a method may include capturing a first image of a slide, and obtaining a classification for the slide based on providing the first image to a machine learning model. Once the classification had been obtained, such a method may continue with determining a parameter set based on the classification for the slide, and capturing a second image of the slide using that parameter set.

Description

ARTIFICIAL INTELLIGENCE SUPPORTED ADAPTIVE GENERATION OF HISTOLOGICAL IMAGESBACKGROUNDFIELD

[0001] The embodiments described herein are generally directed to methods of configuring a whole slide imaging device, and, more particularly, to automatically determining a tissue sample type and configuring a whole slide imaging device based on the automatic tissue sample type determination.RELATED ART

[0002] Digital pathology is an image-based information environment, which is enabled by computer technology that allows for the management of information generated from a physical slide. Digital pathology is enabled in part by virtual microscopy, which is the practice of scanning a specimen on a physical glass slide, and creating a digital slide image that can be stored, viewed, managed, and analyzed on a computer monitor. With the capability of imaging an entire glass slide, the field of digital pathology has exploded, and is currently regarded as one of the most promising avenues of diagnostic medicine in order to achieve even better, faster, and cheaper diagnosis, prognosis, and prediction of important diseases, such as cancer.

[0003] A primary objective for the digital pathology industry is to produce high quality images. Poor image quality can sometimes cause failure modes to be triggered. Failure modes can be triggered by a variety of issues such as missing tissue, an incomplete scan, a scan being out of focus in whole or in part, a stitching error, or other error. Accordingly, there is a need for improved technology for preventing these failure modes by identifying the tissue sample type present in a slide and configuring the whole slide imaging device parameters accordingly.SUMMARY

[0004] The disclosed technology may be implemented in a variety of manners. For example, in some aspects, histological image can be generated using an adaptive method supported by artificial intelligence. Such a method may include capturing a first image of a slide, and obtaining a classification for the slide based on providing the first image to a machine learning model. Once the classification had been obtained, such a method may continue with determining a parameter set based on the classification for the slide, and capturing a second image of the slide using that parameter set.

[0005] Other types of implementations, including in the form of systems and computer readable media for performing such methods, are also possible and will be immediately apparent to those of skill in the art in light of this disclosure. Accordingly, the example method provided in this summary should be understood as being illustrative only, and should not be treated as limiting.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The details of aspects of the disclosed technology, both as to its structure and operation, may be gleaned in part by study of the accompanying drawings, in which like reference numerals refer to like parts, and in which:

[0007] FIG. 1 A illustrates an example processor-enabled device that may be used in connection with the various embodiments described herein;

[0008] FIG. IB illustrates an example line-scan camera having a single linear array, according to an embodiment;

[0009] FIG. 1C illustrates an example line-scan camera having three linear arrays, according to an embodiment;

[0010] FIG. ID illustrates an example line-scan camera having a plurality of linear arrays, according to an embodiment;

[0011] FIG. 2 illustrates an example scanning process

[0012] FIG. 3 illustrates an example adaptive scanning process;

[0013] FIG. 4 illustrates an architecture which may be used in a machine learning model that is trained to classify images;

[0014] FIG. 5 illustrates a layer which may be included in an architecture such as shown in FIG. 4;

[0015] FIG. 6 illustrates a system architecture which may be used in some implementations;

[0016] FIG. 7A provides an example image of a regular tissue sample;

[0017] FIG. 7B provides an example image of a regular tissue sample;

[0018] FIG. 8 provides an example image of a tissue micro array (TMA) tissue sample;

[0019] FIG. 9 provides an example image of a needle biopsy tissue sample;

[0020] FIG. 10 provides an example image of a cytology sample;

[0021] FIG. 11 provides an example image of a ribonucleic acid (RNA) in situ hybridization (ISH) tissue sample; and

[0022] FIG. 12 provides an example of a system which may be implemented based on this disclosure.DETAILED DESCRIPTION

[0023] Disclosed herein is technology which can be used to automatically adapt to requirements of particular slide when generating histological images. After reading this description, it will become apparent to one skilled in the art how to implement the invention in various alternative embodiments and alternative applications. However, although various embodiments of the present invention will be described herein, it is understood that these embodiments are presented by way of example and illustration only, and not limitation. As such, this detailed description of various embodiments should not beconstrued to limit the scope or breadth of the present invention as set forth in the appended claims.

[0024] 1. Example Scanning System

[0025] FIG. 1A is a block diagram illustrating an example processor-enabled slide-scanning system 100 that may be used in connection with various embodiments described herein. Alternative forms of scanning system 100 may also be used as will be understood by the skilled artisan. In the illustrated embodiment, scanning system 100 is presented as a digital imaging device that comprises one or more processors 104, one or more memories 106, one or more motion controllers 108, one or more interface systems 110, one or more movable stages 112 that each support one or more glass slides 114 with one or more samples 116, one or more illumination systems 118 that illuminate sample 116, imaging optics 103 such as one or more objective lenses 120 that each define an optical path 122 that travels along an optical axis, one or more objective lens positioners 124, one or more optional epi-illumination systems 126 (e.g., included in a fluorescencescanning embodiment), one or more focusing optics 128, and one or more line-scan cameras 130, and / or one or more area-scan cameras 132, each of which define a field of view 134 on sample 116 and / or glass slide 114. The various elements of scanning system 100 are communicatively coupled via one or more communication busses 102. Although there may be a plurality of each of the various elements of scanning system 100, for simplicity in the description that follows, these elements will be described in the singular, except when needed to be described in the plural to convey the appropriate information.

[0026] Processor 104 may include, for example, a central processing unit (CPU) and a separate graphics processing unit (GPU) capable of processing instructions in parallel, or a multi core processor capable of processing instructions in parallel. Additional separate processors may also be provided to control particular components or perform particular functions, such as image processing. For example, additional processors may include an auxiliary processor to manage data input, an auxiliary processor to perform floatingpoint mathematical operations, a special-purpose processor having an architecturesuitable for fast execution of signal-processing algorithms (e.g., digital-signal processor), a slave processor subordinate to the main processor (e.g., back-end processor), an additional processor for controlling line-scan camera 130, stage 112, objective lens 120, and / or a display (e.g., a console comprising a touch panel display integral to scanning system 100). Such additional processors may be separate discrete processors or may be integrated into a single processor.

[0027] Memory 106 provides storage of data and instructions for programs that can be executed by processor 104. Memory 106 may include one or more volatile and / or non-volatile computer-readable storage mediums that store the data and instructions. These mediums may include, for example, random-access memory (RAM), read-only memory (ROM), a hard disk drive, a removable storage drive (e.g., comprising flash memory), and / or the like. Processor 104 is configured to execute instructions that are stored in memory 106, and communicate via communication bus 102 with the various elements of scanning system 100 to carry out the overall function of scanning system 100.

[0028] Communication bus 102 may be configured to convey analog electrical signals and / or digital data. Accordingly, communications from processor 104, motion controller 108, and / or interface system 110, via communication bus 102, may include both electrical signals and digital data. Processor 104, motion controller 108, and / or interface system 110 may also be configured to communicate with one or more of the various elements of scanning system 100 via a wireless communication link.

[0029] Motion control system 108 is configured to precisely control and coordinate X, Y, and / or Z movement of stage 112 (e.g., within an X-Y plane), X, Y, and / or Z movement of objective lens 120 (e.g., along a Z axis orthogonal to the X-Y plane, via objective lens positioner 124), rotational movement of a carousel described elsewhere herein, lateral movement of a push / pull assembly described elsewhere herein, and / or any other moving component of scanning system 100. For example, in a fluorescence-scanning embodiment comprising epi-illumination system 126, motion control system 108 may be configured to coordinate movement of optical filters and / or the like in epi- illumination system 126.

[0030] Interface system 110 allows scanning system 100 to interface with other systems and human operators. For example, interface system 110 may include a console (e.g., a touch panel display) to provide information directly to an operator via a graphical user interface and / or allow direct input from an operator via a touch sensor. Interface system 110 may also be configured to facilitate communication and data transfer between scanning system 100 and one or more external devices that are directly connected to scanning system 100 (e.g., a printer, removable storage medium, etc.), and / or one or more external devices that are indirectly connected to scanning system 100, for example, via one or more networks (e.g., an image storage system, a Scanner Administration Manager (SAM) server and / or other administrative server, an operator station, a user station, etc.).

[0031] Illumination system 118 is configured to illuminate at least a portion of sample 116. Illumination system 118 may include, for example, one or more light sources and illumination optics. The light source(s) could comprise a variable intensity halogen light source with a concave reflective mirror to maximize light output and a KG-1 filter to suppress heat. The light source(s) could comprise any type of arc-lamp, laser, or other source of light. In an embodiment, illumination system 118 illuminates sample 116 in transmission mode, such that line-scan camera 130 and / or area-scan camera 132 sense optical energy that is transmitted through sample 116. Alternatively or additionally, illumination system 118 may be configured to illuminate sample 116 in reflection mode, such that line-scan camera 130 and / or area-scan camera 132 sense optical energy that is reflected from sample 116. Illumination system 118 may be configured to be suitable for interrogation of sample 116 in any known mode of optical microscopy.

[0032] In an embodiment, scanning system 100 includes an epi-illumination system 126 to optimize scanning system 100 for fluorescence scanning. It should be understood that, if fluorescence scanning is not supported by scanning system 100, epi-illumination system 126 may be omitted. Fluorescence scanning is the scanning of samples 116 that include fluorescence molecules, which are photon-sensitive molecules that can absorb light at a specific wavelength (i.e., excitation). These photon-sensitive molecules alsoemit light at a higher wavelength (i.e., emission). Because the efficiency of this photoluminescence phenomenon is very low, the amount of emitted light is often very low. This low amount of emitted light typically frustrates conventional techniques for scanning and digitizing sample 116 (e.g., transmission-mode microscopy).

[0033] Advantageously, in an embodiment of scanning system 100 that utilizes fluorescence scanning, use of a line-scan camera 130 that includes multiple linear sensor arrays (e.g., a time-delay-integration (TDI) line-scan camera) increases the sensitivity to light of line-scan camera 130 by exposing the same area of sample 116 to each of the plurality of linear sensor arrays of line-scan camera 130. This is particularly useful when scanning faint fluorescence samples with low levels of emitted light. Accordingly, in a fluorescence-scanning embodiment, line-scan camera 130 is preferably a monochrome TDI line-scan camera. Monochrome images are ideal in fluorescence microscopy because they provide a more accurate representation of the actual signals from the various channels present on sample 116. As will be understood by those skilled in the art, a fluorescence sample can be labeled with multiple florescence dyes that emit light at different wavelengths, which are also referred to as “channels.”

[0034] Furthermore, because the low-end and high-end signal levels of various fluorescence samples present a wide spectrum of wavelengths for line-scan camera 130 to sense, it is desirable for the low-end and high-end signal levels that line-scan camera 130 can sense to be similarly wide. Accordingly, in a fluorescence-scanning embodiment, line-scan camera 130 may comprise a monochrome 10-bit 64-linear-array TDI line-scan camera. It should be noted that a variety of bit depths for line-scan camera 130 can be employed for use with such an embodiment.

[0035] Movable stage 112 is configured for precise X-Y movement under control of processor 104 or motion controller 108. Movable stage 112 may also be configured for Z movement under control of processor 104 or motion controller 108. Movable stage 112 is configured to position sample 116 in a desired location during image data capture by line-scan camera 130 and / or area-scan camera 132. Movable stage 112 is also configured to accelerate sample 116 in a scanning direction to a substantially constantvelocity, and then maintain the substantially constant velocity during image data capture by line-scan camera 130. In an embodiment, scanning system 100 may employ a high- precision and tightly coordinated X-Y grid to aid in the location of sample 116 on movable stage 112. In an embodiment, movable stage 112 is a linear-motor-based X-Y stage with high-precision encoders employed on both the X and the Y axes. For example, very precise nanometer encoders can be used on the axis in the scanning direction and on the axis that is in the direction perpendicular to the scanning direction and on the same plane as the scanning direction. Stage 112 is also configured to support glass slide 114 upon which sample 116 is disposed.

[0036] Sample 116 can be anything that may be interrogated by optical microscopy. For example, glass microscope slide 114 is frequently used as a viewing substrate for specimens that include tissues and cells, chromosomes, deoxyribonucleic acid (DNA), protein, blood, bone marrow, urine, bacteria, beads, biopsy materials, or any other type of biological material or substance that is either dead or alive, stained or unstained, labeled or unlabeled. Sample 116 may also be an array of any type of DNA or DNA- related material, such as complementary DNA (cDNA) or ribonucleic acid (RNA), or protein that is deposited on any type of slide or other substrate, including any and all samples commonly known as microarrays. Sample 116 may be a microtiter plate (e.g., a 96-well plate). Other examples of sample 116 include integrated circuit boards, electrophoresis records, petri dishes, film, semiconductor materials, forensic materials, and machined parts.

[0037] Objective lens 120 is mounted on objective positioner 124, which, in an embodiment, employs a very precise linear motor to move objective lens 120 along the optical axis defined by objective lens 120. For example, the linear motor of objective lens positioner 124 may include a fifty-nanometer encoder. The relative positions of stage 112 and objective lens 120 in X, Y, and / or Z axes are coordinated and controlled in a closed- loop manner using motion controller 108 under the control of processor 104 that employs memory 106 for storing information and instructions, including the computerexecutable programmed steps for overall operation of scanning system 100.

[0038] In an embodiment, objective lens 120 is a plan apochromatic (“APO”) infinity-corrected objective lens which is suitable for transmission-mode illumination microscopy, reflection-mode illumination microscopy, and / or epi-illumination-mode fluorescence microscopy (e.g., an Olympus 40X, 0.75NA or 20X, 0.75 NA). Advantageously, objective lens 120 is capable of correcting for chromatic and spherical aberrations. Because objective lens 120 is infinity-corrected, focusing optics 128 can be placed in optical path 122 above objective lens 120 where the light beam passing through objective lens 120 becomes a collimated light beam. Focusing optics 128 focus the optical signal captured by objective lens 120 onto the light-responsive elements of linescan camera 130 and / or area-scan camera 132, and may include optical components such as filters, magnification changer lenses, and / or the like. Objective lens 120, combined with focusing optics 128, provides the total magnification for scanning system 100. In an embodiment, focusing optics 128 may contain a tube lens and an optional 2X magnification changer. Advantageously, the 2X magnification changer allows a native 20X objective lens 120 to scan sample 116 at 40X magnification.

[0039] Line-scan camera 130 comprises at least one linear array of picture elements 142 (“pixels”). Line-scan camera 130 may be monochrome or color. Color line-scan cameras typically have at least three linear arrays, while monochrome line-scan cameras may have a single linear array or plural linear arrays. Any type of singular or plural linear array, whether packaged as part of a camera or custom-integrated into an imaging electronic module, can also be used. For example, a three linear array (“red-green-blue” or “RGB”) color line-scan camera or a ninety-six linear array monochrome TDI may also be used. TDI line-scan cameras typically provide a substantially better signal-to- noise ratio (“SNR”) in the output signal by summing intensity data from previously imaged regions of a specimen, yielding an increase in the SNR that is in proportion to the square-root of the number of integration stages. TDI line-scan cameras comprise multiple linear arrays. For example, TDI line-scan cameras are available with 24, 32, 48, 64, 96, or even more linear arrays. Scanning system 100 also supports linear arrays that are manufactured in a variety of formats including some with 512 pixels, some with 1,024 pixels, and others having as many as 4,096 pixels. Similarly, linear arrays with avariety of pixel sizes can also be used in scanning system 100. The salient requirement for the selection of any type of line-scan camera 130 is that the motion of stage 112 can be synchronized with the line rate of line-scan camera 130, so that stage 112 can be in motion with respect to line-scan camera 130 during the digital image capture of sample 116.

[0040] In an embodiment, the image data generated by line-scan camera 130 is stored in a portion of memory 106 and processed by processor 104 to generate a contiguous digital image of at least a portion of sample 116. The contiguous digital image can be further processed by processor 104, and the processed contiguous digital image can also be stored in memory 106.

[0041] In an embodiment with two or more line-scan cameras 130, at least one of the line-scan cameras 130 can be configured to function as a focusing sensor that operates in combination with at least one of the other line-scan cameras 130 that is configured to function as an imaging sensor. The focusing sensor can be logically positioned on the same optical axis as the imaging sensor or the focusing sensor may be logically positioned before or after the imaging sensor with respect to the scanning direction of scanning system 100. In such an embodiment with at least one line-scan camera 130 functioning as a focusing sensor, the image data generated by the focusing sensor may be stored in a portion of memory 106 and processed by processor 104 to generate focus information, to allow scanning system 100 to adjust the relative distance between sample 116 and objective lens 120 to maintain focus on sample 116 during scanning. Additionally, in an embodiment, the at least one line-scan camera 130 functioning as a focusing sensor may be oriented such that each of a plurality of individual pixels 142 of the focusing sensor is positioned at a different logical height along the optical path 122.

[0042] In operation, the various components of scanning system 100 and the programmed modules stored in memory 106 enable automatic scanning and digitizing of sample 116, which is disposed on glass slide 114. Glass slide 114 is securely placed on movable stage 112 of scanning system 100 for scanning sample 116. Under control of processor 104, movable stage 112 accelerates sample 116 to a substantially constant velocity forsensing by line-scan camera 130. After scanning a segment of image data, movable stage 112 decelerates and brings sample 116 to a substantially complete stop. Movable stage 112 then moves orthogonal to the scanning direction to position sample 116 for scanning of a subsequent segment of image data (e.g., an adjacent segment). Additional segments are subsequently scanned until an entire portion of sample 116 or the entire sample 116 is scanned.

[0043] In an embodiment, computer-executable instructions (e.g., programmed modules and software) are stored in memory 106 and, when executed, enable scanning system 100 to perform the various functions (e.g., display the graphical user interface, execute the disclosed processes, control the components of scanning system 100, etc.) described herein. In this description, the term “computer-readable storage medium” is used to refer to any media used to store and provide computer-executable instructions to scanning system 100 for execution by processor 104. Examples of these media include memory 106 and any removable or external storage medium (not shown) communicatively coupled with scanning system 100 either directly (e.g., via a universal serial bus (USB), a wireless communication protocol, etc.) or indirectly (e.g., via a wired and / or wireless network).

[0044] FIG. IB illustrates a line-scan camera 130 having a single linear array 140, which may be implemented as a charge-coupled device (“CCD”) or complimentary metal-oxide semiconductor (“CMOS”) array. Single linear array 140 comprises a plurality of individual pixels 142. In the illustrated embodiment, the single linear array 140 has 4,096 pixels 142. In alternative embodiments, linear array 140 may have more or fewer pixels. For example, common formats of linear arrays include 512, 1,024, and 4,096 pixels. Pixels 142 are arranged in a linear fashion to define a field of view 134 for linear array 140. The size of field of view 134 varies in accordance with the magnification of scanning system 100.

[0045] FIG. 1C illustrates a line-scan camera 130 having three linear arrays 140, each of which may be implemented as a CCD array. The three linear arrays 140 combine to form a color array 150. In an embodiment, each individual linear array in color array 150detects a different color intensity, including, for example, red, green, or blue. The color image data from each individual linear array 140 in color array 150 is combined to form a single field of view 134 of color image data. Other types of color line scan cameras, such as 4-line RGB / mono or RGB / NIR, Bayer mask, prism-based 3 R / G / B sensors may also be used in embodiments in which the disclosed technology is used in a color imaging context.

[0046] FIG. ID illustrates a line-scan camera 130 having a plurality of linear arrays 140, each of which may be implemented as a CCD array. The plurality of linear arrays 140 combine to form a TDI array 160. Advantageously, a TDI line-scan camera may provide a substantially better SNR in its output signal by summing intensity data from previously imaged regions of a specimen, yielding an increase in the SNR that is in proportion to the square-root of the number of linear arrays 140 (also referred to as integration stages). A TDI line-scan camera may comprise a larger variety of numbers of linear arrays 140. For example, common formats of TDI line-scan cameras include 24, 32, 48, 64, 96, 120, and even more linear arrays 140.

[0047] 2. Example Scanning Process

[0048] In a system based on the described technology, images may be captured using a scanning process such as shown in FIG. 2. As shown in that figure, scanning may begin with capturing 201 a macro image of the slide to be scanned. This macro image may be an image which has lower resolution than the ultimate image created by scanning the slide, and it may be used for purposes such as identifying where the tissue is located on the slide so that the scanning process can be appropriately targeted. This macro image may be captured 201 by a system such as shown in FIG. 1A using an area scan camera 132, with the line scan camera 130 being used for subsequent (and slower) capture of a higher resolution whole slide image.

[0049] After the macro image has been captured 201, it may be used in a variety of activities which are preparatory for the scan itself. For example, as shown in FIG. 2, a scanning process may include tissue finding 202 in preparation for ultimately scanning the slide.This tissue finding 202 may include steps such as noise correction (e.g., applying various filters, such as gaussian filters, to remove noise from the macro image), dirt removal (e.g., removing aspects of the macro image which appear to be dirt or other extraneous obscurations, such as could be identified using a convolutional neural network or other trained machine learning model, from the macro image), coverslip detection (e.g., identifying where a coverslip is located so that tissue finding can be focused on that portion of a sample), applying minimum object size filters (e.g., applying a blob detection algorithm to identify objects in the macro image, and then removing the objects whose size is below a threshold value) and / or applying black thresholding (e.g., converting the pixels in a macro image from continuous greyscale or color values to quantized values, such as pure black and white) prior to identification of the area on the slide where the tissue can be found.

[0050] In addition to tissue finding 202, the macro image may also be used for focus point placement 203. This may be done by identifying objects in a slide (e.g., the objects which still remain the macro image in a case where minimum object size filters are applied), and treating each of those objects as a focus point for use in ultimately capturing the image. There may also be a focusing 204 step, in which one or more focusing planes could be identified for use when the slide is ultimately scanned. For example, a scanner could determine whether the macro image depicted the focus points with sufficient clarity (e.g., as measured by the contrast between adjacent pixels near the focus points) to exceed a focusing threshold. Then, for any points which were not sufficiently in focus, the scanner could search for focal planes which were suitable for those points (e.g., by searching through a pre-configured range of focal plane heights above the slide surface).

[0051] Similarly, preparation for slide scanning can include selection 205 of a scanning mode. To illustrate, consider the case of a scanner configured to perform real time autofocusing, such as described in U.S. patent 10,481,377, issued on November 19, 2019 for Real-time autofocus scanning, which is hereby incorporated by reference in its entirety. While this real time auto-focusing has a variety of benefits, it may not beappropriate in all cases, and so a scanner which has that capability may also have the capability to perform a scan in point focus mode (e.g., scanning a slide using focal planes determined in advance for particular focus points) as an alternative to scanning in real time focusing mode, and a selection 205 between these modes may be made as part of the preparatory activities in the scanning process. Finally, once the various preparatory activities were complete, slide could be scanned 206, such as by capturing rows of pixels using a line scan camera 130 as described previously in the context of FIGS. 1A-1D.

[0052] 3. Artificial Intelligence Supported Adaptive Scanning

[0053] In a multi-step scanning process such as shown in FIG. 2, the disclosed technology may be used to customize the scanning based on the requirements of a particular situation. This may be done, for example, using an adaptive scanning process such as that shown in FIG. 3. As shown in FIG. 3, an adaptive scanning process may begin with capturing 301 a first slide image, such as a macro captured 201 during the process of FIG. 2. This first slide image may then be used to obtain a classification for the slide which would ultimately be scanned. This may be done by providing the first slide image to a trained machine learning model, such as a deep neural network which would identify features using a set of convolution layers, and then classify the image by providing the identified features to a multi-output dense network trained to function as a classification head. An example of such a network is illustrated in FIGS. 4 and 5, discussed below.

[0054] Turning now to FIG. 4, that figure illustrates a machine learning model which can be used in some embodiments in classifying 302 images in a process such as shown in FIG. 3. In the architecture of FIG. 4, an input image 401 would be analyzed in a series of stages 402a-402n, each of which may be referred to as a “layer,” and which are illustrated in more detail in FIG. 5. As shown in FIG. 5, an input 501 (which, in the initial layer 502a of FIG. 5 would be the input image 401, and otherwise would be the output of the preceding layer) is provided to a layer 502 where it would be processed to generate one or more transformed images 503a-503n. This processing may include convolving the input 501 with a set of filters 504a-504n, each of which would identifya type of feature from the underlying image that would then be captured in that fdter’s corresponding transformed image. For instance, as a simple example, convolving an image with the filter shown in table 1 could generate a transformed image capturing the edges from the input image 501.[ -1 -1 -1 ][ -1 8 -1 ] [ -1 -1 -1 ] Table 1

[0055] As shown in FIG. 5, in addition to generating transformed images 503a-503n a layer may also generate a pooled image 5O5a-5O5n for each of the transformed images 503a- 5O3n. This may be done, for example, by organizing the appropriate transformed image into a set of regions, and then replacing the values in that region with a single value, such as the maximum value for the region or the average of the values for the region. The result would be a pooled image whose resolution would be reduced relative to its corresponding transformed image based on the size of the regions it was split into (e.g., if the transformed image had NxN dimensions, and it was split into 2x2 regions, then the pooled image would have size (N / 2)x(N / 2)). These pooled images 505a-505n could then be combined into a single output image 506, in which each of the pooled images 5O5a-5O5n is treated as a separate channel in the output image 506. This output image 506 can then be provided as input to the next layer as shown in FIG. 4.

[0056] Returning to the discussion of FIG. 4, after a final output image 403 has been created through the various stages 402a-402n of processing, the final output image 403 could be provided as input to a neural network 404. This may be done, for example, by providing the value of each channel of each pixel in the output image 403 to an input node of a densely connected single layer network. The output of the neural network 404 could then be treated a classification of the original input image 401. For example, in the case such as shown in FIG. 4, where a neural network 404 has multiple output nodes each of those output nodes may be treated as corresponding to a potentialclassification. For instance, if a machine learning model having an architecture such as shown in FIGS. 4 and 5 were used to classify a first image based on the type of tissue sample it depicted, then there may be output nodes fortissue sample types such as those described below in table 2. The corresponding classification for the output node with the highest value could be treated as the classification of the image which was originally provided as input (e.g., the first image in the process of FIG. 3).

[0057] A machine learning model such as a model following the architecture discussed in the context of FIGS. 4 and 5 can be trained to make image classifications using images falling into the classes that would be expected to be identified in production (e.g., images of various types of tissue samples, in a case where obtaining 302 a classification of the first slide image was performed by identifying a tissue sample type depicted in that image) annotated with labels indicating the correct classes to which those images belong. This training may include comparing labels applied by the model being trained with the ground truth labels provided by the annotation, and adjusting the values of the machine learning model’s parameters to minimize a loss function (e.g., cross entropy loss) for that comparison. The training can also include splitting the annotated images into multiple subsets, or folds, and then training and evaluating the model multiple times, with a different fold of training images being held back as a validation set each time (i.e., K-fold cross validation). In this way, performance metrics from each training instance can be averaged to verify the model’s generalization performance and, assuming the performance is acceptable, a final trained version of the model (e.g., whichever trained model had the best individual performance) can be used to make inferences (i.e., classify macro images) in production.

[0058] However the step of obtaining 302 a slide classification is performed, once it is complete, the method of FIG. 3 continues with determining 303 a parameter set for the slide. To support this, some implementations of the disclosed technology may utilize a system architecture such as shown in FIG. 6, in which a scanner 601 such as could be used in capturing 301 the first slide image is connected with a scanner administration manager 602 which could provide the appropriate parameter sets based on the slideclassification. For example, in some cases, a scanner 601 may be associated with a setup process in which the person or entity which would be using the scanner could define sets of values that would be used when the scanner was capturing slide images, and those sets of values could be stored in the scanner administration manager 602 as parameter sets (e g., in configuration files) that would be selectively retrieved based on the slide classification. To illustrate how this may take place, consider table 2, which describes how parameter sets may be defined in a case where slides are classified based on the types of tissue samples they include.Table 2

[0059] Finally, once the appropriate parameter set was available (e.g., because it had been retrieved from a scanner administration manager 602), that parameter set could be used in capturing 304 a second image of the slide, for example, by scanning 206 the portion of the slide identified as containing tissue during tissue finding 202 using a line scan camera 130, as described previously in the context of FIGS. 1A and 2. Thus, using a method such as shown in FIG. 3, the disclosed technology can support a system such as shown in FIG. 12, in which slides with various types of tissue samples are provided to a scanner which uses a machine learning model to identify the tissue sample type and select a corresponding configuration (parameter set), and that automatically selected configuration is used to perform the scan in a manner that is adapted to the individual tissue sample type of the input slide.

[0060] Variations on potential implementations of adaptive scanning technology such as disclosed herein are also possible. For example, while the above description gave examples of parameter types which can be used in generating a whole slide image, other parameters may also, or alternatively, be used in some cases. Such parameters may include the width of stripes to use when scanning a slide, sensitivity of algorithms used to detect the coverslip on a slide, and / or type(s) and strength(s) of filters to use in image processing to be applied to a whole slide image after it has been captured. It is also possible that parameter sets can be determined based on classifications other than thetissue sample type classifications noted above. For instance, in some cases, the disclosed technology may be implemented to classify slides based on a type of application in which they would be used (e.g., an image of animal tissue could be classified as being used for veterinary applications, an image of tissue, images of tissues with significant yellowing affixed to a slide with an obsolete mounting media may be classified as being used for archival applications, etc.).

[0061] Similarly, in some cases parameter sets may be determined in manners other than retrieving a previously defined parameter set corresponding to a slide classification. For example, in some cases, an institution utilizing a scanner may define a default parameter set for regular tissue samples (e.g., regular H&E tissue samples), and parameter sets may be automatically generated for other types of tissue samples by modifying the default parameter set to account for the differences between regular tissue samples and other types of tissue samples (e.g., reducing the focus threshold from the focus threshold value in the default parameter set when a slide is classified as having a faintly stained tissue). It is also possible that the techniques used to classify slides may vary between implementations. For example, in some cases machine learning models having architectures which differ from that described in the context of FIGS. 4 and 5 may be used, with more complicated models (e.g., models having additional layers and / or connections between layers) being used to allow for more fine grained distinctions between similar tissue types in contexts where such distinctions are necessary. Classification approaches which do not use networks at all, such as image analysis approaches that distinguish different classes based on image characteristics (e.g., an algorithm which distinguishes between regular H&E stained or immunohistochemistry stained samples and fainter types of samples like RNA ISH or faintly stained samples based on average darkness of pixels in the first image) could be used in some cases, either in addition to, or as alternatives to, network based approaches such as described above.

[0062] The overall scanning process may also be modified in some implementations of the disclosed technology, for example, by incorporating additional steps beyond thosedepicted and described in the context of FIGS. 2 and 3. For instance, in some cases, after a first slide image (e.g., a macro image) has been captured 301, that image may be subjected to a quality evaluation process (e.g., checking if the height of the sample was sufficiently uniform to allow for the complete width of a strip to be within an acceptable range of focus while it was being scanned), and a remedial action could automatically be triggered if the slide did not pass the quality evaluation (e.g., if the sample height was not sufficiently uniform, a flag could be thrown alerting the user of the issue, and potentially suggesting a remediation, such as to replace the slide with another slide that used less mounting media to fix the sample).

[0063] Variations are also possible with respect to the hardware which is used to implement an adaptive scanning process, and how the various tasks associated with that process are split among the various hardware components. For example, while it is possible that a scanner may have multiple cameras (e.g., a line scan camera 130 and an area scan camera 132 as shown in FIG. 1) that could be used to capturing a macro image and scanning a slide, it is also possible that the disclosed technology could be implemented in a system which uses a single camera (e.g., a line scan camera) to both capture a macro image and scan a slide, with different imaging optics (e.g., a separate low magnification photographic lens or a separate low magnification microscope objective for the macro image) which provide different fields of view and magnifications for used to capture a macro image versus scanning a slide. An example of physical hardware which could be used in such a system is provided in U.S. published patent application 2018 / 0188571, for low resolution slide imaging and slide label imaging and high resolution slide imaging using dual optical paths and a single image sensor, the disclosure of which is hereby incorporated by reference in its entirety.

[0064] As an example of another type of variation in how hardware may be used to implement the disclosed technology, in an architecture such as shown in FIG. 6, it is possible that a scanner 601 may both capture an image and classify it, then send the classification to the scanner administration manager 602 to retrieve a corresponding parameter set. However, it is also possible that a scanner 601 could simply capture and image and sendit to the scanner administration manager 602, and the scanner administration manager 602 could both classify the image and determine the appropriate parameter set based on that classification. Similarly, in some cases, a scanner 601 and scanner administration manager 602 may be local to each other (e.g., connected directly, or connected over a local area network), while in others a scanner 601 and scanner administration manager 602 may be remote (e.g., a scanner administration manager 602 may be connected to a scanner 601 over a wide area network connection, such as could be the case for a cloud based system where a scanner administration manager 602 may potentially maintain parameter sets for multiple scanners at multiple organizations which parameter sets may differ from scanner to scanner and / or organization to organization). It is also possible that the technology may be implemented in a manner that eschews the component separation illustrated in FIG. 6, and instead uses a scanner which would maintain its own parameter sets, as opposed to relying on a separate scanner administration manager for this task.

[0065] 4. Additional Non-Limiting Examples

[0066] The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles described herein can be applied to other embodiments without departing from the spirit or scope of the invention. Thus, it is to be understood that the description and drawings presented herein represent a presently preferred embodiment of the invention and are therefore representative of the subject matter which is broadly contemplated by the present invention. It is further understood that the scope of the present invention fully encompasses other embodiments that may become obvious to those skilled in the art and that the scope of the present invention is accordingly not limited. Other variations are also possible, and will be immediately apparent to those of skill in the art in light of this disclosure. For instance, the following examples are provided as concrete (though non-limiting) illustrations of various approaches which could be taken when implementing the disclosed technology.

[0067] Example 1

[0068] A machine for adaptively generating a histological image, the machine comprising a processor and a non-transitory medium storing instructions, executable by the processor, for performing a method comprising: capturing a first image of a slide; obtaining a classification for the slide based on providing the first image to a machine learning model; based on the classification for the slide, determining a parameter set; and using the parameter set, capturing a second image of the slide

[0069] Example 2A

[0070] The machine of example 1, wherein: the machine comprises a camera, first imaging optics and second imaging optics; and the instructions comprise instructions to capture the first image using the camera with the first imaging optics, and the second image using the camera with the second imaging optics.

[0071] Example 3 A

[0072] The machine of example 2A, wherein the camera is a line scan camera.

[0073] Example 2B

[0074] The machine of example 1, wherein: the machine comprises a first camera and a second camera; and the instructions comprise instructions to capture the first image using the first camera, and the second image using the second camera.

[0075] Example 3B

[0076] The machine of example 2B, wherein the first camera is an area scan camera, and wherein the second camera is a line scan camera.

[0077] Example 4

[0078] The machine of any of examples 1-3, wherein: the processor and the non-transitory computer readable medium are comprised by a scanner; the machine learning model istrained to provide the classification for the slide from a set of potential classifications; and the machine further a scanner administration manager which is separate from the scanner and which stores a set of configuration files, wherein each configuration file from the set of configuration files corresponds to a classification from the set of classifications and comprises parameters for the corresponding classification.

[0079] Example 5

[0080] The machine of example 4, wherein each classification from the set of potential classifications is a tissue sample type from a set of potential tissue sample types.

[0081] Example 6

[0082] The machine of claim 5, wherein the set of potential tissue sample types comprises: regular hematoxylin and eosin stained or immunohistochemistry stained tissue; faintly stained tissue; tissue microarray; needle biopsy; cytology sample; and ribonucleic acid (RNA) in situ hybridization (ISH) sample.

[0083] Example 7

[0084] The machine of any of examples 1-6, wherein the parameter set comprises values for a set of parameters comprising: black threshold; noise correction; minimum object size; dirt removal; coverslip detection sensitivity; filter type and strength; scanning mode; focusing density; and focus range.

[0085] Example 8

[0086] A method of adaptively generating a histological image, the method comprising: capturing a first image of a slide; obtaining a classification for the slide based on providing the first image to a machine learning model; based on the classification for the slide, determining a parameter set; and using the parameter set, capturing a second image of the slide.

[0087] Example 9A

[0088] The method of example 8, wherein capturing the first image is performed using a camera with first imaging optics, and capturing the second image is performed using the camera with second imaging optics.

[0089] Example 10A

[0090] The method of example 9A, wherein the camera is a line scan camera.

[0091] Example 9B

[0092] The method of example 8, wherein capturing the first image is performed using a first camera, and capturing the second image is performed using a second camera

[0093] Example 10B

[0094] The method of example 9B, wherein the first camera is an area scan camera, and wherein the second camera is a line scan camera.

[0095] Example 11

[0096] The method of any of examples 8-10, wherein: the machine learning model is trained to provide the classification for the slide from a set of potential classifications; and determining the parameter set comprises selecting a configuration file from a set of configuration files, wherein each configuration file from the set of configuration files corresponds to a classification from the set of classifications and comprises parameters for the corresponding classification.

[0097] Example 12

[0098] The method of example 11, wherein each classification from the set of potential classifications is a tissue sample type from a set of potential tissue sample types.

[0099] Example 13

[0100] The method of example 12, wherein the set of potential tissue sample types comprises: regular hematoxylin and eosin stained or immunohistochemistry stained tissue; faintly stained tissue; tissue microarray; needle biopsy; cytology sample; and ribonucleic acid (RNA) in situ hybridization (ISH) sample.

[0101] Example 14

[0102] The method of any of examples 8-13, wherein the parameter set comprises values for a set of parameters comprising: black threshold; noise correction; minimum object size; dirt removal; coverslip detection sensitivity; fdter type and strength; scanning mode; focusing density; and focus range.

[0103] Example 15

[0104] A non-transitory computer readable medium storing instructions for performing a method of adaptively generating a histological image, the method comprising: capturing a first image of a slide; obtaining a classification for the slide based on providing the first image to a machine learning model; based on the classification for the slide, determining a parameter set; and using the parameter set, capturing a second image of the slide.

[0105] Example 16A

[0106] The medium of example 15, wherein capturing the first image is performed using a camera with first imaging optics, and capturing the second image is performed using a camera with second imaging optics

[0107] Example 17A

[0108] The medium of example 16A, wherein the camera is a line scan camera.

[0109] Example 16B

[0110] The medium of example 15, wherein capturing the first image is performed using a first camera, and capturing the second image is performed using a second camera.

[0111] Example 17B

[0112] The medium of example 16B, wherein the first camera is an area scan camera, and wherein the second camera is a line scan camera.

[0113] Example 18

[0114] The medium of any of examples 15-17, wherein: the machine learning model is trained to provide the classification for the slide from a set of potential classifications; and determining the parameter set comprises selecting a configuration file from a set of configuration files, wherein each configuration file from the set of configuration files corresponds to a classification from the set of classifications and comprises parameters for the corresponding classification.

[0115] Example 19

[0116] The medium of example 18, wherein each classification from the set of potential classifications is a tissue sample type from a set of potential tissue sample types.

[0117] Example 20

[0118] The medium of example 19, wherein the set of potential tissue sample types comprises: regular hematoxylin and eosin stained or immunohistochemistry stained tissue; faintly stained tissue; tissue microarray; needle biopsy; cytology sample; and ribonucleic acid (RNA) in situ hybridization (ISH) sample.

[0119] 5. Interpretation

[0120] None of the examples or illustrations set forth herein should be understood as implying limitations on the scope of any claims included in this document or any related document. Instead, the protection provided by this document or any related document, should be understood as being defined by the relevant document’s claims, when the terms in those claims which are explicitly defined herein are given their explicitdefinitions, and the terms which are not explicitly defined are given their broadest reasonable interpretation as provided by a general purpose dictionary.

[0121] Combinations, described herein, such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof’ include any combination of A, B, and / or C, and may include multiples of A, multiples of B, or multiples of C. Specifically, combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof’ may be A only, B only, C only, A and B, A and C, B and C, or A and B and C, and any such combination may contain one or more members of its constituents A, B, and / or C. For example, a combination of A and B may comprise one A and multiple B’s, multiple A’s and one B, or multiple A’s and multiple B’s.

[0122] A statement that something is “based on” something else should be understood as meaning that that thing is determined at least in part by that which it is “based on.” While the “based on” relationship includes scenarios in which one thing is completely determined by another, the “based on” relationship should not be understood as being limited to only scenarios in which one thing is completely determined by another unless the phrase used is “based exclusively on.”

Claims

CLAIMSWhat is claimed is:

1. A machine for adaptively generating a histological image, the machine comprising a processor and a non-transitory medium storing instructions, executable by the processor, for performing a method comprising: capturing a first image of a slide; obtaining a classification for the slide based on providing the first image to a machine learning model; based on the classification for the slide, determining a parameter set; and using the parameter set, capturing a second image of the slide.

2. The machine of claim 1, wherein: the machine comprises a first camera and a second camera; and the instructions comprise instructions to capture the first image using the first camera, and the second image using the second camera.

3. The machine of claim 2, wherein the first camera is an area scan camera, and wherein the second camera is a line scan camera.

4. The machine of claim 1, wherein: the processor and the non-transitory computer readable medium are comprised by a scanner; the machine learning model is trained to provide the classification for the slide from a set of potential classifications; and the machine further a scanner administration manager which is separate from the scanner and which stores a set of configuration files, wherein each configuration file from the set of configuration files corresponds to a classification from the set of classifications and comprises parameters for the corresponding classification.

5. The machine of claim 4, wherein each classification from the set of potential classifications is a tissue sample type from a set of potential tissue sample types.

6. The machine of claim 5, wherein the set of potential tissue sample types comprises: regular hematoxylin and eosin stained or immunohistochemistry stained tissue; faintly stained tissue; tissue microarray; needle biopsy; cytology sample; and ribonucleic acid (RNA) in situ hybridization (ISH) sample.

7. The machine of claim 1 , wherein the parameter set comprises values for a set of parameters comprising: black threshold; noise correction; minimum object size; dirt removal; coverslip detection sensitivity; filter type and strength; scanning mode; focusing density; and focus range.

8. A method of adaptively generating a histological image, the method comprising: capturing a first image of a slide; obtaining a classification for the slide based on providing the first image to a machine learning model; based on the classification for the slide, determining a parameter set; and using the parameter set, capturing a second image of the slide.

9. The method of claim 8, wherein capturing the first image is performed using a first camera, and capturing the second image is performed using a second camera.

10. The method of claim 9, wherein the first camera is an area scan camera, and wherein the second camera is a line scan camera.

11. The method of claim 8, wherein: the machine learning model is trained to provide the classification for the slide from a set of potential classifications; and determining the parameter set comprises selecting a configuration file from a set of configuration files, wherein each configuration file from the set of configuration files corresponds to a classification from the set of classifications and comprises parameters for the corresponding classification.

12. The method of claim 11, wherein each classification from the set of potential classifications is a tissue sample type from a set of potential tissue sample types.

13. The method of claim 12, wherein the set of potential tissue sample types comprises: regular hematoxylin and eosin stained or immunohistochemistry stained tissue; faintly stained tissue; tissue microarray; needle biopsy; cytology sample; and ribonucleic acid (RNA) in situ hybridization (ISH) sample.

14. The method of claim 8, wherein the parameter set comprises values for a set of parameters comprising: black threshold; noise correction;minimum object size; dirt removal; coverslip detection sensitivity; filter type and strength; scanning mode; focusing density; and focus range.

15. A non-transitory computer readable medium storing instructions for performing a method of adaptively generating a histological image, the method comprising: capturing a first image of a slide; obtaining a classification for the slide based on providing the first image to a machine learning model; based on the classification for the slide, determining a parameter set; and using the parameter set, capturing a second image of the slide.

16. The medium of claim 15, wherein capturing the first image is performed using a first camera, and capturing the second image is performed using a second camera.

17. The medium of claim 16, wherein the first camera is an area scan camera, and wherein the second camera is a line scan camera.

18. The medium of claim 15, wherein: the machine learning model is trained to provide the classification for the slide from a set of potential classifications; and determining the parameter set comprises selecting a configuration file from a set of configuration files, wherein each configuration file from the set of configuration files corresponds to a classification from the set of classifications and comprises parameters for the corresponding classification.

19. The medium of claim 18, wherein each classification from the set of potential classifications is a tissue sample type from a set of potential tissue sample types.

20. The medium of claim 19, wherein the set of potential tissue sample types comprises: regular hematoxylin and eosin stained or immunohistochemistry stained tissue; faintly stained tissue; tissue microarray; needle biopsy; cytology sample; and ribonucleic acid (RNA) in situ hybridization (ISH) sample.

Images