Real-time focusing in slide scanning systems

Abstract

To provide a system and method for overcoming significant problems with real-time focusing found in conventional systems.SOLUTION: Focus points are added to an initialized focus map while acquiring a plurality of image stripes of a sample on a glass slide. For each image stripe, a plurality of frames collectively representing the image stripe may be acquired using both an imaging line-scan camera and a tilted focusing line-scan camera. Focus points representing positions of best focus for trusted frames are added to the focus map. Outlying focus points are removed from the focus map. In some cases, one or more image stripes may be reacquired. Finally, the image stripes are assembled into one composite image of the sample.SELECTED DRAWING: Figure 5A

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims priority to U.S. Provisional Patent Application No. 62 / 883,525, filed on 6 August 2019, and its contents are incorporated herein by reference as if they were fully described herein.

[0002] This application is further related to the following applications, all of which are incorporated herein by reference as if they were fully described herein: International patent application PCT / US2016 / 053581, filed on September 23, 2016, International patent application PCT / US2017 / 028532, filed on April 20, 2017, International patent application PCT / US2018 / 063456, filed on November 30, 2018, International patent application PCT / US2018 / 063460, filed on November 30, 2018, International patent application PCT / US2018 / 063450, filed on November 30, 2018, International patent application PCT / US2018 / 063461, filed on November 30, 2018, International patent application PCT / US2018 / 062659, filed on November 27, 2018, International patent application PCT / US2018 / 063464, filed on November 30, 2018, International patent application PCT / US2018 / 054460, filed on October 4, 2018. International patent application PCT / US2018 / 063465, filed on November 30, 2018, International patent application PCT / US2018 / 054462, filed on October 4, 2018, International patent application PCT / US2018 / 063469, filed on November 30, 2018, International patent application PCT / US2018 / 054464, filed on October 4, 2018. International patent application PCT / US2018 / 046944, filed on August 17, 2018, International patent application PCT / US2018 / 054470, filed on October 4, 2018. International patent application PCT / US2018 / 053632, filed on September 28, 2018, International patent application PCT / US2018 / 053629, filed on September 28, 2018, International patent application PCT / US2018 / 053637, filed on September 28, 2018, International patent application PCT / US2018 / 062905, filed on November 28, 2018, International patent application PCT / US2018 / 063163, filed on November 29, 2018, International patent application PCT / US2017 / 068963, filed on December 29, 2017, International patent application PCT / US2019 / 020411, filed on March 1, 2019, U.S. Patent Application No. 29 / 631,492, filed on December 29, 2017, U.S. Patent Application No. 29 / 631,495, filed on December 29, 2017, U.S. Patent Application No. 29 / 631,499, filed on December 29, 2017, U.S. Patent Application No. 29 / 631,501, filed on December 29, 2017.

[0003] Field of Invention The embodiments described herein generally relate to controlling a slide scanning system, and more specifically to real-time focusing in a slide scanning system. [Background technology]

[0004] Digital pathology is an image-based information environment made possible by computer technology that enables the management of information generated from physical slides. Digital pathology is partially made possible by virtual microscopy, a technique that scans specimens on physical glass slides to create digital slide images that can be stored, displayed, managed, and analyzed on a computer monitor. The ability to image entire glass slides has led to an explosive expansion of the field of digital pathology, and it is now considered one of the most promising tools in diagnostic medicine for better, faster, and less expensive diagnosis, prognosis, and prediction of important diseases such as cancer.

[0005] A primary objective for the digital pathology industry is to reduce scanning time. This reduction can be achieved by switching to real-time focusing during the actual scan. To achieve focused, high-quality image data using real-time focusing during the actual scan, the scanning device must be capable of determining the next Z value for the objective lens (e.g., the distance between the objective lens and the sample). Therefore, what is needed is a system and method that overcomes the significant problems of real-time focusing found in conventional systems. [Overview of the project] [Means for solving the problem]

[0006] A system, method, and non-temporary computer-readable medium for real-time focusing in a slide scanning system are disclosed.

[0007] In one embodiment, a method is disclosed, which includes: initializing a focus map using at least one hardware processor of a scanning system; adding a focus to the focus map by acquiring each of a plurality of frames that collectively represent the image stripes using both an imaging line scan camera and a tilt focusing line scan camera while acquiring a plurality of image stripes of at least a portion of a sample on a glass slide, and adding a focus to the focus map that represents the best focus position for a trusted frame among the plurality of frames; removing all outlier focuses from the focus map; determining, based on the focus error, whether one or more image stripes should be restriped for each of the plurality of frames in the plurality of image stripes; reacquiring one or more image stripes if it is determined that one or more image stripes should be restriped; and assembling the plurality of image stripes into one composite image of at least a portion of the sample.

[0008] Adding focus to the focus map while acquiring multiple image stripes may further include, for each of the multiple image stripes to be acquired except for the last one, determining the direction of the next image stripe to be acquired from that image stripe after acquiring the image stripe. Multiple image stripes can be acquired sequentially by acquiring a reference stripe, sequentially acquiring image stripes from a first side of the reference stripe to a first edge of the scanning region of the sample, and sequentially acquiring image stripes from a second side of the reference stripe opposite to the first side of the reference stripe to a second edge of the scanning region opposite to the first edge of the scanning region.

[0009] The method may further include adding multiple macrofocals to the focus map before starting to acquire multiple image stripes. The method may also include adding one or more macrofocals to the focus map after acquiring one or more of the multiple image stripes. It can further include

[0010] Adding focus to a focus map while acquiring multiple image stripes may further involve determining whether each of the multiple frames in each of the multiple image stripes is reliable. Determining whether a frame is reliable may include: calculating a principal gradient vector, including the average gradient vector, for each column in the frame acquired by an imaging line scan camera; calculating a gradient vector, including the average gradient vector, for each column in the frame acquired by a tilt-focusing line scan camera; identifying the number of analyzable columns in the principal gradient vector; calculating a ratio vector based on the principal gradient vector and the gradient vector; determining whether the frame is analyzable based on the number of analyzable columns and the ratio vector; determining that the frame is unreliable if it is determined that the frame is not analyzable; fitting at least one Gaussian function to the ratio curve represented by the ratio vector if it is determined that the frame is analyzable; identifying the peak of the Gaussian function as the best focus position; identifying the amplitude of the ratio vector at the best focus position as the maximum fit; determining whether the frame is reliable based on the best focus position and the maximum fit; determining that the frame is unreliable if it is determined that the frame is not reliable; and adding the best focus position to the focus map if it is determined that the frame is reliable. Identifying the number of analyzable columns may involve identifying the number of columns in the principal gradient vector that exceed a threshold. Calculating the ratio vector may involve dividing the gradient vector by the principal gradient vector.Determining whether a frame is analyzable may include determining whether the number of analyzable columns exceeds a predefined threshold percentage, and whether the values ​​of the ratio vectors at the parfocal positions are within a predefined range, where the parfocal positions are points on a tilt-focus line scan camera that are parfocal with the imaging line scan camera, and determining that the frame is not analyzable if it is determined that the number of analyzable columns does not exceed a predefined threshold percentage, or that the values ​​of the ratio vectors at the parfocal positions are not within a predefined range, and determining that the frame is analyzable if it is determined that the number of analyzable columns exceeds a predefined threshold percentage and the values ​​of the ratio vectors at the parfocal positions are within a predefined range. Fitting at least one Gaussian function to the ratio curve may include sampling several possible Gaussian functions within the range of mean values ​​and sigma values, and selecting one Gaussian function from among the several possible Gaussian functions that has the smallest difference from the ratio curve, which should be used to identify the best focus position.

[0011] Removing all outlier foci from a focus map may involve calculating the slope for one or more sample points within the focus map in each of four directions away from sample points inside the focus map, and removing the sample point from the focus map if the minimum calculated slope exceeds a predefined threshold.

[0012] Determining whether one or more of a plurality of image stripes should be restriped can be performed by calculating a focus error for each frame in each of the plurality of image stripes, after removing the focus of all outliers from the focus map, by subtracting the actual position of the objective lens during acquisition of the frame from the best in-focus position of that frame within the focus map, and determining to restripe an image stripe if the number of frames having a focus error exceeding a pre-defined threshold exceeds a pre-defined threshold percentage for each of the plurality of image stripes.

[0013] The method can be embodied in the form of an executable software module of a processor-based system, such as a server, and / or in the form of executable instructions stored on a non-transitory computer-readable medium.

[0014] Details of the present invention are partly available by studying the accompanying drawings, both as to its structure and operation, in which like reference numerals refer to like parts.

Brief Description of the Drawings

[0015] [Figure 1A] FIG. is a diagram showing an exemplary processor-corresponding device that can be used in connection with various embodiments described herein, according to one embodiment. [Figure 1B] FIG. is a diagram showing an exemplary line scan camera having a single linear array, according to one embodiment. [Figure 1C] FIG. is a diagram showing an exemplary line scan camera having three linear arrays, according to one embodiment. [Figure 1D] FIG. is a diagram showing an exemplary line scan camera having a plurality of linear arrays, according to one embodiment. [Figure 1E] FIG. is an exemplary side configuration diagram of a line scan camera in a scanning system, according to one embodiment. [Figure 1F] This is an exemplary top view diagram of an imaging sensor relating to the imaging optical path according to one embodiment. [Figure 1G] This is an exemplary top view diagram of a focusing sensor relating to a focusing optical path according to one embodiment. [Figure 1H] This figure shows an exemplary focusing sensor according to one embodiment. [Figure 2] This figure shows exemplary focus errors before and after applying offset correction using a calculated macro focus offset according to one embodiment. [Figure 3] This figure shows an exemplary graph of a low-pass filtered signal and fitting according to one embodiment. [Figure 4] This figure shows a cofocal calculation according to one embodiment. [Figure 5A] This figure shows a process for scanning an image of a sample on a glass slide according to one embodiment. [Figure 5B] This figure shows a process for scanning an image of a sample on a glass slide according to one embodiment. [Figure 5C] This figure shows a process for scanning an image of a sample on a glass slide according to one embodiment. [Figure 6A] This figure shows an exemplary frame of image data acquired by the main imaging sensor according to one embodiment. [Figure 6B] This figure shows an exemplary frame of image data acquired by a tilt focusing sensor according to one embodiment. [Figure 7] This figure shows a process for calculating a ratio vector according to one embodiment. [Figure 8A] This figure shows an example gradient vector for the main imaging sensor and the tilt focusing sensor according to one embodiment. [Figure 8B] This figure shows an exemplary ratio vector for the two gradient vectors in Figure 8A, according to one embodiment. [Figure 9]This figure shows a ratio curve for a tissue sample scanned at a fixed offset from the same focal point, according to one embodiment. [Figure 10] This figure shows an example of a Gaussian fitting process according to one embodiment. [Figure 11] This figure shows a partial set of Gaussian test functions for multiple different mean values ​​and one fixed width, according to one embodiment. [Figure 12] An example of a Gaussian fitting process according to one embodiment, in which the ratio curve has two peaks, is shown. [Figure 13] This figure shows an exemplary set of Gaussian functions according to one embodiment. [Figure 14] This figure shows the exemplary minimum RMS difference and the position of the best-fitting Gaussian function according to one embodiment. [Figure 15] This figure shows the calculation of the error slope according to one embodiment. [Figure 16] This figure shows an example of outlier detection according to one embodiment. [Figure 17A] This figure shows an exemplary heat map representing the focus error according to one embodiment. [Figure 17B] This figure shows an exemplary heat map representing the focus error according to one embodiment. [Modes for carrying out the invention]

[0016] In one embodiment, a system, method, and non-temporary computer-readable medium for real-time focusing in a slide scanning system are disclosed. After reading this specification, it will be clear to those skilled in the art how the invention may be carried out in various alternative embodiments and alternative uses. However, while various embodiments of the invention are described herein, it should be understood that these embodiments are not limiting and are presented only as examples and illustrations. Therefore, this detailed description of various embodiments should not be construed as limiting the scope or extent of the invention as set forth in the appended claims.

[0017] 1. Exemplary scanning system

[0018] Figure 1A is a block diagram showing an exemplary processor-enabled slide scanning system 100 that can be used in connection with the various embodiments described herein. Alternative forms of the scanning system 100 may be used, as will be understood by those skilled in the art. In the illustrated embodiment, the scanning system 100 is presented as a digital imaging device comprising: one or more processors 104, one or more memories 106, one or more motion control devices 108, one or more interface systems 110, one or more movable stages 112 each supporting one or more glass slides 114 equipped with one or more samples 116, one or more illumination systems 118 for illuminating the samples 116, one or more objective lenses 120 each defining an optical path 122 traveling along the optical axis, one or more objective lens positioners 124, one or more optional reflected illumination systems 126 (for example, included in a fluorescence scanning embodiment), one or more focusing optical systems 128, one or more line scan cameras 130, and / or one or more area scan cameras 132 each defining a separate field of view 134 on the sample 116 and / or on the glass slide 114. The various elements of the scanning system 100 are communicably coupled via one or more communication buses 102. Each of the various elements of the scanning system 100 may be plural, but for the sake of simplicity in the following explanation, these elements will be described in the singular form unless it is necessary to describe them in the plural form to convey appropriate information.

[0019] The processor 104 may include, for example, a central processing unit (CPU) and a separate graphics processing unit (GPU) capable of processing instructions in parallel, or it may include a multi-core processor capable of processing instructions in parallel. Additional separate processors may be provided to control specific components or to perform specific functions such as image processing. For example, additional processors may include an auxiliary processor for managing data inputs, an auxiliary processor for performing floating-point mathematical operations, a dedicated processor with an architecture suitable for high-speed execution of signal processing algorithms (e.g., a digital signal processor), a slave processor dependent on the main processor (e.g., a backend processor), an additional processor for controlling the line scan camera 130, the stage 112, the objective lens 120, and / or a display (e.g., a console including a touch panel display integrated into the scanning system 100). Such additional processors may be separate discrete processors or may be integrated into a single processor.

[0020] Memory 106 provides a storage device for data and instructions for programs executable by processor 104. Memory 106 may include one or more volatile and / or non-volatile computer-readable storage media for storing data and instructions. These media may include, for example, random access memory (RAM), read-only memory (ROM), hard disk drives, and / or removable storage drives (including, for example, flash memory). Processor 104 is configured to execute instructions stored in memory 106 and communicate with various elements of the scanning system 100 via the communication bus 102 to perform the overall functions of the scanning system 100.

[0021] The communication bus 102 can be configured to carry analog electrical signals and / or digital data. Therefore, communication from the processor 104, motion control device 108, and / or interface system 110 via the communication bus 102 can include both electrical signals and digital data. The processor 104, motion control device 108, and / or interface system 110 can also be configured to communicate with one or more of the various elements of the scanning system 100 via a wireless communication link.

[0022] The motion control system 108 is configured to precisely control and adjust the X, Y, and / or Z motion of the stage 112 (e.g., in the XY plane), the X, Y, and / or Z motion of the objective lens 120 (e.g., along the Z axis perpendicular to the XY plane via the objective lens positioner 124), the rotational motion of the carousel as described elsewhere herein, the lateral motion of the push / pull assembly as described elsewhere herein, and / or any other movable components of the scanning system 100. For example, in a fluorescence scanning embodiment including an epi-illumination system 126, the motion control system 108 can be configured to adjust the movement of an optical filter or the like in the epi-illumination system 126.

[0023] The interface system 110 enables the scanning system 100 to communicate with other systems and human operators. For example, the interface system 110 may include a console (e.g., a touch panel display) to directly provide information to the operator via a graphical user interface and / or to enable direct input from the operator via a touch sensor. The interface system 110 can also be configured to facilitate communication and data transfer between the scanning system 100 and one or more external devices directly connected to the scanning system 100 (e.g., a printer, a removable storage medium, etc.) and / or one or more external devices indirectly connected to the scanning system 100 via one or more networks (e.g., an image storage system, a scanner management manager (SAM) server, and / or other management servers, an operator station, a user station, etc.).

[0024] The illumination system 118 is configured to illuminate at least a portion of the sample 116. The illumination system 118 may include, for example, one or more light sources and an illumination optical system. The light source may include a variable-intensity halogen light source with a concave mirror for maximizing light output and a KG-1 filter for suppressing heat. The light source may include any type of arc lamp, laser, or other light source. In one embodiment, the illumination system 118 illuminates the sample 116 in transmission mode so that the light energy transmitted through the sample 116 is detected by the line scan camera 130 and / or area scan camera 132. Alternatively or additionally, the illumination system 118 may be configured to illuminate the sample 116 in reflection mode so that the light energy reflected from the sample 116 is detected by the line scan camera 130 and / or area scan camera 132. The illumination system 118 may be configured to be suitable for examining the sample 116 in any known mode of optical microscopy.

[0025] In one embodiment, the scanning system 100 includes an epi-illumination system 126 for optimizing the scanning system 100 for fluorescence scanning. It should be understood that the epi-illumination system 126 may be omitted if fluorescence scanning is not supported by the scanning system 100. Fluorescence scanning is the scanning of a sample 116 containing fluorescent molecules, which are photon-sensitive molecules that can absorb light of a specific wavelength (i.e., excitation light). These photon-sensitive molecules also emit light of higher wavelengths (i.e., emitted light). Because the efficiency of this photoluminescence phenomenon is very low, the amount of light emitted is often very small. Such small amounts of emitted light typically cause conventional techniques for scanning and digitizing the sample 116 (e.g., transmission mode microscopy) to fail.

[0026] In embodiments of the scanning system 100 utilizing fluorescence scanning, it is advantageous to increase the sensitivity of the line scan camera 130 to light by using a line scan camera 130 (e.g., a time-delayed integral (TDI) line scan camera) that includes multiple linear sensor arrays, thereby exposing the same single area of ​​the sample 116 to each of the multiple linear sensor arrays of the line scan camera 130. This is particularly beneficial when scanning weakly fluorescent samples with low levels of emitted light. Therefore, in embodiments of fluorescence scanning, the line scan camera 130 is preferably a monochrome TDI line scan camera. Monochrome images are ideal in fluorescence microscopy because they more accurately represent the actual signals from the various channels present on the sample 116. As will be understood by those skilled in the art, a fluorescent sample can be labeled with multiple fluorescent dyes, each emitting light at a different wavelength, also called "channels."

[0027] Furthermore, since the lower and upper signal levels of various fluorescent samples represent a wide range of spectral wavelengths that the line scan camera 130 should sense, it is desirable that the lower and upper signal levels that the line scan camera 130 can sense are also wide. Therefore, in the fluorescence scanning embodiment, the line scan camera 130 may include a monochrome 10-bit 64 linear array TDI line scan camera. It should be noted that various bit depths can be used for the line scan camera 130 for use with such embodiments.

[0028] The movable stage 112 is configured for precise XY movement under the control of the processor 104 or the motion control device 108. The movable stage 112 can also be configured for Z movement under the control of the processor 104 or the motion control device 108. The movable stage 112 is configured to position the sample 116 to a desired position while image data is being acquired by the line scan camera 130 and / or the area scan camera 132. The movable stage 112 is also configured to accelerate the sample 116 to a substantially constant speed in the scanning direction and then maintain this substantially constant speed while image data is being acquired by the line scan camera 130. In one embodiment, the scanning system 100 can use a highly accurate and tightly coordinated XY grid to assist in positioning the sample 116 on the movable stage 112. In one embodiment, the movable stage 112 is a linear motor-based XY stage with high-precision encoders used in both the X and Y axes. For example, a very precise nanometer encoder can be used on the axis in the scanning direction, as well as on an axis perpendicular to the scanning direction but in the same plane as the scanning direction. The stage 112 is also configured to support a glass slide 114 on which the sample 116 is placed.

[0029] Sample 116 may be anything that can be examined by optical microscopy. For example, a microscope glass slide 114 is commonly used as an observation substrate for specimens containing tissues and cells, chromosomes, deoxyribonucleic acid (DNA), proteins, blood, bone marrow, urine, bacteria, beads, biopsy material, or any other type of biological material or substance, whether dead or alive, stained or unstained, labeled or unlabeled. Sample 116 may be any type of DNA or DNA-related material or protein sequence, such as complementary DNA (cDNA) or ribonucleic acid (RNA), deposited on any type of slide or other substrate containing any sample commonly known as a microarray. Sample 116 may be a microtiter plate (e.g., a 96-well plate). Other examples of sample 116 include integrated circuit boards, electrophoresis recorders, Petri dishes, films, semiconductor materials, forensic materials, and machined parts.

[0030] The objective lens 120 is mounted on an objective lens positioner 124, which, in one embodiment, uses a highly precise linear motor to move the objective lens 120 along the optical axis defined by the objective lens 120. For example, the linear motor of the objective lens positioner 124 may include a 50-nanometer encoder. The relative positions between the stage 112 and the objective lens 120 in the X, Y, and / or Z axes are adjusted and controlled in a closed-loop manner using a motion control device 108, under the control of a processor 104 that uses memory 106 for storing information and instructions, including computer-executable programmed steps for the overall operation of the scanning system 100.

[0031] In one embodiment, the objective lens 120 is a plan apochromatic ("APO") infinity-corrected objective lens suitable for transmission-mode illumination microscopes, reflection-mode illumination microscopes, and / or epi-illumination-mode fluorescence microscopes (e.g., Olympus 40X with 0.75 NA, or Olympus 20X with 0.75 NA). Advantageously, the objective lens 120 can correct for chromatic and spherical aberrations. Because the objective lens 120 is infinity-corrected, a focusing optical system 128 can be positioned in the optical path 122 above the objective lens 120, where the light beam passing through the objective lens 120 becomes a collimated light beam. The focusing optical system 128 focuses the light signal captured by the objective lens 120 to the photoresponse elements of a line-scan camera 130 and / or an area-scan camera 132, and may include optical components such as filters and / or magnification-changing lenses. The objective lens 120, combined with the focusing optical system 128, provides the total magnification for the scanning system 100. In one embodiment, the focusing optical system 128 may include a tube lens and an optional 2X magnification changer. Advantageously, the 2X magnification changer allows the objective lens 120, which is originally 20X, to scan the sample 116 at 40X magnification.

[0032] The line scan camera 130 includes at least one linear array of pixels 142 ("pixels"). The line scan camera 130 can be monochrome or color. A color line scan camera typically has at least three linear arrays, while a monochrome line scan camera can have one or more linear arrays. Any kind of single or multiple linear arrays can also be used, whether packaged as part of the camera or custom-integrated into the imaging electronics module. For example, a 3-linear array ("red-green-blue" or "RGB") color line scan camera or a 96-linear array monochrome TDI can be used. TDI line scan cameras typically provide a significantly better signal-to-noise ratio ("SNR") in the output signal by summing intensity data from previously imaged areas of the sample, resulting in an increase in SNR proportional to the square root of the number of integration stages. TDI line scan cameras include multiple linear arrays. For example, TDI line scan cameras are available with 24, 32, 48, 64, 96, or even more linear arrays. The scanning system 100 also supports linear arrays manufactured in various forms, including 512-pixel, 1024-pixel, and 4096-pixel formats. Similarly, linear arrays with various pixel sizes can also be used with the scanning system 100. A notable requirement for selecting any type of line scan camera 130 is that the movement of the stage 112 can be synchronized with the line rate of the line scan camera 130, so that the stage 112 can move relative to the line scan camera 130 during the acquisition of a digital image of the sample 116.

[0033] In one embodiment, image data generated by the line scan camera 130 is stored in a portion of the memory 106 and processed by the processor 104 to generate a series of digital images of at least a portion of the sample 116. The series of digital images can be further processed by the processor 104, and the processed series of digital images can also be stored in the memory 106.

[0034] In embodiments having two or more line scan cameras 130, at least one of these line scan cameras 130 can be configured to function as a focusing sensor operating in combination with at least one other line scan camera 130 configured to function as an image sensor 130A. The focusing sensor can be logically positioned on the same optical axis as the image sensor 130A, or the focusing sensor can be logically positioned upstream or downstream of the image sensor 130A with respect to the scanning direction of the scanning system 100. In such embodiments having at least one line scan camera 130 functioning as a focusing sensor, the image data generated by the focusing sensor can be stored in part of the memory 106 and processed by the processor 104 to generate focus information that enables the scanning system 100 to adjust the relative distance between the sample 116 and the objective lens 120 to maintain focus on the sample 116 during scanning. Furthermore, in one embodiment, the at least one line scan camera 130 functioning as a focusing sensor can be oriented so that each of the multiple individual pixels 142 of the focusing sensor is positioned at a different logical height along the optical path 122.

[0035] During operation, various components of the scanning system 100 and programmed modules stored in memory 106 enable the automatic scanning and digitization of the sample 116 placed on the glass slide 114. The glass slide 114 is securely positioned on the movable stage 112 of the scanning system 100 for scanning the sample 116. Under the control of the processor 104, the movable stage 112 accelerates the sample 116 to a substantially constant speed for sensing by the line scan camera 130, in which case the speed of the stage 112 is synchronized with the line rate of the line scan camera 130. After scanning the stripes of image data, the movable stage 112 decelerates, bringing the sample 116 to a substantially complete stop. The movable stage 112 then moves perpendicular to the scanning direction to position the sample 116 for scanning subsequent stripes of image data (e.g., adjacent stripes). Subsequently, additional stripes are scanned until part or all of the sample 116 or the entire sample 116 has been scanned.

[0036] For example, during the digital scanning of sample 116, a series of digital images of sample 116 are acquired as multiple consecutive fields of view that are combined together to form an image stripe. Similarly, multiple adjacent image stripes are combined together to form a series of digital images of part or all of sample 116. Scanning of sample 116 may include acquiring vertical or horizontal image stripes. Scanning of sample 116 may be from top to bottom, bottom to top, or both (i.e., bidirectional) and can begin at any point on sample 116. Alternatively, scanning of sample 116 may be from left to right, right to left, or both (i.e., bidirectional) and can begin at any point on sample 116. The image stripes do not need to be acquired in an adjacent or consecutive manner. Furthermore, the resulting image of sample 116 may be an image of the entire sample 116 or an image of only a portion of the sample 116.

[0037] In one embodiment, computer-executable instructions (e.g., programmed modules and software) are stored in memory 106, and when executed, the computer-executable instructions enable the scanning system 100 to perform various functions described herein (e.g., displaying a graphical user interface, executing a disclosed process, controlling components of the scanning system 100, etc.). The term “computer-readable storage medium” as used herein is used to refer to any medium used to store computer-executable instructions and to provide them to the scanning system 100 for execution by the processor 104. Examples of these media include memory 106 and any removable or external storage medium (not shown) that is communicably coupled to the scanning system 100 directly (e.g., via a universal serial bus (USB), wireless communication protocol, etc.) or indirectly (e.g., via a wired and / or wireless network).

[0038] Figure 1B shows a line scan camera 130 having a single linear array 140, which can be implemented as a charge-coupled device ("CCD") array. The single linear array 140 contains a plurality of individual pixels 142. In the illustrated embodiment, the single linear array 140 has 4096 pixels 142. In alternative embodiments, the linear array 140 may have more or fewer pixels. For example, common forms of the linear array include 512, 1024, and 4096 pixels. The pixels 142 are arranged linearly to define the field of view 134 of the linear array 140. The size of the field of view 134 varies depending on the magnification of the scanning system 100.

[0039] Figure 1C shows a line scan camera 130 having three linear arrays 140, each of which can be implemented as a CCD array. The three linear arrays 140 are combined to form a single color array 150. In one embodiment, each individual linear array within the color array 150 detects different color intensities, such as red, green, or blue. The color image data from each individual linear array 140 within the color array 150 are combined to form a single field of view 134 of color image data.

[0040] Figure 1D shows a line scan camera 130 having multiple linear arrays 140, each of which can be implemented as a CCD array. The multiple linear arrays 140 are combined to form a single TDI array 160. Advantageously, the TDI line scan camera can provide significantly better signal-to-noise ratio (SNR) in the output signal by summing intensity data from previously imaged areas of the sample, resulting in an increase in SNR proportional to the square root of the number of linear arrays 140 (also called the integration stage). The TDI line scan camera can include a wider variety of numbers of linear arrays 140. For example, common forms of the TDI line scan camera include 24, 32, 48, 64, 96, 120, and many more linear arrays 140.

[0041] Figure 1E shows an exemplary side view of a line scan camera 130 in a scanning system 100 according to one embodiment. In the illustrated embodiment, the scanning system 100 includes a glass slide 114 with a tissue sample 116, which is placed on a motorized stage 112, illuminated by an illumination system 118, and moved in a scanning direction 170. The objective lens 120 has an optical field of view 134 trained on the slide 114 and provides an optical path 122 for light from the illumination system 118. The light from the illumination system 118 is either light that has passed through the sample 116 on the slide 114, light that has been reflected by the sample 116 on the slide 114, light that has been fluorescently emitted from the sample 116 on the slide 114, or otherwise light that has passed through the objective lens 120. The light travels along the optical path 122 to the beam splitter 174, which allows a portion of the light to pass through the lens 176 and reach the main image sensor 130A. The light may optionally be bent by a mirror 178, as shown in the illustrated embodiment. The image sensor 130A may be, for example, a linear charge-coupled device (CCD).

[0042] Other light travels from the beam splitter 174 through the lens 180 to the focusing sensor 130B. The focusing sensor 130B may also be, for example, a linear CCD. The light traveling to the image sensor 130A and the light traveling to the focusing sensor 130B preferably represent the complete optical field 134 from the objective lens 120, respectively. Based on this configuration of the scanning system 100, the scanning direction 170 of the slide 114 is logically oriented with respect to the image sensor 130A and the focusing sensor 130B such that the optical field 134 of the objective lens 120 passes through the respective image sensor 130A and focusing sensor 130B by a logical scanning direction 172.

[0043] Figure 1F shows an exemplary top view of the imaging sensor 130A with respect to the imaging optical path 122A according to one embodiment. Similarly, Figure 1G shows an exemplary top view of the focusing sensor 130B with respect to the focusing optical path 122B according to one embodiment. As can be seen in Figure 1G, the focusing sensor 130B is inclined at an angle θ with respect to a direction perpendicular to the focusing optical path 122B.

[0044] Figure 1H shows an exemplary focusing sensor 130B according to one embodiment. In the illustrated embodiment, the focusing sensor 130B includes a plurality of sensor pixels 142 within a focusing range (d) (e.g., 20 μm) on a tissue sample. As shown, the focusing sensor 130B can be positioned such that the entire focusing range (d) in the Z-axis is projected by the optical system onto the entire array of focusing sensors 130B in the Y-axis (orthogonal to the X-axis, i.e., the scanning direction 170). The location of each sensor pixel 142 is directly correlated to the Z position of the objective lens 120. As shown in Figure 1H, each dashed line (i.e., p1, p2, ... p) crosses the projected focusing range (d). i ,···p n ) each represents a different focal value and corresponds to the respective focal height (i.e., Z height) of the objective lens 120. p has the optimal focus (e.g., the best contrast index) for a given part of the sample 116. i By using this with the scanning system 100, it becomes possible to determine the optimal focal height for that part of the sample 116.

[0045] The relationship between the focus range (d) projected onto the focus sensor 130B and the focus range (z) on the sample 116 is: d = z × M focusing 2 And here, M focusing is the optical magnification of the focused optical path. For example, z = 20 μm and M focusing If = 20, then d = 8 mm.

[0046] In order for the entire projected in-focus range (d) to be covered by the tilt in-focus sensor 130B that includes the linear array 140, the tilt angle θ satisfies the following relationship: sinθ = d / L where L is the length of the linear array 140 of the in-focus sensor 130B. Using d = 8 mm and L = 20.48 mm, θ = 23.0°. As long as the tilt in-focus sensor 130B covers the entire in-focus range (d), θ and L can be varied.

[0047] The in-focus resolution, or the minimum increment Δz of the movement of the objective lens in the height direction, is a function of the size of the sensor pixel 142, i.e., e = minimum value (ΔL). From the above equation, Δz = e × z / L is derived. For example, when e = 10 μm, L = 20.48 mm, and z = 20 μm, Δz = 0.0097 μm < 10 nm.

[0048] The height Z of the objective lens i and the in-focus position L of the focus i on the in-focus sensor 130B i are related by L i = Z i × M focusing 2 / sinθ where M is a magnification factor.

[0049] <UNK> When the focal height is determined by the average from L1 to L2, according to the analysis of the data from the in-focus sensor 130B, the height of the objective lens 120 needs to be moved from Z1 to Z2 based on Z2 = Z1 + (L2 - L1) × sinθ / M focusing 2 where M is a magnification factor.

[0050] The fields of view (FOV) 134 in the Y-axis of the in-focus sensor 130B and the imaging sensor 130A may be different, but the centers of both sensors 130A and 130B are preferably aligned with each other along the Y-axis.

[0051] 2. Process Overview

[0052] The following describes in detail an embodiment of the process for real-time focusing in a slide scanning system. It should be understood that the described process can be implemented in one or more software modules executed by one or more hardware processors 104 within the scanning system 100. The described process can be implemented as instructions represented by source code, object code, and / or machine code. These instructions may be executed directly by the hardware processor, or alternatively, by a virtual machine operating between the object code and the hardware processor.

[0053] Alternatively, the described processes may be implemented as hardware components (e.g., general-purpose processors, integrated circuits (ICs), application-specific integrated circuits (ASICs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates, or transistor logic), combinations of hardware components, or combinations of hardware and software components. To clearly illustrate hardware and software compatibility, various exemplary components, blocks, modules, circuits, and steps are generally described herein in terms of their functionality. Whether such functionality is implemented as hardware or as software depends on the specific application and design constraints imposed on the overall system. Those skilled in the art will know that the described functionality can be implemented in different ways for each specific application, but such implementation decisions should not be construed as resulting in a departure from the scope of the invention. Furthermore, the grouping of functionality within components, blocks, modules, circuits, or steps is for the sake of simplicity of explanation. Without departing from the invention, specific functionality or steps can be moved from one component, block, module, circuit, or step to another.

[0054] Furthermore, although the processes described herein are shown in a specific arrangement and order of multiple steps, each process may be implemented in fewer steps, more steps, or different steps, and in different arrangements and / or orders of steps. Moreover, it should be understood that any step that does not depend on the completion of another step can be performed before, after, or in parallel with other independent steps, even if these steps are described or illustrated in a specific order.

[0055] In one embodiment, the scanning system 100 uses a focus map to predict the trajectory of the objective lens 120 during scanning of each image stripe. Focal values ​​for the focus map can be measured using two methods: (1) a macro focus point (MFP) method and (2) a real-time focus (RTF) method. In the case of the MFP method, the focus values ​​are calculated before scanning the image stripe and / or between acquisitions of the image stripe, whereas in the case of the RTF method, the focus values ​​are calculated while acquiring the image stripe. Both methods can be used in combination to create a focus map used to predict the focus position of the objective lens 120 during scanning. Advantageously, the RTF method provides far more focus values ​​to the focus map than using the MFP method alone, while adding little to no time to the scanning process.

[0056] The RTF method also provides real-time measurements of focus error. These focus error measurements can be analyzed during the scanning of the sample 116 to correct the trajectory of the objective lens 120 as the image stripe is scanned. This minimizes the focus error at the focal height predicted from the focus map.

[0057] 2.1. MFP method

[0058] In one embodiment, the MFP system includes using a line scan camera 130 to acquire image data along the entire Z-axis (for example, by moving the objective lens 120) while the stage 112 is moving at a constant speed. The image row with the highest contrast within the image data is then identified, and an encoder count (for example, with respect to the objective lens 120) corresponding to that row is calculated using a timing formula. A minimum contrast threshold, representing the noise threshold, can be used to ensure that the focus is above the noise threshold. Historically, this minimum contrast threshold has been approximately 350.

[0059] Historically, MFP systems require a macro focus offset to provide good image quality. Therefore, in one embodiment, the macro focus offset is calculated by testing the focus value in a closed-loop measurement to ensure the MFP system operates correctly. By design, the macro focus offset is required to be zero. However, in practice, there is a systematic error in the Z position of the objective lens 120, as calculated from a given formula and Z-stage adjustment.

[0060] The macro focus offset can be experimentally determined by macro-focusing on a location in the tissue and recording the contrast curve and the encoder count of the maximum contrast. The objective lens 120 can then be moved (i.e., along the Z-axis) to the recorded encoder count, and a second buffer of image data can be recorded along with the corresponding average contrast value. The average contrast value can be compared to the recorded contrast curve, and the distance from the recorded maximum contrast value to the average contrast value can be measured to provide the Z offset to be used as the macro focus offset.

[0061] Figure 2 shows an exemplary focus error before and after applying offset correction using a calculated macro focus offset according to one embodiment. As shown, the average error decreased from 0.6 to 0.0 microns, and the maximum error decreased from 1.0 to 0.4 microns. The impact on image quality was a decrease in the restriping rate (i.e., the rate at which image stripes need to be reacquired due to being out of focus). This offset correction is important because it allows for adding focus where and when needed, and provides ground truth for evaluating the RTF scheme.

[0062] In one embodiment, the MFP parameters are defined with exemplary nominal values ​​as follows and can be stored in a configuration file used by the scanning system 100 for configuration (for example, a “scanner.xml” file defined using Extended Markup Language (XML)):

[0063] Good_Focus_Threshold=350. If the maximum contrast value of a macro focus is less than this threshold, the focusing attempt for that macro focus is considered a failure, and the value for that macro focus is discarded.

[0064] Retry_Count=1. This is the number of times to attempt to refocus if the focusing attempt fails because the maximum contrast value at macro focus is less than the Good_Focus_Threshold value.

[0065] Macrofocus_Pos_Offset = +0.0006. This is the calculated macro focus offset value (in millimeters) due to the Z-motion error of objective lens 120.

[0066] 2.2. RTF method

[0067] In one embodiment, the RTF method utilizes two line scan cameras 130, namely a main imaging sensor 130A (e.g., a trilinear RGB camera) and a single-channel focusing sensor 130B (e.g., a monochromatic camera). Both line scan cameras 130 are aligned so that their respective linear arrays 140 image the same portion of the sample 116 (which may include, for example, tissue). For example, the main imaging sensor 130A may be parallel to the plane of the sample 116 and can function similarly to a trilinear camera used in Aperio ScanScope® products. On the other hand, the focusing sensor 130B is tiltable along the optical Z-axis (e.g., perpendicular to the scanning direction 170) along the linear array 140 of the focusing sensor 130B.

[0068] 2.2.1. design

[0069] Line scan cameras 130A and 130B can be aligned with each other so that when the main image sensor 130A is at its best focus, the point of maximum contrast on the tilt focus sensor 130B is near the center of the tilted linear array 140. The pixel 142 at this point in the tilted linear array 140 of the focus sensor 130B is called the parfocal position. As the objective lens 120 moves up or down relative to the best focus position, the point of maximum contrast on the tilt focus sensor 130B moves left or right relative to the parfocal position. This makes it possible to dynamically determine the direction and amount of the focus error of the main image sensor 130A using the tilt focus sensor 130B. Using the measured focus error, the position of the objective lens 120 can be adjusted in real time so that the main image sensor 130A is always in focus.

[0070] In embodiments of the RTF method, one or more, preferably all, of the following issues can be addressed:

[0071] Changes in tissue contrast across the linear array. In one embodiment, changes in contrast also arise from changes in tissue characteristics across the linear array 140, so the tilt focusing sensor 130B is not the only one used to identify the best focus position. Data from the focusing sensor 130B and data from the main imaging sensor 130A can be combined and the effect of tissue can be removed using the ratio method. The ratio method divides the contrast function of the tilt focusing sensor 130B by the contrast function of the main imaging sensor 130A. This normalized function peaks at the best focus position and removes the effect of tissue.

[0072] Noise and camera alignment errors. In one embodiment, the linear array 140 has 1 × 4096 pixels, and the nominal image pixel dimension is 0.25 microns. At this level of precision, it is impossible to accurately align the main image sensor 130A and the tilt focusing sensor 130B. Furthermore, due to the tilt focusing sensor 130B being tilted with respect to the optical axis, there is a slight change in magnification across the tilt focusing sensor 130B. The contrast value of individual pixels 142 is also noisy because it is calculated by finding the difference in values ​​between adjacent pixels 142. To mitigate these effects, spatial averaging can be used. For example, in one embodiment, the output of each line scan camera 130 is grouped into consecutive frames as it scans the image stripe. One frame is 1000 lines × 4096 pixels. After the pixel contrast value for one frame is calculated, multiple lines are averaged into a single line. A boxcar filter (e.g., 100 pixel width) is then applied across this single line. These operations are performed on frames from both the main imaging sensor 130A and the tilt focusing sensor 130B. These two averaging operations significantly reduce the effects of noise and alignment errors in the ratio method.

[0073] The parfocal position. The value of the parfocal position is known and can be calibrated separately. If there is an error in the parfocal position, the RTF method will shift to a position that is not the optimal focus. Methods for calculating the parfocal position are described elsewhere in this specification.

[0074] Skipped frames. In one embodiment, all other frames are “skipped” frames because the data for these frames is not analyzed for focus errors. While a skipped frame is being acquired, the data for the previous frame is analyzed, and the objective lens 120 is moved to the best focus position for the next non-skipped frame. In other words, the objective lens 120 moves only while a skipped frame is being acquired and remains stationary while a non-skipped frame is being acquired. In normal operation, the frame lag is one frame, which means that the current frame under the objective lens (CFUO) is the frame following the frame being analyzed.

[0075] Over-framing. Over-framing is a term used when the frame lag is greater than one frame. In one embodiment, the scanning of the image stripe proceeds at a constant speed, so a software timer is implemented, which allows for the calculation of the CFUO. The RTF method checks the CFUO when determining the next best focus position. Typically, the CFUO is the skipped frame that follows the frame being analyzed. If not, the RTF method predicts the best focus position for the first unskipped frame following the CFUO and instructs the objective lens 120 to move to that position. Using this strategy, the objective lens 120 is always moved to the best position for its current actual position. The RTF method works properly as long as the frame lag is not too large. If the frame lag becomes too large (i.e., exceeds the value of the Frame_Lag_Stripe_Abort_Threshold parameter), a configuration parameter Frame_Lag_Stripe_Abort_Threshold can be provided to interrupt and rescan the scanning of the image stripe.

[0076] Ratio curve fitting. The ratio method generates a curve that peaks at the best focus position. In one embodiment, a Gaussian function is fitted to the ratio curve, and the peak of the Gaussian function is identified as the best focus position. The quality of the Gaussian function fitting can be evaluated using a set of metrics to certify that the focus estimate is reliable. Only reliable focus values ​​are added to the focus map.

[0077] Focusing only on tissue. Points added to the focus map are required to correspond to the actual location of tissue in the image data. In previous designs, a tissue discovery algorithm was used to calculate a probable tissue map from the macro image, and MFP values ​​were added only to the locations of probable tissue. However, various artifacts on the glass slide 114 (e.g., coverslip edges, plus marks, debris on the coverslip, etc.) can undesirably result in focusing outside the plane of the tissue. In one embodiment, a probable tissue map from a tissue discovery algorithm can be input to the RTF scheme and optionally used to constrain the acceptable locations for focus used by the RTF scheme. The RTF scheme can also analyze each frame to determine whether tissue is present and allow focus only on frames in which tissue is present.

[0078] Texture gaps. For highly reliable estimation of the focus error, sufficient contrast texture must be available across the linear array 140. In one embodiment, a contrast vector for the main imaging sensor 130A is used to determine whether sufficient signal is available to calculate the focus error. Details of this implementation are described elsewhere in this specification.

[0079] Addition of MFP values. The RTF method does not always return a reliable focal value for each tissue frame. This may be due to insufficient tissue in the frame, as well as the result being classified as unreliable by the Gaussian fitting process. In one embodiment, after the image stripe is scanned, the tissue frames are considered sequentially, and frames that are beyond a predetermined distance (stored, for example, as a morphological value of the Rmin parameter) from the nearest reliable focal value are identified by the MFP for focus. These points can be focused and added to the focus map before scanning the next image stripe.

[0080] Rejection of outliers. In one embodiment, after all image stripes have been scanned, the focus map is examined to determine whether any of the foci in the focus map are outliers. Before evaluating the scan quality, outliers can be removed as discussed elsewhere in this specification.

[0081] Rescanning of image stripes. In one embodiment, after all image stripes have been scanned and the focus map is completed, the actual focus position for each tissue frame is compared to the best focus position from the focus map, as described in the restriping process discussed elsewhere in this specification. The final focus map can be created by rescanning image stripes such that 5% of the frames have a focus error exceeding a predetermined threshold.

[0082] Scan initialization. In one embodiment, a focus map for scanning is initialized. A tissue discovery algorithm can identify a reference stripe and three or more MFP values ​​to initiate the scan. The image stripe with the most tissue can be selected as the reference stripe. The MFP focus is measurable at the beginning of the reference stripe and at two or more other locations selected to provide good tissue spacing. The reference stripe is then scanned first using the starting MFP focus value of the stripe, and the focus position is updated using the RTF method. Scanning the reference stripe typically generates a number of focus values, and these focus values ​​are added to the focus map along with the initial MFP focus values.

[0083] Scanning sequence. In one embodiment, after the focus map is initialized, the reference stripe is rescanned. The scan then proceeds to the right or left of the reference stripe until that side of the scanning region is completed. The scan then proceeds from the other side of the reference stripe to the opposite edge of the scanning region. This sequence is chosen so that the image stripe being scanned is as close as possible to the focus value in the focus map in order to maximize the possibility of obtaining an acceptable focus in the first pass.

[0084] 2.2.2. parfocal position

[0085] In one embodiment, to determine the parfocal position, a vertical Ronchi slide is simultaneously imaged by both the main image sensor 130A and the tilt focusing sensor 130B by sweeping the Z range of the objective lens 120 at a constant speed. In this way, buffer pairs of image data are acquired for sensors 130A and 130B.

[0086] In the case of the main image sensor 130A, the average contrast value is calculated for each row of the buffer, and the row with the highest contrast is adopted as the best focus index. An additional check can be added to certify buffer pairs based on the contrast gradient of the image data from the main image sensor 130A. This check can be performed by dividing the buffer of image data from the main image sensor 130A into 40 segments (e.g., approximately 100 columns each) and calculating the best focus index for each segment. If the difference in index between the leftmost segment and the rightmost segment is less than a threshold (e.g., 4), the buffer pair is accepted as having the minimum gradient and is used to estimate the parfocal position. The threshold value of 4 corresponds to a gradient of 0.5 microns / millimeter, which may be a system requirement regarding flatness.

[0087] The tilt focusing sensor 130B does not have a single best focus index (i.e., row). Each column in the image data buffer from the focusing sensor 130B has maximum contrast when the corresponding pixel 142 is at the best focus. Processing the focus buffer involves calculating the gradient for each row, and then finding the row index corresponding to the maximum value for each column. This data can be very noisy, as shown in the illustrative graph of the low-pass filtered signal and fitting in Figure 3. Therefore, due to the asymmetry of the noise, a median filter can be used rather than a mean filter to reduce the noise. A linear fit is then constructed for the filtered index values, as shown by the fit line in Figure 3. The column from the image data acquired by the main imaging camera where the linear fit intersects with the maximum index value is the parfocal point in the tilt focusing sensor 130B.

[0088] The linear fit slope corresponds to the change in Z distance per pixel and is required to calculate the actual distance to move the objective lens 120 to parfocal. However, the left side of the parfocal position appears to have a larger slope than the right side. Therefore, as shown in Figure 3, the linear fit is calculated separately for the data on the left side of the parfocal position and for the data on the right side of the parfocal position. These two slopes provide the left scale factor and right scale factor in microns / pixel units.

[0089] In one embodiment, the output of the parfocal calculation includes the parfocal position, a left scale factor, and a right scale factor. The parfocal position is the position of the parfocal pixel 142 on the tilt focusing sensor 130B. The left scale factor is used to convert the pixel 142 to the left of the parfocal position into microns. Similarly, the right scale factor is used to convert the pixel 142 to the right of the parfocal position into microns.

[0090] Figure 4 shows a parfocal calculation 400 according to one embodiment. The parfocal calculation 400 can be implemented in the form of software instructions executed by the scanning system's processor 104 or an external system. In steps 405A and 405B, synchronized buffers are received from the tilt focusing sensor 130B and the main imaging sensor 130A, respectively. In one embodiment, each buffer contains image data acquired by each sensor during scanning of a vertical Ronkey slide.

[0091] In steps 410A and 410B, the contrast gradient for each row in each buffer is calculated according to a predefined step size. By default, the step size for both buffers can be set to one row to prevent rows from being skipped. Alternatively, the step size may be larger than one row.

[0092] In step 415, the maximum value of the contrast gradient is found with respect to the focusing sensor 130B, and the column index corresponding to that maximum value is identified. Next, in step 420, a median filter (e.g., default = 500) is applied to the points of the contrast gradient. In step 425, a linear fit is found for the median-filtered points. The line representing this linear fit is sent to step 470, under the assumption that a line will also be found for the image data buffer from the main imaging sensor 130A in step 465.

[0093] In step 410B, the contrast gradient calculated from the buffer of the main imaging sensor 130A is averaged across each row in step 430. Then, in step 435, the R, G, and B color channels are averaged. In step 440, the contrast gradient is segmented. By default, the number of segments used in step 440 can be, for example, 40.

[0094] In step 445, the leftmost segment from step 440 is compared with the rightmost segment from step 440. Specifically, the rightmost segment can be subtracted from the leftmost segment, or vice versa. If the absolute value of the difference is greater than or equal to a predetermined threshold T (i.e., "no" in step 450), the buffer pair received in step 405 can be discarded in step 455, and process 400 can be restarted with a new buffer pair. Otherwise, i.e., if the absolute value of the difference is less than a predetermined threshold T (i.e., "yes" in step 450), process 400 proceeds to step 460. In one embodiment, the predetermined threshold T is equal to 4.

[0095] In step 460, the maximum value of the averages calculated in steps 430 and 435 is found, and the row index corresponding to that maximum value is identified. Then, in step 465, a line is calculated at this point.

[0096] If a line is found with respect to the focusing sensor 130B in step 425 and a line is found with respect to the imaging sensor 130A in step 465, then in step 470, the intersection or crossing point of these two lines is found. This intersection is the point of parfocality. Furthermore, in step 475, linear fits are independently found for both the segment to the left of the point of parfocality and the segment to the right of the point of parfocality. Finally, in step 480, the slope of the left fit line is converted to a left scale factor, and the slope of the right fit line is converted to a right scale factor.

[0097] 2.2.3. RTF workflow

[0098] Figures 5A to 5C illustrate a process 500 for scanning an image of a sample 116 on a glass slide 114 according to one embodiment. Many of the illustrated steps are described in more detail elsewhere in this specification. It should be understood that process 500 can be implemented in the form of software instructions executed by the processor 104 of the scanning system 100.

[0099] Process 500 begins in step 510 by acquiring image stripes representing a portion of the image data of sample 116. In step 590, it is determined whether the last image stripe has been acquired. If there are still image stripes to be acquired (i.e., "no" in step 590), the next image stripe is acquired in a further iterative step 510. Otherwise, i.e., there are no image stripes to be acquired (i.e., "yes" in step 590), process 500 proceeds to step 592. In particular, the image stripes can be acquired in any order. Advantageously, the ability of the image stripes to be acquired in any order allows the processor 104 of the scanning system 100 to more effectively construct a focus map during scanning using the focus values ​​acquired by the RTF method.

[0100] In step 592, outliers are removed, as discussed in more detail elsewhere in this specification. In step 594, it is determined whether any of the image stripes need to be restriped, as discussed in more detail elsewhere in this specification. If the image stripes do not need to be restriped (i.e., “no” in step 594), process 500 ends with a complete set of image stripes for at least a portion of the sample 116. Otherwise, i.e., at least one image stripe needs to be restriped (i.e., “yes” in step 594), in step 596, those image stripes are rescanned, and then process 500 ends with a complete set of image stripes for at least a portion of the sample 116. Once a complete set of image stripes is obtained, the processor 104 of the scanning system 100 can align these image stripes and combine them to form a single complete composite image of the entire scanned portion of the sample 116. Furthermore, the processor 104 can compress the composite image using any known compression technique.

[0101] An embodiment of step 510 is shown in more detail in Figure 5B. Specifically, in step 512, a frame of the image stripe is captured at the calculated Z position. Then, in step 520, the captured frame is processed to determine the Z position of the next frame. If there are still frames to be acquired (i.e., "no" in step 580), in a further iteration step 512, the next frame is acquired at the Z position determined in step 520. Otherwise, i.e., there are no frames to be acquired for that image stripe (i.e., "yes" in step 580), process 500 proceeds to step 582.

[0102] In step 582, the Z position with the best focus for an analyzable and reliable frame is added to the focus map. In step 584, an additional macro focus is requested. In step 586, the scanning direction is set. That is, process 500 determines whether to move to the left or right of the current image stripe in order to capture the next image stripe.

[0103] An embodiment of step 520 is shown in more detail in Figure 5C. Buffers of image data representing frames, each captured by both the main imaging sensor 130A and the tilt focusing sensor 130B, are processed in step 522 to generate the position of the maximum value in the buffer of the tilt focusing sensor 130B, the mean square gradient of the buffer of the main imaging sensor 130A, the total number of analyzable columns in the buffer of the main imaging sensor 130A, the weight vector of the main imaging sensor 130A, and the ratio vector (parfocal ratio).

[0104] In step 524, it is determined whether the captured frame is analyzable. For example, the frame is determined to be analyzable if it has sufficient tissue to perform the Gaussian fitting process, as described elsewhere in this specification. If the frame is analyzable (i.e., "yes" in step 524), process 500 proceeds to step 525. Otherwise, i.e., if the frame is not analyzable (i.e., "no" in step 524), process 500 proceeds to step 530.

[0105] In step 525, Gaussian fitting is performed. In step 526, it is determined whether the Gaussian fitting process is reliable (i.e., a good fit) or unreliable (i.e., a poor fit). If the Gaussian fitting result is reliable (i.e., "yes" in step 526), ​​process 500 provides the resulting predicted delta Z (i.e., the predicted change in Z value to maintain focus) to step 528. Otherwise, i.e., if the Gaussian fitting result is unreliable (i.e., "no" in step 526), ​​process 500 sets the frame as unanalyzable and unreliable and proceeds to step 530.

[0106] In step 528, the best focus position is calculated as the sum of the current frame's Z position and the predicted delta Z from the Gaussian fitting process in step 526. Alternatively, in step 530, the best focus position is simply set to the current frame's Z position. In either case, in step 532, the Z position for the next frame is calculated.

[0107] 2.2.4. ratio method

[0108] As discussed with respect to process 500, in one embodiment, image data from the imaging sensor 130A and the focusing sensor 130B are captured within a frame. A single frame contains two buffers: one buffer corresponding to data from the main imaging sensor 130A and another buffer corresponding to data from the tilt focusing sensor 130B. Figure 6A shows an exemplary buffer of image data acquired by the main imaging sensor 130A according to one embodiment, and Figure 6B shows an exemplary buffer of image data acquired by the tilt focusing sensor 130B according to one embodiment. In the illustrated embodiment, each frame consists of two buffers, each with a width of 1000 lines × 4096 pixels.

[0109] Figure 7 shows a process 700 for calculating a ratio vector according to one embodiment. It should be understood that process 700 can be implemented in the form of software instructions executed by the processor 104 of the scanning system 100. Furthermore, process 700 can represent at least a portion of steps 522 and / or 524 in process 500.

[0110] In steps 705A and 705B, frames of image data are received from the main image sensor 130A and the tilt focusing sensor 130B, respectively. Then, in step 710, illumination correction is applied to the image pixels within each frame. For example, illumination correction can utilize sensitivity non-uniformity (PRNU) and / or fixed pattern noise (FPN) techniques. In step 715, the RGB channels within the frame from the main image sensor 130A are each corrected separately and then averaged with equal weighting to form a grayscale frame.

[0111] In step 720, the squared gradient operator is applied to each grayscale frame, i.e., the main imaging frame corrected in step 710A and converted in step 715, and the tilted focus frame corrected in step 710B. The center difference in both horizontal dimensions can be used, with a difference interval of 8 pixels (D8).

[0112] In step 725, the gradient images are averaged along the columns. This converts each frame into a single vector. Then, in step 730, a boxcar filter (e.g., 101 pixels) is applied to each vector to reduce residual noise. Finally, in step 735, the ratio vector is calculated by dividing the pixel values ​​of the two vectors.

[0113] In one embodiment, process 700 is performed on all sets of frames captured by the imaging sensor 130A and the focusing sensor 130B, including skipped frames. In this case, only the frames that were not skipped are analyzed for the optimal focus position, but it is still useful to know whether tissue is present in each frame.

[0114] In addition to the gradient and ratio vectors for each set of frames, it is also possible to calculate the pixel position of the maximum value in the gradient vector and the total number of analyzable columns in the main imaging gradient vector. (For example, in step 524 of process 500) The number of analyzable columns can be used to determine whether there is enough signal to allow further analysis by the Gaussian fitting process.

[0115] Figure 8A shows an exemplary gradient vector for the main imaging sensor 130A (a darker line graph starting and ending at the top) and an exemplary gradient vector for the tilt focusing sensor 130B (a brighter line graph starting and ending at the bottom) according to one embodiment. The structure of the tilt focusing sensor 130B is irregular due to structural variations across frames. Therefore, a single peak cannot be identified in the gradient vector for the focusing sensor 130B.

[0116] Figure 8B shows an exemplary ratio vector from the two gradient vectors in Figure 8A, according to one embodiment. In particular, the ratio vector is much smoother than either of the gradient vectors. As shown, a Gaussian function can be fitted to the ratio vector to enable precise identification of peak 800. Peak 800 in the Gaussian function represents the best focus position for this frame.

[0117] In one embodiment, the output from the ratio method exemplified by process 700 is an input to a Gaussian fitting process and may include one or more of the following:

[0118] Principal gradient vector. The principal gradient vector is the average gradient signal for each column of frames captured by the main imaging sensor 130A.

[0119] Gradient vector. The gradient vector is the average gradient signal for each column of the frame captured by the tilt focusing sensor 130B.

[0120] Ratio vector. The ratio vector is the ratio obtained by dividing each column of the gradient vector by the principal gradient vector.

[0121] The ratio at parfocal points. The ratio at parfocal points is the value of the ratio curve represented by the ratio vector at the parfocal position.

[0122] The baseline ratio. The baseline ratio is the average value of the ratio curve represented by the ratio vector near the end of the ratio vector.

[0123] The number of analyzable columns. The number of analyzable columns is the number of columns in the principal gradient vector whose principal gradient vector exceeds a threshold (for example, MAIN_IMG_COLM_ANALYZABLE_THRESH=50).

[0124] Weight vectors. The principal gradient vectors are normalized to a unit area and used as weight vectors for Gaussian fitting. Columns (i.e., pixels) with little structure (i.e., small gradient values) are of less importance when fitting a Gaussian function to the ratios because these ratio values ​​are very noisy. The larger the gradient value, the more signal there is, and accordingly, the less noise there is in the ratios.

[0125] 2.2.5. Gaussian fitting

[0126] The Gaussian fitting process is represented as step 525 of process 500. The purpose of the Gaussian fitting process is to fit a smooth Gaussian function to the ratio curve and then identify the peak of this Gaussian function (e.g., peak 800 in Figure 8B). This peak represents the calculated best focus position. In one embodiment, the Gaussian fitting process is attempted only when sufficient tissue is present and the ratio curve has acceptable values ​​at the parfocal point. For example, the Gaussian fitting process is performed when the ratio at the parfocal point is between 0.5 and 1.5 and the number of analyzable columns exceeds 85% of the total columns (e.g., "yes" in step 524 of process 500).

[0127] Fitting a Gaussian function to a ratio curve is a nonlinear problem. In one embodiment, an approach to solve this nonlinear problem is to sample a set of possible Gaussian functions and select the Gaussian function that has the smallest root mean square (RMS) difference from the ratio curve.

[0128] The Gaussian curve for each sample has four parameters: amplitude (peak), center (mean), width (sigma), and baseline (offset). The amplitude is parameterized as a function of the distance from the parfocal point. Figure 9 shows the ratio curve for a tissue sample scanned at a fixed offset from the parfocal point. In this example, the parfocal point is at column (i.e., pixel) 1590. In particular, the peak increases as the offset from the parfocal point increases. The peak size is also tissue-dependent. In this way, the rate of increase of the peak from the parfocal point can be estimated for each frame.

[0129] In one embodiment, to scale the Gaussian test function, the position of the maximum value of the slope vector is identified, as shown in the leftmost graph of Figure 10. Using the ratio vector value at this position and the distance from the parfocal to this position, the slope of the fitting can be defined, as shown in the rightmost graph of Figure 10.

[0130] In one embodiment, the Gaussian test function is scaled by a ratio equal to the slope of the fitting, such that the peak increases with distance from the parfocal point. One of these Gaussian test functions is shown in the rightmost graph of Figure 10 as a smooth line approximating the ratio curve. In particular, the best Gaussian function does not need to be centered at the peak position obtained from the ratio curve, because this peak does not necessarily need to be centered and is used only for scaling. The ratio at the baseline of the leftmost graph of Figure 10 is approximately 3.5. This value is estimated from the end of the ratio curve and is used as an offset applied to the Gaussian test function to raise it to an appropriate level.

[0131] In one embodiment, a set of mean and sigma values ​​is used to generate a Gaussian test function. The mean may range from 100 to 3996 (i.e., 4096-100) in increments of 25. The sigma value may range from 500 to 1200 in increments of 100. Figure 11 shows a partial set of Gaussian test functions for several different mean values ​​and one fixed range.

[0132] A second set of modified Gaussian functions (one-sided only) is also added to this set. This is because the ratio curve may not have a single well-defined shape corresponding to a single peak. For example, the ratio curve may be wider when there are two peaks, as shown in Figure 12. Subjective image quality has been empirically found to be best when the rightmost peak is in focus. In the case shown in Figure 12, this rightmost peak is also the largest and most prominent peak. The problem with symmetric Gaussian functions is that they can result in a wide peak centered between the two peaks. In this case, neither set of features is in focus, the image will appear "soft," and will be defocused throughout the frame. The rightmost peak represents the direction away from the glass slide 114, and therefore this set of Gaussian functions prefers to focus on the upper features in the tissue section when two possible depths of focus are present.

[0133] Figure 13 shows two sets of Gaussian functions (i.e., symmetric and one-sided) and their mean values ​​corresponding to the parfocal positions. The complete complementation of the Gaussian test functions, including possible solutions, lies in the set of mean and sigma values ​​described above. Each of these Gaussian functions is normalized with respect to amplitude and baseline as described above. The RMS difference is then calculated for each Gaussian test function against the ratio vector. The Gaussian function associated with the smallest RMS difference value is selected as the best-fitting Gaussian function. Point 1400 in Figure 14 indicates the smallest RMS difference value and the position of the best-fitting Gaussian function.

[0134] In one embodiment, the Gaussian fitting process returns two numbers: the best focus position and the maximum fit value. The best focus position is the column (i.e., pixel) corresponding to the best focus. The focus error is proportional to the difference between this value and the parfocal point. The maximum fit value is the amplitude of the ratio vector at the optimal focus position.

[0135] In one embodiment, the return values ​​from the Gaussian fitting process are analyzed to determine whether these return values ​​are reliable. Only reliable values ​​are added to the focus map to be used for scanning subsequent image stripes. For example, the slope of the error is calculated and compared with the slope of the fitting defined above with respect to the Gaussian fitting. For the return values ​​to be reliable, these two slopes should be equal. Otherwise, the return values ​​should not be trusted. The slope of the error can be calculated as follows, as shown in Figure 15:

[0136] Error slope = (Maximum fit - Ratio at parfocal point) / (Best focus position - Parfocal point)

[0137] There are two possible cases:

[0138] Case 1: The error slope and the fitting slope have different signs (i.e., on both sides of the parfocal point). In this case, the return value is reliable if the absolute value of (maximum fit - parfocal ratio) / (parfocal ratio) is less than 0.20. These slopes are noisy near the parfocal point. Therefore, these slopes are not used to invalidate the Gaussian result.

[0139] Case 2: The slope of the error and the slope of the fitting have the same sign (i.e., on the same side of the same focus). In this case, the return value is reliable if (slope of fitting - slope of error) / (slope of fitting + slope of error) is less than 0.5. The difference between these slopes is equivalent to the average of these two slopes.

[0140] 2.2.6. Frame analysis possible score

[0141] In one embodiment, each frame of the image data receives one of the following status scores (for example, in step 524 of process 500):

[0142] NonAnalyzable (e.g., =-2): The frame does not have any structure.

[0143] NonAnalyzableButHasTissue (e.g., =-1): The frame has tissue, but it is insufficient for the Gaussian fitting process. In one embodiment, the frame has tissue if the mean of all columns with respect to the principal gradient vector is greater than the value of the MAIN_IMG_COLM_ANALYZABLE_THRESH parameter (e.g., 50).

[0144] AnalyzableButUntrustable(e.g., =0): The results of the Gaussian fitting process are not reliable.

[0145] AnalyzableButSkipped(e.g., =1): The frame has enough structure for the Gaussian fitting process, but it was skipped.

[0146] AnalyzableAndTrustable(e.g., =2): The Gaussian fitting process returned a reliable result. This point is added to the focus map that should be used for focusing when scanning subsequent image stripes.

[0147] MFPFrame (e.g., =3): A frame has a macro focus. In one embodiment, only one focus value is allowed per frame, so a frame that has received a macro focus before scanning begins will not receive an RTF value.

[0148] 2.2.7. Outlier rejection

[0149] After all image stripes have been scanned, the focus map is completed. In one embodiment, at this point, the focus map is analyzed to determine whether any point within the focus map (either an RTF point or an MFP point) is an outlier. This determination is represented as step 592 of process 500.

[0150] Outliers can be identified by considering the surface inclination away from the sample point. The inclination can be calculated in four directions on the surface away from each sample point: up, down, left, and right. If the minimum inclination exceeds a threshold, that point can be designated as an outlier.

[0151] Figure 16 shows an example viewed laterally along the focal plane. Two points, 1610 and 1620, are identified as outliers. Therefore, these two points, 1610 and 1620, are removed from the focal map (for example, in step 592 of process 500) before testing the stripe for possible restriping (for example, in step 594 of process 500).

[0152] 2.2.8. Restriping

[0153] In one embodiment, the focus error for each frame containing tissue is calculated by subtracting the actual position of the objective lens 120 during scanning of that frame from the best in-focus position for that frame, as identified from the final focus map. (For example, represented as step 594 of process 500) This calculation is performed after potential outliers have been removed from the focus map (for example, in step 592 of process 500). Figure 17A shows the focus errors after all image stripes have been scanned in the first pass. The red and dark blue frames represent positive and negative errors, respectively, while the green frames represent very small errors. Stripe seven is shown to have a dark red area, and therefore this image stripe is selectable for restriping. Figure 17B shows the focus errors after stripe seven has been restriped. After restriping, most of stripe seven is green, indicating that the objective lens 120 is substantially in agreement with the focus map.

[0154] Since the objective lens 120 moves in a stepwise manner only for frames that have not been skipped, the skipped frames generally have a small focus error after restriping. If a large focus error remains after restriping, this indicates a large tilt along the scanning axis, i.e., generally, insufficient focus. Final image quality evaluation can be performed from the heatmap after restriping (shown in the example in Figure 17B).

[0155] In one embodiment, the decision of whether to restripe (for example, in step 594 of process 500) is made for each image stripe based on the number and size of focus errors for that image stripe. For example, if 5% of the frames in an image stripe exceed a defined threshold (for example, stored as the Focus_Quality_Restripe_Threshold parameter), the image stripe is restriped. This threshold may be a setting that can be adjusted to a level that matches the user's preference. Naturally, restriping all image stripes will yield the best image quality. However, this will also double the scanning time.

[0156] 2.2.9. Image quality score

[0157] Image quality is primarily a function of focusing accuracy, provided that the focal point is measured on actual tissue and not on artifacts (e.g., the edges of the coverslip, air bubbles, dust on the coverslip, etc.). Some tissues may have significant tilt, making it difficult to focus on the entire frame. This can result in insufficient image quality, but there is little that can be done to solve this problem.

[0158] In one embodiment, each scanned image is given a binary image quality score, i.e., pass or fail. A "fail" score is equivalent to a significant portion of slide 114 being very out of focus. The image quality score of "fail" can be based on the following two indicators:

[0159] The percentage of bad frames. If the percentage of bad frames exceeds the Image_Quality_Bad_Frames_Threshold parameter, a failure is reported. In one embodiment, the percentage of bad frames is generally less than 5%, so if this percentage is higher than 5%, individual image stripes are restriped.

[0160] Average tilt. The average tilt is calculated from the focus map. Slides with a tissue tilt of less than 1 micron per millimeter can be expected to have good image quality. If the tilt exceeds the Image_Quality_Tilt_Threshold parameter, a failure can be reported.

[0161] In one embodiment, an image that has received a passing quality score may still be deemed unacceptable by the operator of the scanning system 100. Quality improvement can be achieved by reducing the Focus_Quality_Restripe_Threshold parameter, but this results in an increase in the number of image stripes rescanned to improve quality. Failed slides and slides deemed by the operator to have insufficient quality can be rescanned using the focusing method set in ReScan. The ReScan workflow adds additional MFP points at the start of the scan and restripes all image stripes, which naturally takes a relatively long time.

[0162] 2.2.10. RTF parameters

[0163] In one embodiment, the RTF method utilizes a set of parameters, some of which can be adjusted to improve performance and to suit various samples 116 and / or glass slides 114. Many of these parameters may be configurable at runtime from a configuration file (e.g., “scanner.xml” or other files stored in memory 106), while others can be fixed in the software. Examples of both types of parameters are identified and described below.

[0164] 2.2.10.1. Fixed parameters

[0165] Certain parameter values ​​can be fixed in the software code. These parameter values ​​can be determined based on scanning test slides and the desired algorithm performance. Exemplary and non-limiting examples of such fixed parameters are described below:

[0166] MAIN_IMG_COLM_ANALYZABLE_THRESH. An average principal gradient vector value greater than this threshold indicates the presence of tissue. An exemplary value for this parameter is 50. In one embodiment, 85% of the columns of principal gradient vectors must exceed this value in order to proceed with the Gaussian fitting process.

[0167] Parfocal threshold. In one embodiment, if the parfocal ratio is greater than 0.5 and less than 1.5 (i.e., 0.5 < parfocal ratio < 1.5), the frame can proceed to the Gaussian fitting process.

[0168] Sample values ​​for Gaussian fitting. These values ​​are the center point (mean) and width (sigma) for testing as candidates for the best-fitting Gaussian function. In one embodiment, the mean ranges from 100 to 3996 (4096-100) in increments of 25, and the sigma values ​​range from 500 to 1200 in increments of 100.

[0169] 2.2.10.2. Configurable parameters

[0170] Configurable parameter values ​​can be stored in an XML file (e.g., “scanner.xml”, or other file stored in memory 106) used to hold various parameters required to configure the scanning system 100 to operate. Exemplary and non-limiting examples of such configurable parameters, along with their nominal values, are described below:

[0171] Focusing_Method. This parameter can be set to one of the following values:

[0172] RTF: This is the default method. When the RTF method is set as the focusing method, a predetermined number (e.g., 3) macro focus points are used to start the scan, and then the RTF method and additional macro focus points are used to create a focus map during the scan. A probable tissue map from the tissue discovery algorithm can be used to identify acceptable frames for focus. Furthermore, outlier focus points are identified and removed, and restriping is performed on image stripes that exceed the restriping threshold.

[0173] PointFocus: When the MFP method is set as the focusing method, only macro focus is used to create the focus map. These points are evaluated before the image stripes are acquired, so no restriping is performed.

[0174] ReScan: This method is time-consuming but aims to provide good image quality for slide 114, which did not pass the default RTF method. The entire available scanning area is scanned. Therefore, the tissue discovery algorithm is not used to constrain the focus position during scanning. The scan begins with a complete completion of macro focus (equivalent to PointFocus), and all image stripes are restriped.

[0175] Parfocal. The value of this parameter is the pixel position on the tilt focusing sensor 130B that corresponds to the parfocality with the main imaging sensor 130A. By default, the value of this parameter can be set to 1766.

[0176] Stage_Tilt_Scan. This may be a parameter available for future use and can have a default value of 0.0.

[0177] Stage_Tilt_Index. This may be a parameter available for future use and can have a default value of 0.0 or greater.

[0178] Image_Data_Number_Rows. The value of this parameter is the number of rows that make up a single frame. By default, the value of this parameter can be set to 1000.

[0179] Image_Data_Number_Columns. The value of this parameter is the number of columns that make up a single frame, and must be less than or equal to the number of pixels (142) in a line scan camera (130). By default, the value of this parameter can be set to 4096.

[0180] Image_Data_Number_Color_Channels. The value of this parameter is the number of color channels. For RGB data, the value of this parameter is always 3, representing the three color channels: red, green, and blue. Therefore, by default, the value of this parameter can be set to 3.

[0181] Z_Offset. The value of this parameter is the offset for the focal point when scanning in the X / Y method (i.e., both MFP and RTF methods). This value can be experimentally adjusted by scanning a tissue sample 116 on slide 114 with a series of values ​​to determine which value provides the sharpest image. Alternatively, a test script can be used to scan a small area (e.g., 1 mm × 1 mm) with a sequence of offset values ​​and calculate the average contrast for each offset value in the sequence. The offset value with the highest contrast corresponds to the optimal value for this parameter. By design, this offset value is required to be zero. However, it has been found that a systematic error in the Z position may exist when stage 112 is moving compared to when stage 112 is stationary. By default, the value of this parameter can be set to 0.0005 millimeters.

[0182] Z_Scaling_Factor_Left. The value of this parameter is the scale factor, multiplied by 0.01 as the unit of microns per pixel. The scale factor has units of encoder counts (relative to the objective lens 120) per pixel, if 100 counts per micron. This parameter is used to convert pixels to microns along the tilt focusing sensor 130B and is applied to the Z distance in the downward direction (i.e., towards the glass slide 114). By default, the value of this parameter can be set to 0.38832.

[0183] Z_Scaling_Factor_Right. The value of this parameter is the scaling factor, multiplied by 0.01 as the unit of microns per pixel. The scaling factor has units of encoder counts (relative to the objective lens 120) per pixel, if 100 counts per micron. This parameter is used to convert pixels to microns along the tilt focusing sensor 130B and is applied to the Z distance in the upward direction (i.e., away from the glass slide 114). By default, the value of this parameter can be set to 0.31943.

[0184] Issue_Move_Every_X_Frames_Modulo. The scanning workflow involves moving the objective lens 120 only on specific frames. Frames where no movement of the objective lens 120 occurs are analyzed using the RTF method. By default, the value of this parameter can be set to 2, which means that every other frame is skipped.

[0185] Frame_Lag_Stripe_Abort_Threshold. The value of this parameter is the number of frames that RTF processing is allowed to lag behind the position of objective lens 120. A larger value effectively disables this feature. If the frame lag exceeds the value of this parameter, the scanning of the image stripes is aborted and a rescan of the image stripes is initiated. By default, the value of this parameter can be set to 300.

[0186] Debugging. This parameter takes a boolean value. By default, this parameter can be set to false. When this parameter is set to true, the actual frame image data is output.

[0187] Debug_Frame_Number. The value of this parameter is the number of frames that should be output when the Debugging parameter is set to true. By default, the value of this parameter can be set to 1.

[0188] Debug_Stripe_Number. The value of this parameter is the number of image stripes that should be output when the Debugging parameter is set to true. Only a single frame can be output to avoid interference with subsequent RTF processing and to prevent data changes from normal operation that may be caused by writing information to memory. By default, the value of this parameter can be set to 3.

[0189] Focus_Quality_Restripe_Threshold. The value of this parameter is the encoder count. If a certain percentage of frames containing tissue within the image stripe (e.g., 5%) have a focus error exceeding this value, the image stripe will be restriped. The focus error can be calculated by comparing the actual Z position of objective lens 120 with the final focus map. Setting this parameter to a high value effectively disables restriping, while setting it to a low value will rescan all image stripes containing tissue. By default, the value of this parameter can be set to 90.

[0190] Do_Restripe. The value of this parameter is a boolean. If the value of this parameter is true, restriping is enabled. Otherwise, i.e., if the value of this parameter is false, restriping is disabled and therefore not performed (for example, steps 594 and 596 of process 500 are skipped). In addition, if the value of this parameter is false, other related functions (for example, outlier rejection in step 592 of process 500) may also be skipped. By default, the value of this parameter can be set to true.

[0191] Do_Outlier_Rejection. The value of this parameter is a boolean. If the value of this parameter is true, the focus map is analyzed for outliers (for example, in step 592 of process 500). All outliers found are discarded, and the focus map is recalculated without the discarded outliers. This is done before determining whether to perform restriping (for example, in step 594 of process 500). If the value of this parameter is false, the focus map is not analyzed for outliers, and outlier rejection is not performed (for example, step 592 of process 500 is skipped). By default, the value of this parameter can be set to true.

[0192] Outlier_Rejection_Threshold. The value of this parameter is in microns per millimeter. In the embodiment of step 592 of process 500, for each point in the focus map, four slopes are calculated along the surface: up, down, right, and left. If the minimum slope is greater than the value of this parameter, the point is marked as an outlier and is not used in the focus map thereafter. The aim is that outlier points have large slopes in all directions away from that point. By default, the value of this parameter can be set to 2.0.

[0193] Focus_On_Probable_Tissue. This parameter is a Boolean value. When this parameter is true, a probable tissue map from the tissue discovery algorithm is used as a mask for acceptable focus positions in the RTF method. The RTF method is very sensitive to detecting frames containing tissue, and as a result, focus points may be added to the focus map even if they do not actually exist on tissue. Using a probable tissue mask prevents this. When this parameter is false, the probable tissue map is not used as a mask. By default, this parameter can be set to true.

[0194] Request_Additional_MFPs. The value of this parameter is a Boolean. If the value of this parameter is true, additional macro focus can be requested after each image stripe has been scanned (for example, in step 584 of process 500). These additional macro focus are used to provide additional focus values ​​to the focus map when the RTF method cannot reliably focus. If the value of this parameter is false, no additional macro focus is requested after each image stripe has been scanned. By default, the value of this parameter can be set to true.

[0195] Focus_Point_R_Min. The value of this parameter is in millimeters. If the value of the Request_Additional_MFPs parameter is true, and the frame containing tissue is further from a reasonable focal value than the value of this Focus_Point_R_Min parameter, a macro focus is requested for the center of the frame (for example, in step 584 of process 500). These additional macro focuses are focused and added to the focus map before scanning the next image stripe. By default, the value of this parameter can be set to 2.

[0196] Enable_PredictedZ_Offset. This parameter is a Boolean value. When this parameter is true, the predicted value from the focus map is subtracted from the best focus value calculated using the RTF method. The resulting difference is added to the focus value in the focus map predicted for the next frame. This feature aims to improve focusing accuracy by eliminating the trend of focus error and to reduce the restriping rate. When this parameter is false, this feature is disabled. By default, this parameter can be set to true.

[0197] Image_Quality_Bad_Frames_Threshold. This parameter is a percentage. If the percentage of bad frames exceeds this value, the image quality score will be set to "Fail". By default, this parameter can be set to 10%.

[0198] Image_Quality_Tilt_Threshold. The value of this parameter is in microns per millimeter. If the average tilt of the tissue exceeds the value of this parameter, the image quality score will be set to "Fail". By default, the value of this parameter can be set to 5 microns / mm.

[0199] The above description of the disclosed embodiments is provided to enable any person skilled in the art to create or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles described herein are applicable to other embodiments without departing from the spirit or scope of the invention. Accordingly, it should be understood that the descriptions and drawings presented herein represent currently preferred embodiments of the invention and are therefore representative of the subject matter broadly intended by the invention. It should be further understood that the scope of the invention fully encompasses and is therefore not limited to other embodiments which will be obvious to those skilled in the art.

[0200] The combinations described herein, such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof,” include any combination of A, B, and / or C, and may include multiple A's, multiple B's, or multiple C's. Specifically, combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof” may be A only, B only, C only, A and B, A and C, B and C, or A and B and C, and any such combination may contain one or more members of A, B, and / or C that are its components. For example, the combination of A and B may include one A and multiple B's, multiple A's and one B's, or multiple A's and multiple B's.

Claims

1. It is a method, The method described above uses at least one hardware processor of the scanning system, Initializing the focus map, While acquiring multiple image stripes of at least a portion of the sample on the glass slide, for each of the multiple image stripes, Using both the imaging sensor and the focusing sensor, each of the multiple frames that collectively represent the image stripe is acquired. The focus map is updated to include the focus representing the best focus position for the trusted frame among the multiple frames mentioned above. This adds a focus to the aforementioned focus map, Removing all outlier foci from the aforementioned focus map, Based on the focus error of each of the multiple frames within the multiple image stripes, it is determined whether one or more of the acquired multiple image stripes should be rescanned. For each of the multiple image stripes that has been determined to require rescanning, each of the multiple frames that collectively represent the image stripe is reacquired, and the multiple frames previously acquired for the image stripe are replaced with the reacquired multiple frames. The plurality of image stripes are assembled to form a single composite image of at least a portion of the sample. Includes, Adding focus to the focus map while acquiring the aforementioned multiple image stripes is, For each of the multiple image stripes to be acquired, except for the last one, after acquiring the image stripe, the direction of the next image stripe to be acquired from the multiple image stripes is determined from the acquired image stripe. Further including, method.

2. The aforementioned multiple image stripes, in order, Obtaining the reference stripe, The image stripe is acquired sequentially from the first side of the reference stripe to the first edge of the scanning area of ​​the sample, The image stripe is acquired sequentially from the second side of the reference stripe, which is opposite to the first side of the reference stripe, to the second edge of the scanning region, which is opposite to the first edge of the scanning region. Obtained by, The method according to claim 1.

3. A method, The method described above uses at least one hardware processor of the scanning system, Initializing the focus map, While acquiring multiple image stripes of at least a portion of the sample on the glass slide, for each of the multiple image stripes, Using both the imaging sensor and the focusing sensor, each of the multiple frames that collectively represent the image stripe is acquired. The focus map is updated to include the focus representing the best focus position for the trusted frame among the multiple frames mentioned above. This adds a focus to the aforementioned focus map, Removing all outlier foci from the aforementioned focus map, Based on the focus error of each of the multiple frames within the multiple image stripes, it is determined whether one or more of the acquired multiple image stripes should be rescanned. For each of the multiple image stripes that has been determined to require rescanning, each of the multiple frames that collectively represent the image stripe is reacquired, and the multiple frames previously acquired for the image stripe are replaced with the reacquired multiple frames. The plurality of image stripes are assembled to form a single composite image of at least a portion of the sample. Includes, The aforementioned method, Before starting to acquire the aforementioned multiple image stripes, add multiple macro focus points to the focus map. Further including, method.

4. A method, The method described above uses at least one hardware processor of the scanning system, Initializing the focus map, While acquiring multiple image stripes of at least a portion of the sample on the glass slide, for each of the multiple image stripes, Using both the imaging sensor and the focusing sensor, each of the multiple frames that collectively represent the image stripe is acquired. The focus map is updated to include the focus representing the best focus position for the trusted frame among the multiple frames mentioned above. This adds a focus to the aforementioned focus map, Removing all outlier foci from the aforementioned focus map, Based on the focus error of each of the multiple frames within the multiple image stripes, it is determined whether one or more of the acquired multiple image stripes should be rescanned. For each of the multiple image stripes that has been determined to require rescanning, each of the multiple frames that collectively represent the image stripe is reacquired, and the multiple frames previously acquired for the image stripe are replaced with the reacquired multiple frames. The plurality of image stripes are assembled to form a single composite image of at least a portion of the sample. Includes, The aforementioned method, After obtaining one or more of the aforementioned multiple image stripes, one or more macrofocals are added to the focus map. Further including, method.

5. A method, The method described above uses at least one hardware processor of the scanning system, Initializing the focus map, While acquiring multiple image stripes of at least a portion of the sample on the glass slide, for each of the multiple image stripes, Using both the imaging sensor and the focusing sensor, each of the multiple frames that collectively represent the image stripe is acquired. The focus map is updated to include the focus representing the best focus position for the trusted frame among the multiple frames mentioned above. This adds a focus to the aforementioned focus map, Removing all outlier foci from the aforementioned focus map, Based on the focus error of each of the multiple frames within the multiple image stripes, it is determined whether one or more of the acquired multiple image stripes should be rescanned. For each of the multiple image stripes that has been determined to require rescanning, each of the multiple frames that collectively represent the image stripe is reacquired, and the multiple frames previously acquired for the image stripe are replaced with the reacquired multiple frames. The plurality of image stripes are assembled to form a single composite image of at least a portion of the sample. Includes, Adding focus to the focus map while acquiring the aforementioned multiple image stripes is, For each of the multiple frames in each of the multiple image stripes, determine whether that frame is reliable. Further including, method.

6. Determining whether the aforementioned frame is trustworthy is: For each column within the frame acquired by the aforementioned image sensor, calculate the principal gradient vector including the average gradient vector, For each column in the frame acquired by the focusing sensor, calculate a gradient vector including the average gradient vector, To identify the number of analyzable columns in the aforementioned principal gradient vector, Based on the principal gradient vector and the slope gradient vector, the ratio vector is calculated, Based on the number of analyzable columns and the ratio vector, it is determined whether the frame is analyzable. If it is determined that the frame is not analyzable, then it is determined that the frame is unreliable. If it is determined that the aforementioned frame is analyzable, Fitting at least one Gaussian function to the ratio curve represented by the ratio vector, Identifying the peak of the aforementioned Gaussian function as the best focus position, Identifying the amplitude of the ratio vector at the best focus position as the maximum fit value, Based on the best focus position and the maximum fit, it is determined whether the frame is reliable. If it is determined that the frame is not reliable, then it is determined that the frame is unreliable. If it is determined that the frame is reliable, the best focus position is added to the focus map. including, The method according to claim 5.

7. Identifying the number of analyzable columns is, This includes identifying the number of columns in the principal gradient vector that exceed a threshold. The method according to claim 6.

8. Calculating the aforementioned ratio vector is, Divide the aforementioned gradient vector by the aforementioned main gradient vector. including, The method according to claim 6.

9. Determining whether the aforementioned frame is analyzable is: This involves determining whether the number of analyzable columns exceeds a predefined threshold percentage, The method involves determining whether the value of the ratio vector at the cofocal position falls within a predefined range, wherein the cofocal position is a point on the focusing sensor that is in focus with the imaging sensor. If it is determined that the number of analyzable columns does not exceed the predefined threshold percentage, or that the value of the ratio vector at the same focal position is not within the predefined range, then it is determined that the frame is not analyzable. The frame is determined to be analyzable when it is determined that the number of analyzable columns exceeds the predefined threshold percentage and the value of the ratio vector at the cofocal position is within the predefined range. including, The method according to claim 6.

10. Fitting at least one Gaussian function to the ratio curve is Sampling multiple possible Gaussian functions within the range of the mean and the range of the sigma value, From among the multiple possible Gaussian functions, select one Gaussian function that has the smallest difference from the ratio curve, which should be used to identify the best focus position. including, The method according to claim 6.

11. A method, The method described above uses at least one hardware processor of the scanning system, Initializing the focus map, While acquiring multiple image stripes of at least a portion of the sample on the glass slide, for each of the multiple image stripes, Using both the imaging sensor and the focusing sensor, each of the multiple frames that collectively represent the image stripe is acquired. The focus map is updated to include the focus representing the best focus position for the trusted frame among the multiple frames mentioned above. This adds a focus to the aforementioned focus map, Removing all outlier foci from the aforementioned focus map, Based on the focus error of each of the multiple frames within the multiple image stripes, it is determined whether one or more of the acquired multiple image stripes should be rescanned. For each of the multiple image stripes that has been determined to require rescanning, each of the multiple frames that collectively represent the image stripe is reacquired, and the multiple frames previously acquired for the image stripe are replaced with the reacquired multiple frames. The plurality of image stripes are assembled to form a single composite image of at least a portion of the sample. Includes, Removing all outlier foci from the aforementioned focus map means, for one or more sample points in the focus map, In each of the four directions, the inclination away from the sample point within the focus map is calculated, If the calculated minimum value of the slope exceeds a predefined threshold, the sample point is removed from the focus map. including, method.

12. A method, The method described above uses at least one hardware processor of the scanning system, Initializing the focus map, While acquiring multiple image stripes of at least a portion of the sample on the glass slide, for each of the multiple image stripes, Using both the imaging sensor and the focusing sensor, each of the multiple frames that collectively represent the image stripe is acquired. The focus map is updated to include the focus representing the best focus position for the trusted frame among the multiple frames mentioned above. This adds a focus to the aforementioned focus map, Removing all outlier foci from the aforementioned focus map, Based on the focus error of each of the multiple frames within the multiple image stripes, it is determined whether one or more of the acquired multiple image stripes should be rescanned. For each of the multiple image stripes that has been determined to require rescanning, each of the multiple frames that collectively represent the image stripe is reacquired, and the multiple frames previously acquired for the image stripe are replaced with the reacquired multiple frames. The plurality of image stripes are assembled to form a single composite image of at least a portion of the sample. Includes, Determining whether one or more of the aforementioned multiple image stripes should be rescanned is done after removing all outlier foci from the focus map, For each of the multiple frames in each of the multiple image stripes, the focus error for the frame is calculated by subtracting the actual position of the objective lens during frame acquisition from the best focus position of that frame within the focus map. For each of the aforementioned plurality of image stripes, if the number of frames having a focus error exceeding a predefined threshold exceeds a predefined threshold percentage, it is determined to rescan the image stripe. including, method.

13. A scanning system for carrying out the method according to any one of claims 1 to 12, The scanning system is Image sensor and Focus sensor and, At least one hardware processor, One or more software modules and Includes, The one or more software modules are executed by the at least one hardware processor, Initialize the focus map, While acquiring multiple image stripes of at least a portion of the sample on the glass slide, for each of the multiple image stripes, Using both the imaging sensor and the focusing sensor, each of the multiple frames that collectively represent the image stripe is acquired. The focus map is updated to include the focus representing the best focus position for the trusted frame among the multiple frames mentioned above. By doing so, a focus is added to the aforementioned focus map, Remove all outlier foci from the aforementioned focus map. Based on the focus error of each of the multiple frames within the multiple image stripes, it is determined whether one or more of the acquired multiple image stripes should be rescanned. For each of the multiple image stripes that is determined to require rescanning, each of the multiple frames that collectively represent the image stripe is reacquired, and the multiple frames previously acquired for the image stripe are replaced with the reacquired multiple frames. The plurality of image stripes are assembled to form a single composite image of at least a portion of the sample. A scanning system configured in such a way.

14. A non-temporary computer-readable medium storing instructions for carrying out the method described in any one of claims 1 to 12, When the aforementioned instruction is executed by the processor of the scanning system, the processor will: Initialize the focus map, While acquiring multiple image stripes of at least a portion of the sample on the glass slide, for each of the multiple image stripes, Using both the imaging sensor and the focusing sensor, each of the multiple frames that collectively represent the image stripe is acquired. The focus map is updated to include the focus representing the best focus position for the trusted frame among the multiple frames mentioned above. By doing so, a focus is added to the aforementioned focus map, Remove all outlier foci from the aforementioned focus map. Based on the focus error of each of the multiple frames within the multiple image stripes, it is determined whether one or more of the multiple image stripes should be rescanned. Using both the imaging sensor and the focusing sensor, for each of the multiple image stripes that is determined to require rescanning, each of the multiple frames that collectively represent the image stripe is reacquired. The plurality of image stripes are assembled to form a single composite image of at least a portion of the sample. A non-temporary computer-readable medium.

Images