Digital imaging system and method

Abstract

To provide an automated system and a method for evaluating a specimen attached to a substrate such as a microscope slide.SOLUTION: An automatic digital imaging system 100 includes an imager 104 for acquiring a plurality of micro images of a specimen mounted on a substrate. The specimen includes a plurality of objects dispersed in a three-dimensional volume, and generates a whole specimen image of the specimen using the micro images. The object included in the specimen is rendered substantially in focus in the whole specimen image, regardless of a z-axis depth of each object in the specimen. The whole specimen image is saved in storage media for later review by a cytologist using a computer-controlled review station 108 that includes a display and a user interface. The cytologist reviews and classifies the saved entire specimen image by the user interface of the review station.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This patent application claims priority to U.S. Provisional Application No. 62 / 755,151, entitled "DIGITAL IMAGING SYSTEM AND METHOD," filed November 2, 2018, and U.S. Provisional Application No. 62 / 755,291, entitled "DIGITAL IMAGING SYSTEM AND METHOD," filed November 2, 2018, each of which is incorporated by reference in its entirety into this specification.

[0002]

[0002] The present disclosure relates generally to digital imaging systems and methods, and more particularly to digital imaging systems and methods for acquiring digital images of specimens, such as cytological (cell) specimens and pathological (solid) tissue specimens, that can be displayed and viewed on a computer monitor by a cytologist and / or cytopathologist. [Background technology]

[0003]

[0003] Cytology is a branch of biology that deals with the study of cell formation, structure, and function. As applied in a laboratory setting, cytologists, cytologists, and other medical professionals make medical diagnoses of a patient's condition based on the visual examination of a sample of the patient's cells. Such samples are referred to herein as "cytological" specimens. A typical cytological procedure is the "Pap smear," in which cells are scraped from a woman's cervix and analyzed to detect the presence of abnormal cells that are precursors to the development of cervical cancer. Cytological procedures are also utilized to detect abnormal cells and disease in other parts of the human body.

[0004]

[0004] Cytological techniques have been widely adopted because collection of cell samples for analysis is generally less invasive than traditional surgical pathology procedures such as biopsy. In a biopsy, a solid tissue sample (referred to herein as a "pathology" specimen) is excised from a patient using a specialized biopsy needle having a spring-loaded translatable stylet, fixed cannula, or the like. Cell samples are obtained from the patient by a variety of techniques including, for example, scraping or swabbing an area or using a needle to aspirate fluid from the thoracic cavity, bladder, spinal canal, or other suitable area. The obtained cell sample is typically placed in a preservative solution and then extracted from the solution and transferred to a glass slide. A fixative is applied to the cell sample to keep the cells in place on the slide for easy subsequent staining and examination.

[0005]

[0005] It is generally desirable for cells on a slide to have a proper spatial distribution so that individual cells can be examined. A monolayer of cells is usually preferred. Thus, to prepare a cytological specimen from a liquid sample containing many cells (e.g., tens of thousands), it is usually necessary to first separate the cells from each other by mechanical dispersion, liquid shear, or other techniques so that a thin monolayer of cells is collected and placed on a slide. In this way, the cytologist can more easily identify the presence of abnormal cells in a patient sample. Cells may also be counted to ensure that an appropriate number of cells have been evaluated.

[0006]

[0006] Specific methods and apparatus for producing a thin monolayer of cells from a liquid sample container and then transferring the thin layer to a "specimen slide" convenient for visual inspection are disclosed in U.S. Patent Nos. 5,143,627, 5,240,606, 5,269,918, 5,282,978, 6,562,299, 6,572,824 and 7,579,190. All publications cited herein are incorporated by reference in their entirety. According to one method disclosed in these patents, patient cells suspended in a preservative fluid and contained in a sample container are dispersed using a spinning sample collector that is inserted into the container. A controlled vacuum is applied to the sample collector to draw the liquid through a screen filter until the desired amount and spatial distribution of cells is collected against the filter. The sample collector is then removed from the sample container and the filter portion is pressed against the slide to transfer the collected cells to the slide in substantially the same spatial distribution as they were collected. Devices manufactured according to the teachings of one or more of these patents have been commercially successful, such as the ThinPrep™ 2000 processor (in which specimen slides are processed one at a time from patient samples) and the ThinPrep™ 5000 processor (in which specimen slides are processed in batches from patient samples) manufactured and sold by Hologic, Inc., Marlborough, Massachusetts. See also U.S. Patent Nos. 7,556,777 and 7,771,662.

[0007]

[0007] Once the specimen slide is prepared, the specimen may be visually inspected by a cytologist, usually under magnification, with or without various illumination sources. Additionally or alternatively, an automated slide imaging system may be used to assist in the cytological examination process. For example, an automated slide imaging system may capture images of all, or substantially all, of the individual cells in a cytological specimen fixed on a slide, and perform a preliminary evaluation of the cells using image processing techniques to present the potentially most relevant cells on the slide to the cytologist for closer inspection. Examples of such imaging systems are disclosed in U.S. Patent Nos. 7,587,078, 6,665,060, 7,006,674, 7,369,304, and 7,590,492. Depending on whether the actual specimen slide is inspected under magnification or a magnified image of the specimen is inspected, the specimen is typically classified by the cytologist as either "normal" or "abnormal," with abnormal samples typically falling into one of the major categories defined in the "Bethesda System for Reporting Cervical / Vaginal Cytological Diagnosis." The categories include low-grade squamous intraepithelial lesion (LSIL), high-grade squamous intraepithelial lesion (HSIL), squamous cell carcinoma, adenocarcinoma, atypical glandular cells (AGUS), adenocarcinoma in situ (AIS), and atypical squamous cells (ASC). Additional information on the classification of cytologic specimens is widely available, such as in "The Yokohama System for Reporting Endometrial Cytology," Diagnostic Cytopathology, May 2018, Vol. 46(5), pp. 400-412, and "Reporting and Guidelines for Non-Gynecologic Cytopathology Specimens," Archives of Pathology and Laboratory Medicine, November 2009, Vol. 133, No. 11, pp. 1743-1756.

[0008] However, there are many drawbacks associated with conventional systems and methods for acquiring digital images of biological specimens. For one, conventional systems and methods suffer from slow acquisition times due to the time it takes to scan the entire specimen. In addition, conventional systems and methods typically provide only a single focal plane across the specimen. Biological specimens, whether cytological or pathological, are in fact three-dimensional (i.e., have depth). Because of this, the depth of field of the image is very limited due to the high magnification and focal aperture required to acquire digital images of biological specimens. Thus, portions of the specimen that are outside the depth of field of the focal plane are out of focus or do not appear in the image. To acquire focused digital images at multiple different depths of the specimen, it is necessary to adjust the focal plane, such as by moving the specimen or the camera or by adjusting the focusing lens. However, this requires additional scanning of the specimen for each focal plane, further slowing down the acquisition time. Summary of the Invention

[0009]

[0009] Disclosed and described herein are embodiments of an improved automated system for evaluating a specimen (e.g., a cytological or pathological specimen) mounted on a substrate. Typically the substrate is a microscope slide, and thus, although the embodiments are described using a slide as the substrate, it should be understood that the automated systems and methods disclosed herein are not limited to the use of slides, and may utilize any suitable substrate. The specimen may include any type of specimen that is digitally imaged under microscopic magnification, such as biological or chemical specimens, including cytological specimens, tissue specimens, and the like. As used herein, the term "specimen" may apply to the entire specimen or a portion thereof, depending on the context.

[0010] An exemplary automated slide imaging system constructed in accordance with the disclosed invention includes an imager configured to acquire images of a specimen attached to a surface of a slide, the specimen including a plurality of objects distributed within a three-dimensional volume, the imager configured to generate a whole specimen image from the acquired images, the objects being depicted in focus within the whole specimen image regardless of the individual positions of each object within the three-dimensional volume, the three-dimensional specimen volume having a length, a width, and a thickness, the thickness defining a z-axis relative to the slide surface, and each object of the specimen (e.g., individual cells or tissue structures, as the case may be) disposed at a different position along the z-axis. The acquired images include a macro-image of the specimen and a plurality of micro-images of the specimen, the macro-image including one or more fiducial markers disposed on the slide surface, and the imager configured to acquire the micro-images based at least in part on the relative positions and boundaries of the specimen on the slide surface determined from the macro-image.

[0011] In an exemplary embodiment, the imager includes a first imaging platform configured to seat the slide, a first camera configured to capture a macro image when the slide is seated on the first imaging platform, a second imaging platform further configured to seat the slide, and a second camera configured to capture a micro image of a specimen attached to the slide mounted on the second imaging platform. The imager is configured to automatically move at least one of the second camera and the second imaging platform relative to the other to capture the micro images, where an optical axis of the second camera forms a non-orthogonal angle with the second imaging platform, and the imager is configured to capture the micro images of the specimen at the same z-axis of the three-dimensional specimen volume. In particular, the slide to be imaged may have a thickness, and the micro images may include at least a portion of the slide below a surface. The specimen may be covered with a cover slip that is sufficiently transparent to capture a micro image of the specimen through the cover slip, the cover slip having a thickness, and each of the micro images includes at least a portion of the depth of the cover slip. In an exemplary embodiment, the specimen slide has a width defining an x-axis and a length defining a y-axis, and the imager is configured to translate the slide along the y-axis relative to the second camera as the second camera acquires micro-images at each y-axis position, each micro-image including the entire x-axis width of the specimen based on a boundary of the specimen determined at the slide surface.

[0012] In an exemplary embodiment, the imager has one or more slide holder receptacles, each slide holder receptacle configured to receive a slide holder including a plurality of slots, each slot configured to hold an individual slide, and a robotic arm assembly configured to (i) remove a slide from a slide holder slot of the slide holder receptacle, (ii) move and seat the slide to the first imaging platform for acquiring a macro image, (iii) remove the slide from the first imaging platform again, (iv) move and seat the slide to the second imaging platform for acquiring a micro image, and (v) remove the slide from the second imaging platform again. The robotic arm assembly is further configured to (vi) transfer the slide to a slide holder that is the same or different from the slide holder from which it was removed, and (vii) release the slide into the slot of the respective same or different slide holder, where the slot of the respective same or different slide holder is the same slot from which the slide was removed by the robotic arm assembly.

[0013] In an exemplary embodiment, the imager includes an image processor configured to generate a whole specimen image from a micro image, the image processor identifying a best focus image for each individual object in the micro image, and the best focus images of the objects are incorporated into the whole specimen image. The image processor is preferably further configured to identify objects of interest (e.g. individual cells or tissue structures) within the specimen and to store images of the identified objects of interest together with the whole specimen image. The macro image of the specimen includes an image of a barcode on a slide surface, in which case the imager is preferably configured to obtain information about the specimen from the barcode.

[0014]

[0014] In an exemplary embodiment, the system further includes a review station including a display monitor, a user interface, and a processor functionally coupled to each display monitor and user interface, the processor configured to display the entire specimen image on the display monitor along with individual images of each object within the specimen image.

[0015]

[0015] Without limitation, the review station may be configured to allow a user of the system to select a whole specimen image from a list of stored whole specimen images using a user interface.

[0016]

[0016] Without limitation, the system may be configured to allow a user of the system to use a user interface to ultimately characterize or otherwise transfer the entire specimen image for secondary review.

[0017] Without limitation, the list of stored whole specimen images that have not been conclusively characterized or otherwise forwarded for secondary review may be organized and displayed in a number of different formats based on input received via the user interface. The system may be further configured to allow an authorized third party to input, in whole or in part, the list of stored whole specimen images for review by a particular system user.

[0018] In an exemplary embodiment, the system is configured to allow a system user to add annotations to the whole specimen image and / or to a data file associated with the whole specimen image, which annotations are available for review along with the whole specimen image by subsequent system users. For example, annotations may be associated with individual objects in the whole specimen image and may be in the form of electronic markings made on the whole specimen image or an image of a portion of the whole specimen image.

[0019] In an exemplary embodiment, the system is configured to, in response to a system user prompt associated with a respective object in the whole specimen image, display one or more additional objects sharing one or more characteristics with the respective object, where the one or more additional objects (e.g., cells or tissue structures, if applicable) may be from the whole specimen image and / or from a library containing previously classified objects. Without limitation, the system may be configured to allow data regarding the whole specimen image or individual objects therein to be entered via a user interface and stored in a data file associated with the whole specimen image.

[0020]

[0020] In an exemplary embodiment, the system is configured to display a magnified view of at least a portion of the entire specimen image and to automatically scan at least a portion of the displayed entire specimen image. In such an embodiment, the system preferably automatically scans in a user-selected scan pattern, including but not limited to a serpentine pattern, a row-by-row pattern, and a column-by-column pattern. In such an embodiment, the system is preferably configured to allow a system user, via a user interface, to set the magnification level at which the system displays the scanned at least a portion of the entire specimen image. In such an embodiment, the system is preferably configured to allow a system user, via a user interface, to stop and start a scan at a displayed position of the scan, as well as to set the scan speed. In such an embodiment, the system is configured to allow a system user to pause the scan at each object as it is displayed during the scan.

[0021]

[0021] In an exemplary embodiment, the system is configured to display a review screen on a display monitor, the review screen including a main image panel in which an entire specimen image is displayed, and an object panel containing separate images of individual objects within the specimen image, and a system user can zoom in / out and / or pan the entire specimen image in the main image panel via a user interface. Without limitation, the system may be configured to display an area of ​​the entire specimen image containing the respective object in the main image panel upon a system user selecting an individual image of an object in the object panel via a user interface. Without limitation, the system is configured to allow a system user to select an object within the displayed entire specimen image via a user interface, where the system displays an image of the selected object in a system user selected image panel.

[0022]

[0022] In accordance with a further aspect of the invention disclosed herein, there is provided a method for generating a whole specimen image of a specimen attached to a surface of a slide, wherein the specimen includes a plurality of objects distributed within a three-dimensional volume, the method comprising the steps of: (i) acquiring a macro-image of the specimen; (ii) acquiring a plurality of micro-images of the specimen based at least in part on the macro-image; and (iii) generating the whole specimen image by processing the micro-images using an image processor, wherein the objects are depicted substantially in focus in the whole specimen image regardless of individual positions of each object within the three-dimensional volume, the three-dimensional volume having a length, a width, and a thickness, the thickness defining a z-axis relative to the slide surface, and each object of the specimen (e.g., a cell or tissue structure) being disposed at a different position along the z-axis.

[0023]

[0023] Without limitation, the macro-images are acquired using a first camera and the micro-images are acquired using a second camera, at least one of the second camera and the slide is automatically moved relative to the other when acquiring the micro-images, an optical axis of the second camera forms a non-orthogonal angle with the slide, and the micro-images are acquired on the same z-axis as the three-dimensional volume. The micro-images can then be processed using an image processor to determine a best-focus image for each individual object in the micro-image and to generate a whole specimen image by incorporating the best-focus image for each object into a whole specimen image.

[0024]

[0024] The macro-image preferably captures one or more fiducial markers disposed on a slide surface, and the method further includes identifying a relative position and boundary of a specimen on the slide surface based at least in part on the one or more fiducial markers, and wherein the micro-images are acquired based at least in part on the relative position and boundary of the specimen on the slide surface identified from the macro-image. The micro-images may be acquired by translating the slide along a y-axis relative to the second camera as the second camera acquires micro-images at respective y-axis positions, each micro-image comprising an entire x-axis width based on the boundary of the specimen identified on the slide surface.

[0025]

[0025] The method may further include identifying objects of interest within the specimen (eg, individual cells or tissue structures) and storing images of the identified objects of interest along with the entire specimen image.

[0026]

[0026] The method may further include the step of reviewing the entire specimen image, including images of individual objects therein, using a computer-controlled review station including a display and a user interface.

[0027]

[0027] Other and further features and advantages of the disclosed embodiments of the automated imaging and review system are illustrated in the accompanying drawings and are described in the detailed description of the drawings. [Brief description of the drawings]

[0028]

[0028] The above and other aspects of the embodiments will be described in further detail with reference to the accompanying drawings, in which like reference numbers refer to like elements and the description of like elements applies to all described embodiments wherever relevant. [Figure 1]

[0029] FIG. 1 is a block diagram of an automated digital imaging system, according to one embodiment. [Diagram 2]

[0030] FIG. 2 is a block diagram of the automated digital imaging system of FIG. 1 showing a single imager and review station, according to one embodiment. [Diagram 3]

[0031] FIG. 3 illustrates a specimen slide for use in the automated digital imaging system of FIG. 1, according to one embodiment. [Figure 4]

[0032] FIG. 4 is a front perspective view of a digital imager of the digital imaging system of FIG. 2, according to one embodiment. [Diagram 5]

[0033] FIG. 5 is a side perspective view of a slide carrier of the digital imager of FIG. 4 according to one embodiment. [Figure 6]

[0034] FIG. 6 is a side perspective view of the slide carrier of FIG. 5 with a slide rack loaded with slides attached to the slide carrier, according to one embodiment. [Figure 7]

[0035] FIG. 7 is an enlarged front perspective view of a slide carrier bay of the digital imager of FIG. 4, according to one embodiment. [Figure 8]

[0036] FIG. 8 is an end view of the slide carrier of FIGS. 5 and 6 according to one embodiment. [Figure 9]

[0037] FIG. 9 is a rear perspective view of the digital imager of FIG. 4, according to one embodiment. [Figure 10]

[0038] FIG. 10 is a front perspective view of the digital imager of FIG. 4 with the enclosure panel removed to show the internal components, according to one embodiment. [Figure 11]

[0039] FIG. 11 is a schematic diagram of an imager computer of the digital imager of FIG. 4, according to one embodiment. [Figure 12]

[0040] FIG. 12 is a schematic diagram of a chassis / enclosure assembly of the digital imager of FIG. 4, according to one embodiment. [Figure 13]

[0041] FIG. 13 is a schematic diagram of a slide carrier loading deck of the digital imager of FIG. 4, according to one embodiment. [Figure 14]

[0042] FIG. 14 is an enlarged front view of the slide handler of the digital imager of FIG. 4, according to one embodiment. [Figure 15]

[0043] FIG. 15 is a side view of the slide handler of FIG. 14, according to one embodiment. [Figure 16]

[0044] FIG. 16 is a front perspective view of the slide handler of FIG. 14 according to one embodiment. [Figure 17]

[0045] FIG. 17 is a schematic diagram of a slide handling gantry of the slide handler of FIG. 14, according to one embodiment. [Figure 18]

[0046] FIG. 18 is a schematic diagram of a gantry robot motion controller for the slide handler of FIG. 14 according to one embodiment. [Figure 19]

[0047] FIG. 19 is a front perspective view of the digital imager of FIG. 4 with the panels, slide handler, and slide carrier deck removed to show the imaging station, according to one embodiment. [Figure 20]

[0048] FIG. 20 is a partial front view of the digital imager of FIG. 4 (and a front cross-sectional view of the vibration absorber) with the panel, slide handler, and slide carrier deck removed to show the imaging station, according to one embodiment. [Figure 21]

[0049] FIG. 21 is a front view of an image capture subassembly of an imaging station for the digital imager of FIG. 4, according to one embodiment. [Figure 22]

[0050] FIG. 22 is a cross-sectional front view of the image capture subassembly of FIG. 21 according to one embodiment. [Figure 23]

[0051] FIG. 23 is an enlarged top view of the slide imaging station of the digital imager of FIG. 4, according to one embodiment. [Figure 24]

[0052] FIG. 24 is an enlarged perspective view of the slide imaging station of the digital imager of FIG. 4 according to one embodiment. [Figure 25A]

[0053] FIG. 25A is a diagram showing the imaging plane of a conventional imaging system, which is a nominally parallel plane of the slide (and the specimen thereon). [Figure 25B]

[0054] FIG. 25B is a diagram illustrating a tilted imaging plane of the digital imager of FIG. 4, according to one embodiment. [Figure 25C]

[0055] FIG. 25C illustrates a tilted imaging plane of the digital imager of FIG. 4 for imaging the entire depth (Z dimension) of a specimen on a slide, according to one embodiment, and shows dimensions for determining an appropriate camera tilt angle. [Figure 26]

[0056] FIG. 26 is a schematic diagram of an imaging station of the digital imager of FIG. 4, according to one embodiment. [Figure 27]

[0057] FIG. 27 is a front view of the illumination subsystem of the digital imager of FIG. 4, according to one embodiment. [Figure 28]

[0058] FIG. 28 is a cross-sectional front view of the illumination subsystem of FIG. 27 according to one embodiment. [Figure 29]

[0059] FIG. 29 is a schematic diagram of a system interface board of the digital imager of FIG. 4, according to one embodiment. [Diagram 30]

[0060] FIG. 30 is an expanded perspective view of a macrovision station of the digital imager of FIG. 4, according to one embodiment. [Diagram 31]

[0061] FIG. 31 is a front view of the macrovision station of the digital imager of FIG. 4, according to one embodiment. [Diagram 32]

[0062] FIG. 32 is a side cross-sectional view of a macrovision station of the digital imager of FIG. 4, according to one embodiment. [Diagram 33]

[0063] FIG. 33 is a schematic diagram illustrating a serpentine scan pattern used by the digital imager of FIG. 4 to scan a specimen slide, according to one embodiment. [Diagram 34]

[0064] FIG. 34 is a block diagram of the workflow subsystems and digital imaging display system (review station) of the automated digital imaging system of FIG. 1, according to one embodiment. [Diagram 35]

[0065] FIG. 35 illustrates a workflow scenario for reviewing digital images of specimen slides, according to one embodiment. [Diagram 36]

[0066] FIG. 36 is a partial view of a worklist screen displayed by a digital image display system during the review process, according to one embodiment. [Figure 37]

[0067] FIG. 37 is a partial view of a worklist screen displayed by a digital image viewing station selection box for selecting a case to review, according to one embodiment. [Figure 38]

[0068] FIG. 38 is a schematic illustrating a screen displayed by a digital image viewing station selection box for selecting a case to review, according to one embodiment. [Figure 39]

[0069] FIG. 39 illustrates the ability of a digital image viewing station to switch between different worklist screens, according to one embodiment. [Diagram 40]

[0070] FIG. 40 illustrates a worklist screen having a simple list format displayed on a digital image display system according to one embodiment. [Diagram 41]

[0071] FIG. 41 illustrates a worklist screen with a tile / grid format displayed on a digital image display system, according to one embodiment. [Diagram 42]

[0072] FIG. 42 illustrates a worklist screen with sorted column tile / grid format displayed on a digital image display system according to one embodiment. [Diagram 43]

[0073] FIG. 43 illustrates a worklist screen with a details panel displayed on a digital image display system according to one embodiment. [Diagram 44]

[0074] FIG. 44 illustrates a worklist screen with graphical status indicators displayed on a digital image display system, according to one embodiment. [Diagram 45]

[0075] FIG. 45 shows a review screen for a GYN ("gynecology") case (single slide case) displayed on a digital image display system, according to one embodiment. [Diagram 46]

[0076] FIG. 46 shows a review screen for a NGYN ("Non-Gynecologic") case (multiple slide case) displayed on a digital image display system, according to one embodiment. [Figure 47]

[0077] FIG. 47 shows another review screen for a NGYN case (multiple slides) displayed on a digital image display system, according to one embodiment. [Figure 48]

[0078] FIG. 48 shows a completion screen for a typical GYN case (single slide case) displayed on a digital image display system according to one embodiment. [Figure 49]

[0079] FIG. 49 shows a completion screen for an abnormal GYN case (single slide case) displayed on a digital image display system according to one embodiment. [Figure 50]

[0080] FIG. 50 illustrates a post-case completion worklist screen displayed on a digital image display system, according to one embodiment. [Figure 51]

[0081] FIG. 51 illustrates a worklist screen after case completion displayed on a digital image display system with the completed case hidden from the worklist, according to one embodiment. [Figure 52]

[0082] FIG. 52 illustrates another workflow scenario for reviewing digital images of specimen slides, according to one embodiment. [Figure 53]

[0083] FIG. 53 illustrates yet another workflow scenario for reviewing digital images of specimen slides, according to one embodiment. [Figure 54]

[0084] FIG. 54 illustrates yet another workflow scenario for reviewing digital images of specimen slides, according to one embodiment. [Figure 55]

[0085] FIG. 55 provides an overview of the workflow roles and scenarios shown in FIGS. 35 and 52-54, according to one embodiment. [Figure 56]

[0086] 56A-56C are diagrams illustrating the various administrators / management layers with respect to the workflow roles and scenarios shown in FIGS. 35 and 52-54. [Figure 57]

[0087] FIG. 57 shows another review screen displaying a digital image of the entire specimen area displayed on a digital image display system, according to one embodiment. [Figure 58]

[0088] FIG. 58 illustrates another review screen configured to allow a reviewer to add comments and mark / draw on the image, according to one embodiment. [Figure 59]

[0089] FIG. 59 illustrates another review screen configured to allow a reviewer to add recorded audio comments to a digital image case, according to one embodiment. [Figure 60]

[0090] FIG. 60 shows another review screen configured in one embodiment to allow a first reviewer to share the first reviewer's screen with a second reviewer on a different display system 108, and to give control to the second reviewer to control the review screen 434 on the first reviewer's screen. [Figure 61]

[0091] FIG. 61 illustrates another review screen configured to allow a reviewer to bookmark cases of digital images, according to one embodiment. [Figure 62]

[0092] FIG. 62 shows a graphical representation of an image analysis algorithm for digital cytology, according to one embodiment. [Figure 63]

[0093] FIG. 63 illustrates a block diagram of an exemplary workflow server with backup and archive functionality, according to one embodiment. [Figure 64]

[0094] FIG. 64 shows a flowchart of a method for archiving slides using an archive service module according to one embodiment. [Figure 65]

[0095] FIG. 65 illustrates an example of a calibration table for performing distortion correction of a digital image, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029]

[0096] 1 and 2 show an automated digital imaging system 100 for digitally imaging a plurality of samples on a substrate 102 (see FIG. 3 ) on which the specimens are disposed. For example, the substrate 102 is a microscope slide as shown in the described embodiment, but may be any other suitable substrate 102, such as a microplate, a microarray, or other suitable medium. Although the described embodiment of the digital imaging system 100 utilizes a microscope slide and thus the substrate 102 is referred to as a slide, it should be understood that the digital imaging system 100 is not limited to using slides and may utilize any suitable substrate 102. Thus, as used herein, the term “slide” shall mean any suitable substrate for fixing specimens thereon, including a microplate, a microarray, or other suitable medium. As used herein, the term “microscope slide” shall mean a thin, flat piece of glass or plastic used to hold objects for examination under a microscope. The specimens may include any type of specimen that is digitally imaged under microscopic magnification, such as biological or chemical specimens, including both cytological specimens and pathological tissue specimens. For purposes of describing the imaging system 100, the specimen will be taken to be a cytological specimen that contains individual cells distributed within a volume of some dimension, however, it should be understood that the system can be used equally well with different types of specimens, such as pathology specimens, as it can be used with cytological specimens.

[0030]

[0097] The automated digital imaging system 100 is designed to acquire high-resolution digital images of the entire specimen (or specimens) on a slide 102, enabling review of the specimen using the digital image of the specimen rather than the physical glass slide 102. An optical microscope is not used in the typical review workflow of the digital imaging system 100. Instead, a digital image of the entire specimen on the slide 102 is acquired and made viewable on a computer monitor 109 at a review station 108 (see FIG. 2). The digital imaging system 100 may also include an image processing software application 137 (installed on the imager computer 105 of the imager 104 and / or the workflow subsystem 106) that embodies image processing algorithms configured to analyze the digital image of the slide specimen and identify objects of interest (OOI), i.e., individual cells in the case of a cytological specimen, or individual tissue structures in the case of a pathological solid tissue specimen. The identified OOIs are then used to assist a user (e.g., a cytologist or cytopathologist) in quickly and accurately reviewing digital images of slide specimens, as described in the above-referenced U.S. Patent Nos. 7,587,078, 6,665,060, 7,006,674, and 7,590,492. The term "OOI" means that an object has been determined to have characteristics that indicate that the object (individual cells or tissue structures, as the case may be) may be useful for consideration in the diagnosis or treatment of a patient, and does not necessarily mean that such object has been determined to be of real interest or practical importance to a cytologist or cytopathologist.

[0031]

[0098] The automated digital imaging system 100 includes one or more imager subsystems 103 (each including an imager 104 and an imager computer 105) and one or more review stations 108 operatively connected to a workflow subsystem 106 (also referred to as a "workflow server 106"). Generally, each imager 104 is a bench-top system for capturing digital images of slides 102. The imagers 104 process batches of slides in an automated manner. The workflow server 106 is a computer server that functions primarily as a mass disk storage system for storing and managing digital images and associated slide data. The workflow server 106 also includes software for image processing and data management, and may also provide networking capabilities. The review station 108 is a workstation including a computer and monitor for accessing the digital images from the workflow server and displaying the digital images for diagnostic review of the images (OOI and / or whole specimen images).

[0032]

[0099] 1, the imager subsystem 103 is in data communication with the workflow subsystem 106 via a communications network, which may include one or more of a local area network (LAN, e.g., Ethernet), a wide area network (WAN), the Internet (e.g., a virtual private network (VPN)), or other suitable networks. Similarly, the review station 108 is in data communication with the workflow subsystem 106 via a communications network, which may include one or more of a local area network (LAN, e.g., Ethernet), a wide area network (WAN), the Internet (e.g., a virtual private network (VPN)), or other suitable networks.

[0033]

[0100] The imager 104 in the described embodiment is designed to operate with a microscope slide 102. Referring to FIG. 3, an exemplary embodiment of the microscope slide 102 is shown. The microscope slide 102 is a rectangular glass plate 110 (or other suitable material) having a slide identification area 112, a specimen area 114, and fiducial marks 116. The microscope slide 102 may be a standard size microscope slide, approximately 75 mm by 25 mm, or other suitable size. The microscope slide 102 may have beveled corners to facilitate handling and positioning of the slide 102. The specimen area 114 may be a circle having a diameter of up to approximately 22 mm. The entire specimen area on the microscope slide 102 may also be imaged. The slide identification area 112 may be up to approximately 25-28 mm in length. The slide identification portion 112 may be printed with a bar code, ID number, and / or other information. The specimen area 114 is left as a transparent area of ​​the glass slide. The fiducial marks 116 may be used by the imager 104 as reference points on the slide 102 to identify the position and / or orientation of the slide 102 and its features relative to the imager 104. The specimen 119, which includes multiple objects distributed within a three-dimensional volume, is fixed to the slide 102, typically within the specimen area 114, although in some cases the specimen may extend outside the specimen area 114. The three-dimensional volume has a length (l), a width (w), and a thickness or depth (d). The thickness (d) defines a z-axis relative to the surface of the slide 102. The three-dimensional volume may be substantially uniform or non-uniform in shape, including thickness, length, and / or width. The volume of the specimen 119 is uniform, i.e., one or more dimensions of the specimen vary by less than 10%, or less than 15%, or less than 20%. The specimen 119 may be any suitable specimen, such as a cytological specimen, where the objects are cells, a solid tissue sample, where the objects are tissue structures, etc.

[0034]

[0101] As shown in FIG. 3, a cover slip 115 can be used to cover the specimen 119 in the specimen area 114. The specimen cover slip 115 is sufficiently transparent to capture micro-images of the specimen through the cover slip 115. In other words, the cover slip 115 does not impede the imager 104 in capturing micro- and / or macro-images through the cover slip 115 using the imaging station 190 and / or the macro vision station 232. The cover slip 115 serves to preserve the specimen 119, protect it from contaminating itself and others, and hold the specimen 119 flat in place. The cover slip 115 has a thickness 117.

[0035]

[0102] 4 and 5, slide carriers 118 are used to load multiple slides 102 into the imager 104. As shown in FIG. 4, the imager 104 has a capacity of ten slide carriers 118, but may be configured to hold any suitable number of slide carriers 118. Each slide carrier 118 has two rack holders 202, each of which receives a slide rack 120. Each slide rack 120 may hold twenty slides 102, or forty slides 102, or any other suitable number of multiple slides 102. The slide racks 120 may be standard off-the-shelf racks, such as Sakura® racks or Leica® racks. Thus, each slide carrier 118 holds forty slides, and the total capacity of the imager 104 is 400 slides (ten slide carriers 118 x 40 slides per carrier 118). The slide carriers 118 have handles 122 for carrying the slide carriers 118 and inserting and removing the slide carriers 118 from the imager 104. 7 and 8, slide carrier 118 has a T-slot 124 that is slidingly received on a T-rail 126 attached to a slide carrier deck 128. Slide carrier 118 is installed on imager 104 by aligning T-slot 124 with T-rail 126 and sliding slide carrier 118 onto slide carrier deck 128. Slide carrier 118 is removed from imager 104 by pulling slide carrier 118 straight out from slide carrier deck 128.

[0036]

[0103] 2 and 11, the imaging subsystem 103 includes an imager 104 and an imager computer 105. The imager 104 and the imager computer 105 communicate data with each other via a network connection, such as an Ethernet connection (e.g., a 10GE optical connection). The imager computer 105 may be any suitable computer, such as a personal computer having one or more computer processors (CPUs) and graphics processors (GPUs). For example, one suitable computer for the imager computer 105 is shown diagrammatically in FIG. 11. The imager computer 105 may have one or more of the following specifications:

[0104] a. Motherboard 130 - dual processor server class. It is understood that this dual processor configuration is appropriate for the software intended to run on the imager subsystem 103;

[0105] b.CPU-Intel(TM) Xeon(TM) class or above;

[0106] c. At least 64Gb of DRAM;

[0107] d. Dedicated GPUs 132, such as two Nvidia™ RTX-5000 or higher, to ensure sufficient processing speed for image capture and generation of a slide data set (SDS 304, see FIG. 34) for each slide imaged by the imager subsystem 103;

[0108] e. a local hard drive 133 for storing software programs 135, including an image processing software application 137 (described in more detail below);

[0109] f. Local mass data storage, such as hard drives, solid state drives, etc.134;

[0110] g. Network and USB input / output (I / O) 136;

[0111] h.Power supply 138;

[0112] i. Uninterruptible power supply (UPS) 140;

[0037]

[0113] The input / output 136 includes one or more of the following:

[0114] Imager VGA video, touchscreen USB, and computer audio are provided to the imager 104. The imager 104 receives VGA video from the imager computer 105 and drives the imager display monitor 168. A dedicated imager computer USB port provides I / O to the touchscreen of the imager display monitor 168. The imager computer 105 provides audio to the imager 104 where it is amplified. The imager speaker is connected to an audio amplifier.

[0115] b.USB ports are provided for manufacturing and service support of the keyboard and mouse. Under normal operation the keyboard and mouse are not attached and are not required for normal system operation of the imager. An additional USB port is provided on the front of the imager for easy user access.

[0116] c. Imager computer power is obtained from the mains via the imager IEC entry connector, or from an external Smart UPS 140 (if connected).

[0117] d. A dedicated USB port is provided to communicate with a smart UPS. In the event of a mains power failure, the UPS communicates the event to the imager computer 105, which initiates a controlled power down such that the integrity of the computer software, data, and configuration is maintained.

[0038]

[0118] As noted above, the imager computer 105 is in data communication with the workflow subsystem 106 via a communications network, such as a 1 Gb Ethernet wired network connection.

[0039]

[0119] Turning to FIGS. 4, 9 and 10, the imager 104 is a bench-top digital imaging device and serves as an input device for the overall digital imaging system 100. The imager 104 captures digital images of a batch of slides 102 loaded into the imager 104 in a slide carrier 118 in an automated manner with little or no user intervention. The imager 104 includes a chassis assembly 142, a base assembly 148, and an enclosure assembly, which provide a framework within which the imager components are mounted and housed. The chassis / enclosure assembly 142 can be configured to be a bench-top instrument having dimensions of 36 inches wide by 28 inches deep by 25 inches high or less. The imager 104 can have a weight of 200 pounds or less without slides 102 or slide carriers 118 loaded therein. These are only preferred dimensions and weights and do not necessarily limit the imager 104, except as specified in the claims. Thus, the imager 104 may have other maximum dimensions and weights consistent with the intended use.

[0040]

[0120] The imager 104 includes a chassis assembly 142 attached to a base assembly 148 and an enclosure assembly 146. The imaging station 190 includes a base 144 and an anti-vibration imager base plate 152 (see FIG. 19 ) on which the imager components are attached. The imaging station 190 is attached to the base assembly 148. The enclosure assembly 146 includes an enclosure left panel 154, an enclosure right panel 156, an enclosure rear panel 158, an enclosure top panel 159, an enclosure left front window 160, an enclosure right front window 162, an enclosure right panel 164, and a slide carrier deck door 166. Each of the enclosure components is attached to the chassis assembly 142 and / or another enclosure component. The enclosure left front window 160 is transparent or translucent so that the internal imager components are visible through the enclosure left front window 160. The enclosure left front window 160 is pivotally mounted so that it can be opened and closed to provide access to the internal imager components. The enclosure right front window 162 is transparent or translucent so that other internal imager components are visible through the enclosure right front window 162. The enclosure left panel 154 has a left window 170 through which the internal imager components can be viewed. The enclosure top panel 159 may also have a top window 172 through which the internal imager components are visible. The enclosure display panel 168 is pivotally mounted so that it can be opened and closed to provide access to the internal imager components. The slide carrier deck door 166 is pivotally mounted so that it can be opened and closed to provide access to the slide carrier deck 128 for loading and unloading the slide carrier deck 128 (see FIGS. 4 and 10).

[0041]

[0121] The enclosure assembly 146 also provides inputs and outputs for the imager 104, user access, and environmental control. Figure 12 is a block diagram of the enclosure assembly 146. The enclosure assembly 146 provides air management to ensure there is sufficient airflow inside the imager 104, to maintain the internal operating temperature below a maximum (e.g., below 40°C), and to maintain a positive internal pressure (when the slide carrier deck door 166 is closed). The enclosure assembly 146 provides external signal and power connections.

[0042]

[0122] The imager display monitor 168 may be an LCD touchscreen display or other suitable display. The imager display monitor 168 indicates operation and status of the imager 104 and also provides operator control via a touchscreen (or other input device). As shown in FIG. 12, the imager display monitor 168 is connected to the imager computer 105 through a bulkhead connector in the enclosure assembly 146. The imager computer 105 provides VGA video, touchscreen USB, and computer audio to the imager 104. The imager 104 receives VGA video from the imager computer 105 and drives the imager display monitor 168. A dedicated imager computer USB port provides I / O to and from the touchscreen of the imager display monitor 168. The imager computer 105 also provides audio to the imager 104 where it is amplified. The imager speaker is connected to an audio amplifier.

[0043]

[0123] 12, the enclosure assembly 146 also includes a window sensor for detecting the open or closed state of the enclosure left front window 160, and a window lock for locking the enclosure left front window 160 in a closed position during imaging operations of the imager 104. The enclosure assembly 146 also includes a door sensor for detecting the open or closed state of the slide carrier deck door 166, and a door lock for locking the slide carrier deck door 166 in a closed position during imaging operations of the imager 104.

[0044]

[0124] Turning now to the internal components of the imager 104, a slide carrier deck 128 is disposed near the bottom of the imager on the base 148. The slide carrier deck 128 has a plurality of slide carrier bays, such as ten slide carrier bays, or twenty slide carrier bays, or any other suitable number of slide carrier bays, each configured to receive a respective slide carrier 118. The slide carrier deck 128 is open at the top, providing access to the slides 102 contained in slide carriers 118 that have been loaded onto the slide carrier deck 128 by a slide handler 176. The slide handler 176 removes the slides 102 from the slide carriers 118, moves the slides 102 to various stations of the imager 104, and inserts the slides 102 back into the slide carriers 118.

[0045]

[0125] 13, slide carrier deck 128 has a carrier lock 173 and a carrier presence sensor 175 for each slide carrier bay 174. Carrier lock 173 may be a solenoid operated lock or other suitable powered locking mechanism, and carrier presence sensor 175 may be an optical sensor or other suitable sensor for detecting the presence of slide carrier 118 in the respective slide carrier bay 174. Carrier lock 173 and carrier presence sensor 175 are in data communication with imager computer 105 via system interface board 182 (see FIG. 29, SIB, described below).

[0046]

[0126] 10 and 14-16, slide handler 176 automatically moves slides 102 between various stations of imager 104, including between slide carrier 118, imaging station 190, and macrovision station 232. Slide handler 176 includes support platform 180 and slide handling gantry 178 movably coupled to support platform 180. Slide handling gantry 178 is a pick-and-place robot capable of moving slide gripper 184 in three degrees of freedom: lateral motion (X-axis) and vertical motion (Z-axis) (as shown in FIG. 14), and back-and-forth motion (Y-axis) (as shown in FIG. 15). Slide handling gantry 178 can also rotate slide gripper 184 about the theta axis (see FIG. 15) and open and close slide gripper 184 to grip and release slide 102. The slide handling gantry 178 has motion mechanisms for providing movement and / or actuation of the slide handling gantry, including an X-axis mechanism 254, a Y-axis mechanism 264, a Z-axis mechanism 274, a theta-axis mechanism 290, and a gripper mechanism 185. The X-axis mechanism 254 includes a motor 256, an encoder 258, a home sensor 260, and a drive mechanism 262. The Y-axis mechanism 264 includes a motor 266, an encoder 268, a home sensor 270, and a drive mechanism 272. The Z-axis mechanism 274 includes a motor 276, an encoder 278, a home sensor 280, and a drive mechanism 282. The theta-axis mechanism 290 includes a theta motor 292, a theta encoder 294, a theta home sensor (not shown), and a theta drive mechanism (not shown). Each motion mechanism includes a motor, an encoder, a home sensor, and a drive mechanism to meet the travel, transit acceleration, velocity, and payload requirements of the motion envelope. Gripper 184 also has an optical sensor 188 (slide inventory sensor 188 ) that is used to detect the presence or absence (inventory) of slide 102 in slide carrier 118 and the presence or absence of slide rack 120 in slide carrier 118 .Gripper 184 may be an off-the-shelf part designed to grip slide 102 from the end of the slide, such as those manufactured by SMC™ Electric Grippers™ of Los Angeles, California.

[0047]

[0127] As shown in Figure 17, the slide handling gantry 178 is operatively connected to a gantry robot motion controller 186 which controls the motion of each motion mechanism. The gantry robot motion controller 186 is in data communication with the imager computer 105 via a controller area network (CAN), which transmits data and commands to the gantry robot motion controller 186 to control the operation of the slide handling gantry 178. Figure 18 shows a schematic diagram of the gantry robot motion controller 186.

[0048]

[0128] 10 and 19-26, the imager 104 has an imaging station 190 for capturing digital micro-images of the slides 102 (i.e., the specimens 119 on each slide 102) using a digital camera 192, a tube lens assembly 194, a mirror 196, and an objective lens 198. The camera 192 uses a Sony CMOS Pregius GenII sensor and has a resolution of 4096×2160 pixels. The light path provides an optical resolution of 0.255 μm / pixel. Because the field of view ("FOV") of the camera is much smaller than the entire specimen area 114 of the slide 102, the imager 104 must take many micro-images to capture the entire specimen area 114. For example, the field of view of the camera 192 may be 1044 μm×550 μm, or approximately 1 mm×0.5 mm, while the specimen area 114 of the slide 102 may have a diameter of 21-22 mm. Based on these specifications, camera 192 requires approximately 885 non-overlapping images to image the entire specimen area 114. However, to obtain a focused image of all objects within the three-dimensional volume of the specimen 119, a step size equal to the image sub-area equal to the depth of field of imaging station 190 is used, resulting in overlapping images, so many more frames are captured. For example, as described below, a micro-image is taken every 38.25 μm=~1 / 14 of the frame height, so the actual number of images captured during a scan of the entire specimen area is approximately 11,000, or more than 10,000, or more than 5,000, or between 5000 and 15000. Camera 192 acquires micro-images while XY slide stage 210 moves specimen 119 at a constant speed. In this example, camera 192 takes an image approximately every 38 μm of stage movement in the scan direction, and therefore all camera fields of view contain 14 images per FOV. The XY slide stage 210 provides relative motion between the field of view of the camera 192 and the slide 102 so that the camera 192 can scan the entire specimen area 114 of the slide 102 and take multiple individual digital images that, when combined, cover the entire specimen area 114.

[0049]

[0129] As described herein, the individual micro-images can be combined using an image processing software application 137 (see FIG. 11) to create a composite image of the entire specimen area 114.

[0050]

[0130] As shown in Figures 19-22, the imaging station 190 includes a high resolution digital camera 192 that is part of an image acquisition assembly 191 (which is a microscope camera). The image acquisition assembly 191 can be angled from 0° to 3°, or 0° to 10°, or 0° to 5°. The camera 192 is in network data communication with the imager computer 105 via a high speed network, such as a 10GE Ethernet optical network, or other suitable high speed network capable of transmitting high resolution micro images from the camera 192 to the imager computer 105 with sufficient throughput during the image scanning process.

[0051]

[0131] A tube lens 194 is placed in the optical path of the camera 192. The tube lens 192 in combination with the objective lens 198 sets the optical magnification of the camera 192. A first fold mirror 196 is placed in the optical path of the camera 192 after the tube lens 194 and is angled at approximately 45° with respect to the optical imaging axis of the camera. The angle and position of the first fold mirror 196 can change the co-linearity of the optical axis of the objective lens and the center of the camera 192. The first fold mirror 196 is in the imaging path and must be of sufficient quality (e.g., reflectivity greater than 90% and precision flatness greater than λ / 4) or it will directly impact image quality by blurring light at the detector over all or part of the field of view and / or cause distortion.

[0052]

[0132] An objective lens 198 follows the optical path of the first fold mirror 196. The optical axis of the objective lens 198 is angled at approximately 90° from the optical axis of the camera by the first fold mirror 196. The objective lens 198 is attached to an imaging Z stage 200. The Z stage 200 has threads for mounting the objective lens 198. The Z stage 200 can controllably move the objective lens 198 to adjust the focus of the camera 192 on the slide 102 (i.e., adjust the focal plane relative to the slide 102). The position of the Z stage can be set via a Z stage controller. The slide 102 (i.e., the specimen 119 on the slide 102) is positioned at the front focal plane of the objective lens 198, which projects an image of the slide 102 to the camera 192 via the first fold mirror 196 and the tube lens 194.

[0053]

[0133] 21 and 22, all of the components of the image acquisition subassembly 191, including the digital camera 192, the tube lens 194, the first folding mirror 196, the objective lens 198, and the Z-stage 200, are mounted on an image acquisition base plate 201. The image acquisition base plate 201 has a number of image subassembly alignment adjustments 203 that allow for positional and / or angular adjustment of the image acquisition subassembly 191 with respect to other components of the imaging station 190, as well as relative positional and / or angular adjustment of the components of the image acquisition subassembly 191.

[0054]

[0134] In Figures 19-20 and 23-24, the slide 102 is behind the objective lens 198 in the optical path of the camera 192. The slide 102 is held on a slide holder 202. The slide holder 202 has a slide recess 204 for holding the slide 102 in a horizontal position on the slide holder 202. The slide holder 202 has a first slide justifier arm 206 that is spring loaded to bias the slide 102 in the Y direction within the slide recess 204. The slide holder 202 has a second slide justifier arm 208 that is spring loaded to bias the slide 102 in the X direction within the slide recess 204. The first and second slide justifier arms 206, 208 hold the slide 102 securely in the proper position on the slide holder 202 so that the slide 102 does not move during the scanning operation. The imaging station 190 has a slide release bracket 209 attached to an XY stage platform 214 that is stationary relative to the slide holder 202. The slide holder 202 is moved relative to the slide release bracket 209 to operate the first and second slide justifier arms 206, 208 to release or engage the first and second slide justifier arms 206, 208 to retain or release the slide 102 in the slide holder 202.

[0055]

[0135] As best shown in FIG. 20, the optical axis of the objective lens 198 is tilted in the scan direction at a tilt angle 205 relative to the orthogonality of the plane of the slide 102 (and slide recess 204) (see FIG. 33). In other words, the resulting optical axis of the camera 192 and the optics at the surface of the slide (also referred to simply as the "optical axis of the camera 192") are non-orthogonal to the plane of the slide 102. As described in more detail herein, the tilt angle 205 allows the imager 104 to acquire a volumetric image of the specimen 119 on the slide 102 (i.e., an image that spans the depth of the specimen 119). In other words, the micro-image includes focused images of features at different depths of the specimen 119 on the slide 102, rather than just a single focal plane as would be the case for an image taken at an orthogonal angle of the specimen 119. The imaging station 190 may be configured to acquire micro-images, each of which includes at least a portion of the depth of the slide 102 below the surface of the slide 102. If a coverslip 115 is used on the slide 102 , the imaging station 190 may be configured to acquire micro-images, each of which includes at least a portion of the depth of the coverslip 115 .

[0056]

[0136] As described herein, these micro-images can be processed by an image processing software application 137 (see FIG. 11) to stitch the micro-images together and flatten the images into a two-dimensional composite image of the specimen 119 that is in focus at all depths of the specimen 119. For example, U.S. Patent Application Publication No. 2009 / 0295963 describes a method for stitching images together to form a single digital image 302 (see, e.g., FIG. 34, which shows a block diagram of the digital image 302). As shown in FIG. 25, the appropriate tilt angle 205 for imaging the entire depth (Z dimension) of the specimen 119 on the slide 102 is a function of the thickness (d) of the specimen and the projected field of view (d1) of the camera 192 on the slide 102. The projected field of view (d1) is the in-plane field of view of the camera sensor divided by the combined optical magnification of the camera 192 (including, e.g., the tube lens 194, the objective lens 198, and other magnifications in the optical path). As shown in FIG. 25C, the tilt angle 205 is the arcsine (d / d1). As an example, for a specimen depth (d) of 10 micrometers, an in-plane field of view (d1) of 8 millimeters, and a combined optical magnification of 40x (i.e., d1=8 millimeters / 40=0.20 mm), the tilt angle 205 is arcsine [0.01 mm / (8 mm / 40)], which is about 2.86 degrees. In another example shown in FIG. 25C, for a specimen depth (d) of 24 micrometers and an effective in-plane field of view (d1) of 0.5 millimeters (e.g., a sensor field of view of 20 millimeters and an optical magnification of 40x), the tilt angle 205 is 2.75 degrees. In general, for typical specimen depths, camera sensor dimensions, and magnifications, the tilt angle 205 is typically between about 2 degrees and about 10 degrees.

[0057]

[0137] The slide holder 202 has a slide holder base plate 212 attached to an XY slide stage 210. The slide holder base plate 212 has a slide holder angle adjustment unit 213 so that the orientation of the slide holder base plate 212 can be adjusted relative to the XY slide stage 210 (and also relative to the optical axis of the objective lens 198).

[0058]

[0138] The XY slide stage 210 moves the slide holder 202 and slide 102 in a scanning pattern during an imaging procedure to acquire images that cover the entire specimen area 114 (or other area of ​​interest) of the slide 102. As shown in FIG. 26, the XY slide stage 210 includes motors, encoders, and limits of motion in each of the X and Y axes. The XY slide stage 210 is connected to an XY stage driver 211, which is connected to an XY stage controller 215 for controlling the movement of the XY slide stage 210. The XY stage controller 215 is in data communication with the imager computer 105 via an Ethernet network connection, allowing the imager computer 105 to control the operation of the XY slide stage 210.

[0059]

[0139] The XY slide stage 210 is attached to an XY stage platform 214, which is attached to the imager base plate 152. The XY stage platform 214 has an XY stage angle adjustment portion 218, which allows the orientation of the XY stage platform 214 relative to the imager base plate 152 to be adjusted.

[0060]

[0140] Thus, the slide holder angle adjuster 213 and the XY stage angle adjuster 218 can be adjusted to properly align the movement of the slide holder 202 and the XY slide stage 210 horizontally or at a desired angle relative to the optical axis of the objective lens 198 (i.e., the optical axis of the camera 192). Typically, the movement of the slide holder 202 and the slide stage 210 are aligned to be parallel to provide a constant focal height of the micro-images during scanning, such that the image acquisition assembly 191 acquires micro-images of the specimen 119 at the same height (e.g., a single z-axis focal height) relative to the surface of the slide 102.

[0061]

[0141] The imager base plate 152 is isolated from the scanner chassis 144 via multiple (in this case four) anti-vibration mounts 221. The anti-vibration mounts 221 provide vibration isolation for the imaging station 190.

[0062]

[0142] The objective lens 198 is focused onto the slide 102 held in a slide holder 202 using a Z stage 200, as shown in Figures 19-22 and 24. As shown in Figure 26, the Z stage 200 includes a piezo focus n point controller 217 and a piezo focus n point stage 219 for controlling the Z stage and thus the focus of the objective lens 198.

[0063]

[0143] As shown in FIGS. 23 and 24 , a microscope calibration component 207 (also referred to as a “microscope calibration target”) is disposed within the slide holder base plate 212 such that the microscope calibration component 207 moves with the XY slide stage 210. The microscope calibration component 207 may be directly attached to the slide holder base plate 212 or may be integral with the slide holder base plate 212 such that the microscope calibration component 207 is defined in the slide holder base plate 212 and cannot be removed therefrom. For example, the microscope calibration component 207 may be formed or etched within or through the slide holder base plate 212. Alternatively, the microscope calibration component 207 may be a separate component from the slide holder base plate 212 that is attached to the slide holder base plate 212, such as being mounted in a cavity defined by the slide holder base plate 212. The microscope calibration component 207 may be configured to perform both positional and optical calibration of the microscope camera assembly 191 (also referred to as the image acquisition subassembly 191). Alternatively, the microscope calibration component 207 may be configured to perform only an optical calibration of the microscope camera subassembly 191. For positional calibration, the calibration component 207 is configured to measure or determine positional calibration parameters including one or more of the "X" position of the XY slide stage 210, the "Y" position of the XY slide stage 210, the "Z" position of the XY slide stage 210, camera and stage alignment, changes in one or both of the "X" and "Y" positions of the XY slide stage 210 during microimaging of the slide 102, and / or review and changes in the "Z" position of the XY slide stage 210 during microimaging of the slide 102.For optical calibration, the calibration component 207 is configured to measure or determine optical calibration parameters of the microscope camera subassembly 191, including, for example, one or more of grayscale linearity, magnification, signal-to-noise ratio, illumination changes, modulation transfer function (MTF), checking non-moving pixels or elements of the digital camera 192 (e.g., CMOS or charge-coupled device (CDD)), illumination evenness or uniformity, detection of artifacts such as dust or dirt that degrade the signal-to-noise ratio. The microscope camera subassembly 191 is then calibrated and adjusted as necessary based on the positional and / or optical calibration parameters measured or determined using the calibration component 207. An example of a suitable microscope calibration component 207 is disclosed in U.S. Pat. No. 7,848,019 B2.

[0064]

[0144] 27 and 28, the imaging station 190 also includes an illumination module 222 for providing sufficient uniform light to the slide 102 for the objective lens 198 to collect in order for the camera 192 to create a high quality image. The illumination module 222 includes a light source 224 (e.g., LED 224), a first lens 226, a second lens 227, a third lens 229, a first aperture 223, a second aperture 225, a second folding mirror 228, a tube lens mount 237, and an adjustable focusing lens 230 necessary to project the LED light onto the bottom of the slide 102. The illumination module 222 projects uniform light to achieve uniform illumination at the focal plane of the specimen. The term "uniform light" means light that is substantially uniform in color and brightness. The term "uniform illumination" means that the specimen 119 is substantially illuminated with light that is substantially uniform in color and brightness. The illumination module 222 is powered by a mains power supply wired through the SIB. The light level of the illumination module 222 is set with feedback from the camera to achieve a target background brightness. The illumination module 222 interfaces with a lighting interface on the SIB 182 (described below). The illumination module 222 has a Y adjustment screw for adjusting the illumination in the Y direction in the XY plane relative to the image acquisition assembly 191, and an X adjustment screw 235 for adjusting the illumination in the X direction relative to the image acquisition assembly 191 in the XY plane.

[0065]

[0145] The illumination module 222 contains light sources 224 (e.g., LEDs) whose power, spectral distribution, and radiance profile are the starting point for the uniformity, quality, and quantity of all the illumination required to generate high quality micro-images.

[0066]

[0146] A scanning process for obtaining micro-images of the slide 102 using the scanning station 190, including focused images of features at different depths of the specimen 119 on the slide 102, will now be described. As discussed above and shown in FIG. 25, the imaging station 190 captures micro-images of the specimen 119 on the slide 102 by tilting the focal plane of the camera 192 with respect to the surface of the specimen 119 on the slide 102 such that the focal plane extends through the depth of the slide 102. Thus, each micro-image captured by the camera 192 includes an image of the depth of the specimen 119 on the slide 102, since it is taken at an angle with respect to the surface of the specimen 119 (i.e., at an angle with respect to the surface of the slide 102). As shown in FIG. 33, the slide 102 is moved by the XY slide stage 210 to scan the field of view of the camera 192 over the entire specimen area 114 (or a predetermined area of ​​the entire actual specimen 119, such as when the actual specimen 119 covers an area different from the specimen area 114 and the boundaries of the actual specimen 119 are pre-determined). The XY slide stage 210 moves the slide 102 in a serpentine path back and forth, capturing micro-images of a swath of the specimen 119 at each pass. To capture the swath, the XY slide stage 210 moves the slide 102 continuously, and the camera 192 is triggered by the XY slide stage 210 to capture an image when it reaches each trigger point according to the position of the stage encoder. The serpentine path, with the start of each successive swath close to the end of the previous swath, minimizes the time required to scan the entire specimen 119. As the slide 102 moves along the swath, the camera 192 captures micro-images of the specimen 119.

[0067]

[0147] To maintain acceptable focus during imaging scans, a best focus plane for the slide 102 can be estimated using measurements of fiducial marks 116 printed on the slide 102 and an offset defined by measurements of the best focus offset of the specimen 119 from the fiducial plane. Coefficients defining the offset plane are then uploaded to the XY stage controller 215, which maintains the position of the Z stage 200 according to those coefficients during the scan. During imaging scans, scan quality is monitored in a variety of ways, including focus quality metrics, stage following errors, and image quality metrics. Image quality metrics can include one or more of sharpness, noise, dynamic range, tone reproduction, contrast, color accuracy, distortion, vignetting, etc., which may be measured by digital imaging algorithms known by those skilled in the art.

[0068]

[0148] As shown in FIG. 25A, in a conventional imaging system, the imaging plane is nominally parallel to the plane of the slide (and the specimen 119 on the slide) (i.e., parallel within manufacturing tolerances). A single best focal plane is captured for each adjacent field of view, spaced about the length of the camera's field of view apart (the micro-images may overlap slightly to aid in stitching the images together). The micro-images are then stitched together to create a composite image of the entire specimen 119. Note that due to different heights of objects on the glass slide within the three-dimensional volume of the specimen 119, not all objects are captured in focus in a single imaging run.

[0069]

[0149] In contrast, as shown in FIG. 25B, the image capture method of the imager 104 utilizes an imaging plane that is tilted relative to the specimen 119 on the slide 102. Because the imaging plane is tilted relative to the specimen 119 on the slide 102, different sub-regions of the camera pixel array capture images of the specimen at different heights. Thus, with the camera at imaging position #1, object A is captured in focus, but in that same camera frame, object B is near the right edge of the frame and is significantly out of focus. Instead of stepping the camera position by the length of the field of view in the scan direction between images, a step size is used that is equal to the image sub-region that matches the depth of field of the imaging optics. The series of images are overlapping. Object A is in focus when the camera is at position #1, but object B is captured in focus when the camera reaches position #4. In this way, all objects within a height range equal to the difference between the left and right edges of the imager's focal plane are captured in focus in an image.

[0070]

[0150] Referring to FIG. 25C, in one example, the camera pixel array may be divided into 14 sub-regions representing different focal planes across a sample depth of 24 μm. Once the same sample area (tile) is captured in each camera sub-region, the sub-regions are vertically reconstructed to create a focus stack of tiles. The stack is then converted into a single-plane merged focus image by comparing each pixel with adjacent pixels in the same plane and selecting the pixel in the plane with the best focus quality using an algorithm for determining pixel focus with a focus metric as taught in U.S. Pat. No. 7,769,219. Examples of identifying pixels and other objects with the best focus quality are also described in U.S. Pat. No. 7,736,304, supra. Techniques for determining best focus for slide fiducial marks are taught in U.S. Pat. No. 8,116,550. The number of sub-regions is merely an example, and the digital imaging system 100 may use any suitable number of sub-regions for image processing.

[0071]

[0151] Below, an exemplary method of processing images by GPU 132 to obtain a merged image of the entire specimen 119 on slide 102 is described. Image processing involves many steps to take raw images and convert them into a final merged compressed image. Color camera 192 may utilize a Bayer mask (a color filter array for placing RGB color filters over a grid of photosensors). In such a case, the first step of the image processing algorithm is to de-Bayer the image and convert it into an RGB image (a bitmap image that holds the red / green / blue values ​​for each pixel). The image is then corrected for uniformity and distortion using a pre-mapped calibration table. In one example, the calibration table is an array the size of the camera image. This table contains an entry for each pixel location of the corrected image result. Each entry indicates a 2x2 pixel sub-region of the source image and the weighting factors that are applied to those source pixels and summed to obtain the resulting pixel value. This allows the corrected image to represent (by interpolation) the sub-pixel shifts required to correct for small optical distortions measured during the calibration process. FIG. 65 shows an example of a pre-mapped calibration table for position distortion correction of 1 / 2 pixel top, bottom, left, right, and right.

[0072]

[0152] Once all subregion images for a tile (e.g., 14 subregions) have been captured, a merged image is calculated. The difference between each pixel and a particular pixel in its neighboring region is calculated. This difference is then weighted based on the pixel's value to determine the relative merit of that particular plane for that particular pixel, generating a tile of plane value values ​​(e.g., 0-13). A moving average is then applied to the tiles of plane values ​​to improve the transition between objects in the final image. Pixels of the merged image are then selected based on their associated plane values. The merged images are then tiled to generate swaths. Once a complete swath is created, it is stitched back to the previous swath. Once the image of the entire specimen 119 has been stitched together, the entire image is compressed at a ratio of approximately 20:1 using JPEG, JPEG2000, or other suitable compression algorithm. A pyramid is then created for the image, which, along with the slide metadata, is transferred to the workflow subsystem 106 where it is accessible for review by the review station 108. Slide metadata refers to non-image information related to the whole specimen image, such as the slide identifier, which is a barcode read from the slide, the date and time of image capture, etc. Slide metadata may include a list of locations of OOI 308 identified by image analysis algorithms. An example of a "pyramid" is a set of lower resolution images obtained by repeatedly reducing the size of the original full whole specimen image by 50% until a preset minimum size is reached. The purpose of the pyramid is to support more efficient rendering on the review screen. If a reviewer is looking at a zoomed out, lower resolution version of the image, the data can be provided by accessing the appropriate level of pyramid image data, which requires less data transfer than sending the full resolution image and then reducing it in size at the review station 108.

[0073]

[0153] As shown in FIGS. 30-32, the imager 104 also has a macro vision station 232 for acquiring identification information of each slide 102 and for acquiring unmagnified macro images of the entire slide and / or specimen area 114 (e.g., for acquiring information outside the specimen area 114). The macro vision station 232 includes a first slide holder 234 in an imaging position within an imaging area of ​​a macro camera 231 (e.g., a digital camera having a CCD or other imaging sensor) of the macro vision station 232, and a second slide holder 236 in a queue position outside the imaging area of ​​the macro vision station 232. A bottom illumination module 238 having a PCB and LEDs is disposed below the first slide holder 234 to illuminate the bottom of the slide 102 disposed in the first slide holder 234. A diffuser 244 may be disposed between the bottom illumination module 238 and the first slide holder 234. A top illumination module 240 having a PCB and LEDs is positioned below the first slide holder 234 to illuminate the top of the slide 102 placed in the first slide holder 234. The macro vision station 232 has a macro camera 231 and a second folding mirror 242 that reflects the optical axis of the macro camera 231 to the slide 102 in the first slide holder 234. The macro camera 231 is mounted using a camera adjustment 246 to allow the orientation of the macro camera 231 to be adjusted.

[0074]

[0154] The macro vision station 232 acquires a macro image of the slide 102, including the barcode identification information and / or other information on the slide 102, and an image of the entire slide 102 and / or the specimen area 114, and transmits the macro image to the imager computer 105. The macro image includes one or more fiducial marks 116 and the specimen area 114 and / or the specimen 119. The imager computer 105 is configured to determine a relative position and boundaries of the specimen area 114 and / or the specimen 119 on the surface of the slide 102 based at least in part on the one or more fiducial marks 116. The relative position and boundaries of the specimen area 114 and / or the specimen 119 are provided to the imaging station 190. The imaging station 190 (e.g., image acquisition subassembly 191) is configured to collect a micro image based at least in part on the relative position and boundaries of the specimen area 114 and / or the specimen 119 on the slide 102. In other words, the imaging station 190 utilizes the relative position and boundaries of the bounding specimen area 114 and / or specimen 119 to determine the scanning position of the camera 192 when acquiring micro images so as to avoid overscanning beyond the edges of the specimen area 114 and / or specimen 119, which would waste time and storage. The macro vision station 232 also interfaces with the SIB 182, which interfaces with the imager computer 105 to control the operation of the macro vision station. The macro camera 231 is in data communication with the imager computer 105 via a communications network, such as a 100baseT Ethernet network.

[0075]

[0155] A schematic diagram of the SIB 182 is shown in Figure 29. The SIB consolidates external signals transmitted between the imager computer 105 and the imager 104 and redistributes them to the appropriate components and subassemblies of the imager 104, and to the appropriate inputs of the imager computer 105.

[0076]

[0156] The complete operation of the imager subsystem 103 to image the slides 102 and create a single merged image of the entire specimen 119 on each slide 102 will now be described. An operator manually loads each of the slides 102 to be scanned into the slide carriers 118. If the slides 102 are provided in a slide rack 120, this simply involves inserting the slide rack 120 into the slide carrier 118. If the slides 102 are not in the slide rack 120, the operator first loads the slide 102 into the slide rack 120 and then inserts the slide rack 120 into the slide carrier 118. The slide carrier 118 can be used for storage of the slides 102 before and after imaging, and for transporting and loading the slides 102 into the imager 104.

[0077]

[0157] The slides 102 in the slide carriers 118 are manually inspected to ensure that the slides 102 are properly oriented in the slide carriers 118 and that the printed sides of the slide identification portions 112 are all facing backwards (towards the back of the slide carriers 118). With the imager 104 in idle or pause mode, the operator opens the slide carrier deck door 166 and manually inserts slide carriers 118, each loaded with up to 40 slides, into each slide carrier bay 174. Carrier presence sensors 175 in each slide carrier bay 174 each detect whether a slide carrier 118 is installed and signal the imager computer 105 accordingly. The carrier lock 173 of the slide carrier 118 currently being actively imaged by the imager 104 is actuated to lock the active slide carrier 118 in place when the imager 104 removes and images a slide 102 from the active slide carrier 118. Other slide carriers 118 not actively imaged by the imager 104 are not locked (considered inactive).

[0078]

[0158] When a new slide carrier 118 is inserted into a slide carrier bay 174 and its carrier presence sensor 175 signals that the slide carrier 118 is present, the imager 104 schedules an inventory of such slide carrier 118. The imager 104 performs an inventory on each of the slide carriers 118 mounted to the imager 104 using an optical slide inventory sensor 188. The slide handling gantry 178 scans the sensor 188 on each of the slide carriers 118 to determine which of the possible slide slots in the slide rack 120 has a slide 102 inserted into it. This slide inventory information is provided to the imager computer 105 and used when each slide carrier 118 is activated and ready to be scanned, so that the slide handling gantry 178 only moves to slide carrier 118 slots in which a slide 102 is present.

[0079]

[0159] When each of the slide carriers 118 is scheduled for imaging, the slide handling gantry 178 moves to the first slide 102 in the slide carrier 118, activates the gripper 184 to grip it, and moves the slide 102 to the macro vision station 232. The first slide 102 is then placed on the first slide holder 234 in the imaging position of the macro vision station 232. The macro vision station 232 acquires multiple macro images of the first slide 102 using the macro camera 231, creates a complete specimen image (i.e., a macro image that includes a single image of the entire specimen 119), and reads the printed identification information (e.g., a barcode). This information becomes part of the slide data set (SDS) record of the first slide 102. As described herein, the macro image also includes the fiducial marks 116 and one or more of the specimen area 114 and / or the specimen 119. The imager computer 105 uses the macro image to determine the relative positions and boundaries of the specimen area 114 and / or specimen 119 on the surface of the slide 102 based at least in part on the one or more fiducial marks 116. The relative positions and boundaries of the specimen area 114 and / or specimen 119 are provided to the imaging station 190.

[0080]

[0160] When the imaging station 190 is ready to accept a new slide 102, the slide handling gantry 178 moves the first slide from the first slide holder 234 in the imaging position of the macrovision station 232 to the imaging station 190 and places the first slide 102 on the slide holder 202 of the imaging station 190. While loading the first slide 102 into the slide holder 202, the XY slide stage 210 can be positioned such that the slide release bracket 209 actuates the first and second slide justifier arms 206, 208 to the release position so that the first slide 102 is placed on the slide holder 202. Once the first slide 102 is placed on the slide holder 202, the XY slide stage 210 moves the slide holder 202 such that the first and second slide justifier arms 206, 208 are biased to their respective engaged positions, thereby biasing and holding the first slide 102 in place within the slide holder 202 during scanning.

[0081]

[0161] After the first slide 102 has been moved from the first slide holder 234 in the imaging position of the macro vision station 232 to the slide holder 202 of the imaging station 190, the slide handling gantry 178 can then move the next slide 102 in the slide carrier 118 (i.e., the second slide 102) to the first slide holder 234 in the imaging position of the macro vision station 232. The macro vision station 232 can perform a process of acquiring a macro image in a manner similar to that described for the first slide 102 in parallel (i.e., simultaneously) with the imaging station 190 acquiring a micro image of the specimen 119, as described below. The loading of the second slide into the first slide holder 234 of the macro vision station 232, and the macro imaging of the second slide 102 can occur while the imaging station 190 is imaging the first slide 102. After the macro vision station 232 completes macro imaging of the second slide 102, the second slide 102 must typically wait at the macro vision station 232 until the imaging station 190 completes micro imaging of the first slide 102 and the slide handling gantry 178 slides the first slide 102 from the slide holder 202 to the second slide holder 236 in a queue position (described in more detail below).

[0082]

[0162] The imaging station 190 then scans the first slide 102 to capture multiple micro-images of the specimen 119 on the first slide 102. As described above, while the digital camera 192 takes the multiple micro-images, the first slide 102 is moved continuously in a serpentine path by the XY slide stage 210 to move the first slide 102 under the objective lens 198. The imager 104 coordinates the movement of the XY slide stage 210 and the Z-axis movement (focus) to image the slide 102. While the first slide 102 is being scanned, image data from the camera 192 is streamed to the imager computer 105. The micro-images are reconstructed into segmented image data from the camera 192 and streamed to the imager computer 105. Because the imaging station 190 acquires micro-images based at least in part on the relative positions and boundaries of the specimen area 114 and / or specimen 119 on the slide 102, wasted time and storage caused by scanning and imaging outside of the specimen area 114 and / or specimen 119 is minimized.

[0083]

[0163] After the first slide 102 is imaged, the slide handling gantry 178 moves the first slide 102 to the second slide holder 236 in a queue position. This queue position is the cue point when the imaging operation is completed. This function is used to optimize the throughput of the slide handling gantry 178 by placing slides 102 that have completed the imaging process near the slide holder 202 of the imaging station 190, rather than returning the first slide 102 to the slide carrier 118. The first slide 102 remains in the queue position while the slide handling gantry 178 moves the next slide 102 (e.g., the second slide 102) to be micro-imaged by the imaging station 190 from the first slide holder 234 in the imaging position of the macrovision station 232 to the slide holder 202 of the imaging station 190. If not in the queue position, the first slide 102 must be returned to the slide carrier 118 before the next slide 102 can be placed in the slide holder 202 of the imaging station 190, a slower process since the slide carrier 118 is farther away from the slide holder 202 than the second slide holder 236. After the next slide 102 to be micro-imaged by the imaging station 190 has been moved to the slide holder 202, the slide handling gantry 178 is available to move the first slide 102 from the second slide holder 236 back into the slide carrier 118 (i.e., while the imaging station 190 is imaging the next slide 102). Once the second slide 102 has been moved onto the imaging station 190, the slide handling gantry 178 moves the third slide 102 from the active slide carrier 118 to the macrovision station 232 and places it onto the first slide holder 234, which then images the third slide 102. This process is repeated for all slides 102 in the active slide carrier 118. In this manner, the overall throughput of the imager 104 is increased by using the second slide holder 202 queue position.

[0084]

[0164] The imager computer 105 receives the camera micro images from the imaging station 190 and reconstructs them, which are then reconstructed into focal plane images as described above and combined into a single best focus plane as described above. This is all done in the GPU, as well as stitching / aligning adjacent swaths. The resulting images are streamed from the GPU to the CPU, compressed and saved to disk.

[0085]

[0165] The imager computer 105 also utilizes image analysis algorithms (described below) to scan the image data to identify objects of interest (OOI) and generate OOI location data, which identifies the location of the OOI within the image. This data also becomes part of the SDS 304. Each SDS 304 is processed to generate an optimized data set ("pyramid") that is used for image display at the review station 108. The pyramid data (a compressed subset of the SDS) is designed to speed up access to slide data for recorded OOIs. The pyramid data is also added to the SDS. Once image processing and pyramid generation are complete, the compressed SDS is sent to the workflow subsystem 106 and moved to an active storage partition on a NAS disk array dedicated to active data (see FIG. 1).

[0086]

[0166] The image analysis algorithm may identify OOI using any suitable process. As an example, the image analysis algorithm analyzes the digital image 302, such as by scanning the digital image 302 and performing primary and secondary segmentation. The primary and secondary segmentation measures, identifies, and / or extracts various characteristics for each of the individual and clustered objects in the digital image 302. Certain characteristics are known to be relevant to object classification and can be utilized, such as to classify as normal, abnormal, diseased, healthy, precancerous, cancerous, etc. For example, in the case of a digital image 302 of a cytological sample, characteristics may include cell size, nuclear to cytoplasmic area ratio, nuclear corrected integrated optical density, cytoplasmic vacuolization, darkness, etc., which may be used to classify cells as precancerous, cancerous, normal, and / or abnormal. Primary and secondary segmentation are described in U.S. Patent Application Publication No. 2004 / 0253616. The algorithm then calculates an object score for each object based on the measurements of the respective characteristics. Based on this score, the algorithm identifies or selects objects that are considered OOI 308 and clusters the objects (see, for example, FIG. 34).

[0087]

[0167] The imager computer 105 may analyze the determined characteristics of each OOI 308 and identify OOIs 308 on the same digital image 302 (i.e., the same sample) that are similar using an OOI matching algorithm. The OOI matching algorithm compares the determined characteristics of each of the OOIs 308 to identify OOIs 308 that have similar determined characteristics, such as having one or more characteristics that are within respective predetermined ranges of each other. The OOI matching algorithm may also be utilized to determine whether an OOI 308 on the digital image 302 is similar to an OOI 308 in a library of OOIs 308. The library of OOIs 308 may be stored in a database of a laboratory information system (LIS) 408 (see FIG. 1 ) or in a database stored in another data storage device or system in network communication with the imager computer 105. The database of OOIs 308 includes a library of OOI images and / or digital images 302 of previously classified and characterized objects (e.g., cells). Each object in the library of OOI images and / or digital images 302 has previously identified characteristics that can be compared to the characteristics of each of the OOI 308 on the digital images 302 being analyzed by the imager computer 105. The objects in the library images may be pre-classified as abnormal, normal, pre-cancerous, cancerous, malignant, pre-malignant, benign, etc.

[0088]

[0168] FIG. 62 shows a graphical representation of an image analysis algorithm 500 for digital cytology. In step 502, the digital image 302 is divided into frames. In step 504, the frames are processed by a frame processor running in parallel threads. The frame processor segments the frames, such as using a watershed method applied to the G channel of RGB. The frame processor further computes object features and performs convolutional neural network (CNN) inference (e.g., using a 192x192 window, color corrected MobileNet) running in parallel on the CPU and GPU (graphics processing unit) of the imager computer 105. In step 506, the slide is processed and OOIs are selected and grouped based on the identified features in step 508.

[0089]

[0169] The automated digital imaging system 100 may include a machine learning diagnostic module configured to utilize machine learning to characterize and / or diagnose objects on the digital images 302. The machine learning diagnostic module may be installed and executed on the imager computer 105, the review station computer 111, or other suitable computer system. The machine learning diagnostic module may perform a process to characterize objects on the digital images 302 and diagnose a patient based on the characterization. For example, as shown in one exemplary process, the process includes creating a training set of characterized / diagnosed objects. The training set is created by imaging a large number of specimen slides 102, such as hundreds, thousands, or tens of thousands of slides 102, to obtain a digital image 302 of each specimen slide 102. OOIs are then identified in each of the digital slides 102 as described herein. The OOIs are independently classified for classification by multiple clinicians, such as three, four, five, or more clinicians (e.g., cytologists and / or pathologists in the case of cytological specimens). For example, the objects may be classified as HSIL, LSIL, normal, abnormal, diseased, healthy, precancerous, cancerous, etc. "HSIL" is an acronym for high grade squamous intraepithelial lesion, and "LSIL" means low grade squamous intraepithelial lesion. A lesion is an area of ​​abnormal tissue, and high and low grades refer to the likelihood that it will progress to cancer. A machine learning diagnostic module then determines multiple characteristics of each OOI, such as by using image analysis algorithms described herein or similar algorithms. For example, for a digital image 302 of a cytological sample, the characteristics may include cell size, nuclear to cytoplasmic area ratio, nuclear corrected integrated optical density, cytoplasmic vacuolization, darkness, etc., which may be used to classify cells as precancerous, cancerous, normal, and / or abnormal.

[0090]

[0170] The machine learning diagnosis module then uses a pattern recognition algorithm to determine a characterization / diagnosis relationship between the features identified by the image analysis algorithm and the characterization / diagnosis determined by the clinician. The characterization / diagnosis relationship may be an algorithm, function, equation, or other suitable relationship.

[0091]

[0171] The machine learning diagnostic module can then use the characterization / diagnosis relationships to characterize / diagnose objects on the new specimen slide 102. The imaging station 103 generates a digital image 302 of the slide 102. Image analysis algorithms are used to identify the OOIs on the digital image 302. The machine learning diagnostic module then determines a number of characteristics of each OOI, such as by using the image analysis algorithm. Finally, the image analysis algorithms characterize / diagnose each OOI, using the characterization / diagnosis relationships. The characterization / diagnosis of each OOI is also added to the SDS 304.

[0092]

[0172] The machine learning diagnostic module can continue to learn and optimize the characterization / diagnosis relationships by adding digital images 302 from new specimen slides 102 to the training set and then running a process to generate the characterization / diagnosis relationships using pattern recognition algorithms.

[0093]

[0173] The imager computer 105 may also include a slide statistics module to identify slide statistics such as the number of objects on a slide (e.g., cell count), the number of borders, etc. The imager computer 105 may utilize a similar process to determining the OOI to determine the slide statistics. Similar to determining the OOI, the imager computer scans the digital image 302, identifies individual objects, clustered objects, and borders, and counts the number of objects, clustered objects, and borders. The slide statistics are added to the SDS 304.

[0094]

[0174] The imaging subsystem 103 takes images of all slides 102 in each slide carrier 118 loaded into the imager 104 and processes the images before completing the process. Once imaging is complete, the slide carriers 118 are removed from the slide carrier deck 128. The slides 102 can then be removed from the slide carriers 118 and physically stored per lab procedures or discarded if no longer needed.

[0095]

[0175] As described above, the processed digital images including their respective SDSs are sent to the workflow subsystem 106 where they are stored. The workflow subsystem 106 stores the digital images in active storage where they can be accessed by the review station 108. The workflow subsystem 106 includes a workflow subsystem management application 248 (see FIG. 1 ) configured to manage the workflow of reviewing the digital images by a reviewer, such as a cytologist and / or a cytopathologist. For example, when a reviewer logs into the review station 108, the workflow subsystem 106 sends to the review station 108 a worklist of cases including digital images for the reviewer. The review station 108 displays the worklist to the reviewer. The reviewer can then manage the worklist, select a case to work on, review the digital images for the selected case, complete the case, and / or track completed and ongoing cases.

[0096]

[0176] Upon selecting a case for review, the review station 108 accesses the selected case from the workflow subsystem 106, which transmits the digital images and SDS to the review station 108. The review station 108 displays the digital images on various review screens. The user interface of the review station 108 allows the reviewer to select different digital images (e.g., a case may contain multiple digital images or a case may contain a single digital image). The review screens display the selected digital image and an image of the OOI. The OOIs may be displayed as thumbnails to display multiple OOIs at once on the review screen, allowing the reviewer to select an OOI to see a full-sized view of the OOI. Once the reviewer has reviewed the digital images for a case, the review station 108 displays a completion screen that allows the reviewer to dispose of the case, such as marking the case as "complete" or "for review by a second reviewer (e.g., a cytopathologist)" or "complete later" (if the reviewer has not yet completed a full review of the case).

[0097]

[0177] Upon completion of the case, including a second review (such as by a cytopathologist) if necessary, the case is entered into a laboratory information system (LIS) 250 (see FIG. 1). The workflow subsystem 106 may have a LIS communication interface 252 to send the completed case to the LIS, or it may be entered into the LIS manually. Clinicians can then access the case results via the LIS 250.

[0098]

[0178] 34-62, embodiments of systems and methods for displaying digital images generated by automated digital imaging system 100 are described in further detail. The digital display systems and methods enable reviewers of biological specimens, such as cells, tissues, and other objects, to more efficiently and accurately review digital images of the specimens and identify, characterize, and / or diagnose attributes within the specimens. Thus, the digital display systems and methods provide certain improvements to the way computers operate and impart certain capabilities to display systems for biological specimens that solve various problems in computerized digital microscopy of biological specimens.

[0099]

[0179] 34, the workflow subsystem 106 and digital imaging display system 108 (also referred to as the "review station 108") of the digital imaging system 100 are shown in further detail. As described above, the imager computer 105 uses image processing algorithms (e.g., software applications 135 installed on the imager 104 and / or workflow subsystem 106) to analyze the digital images of the slide specimen 119 and identify objects of interest (OOIs) 308 (see FIG. 36). The identified OOIs 308 can then be used to assist a user (e.g., a cytologist or cytopathologist) in quickly and accurately reviewing the digital images of the slide specimen 119.

[0100]

[0180] As described above, the workflow server 106 is a computer server that functions primarily as a mass disk storage system for storing and managing the digital images and associated slide data stored in the SDS 304 data record for each slide 102. The workflow server 106 may also include software for image processing and data management and provide network capabilities. The SDS 304 includes the stitched digital images 302, slide information 306 captured by the imager 104 for each slide 102, and OOI position data 308. Each SDS 304 is processed by the imager computer 305 to generate an optimized and augmented data set ("pyramid") that is used for image display on the digital display system 108. As described herein, the pyramid data (a subset of the SDS) is designed to speed up access of slide data for recorded OOIs. The pyramid data may also include information that improves the display of the digital image 302 of the entire specimen by providing coordinate and display information for panning and zooming to areas of the digital image 302 that do not have OOIs (particularly non-gynecological slides). Pyramid data is also added to the SDS. Once image processing and pyramid generation is complete, the compressed SDS 304 is sent to the Workflow Subsystem 106 and moved to an active storage partition 107 on a NAS disk array dedicated to active data (see FIG. 1).

[0101]

[0181] The digital display system 108 is a workstation including a computer 111 and a monitor 109 for accessing the digital images from the workflow server and displaying the digital images for diagnostic review of the images (OOI 308 and / or whole specimen image 302). As shown in FIG. 1, the imager subsystem 103 is in data communication with the workflow subsystem 106 via a communication network, which may include one or more of a local area network (LAN, e.g., Ethernet), a wide area network (WAN), the Internet (e.g., a virtual private network (VPN)), or other suitable networks. Similarly, the review station 108 is in data communication with the workflow subsystem 106 via a communication network 113, which may include one or more of a local area network (LAN, e.g., Ethernet), a wide area network (WAN), the Internet (e.g., a virtual private network (VPN)), or other suitable networks.

[0102]

[0182] The digital image display system 108 is configured to access the SDS 304 of each slide 102 from the workflow subsystem to allow a reviewer (e.g., a cytologist and / or a cytopathologist) to review the digital image of the slide 102 on the monitor 109. With reference to FIG. 34, each digital image display system 108 includes a computer 111 and a computer monitor 109. The computer 111 can be any suitable computer having a microprocessor (CPU), memory, storage, and a network adapter. The computer 111 may have a graphics processing unit (GPU) to improve the speed of processing large digital image files. The digital image display system 108 can also include input devices such as a keyboard, mouse, touchpad, etc. The monitor 109 can be a touch screen monitor where the touch screen is an input device. As described herein, the digital image display system 108 is in data communication with the workflow subsystem 106 via a communication network 113.

[0103]

[0183] The monitor 109 may be a color monitor specially configured for displaying the digital image 302 of the specimen slide 102. For example, the monitor 109 may have a particular resolution and / or color calibration for displaying the specimen slide 102 for use in pathology or other particular applications. For example, in cytopathology, color may be used to distinguish different types of cells, and particular types of colored stains and / or reagents may be used to prepare the specimen for review by a reviewer who marks the specimen. For example, U.S. Pat. No. 6,661,501 describes various stains and their manufacturing methods, methods for staining cells for cytological or histological analysis to contrast the nuclear portion of the cell with the cytoplasmic portion, and systems and methods for illuminating cytological samples, which are suitable for preparing cytological specimens for use in the digital imaging system 100. In order for the reviewer to properly review the slide 102, the effects of the stains and / or reagents must be detectable on the digital image when displayed on the monitor 109. In other words, the digital imaging system 100 can be configured so that the digital image of the slide 102 viewed on the monitor 109 appears the same or substantially the same as the appearance of the actual slide 102 using a microscope-based system to view the slide 102.

[0104]

[0184] 35, an exemplary workflow path 310 for reviewing digital images 302 of specimen slides 102 using the digital image display system 108 is shown. A reviewer accesses a case 312 from a number of cases 312 stored in a case management system (a component of the workflow subsystem management module 314 of the workflow subsystem 106). Each case 312 includes or is associated with one or more SDSs 304, each having one or more digital images 302. A first reviewer 316 (e.g., a cytologist) pulls up a case 312 on the digital image display system 108.

[0105]

[0185] As shown in FIGS. 36-44, the display system 108 can display a worklist of cases 312 to be reviewed from which the first reviewer 316 can select for review. As shown in FIG. 36, the display system 108 generates a navigation menu 318 and displays it on the monitor 108 of the display system 109. FIG. 36 shows that the navigation menu 318 can be collapsed to provide space on the monitor 109 to display other information. Upon selecting any one of the selections in the navigation menu 318, the display system 108 displays a list 320 of the cases 312 in that menu selection. As shown in FIG. 37, the list 320 includes check boxes 322 that the reviewer 316 can select to review next. The list 320 also includes a "check all" box to select all of the cases 312 in the list 320.

[0106]

[0186] FIG. 38 provides an overview of the different screens displayed by the display system 108 during use of the display system 108, including different screen options and different screen layouts depending on the type of case 312. At the leftmost part of FIG. 38, the display system 108 displays a login screen 324. After the user logs in, the display system 108 displays one of the worklist screens 326, 328, or 330. The different worklist screens 326, 328, 330 have different display formats. The worklist screen 326 displays the worklist in a simplified list format with different columns for different information for each listed case 312, as shown in more detail in FIG. 40. The worklist screen 328 displays the worklist in a tile / grid format with a two-dimensional arrangement of tiles, as shown in more detail in FIG. 41. The worklist screen 330 displays the worklist in a Kanban format with the cases 312 displayed as tiles in separate columns depending on the status of each case, as shown in more detail in FIG. 42. After a user selects a case 312 from one of the worklist screens, the display system 108 displays the selected case on a review screen 332 for gynecological cases (GYN) or on a review screen 334 for non-gynecological cases (NGYN). Because gynecological cases typically include only a single slide 102, the review screen 332 is formatted to display a screen for reviewing digital images of the single slide 102. In contrast, because non-gynecological cases typically include multiple slides 102, the review screen 334 is formatted to display a screen for reviewing digital images of multiple slides 102 simultaneously. Thus, references throughout the description and drawings to GYN cases may be considered to refer to cases having a single slide 102 and NGYN to cases having multiple slides 102.

[0107]

[0187] 38, after the user completes the review of the review screens 332, 334, the display system 108 displays a review completed screen 336 or 338 depending on whether it is a GYN or NGYN case. After the user exits the review completed screens 336, 338, the display system 108 displays a worklist completed screen 340 or a saved worklist screen 342. The worklist completed screen 340 displays a list of cases 312 that have been completed by the user, while the saved worklist screen 342 displays a list of cases in progress 312, which means cases that have been started but not yet completed.

[0108]

[0188] 39 illustrates that the display system 108 may be configured to toggle between different worklist screens 326, 328, 330. For example, the worklist screens 326, 328, 330 may include a worklist screen command 344 that a user can select with a cursor or other input command using an input device to direct the display system 108 to display the desired worklist screen.

[0109]

[0189] Turning to FIG. 40, the worklist screen 326 is shown in detail. The worklist screen 326 shows the user's worklist of cases 312 in a simplified list with each case 312 on its own row, and information about each case is shown in different columns in each row. The "MY WORKLIST" title includes a case count 351 indicating the number of cases 312 in the worklist screen 330. For each case 312, the worklist screen 326 includes the following information columns: slide number indicator, access number, case type, status, and deadline. The slide number indicator indicates whether the case 312 has a single slide, indicated by a slide icon 352, or multiple slides, indicated by a file folder icon 354 indicating the number of slides in the case 312. The worklist screen 326 also includes a filter function 346 to filter the cases 312 in the displayed list to the display system 108. The worklist screen 326 also displays a completed reviews counter 348 indicating the number of reviews the user has completed that day. The worklist screen 326 also has a search box 350 in which a user can enter a search query, whereupon the display system 108 performs a search of cases 312 for the search query. The display system 108 may be configured to search the user's worklist, the entire database of cases 312 in active storage 107, or any other desired database of cases 312.

[0110]

[0190] 41 shows details of the worklist screen 328. The worklist screen 328 displays a worklist of a user's cases 312 in a tile / grid format. Each tile contains the same or similar information as the columns in the worklist 326, including a slide number indicator, accession number, case type, status, and due date. The worklist screen 328 also includes a case count 351, a filter function 346, a review completion counter 348, and a search box 350, similar to the worklist screen 326.

[0111]

[0191] FIG. 42 shows the Worklist screen 330 in more detail. The Worklist 330 shows a user's worklist of cases 312 in a Kanban format, with the cases 312 displayed as tiles in separate columns depending on the status of each case. The tiles are substantially the same as the tiles in the Worklist screen 326. The tiles are arranged in three columns: Awaiting Primary Review column 356, In Review column 358, and Primary Review Complete column 360. Each tile contains the same or similar information as the columns in the Worklist 326, including a slide count indicator, accession number, case type, status, and due date. The cases 312 in the Primary Review Complete column 360 are displayed with less contrast (e.g., grayed out) than the cases 312 in the Awaiting Primary Review column 356 and In Review column 358 to allow the user to see progress without distracting from the task to be completed. The worklist screen 330, like the worklist screens 326, 328, also includes a case count 351, a filter function 346, a completed reviews counter 348, and a search box 350.

[0112]

[0192] FIG. 43 illustrates a details panel 361 that may be displayed by the display system 108 on any of the worklist screens 326, 328, 330. Although this feature is only shown for the worklist screen 326, it may be used on the worklist screens 328 and 330 as well. The worklist screen 326 includes a details button 362. When a user selects one of the cases 312 in the worklist, such as by clicking on the case 312, the display system 108 highlights the case in the list, as shown in FIG. 45. When a user then clicks on the details button 362, the display system 108 displays a details panel 360 that includes additional details of the selected case 312, including, for example, a progress indicator 364, an imaging date 366, the name of the reviewer responsible for the case 312, and a case history 368. The width of the details panel 360 may be adjusted to make it wider or narrower, which in turn narrows or widens the main list, respectively.

[0113]

[0193] Figure 44 illustrates the use of a graphical status indicator 370 to display the status of a case 312 in any of the worklist screens 326, 328, 330. The graphical status indicator 370 in Figure 44 for the worklist screen 326 is a colored line along the left side of each column that has a particular status, e.g., a status of "In Progress." The tiles of the worklist screens 328, 330 may similarly include colored lines or colored borders to provide a convenient, easy-to-see status indicator.

[0114]

[0194] Returning to FIG. 35 , once the reviewer 316 has determined which case 312 to review from the worklist screens 326, 328, 330, the reviewer 316 selects the case 312 from the worklist screens 326, 328, 330. Upon receiving the selection of the case 312, the display system 108 accesses the SDS 304 for the selected case 312 from the active storage 107 of the workflow subsystem 106 and loads the case 312 into the display system 108. Upon loading the SDS 304 for the selected case 312, the display system 108 displays a review screen for the case 312. The review screen may differ depending on whether the selected case 312 is a GYN case or a NGYN case.

[0115]

[0195] 45 shows an exemplary GYN review screen 332. The GYN review screen 332 has a case identification panel 374 at the top of the screen. The case identification panel 374 includes the accession number, case type (e.g., GYN or NGYN), review due date, imaging date, patient information (e.g., name, date of birth, age, etc.), and reviewer identification. The GYN review screen 332 has an OOI panel 376, a main image panel 378, and a reviewer select image panel 380.

[0116]

[0196] The display system 108 displays the digital images 302 of the selected case 312 in a main image panel 378. The user can zoom and pan around the digital images 302 in the main image panel 378 using commands displayed on the GYN review screen 332 and / or commands selectable with an input device of the display system 108. The display system 108 also displays a region of the OOI image 382 when selected as described below.

[0117]

[0197] The display system 108 may have a scan entire digital image feature where the system scans the entire digital image 302 and displays it in the main image panel 378. The scan entire digital image feature may be a selectable feature having a selectable button on the review screen 332. The scan entire digital image feature allows a user to set a zoom level at which the digital image 302 is automatically scanned, such as by using a zoom command or a selectable command displayed on the review screen 332. The scan entire digital image feature scans the entire digital image 302 at a selected zoom level and displays it in the main image panel 378. The scan pattern may be a serpentine pattern similar to the scan pattern for imaging the slide 102, a row-by-row scan in the same direction (e.g., left to right), a column-by-column scan, or other suitable scan pattern. The scan entire digital image feature may also include user-selectable commands that allow a user to stop (i.e., pause) and start the scan, set a stop time, set the speed of the scan, and / or set the number of regions of the slide (e.g., 2000 most important regions, 20 most important regions, etc.). The scan entire digital image function may also have a user-selectable option to stop at every object (e.g., each cell or other object) on the digital image 302. The stop at every object option allows the user to stop for a set stop time or until the user selects a selectable option to continue scanning.

[0118]

[0198] The OOI panel 376 includes multiple OOI images 382 of areas of the digital image 302 that correspond to the OOI location data 308 of each OOI in the SDS 304. The OOI images 382 are reduced (zoomed out) images of the digital image 302. When a user selects one of the OOI images 382, ​​the area of ​​the digital image 302 of the selected OOI image 382 is displayed in the main image panel 378 at an enlarged size (i.e., enlarged compared to the OOI image 382). The main image panel 378 still displays the digital image 302, although zoomed and moved to initially display the area of ​​the OOI image 382, ​​so that the user can zoom and pan around the area of ​​the selected OOI image 382 in the same manner as described for the digital image 302. The OOI images may also include a show similar command 384, such as a "+ sign" on the OOI image 382. When a user selects a search similar command 384 for a selected OOI image 382, ​​the display system 108 displays additional OOI images 382 that have similar characteristics to the selected OOI image 382. For example, the display system 108 may display three to six additional OOI images 382 included in the SDS as similar to the selected OOI image 382. The additional OOI images 382 include objects, such as cells, that share one or more characteristics with the objects in the selected OOI image 382. The one or more additional OOI images 382 may be obtained from the same whole specimen digital image 302 or may be obtained from a library of images that include previously classified objects that share one or more characteristics with the objects in the selected OOI image 382.

[0119]

[0199] The reviewer selected images panel 380 is an area of ​​the GYN review screen 332 where the reviewer 316 can place one or more OOI images 382, ​​other images that the reviewer has added as OOI, or any other images / locations on the digital image 302 that the reviewer has selected as being of particular interest. For example, an OOI image 382 may be selected for review by a second reviewer 317, such as a pathologist.

[0120]

[0200] 46 illustrates an exemplary NGYN review screen 334. As described herein, the primary difference between a GYN case and an NGYN case 312 is that a GYN case includes a digital image 302 of a single slide 102, whereas an NGYN case 312 includes one digital image 302 for each of multiple slides associated with the case. The NGYN review screen 334 has many of the same features as the GYN review screen 332, such as a main image panel 378 and a reviewer select image panel 380.

[0121]

[0201] Additionally, the NGYN review screen 334 includes a slide list 388 listing each slide 102 for which the NGYN case 312 has a digital image 302. Each slide 102 of the NGYN case 312 is represented by a slide icon 387 in the slide list 388 with an identifier, such as a letter or number. When the reviewer selects a slide 102 in the slide list 388, the display system 108 displays the digital image 302 of the selected slide 102 in the main image panel 378. The selected slide 102 may be highlighted in the slide list 388 to indicate that it is the currently displayed slide 102. The display system 108 allows the reviewer to interact with the digital images 302 in the same manner as for the GYN case 312, as described above. The reviewer can select each of the slides 102 in the slide list 388, and the display system 108 repeats the display process for each of the slides 102. The NGYN review screen 334 may also include a slide count 392 indicating the number of slides of the NGYN case 312 reviewed and the total number of slides of the NGYN case 312.

[0122]

[0202] FIG. 47 shows another example of an NGYN review screen 394 that is similar to the NGYN review screen 334, except that the slide list 388 is wider and the slide icons 387 are larger to better show each individual slide 102.

[0123]

[0203] Any of the review screens, including the GYN review screen 332 and the NGYN review screens 334, 394, may also display other data from the SDS of the digital image 302 displayed on the review screen. The review screens may display the characteristics identified for one or more OOIs. The review screens may include a selectable option for each OOI image 382 that, when selected, causes the identified characteristics for that OOI image 382 to be shown. The identified characteristics may be displayed in a pop-up window. The review screens may also display machine learning characterizations / diagnosis associated with the OOI image 382. The review screens may include a selectable option for each OOI image 382 that can be selected to display the characterizations / diagnosis for the OOI image 382, ​​such as in a pop-up window. The review screens may display slide statistics from the SDS 304. The display of the slide statistics may be in response to a selectable option to display the slide statistics in a pop-up window, or the slide statistics may be automatically displayed on the case identification panel 374 or elsewhere on the review screen.

[0124]

[0204] 58-61 show review screens with additional functionality such as adding comments to the OOI image 382 and / or digital image 302, sharing the screen, and bookmarking the OOI image 382 and / or digital image 302. It should be understood that any one or more of these additional features and graphical user interfaces may be included in any of the review screens shown and described herein.

[0125]

[0205] FIG. 58 illustrates an exemplary GYN review screen 420 configured to allow a reviewer to add comments and / or mark / draw on the OOI image 382 and / or digital image 102. The review screen 420 is similar to the review screen 332 of FIG. 47 and has most of the same features as the review screen 332 of FIG. 47. As shown in FIG. 58, the review screen 420 includes a comment panel 422. The review screen 420 has an add comment command 424. When the reviewer selects the add comment command 424, the review screen 420 allows the reviewer to select a comment area 432 of the OOI image 382 in the OOI panel 376 or on the OOI image 382 or digital image 302 displayed in the main image panel 378. The reviewer can then add a text comment 426 to the comment panel 422. In this manner, the reviewer can make comments directed to a particular object on the OOI image 382 or digital image 302. The comment panel 422 displays a comment ID 428 (e.g., numbers or letters), a reviewer ID 430 (e.g., the reviewer's name or other identifier), and comment text 426. The reviewer can add comments by selecting the add comment command 424 and repeating the process. The comments, including the comment text 426, the comment area 432, the comment ID 428, and the reviewer ID 430, are stored in the SDS 304 for the respective slide 102. The comment panel 422 allows other reviewers to see the comments of the previous reviewer and add comments using the same process. The comments of the other reviewers will have different comment IDs 428 and / or reviewer IDs 430. A reviewer can add a global slide-level comment by skipping the selection of a comment area and simply including the text comment 426 in the comment panel 422. Alternatively, the review screen 420 may include a separate global slide-level comment panel similar to the comment panel 422.

[0126]

[0206] FIG. 59 illustrates an exemplary GYN review screen 450 configured to allow a reviewer to add recorded audio comments (e.g., dictated comments) to the case 312 of the digital image 102 and / or mark / draw on the OOI image 382 and / or digital image 102. The review screen 450 is similar to the review screen 332 of FIG. 47 and has most of the same features as the review screen 332 of FIG. 47. As shown in FIG. 59, the review screen 450 includes a dictation comment command 452. When the reviewer selects the dictation comment command 452, the review screen 450 allows the reviewer to record dictation comments. For example, the review screen 450 displays a dictation comment window 454. The dictation comment window 454 has a dictation control bar 456. The dictation control bar 456 has a record button 458, a play button 458, a pause button 460, and a navigation slide 462. The record button 458 starts recording an audio comment. A play button 458 plays the audio comment. A pause button 460 pauses either the recording or playback. A navigation slide 462 allows the reviewer to go forward or backward within the recorded comments. The dictation comment command 452 may also allow the reviewer to mark the comment area 432, similar to the review screen 420. Thus, the reviewer can direct comments to specific objects on the OOI image 382 or digital image 302. The dictation comment window 454 displays, for each audio comment, a comment ID 428 (e.g., numbers or letters), a reviewer ID 430 (e.g., the reviewer's name or other identifier), and a dictation control bar 456. The reviewer can add additional dictation comments by selecting the dictation comment command 452 and repeating the process. The audio comments, including the recorded audio, the comment area 432, the comment ID 428, and the reviewer ID 430, are stored in the SDS 304 for the respective slide 102. A dictation comment window 454 allows other reviewers to play back the previous reviewer's audio comments and add additional audio comments using the same process.Comments from other reviewers have a different comment ID 428 and reviewer ID 430.

[0127]

[0207] FIG. 60 illustrates an exemplary GYN review screen 434 configured to enable a first reviewer to share the first reviewer's screen with a second reviewer on a different display system 108 and to allow the first reviewer to give the second reviewer control of the review screen 434 on the first reviewer's screen. In other words, the second reviewer's display system 108 displays the same review screen 434 that is displayed on the first reviewer's display system 108, and the second reviewer can control the review screen 434 on the first reviewer's screen using the second reviewer's display system 108. The review screen 434 has a screen share command 436. When the first reviewer selects the screen share command 436, the review screen 434 displays a screen share window 438, such as a pop-up window. The screen share window 438 requests the second reviewer's screen share ID. The first reviewer enters the screen share ID (e.g., name, number, code, etc.) of the second reviewer and presses enter to send the screen share ID to the workflow subsystem 106. The workflow subsystem 106 verifies the screen share ID and then shares the review screen 434 from the first reviewer's display system 108 with the second reviewer's display system 108. The review screen 434 also has a share control command 440 that allows the first reviewer to pass control of the review screen 434 to the second reviewer. The screen share command 436 can be a toggle command that allows the first reviewer to share and unshare the review screen 434 by selecting the command 436. Similarly, the share control command 440 can be a toggle command that allows the first reviewer to share and unshare control of the review screen 434. Selection of screen share command 436 may also activate a chat window 442, which may be displayed in any open space on review screen 434 or in a movable pop-up window. In chat window 442, a first reviewer and a second reviewer may text chat in chat window 442.Additionally, the review screen 434 may enable audio or audio / video communication between the first reviewer and the second reviewer via the workflow subsystem to provide audio or audio / video communication between the reviewers. Additionally, the review screen 434 may be configured to allow additional reviewers to share the review screen 434 of the first reviewer by repeating the share command for the additional reviewers.

[0128]

[0208] FIG. 61 illustrates an exemplary GYN review screen 444, configured to allow, for example, a reviewer to bookmark a case 312 in a digital image 302 to create a reference library of interesting cases 312 and digital images 302. The review screen 444 is similar to the review screen 332 in FIG. 47 and has most of the same features as the review screen 332 in FIG. 47. As shown in FIG. 61, the review screen 444 has a bookmark command 446. When a reviewer selects the bookmark command 446, the review screen 444 displays a bookmark comment window 448 that allows the reviewer to enter a comment. For example, the reviewer can enter comment text 426 about why the reviewer bookmarked the case 312 as being of interest. The bookmark command 446 may also allow the reviewer to mark the comment area 432, similar to the review screen 420. The bookmark comment window may include a comment ID 428 and a reviewer ID 430, similar to the review screen 420. Additionally, a bookmark record is also recorded in the SDS 304 for the digital image 302, including the comment text 426, the comment ID 428, the reviewer ID, and the comment area 432. The workflow subsystem 106 assigns the bookmarked case 312 to a reference library, such as a reference library of interesting cases.

[0129]

[0209] After a reviewer selects and reviews a slide 102 in the slide list 388, the display system 108 provides a visual indicator 390 that indicates that each slide 102 has been reviewed. For example, the slide list 388 shown in Figure 48 has a dot 390 on top of each slide icon 387 in the slide list 388. The dot 390 indicates whether each slide 102 has been reviewed by a reviewer.

[0130]

[0210] FIG. 57 shows yet another example of a review screen 414, which is similar to the review screen 332 of FIG. 47 and has most of the same features as the review screen 332 of FIG. 47. The main difference is that the review screen 414 displays the entire specimen area of ​​the digital image 302 in the main image panel 378. In other words, the image in the main image panel 378 is fully zoomed out and the entire digital image 302 is displayed. The review screen 414 also has a zoom slider 416 that adjusts the zoom of the image in the main image panel 378. Any of the other review screens 332, 334, and 394 may also display the entire digital image 302 fully zoomed out in the main image panel 378 and may also include a zoom slider 416.

[0131]

[0211] After the reviewer 316 has reviewed the case 312, the reviewer 316 selects a review complete command 396 on the review screen 332, 334, 394 (see Figures 45-47). Upon receiving the review complete command 396, the display system 108 displays a completion screen. Figure 48 shows an example of a completion screen 336 for a GYN case 312 where the reviewer 316 has determined that the specimen is normal upon review. The completion screen 336 may include the same functionality as the GYN review screen 332, and may further include a completion pane 399 that is displayed on the main image panel 378. The reviewer 316 selects a "Complete as normal" command 400. Upon receiving this successful completion command, the display system 108 sends a successful completion signal to the workflow subsystem 106, updating the case 312 as determined to be normal. As shown in the workflow diagram of Figure 35, the selected case 312 is then flagged to be filed as a completed case or routed to quality control. The percentage of reviewed cases completed as normal by the first reviewer 316 is sent to quality control to be reviewed by another reviewer as a quality control check. The quality control check may determine if the reviewer 316 erroneously completed one or more cases 312 as normal.

[0132]

[0212] The completion window 399 also includes a "Finish Later" command 401 if the reviewer 316 does not complete their review of the selected case 312. Upon receiving the Complete Later command 401, the display system 108 marks the case 312 as in progress and leaves the case in the reviewer's worklist as an in progress case.

[0133]

[0213] FIG. 49 shows an example of a completed screen 336 for a GYN case 312 when the reviewer 316 determines that the specimen is abnormal after review. The completed screen 336 may include the same functionality as the GYN review screen 332, and may further include a completed pane 399 displayed on the main image panel 378. If the first reviewer 316 determines that the specimen slide 102 is abnormal, the first reviewer 316 selects the Send to Pathologist command 404. The workflow subsystem 106 then schedules the case 312 for review by the pathologist 317, as shown in the workflow diagrams of FIGS. 35 and 52-54. The pathologist then logs into the display system 108, as shown in FIG. 35, which displays a worklist screen, a review screen, and a completed screen, allowing the pathologist to select, review, and complete the case in essentially the same manner as the first reviewer 316.

[0134]

[0214] When the reviewer is finished reviewing the case, the reviewer can return to the worklist screen. FIG. 50 shows an example of a worklist screen 340 showing a details panel 361 of a case 312 that was completed as an anomaly by the first reviewer 316 and assigned to a pathologist for pathologist review. By default, a completed case 312 remains in the reviewer's worklist until the reviewer chooses to hide the completed case 312. The worklist screen 340 of FIG. 50 displays a hide case command 406 to hide a selected completed case from the worklist screen 340. The default can be changed to apply a rule to automatically hide completed cases. FIG. 51 shows the worklist screen 340 with the completed case 312 hidden.

[0135]

[0215] The comments, review results, and status of each case 312 are stored in a slide data file in the workflow server 106 (e.g., database server 530, FIG. 63). The slide data file may include the digital images 302, the SDS 304, the slide information 306, and / or the pyramid / OOI 308, or it may be a different data file.

[0136]

[0216] As shown in FIG. 35, the disposition of each case 312 may then be entered into a laboratory information system (LIS) 408 that can be accessed by a clinician 412 on a clinician computer 410 in data communication with the LIS 408 .

[0137]

[0217] 35 and 52-56 illustrate several variations of workflows for review of a case 312 by a first reviewer 316 (e.g., a cytologist) and a second reviewer 317 (e.g., a pathologist) using the digital image display system 108 and workflow subsystem 106. In the workflows illustrated in FIGS. 35 and 52-54, the terms "pull" and "push" indicate whether a reviewer pulls a case 312 from among the imaged cases waiting for review, or whether a case is assigned (pushed) to a reviewer by a manager or an automated process (e.g., a rules-based algorithm using rules for case type, case outcome, etc.). FIG. 55 includes a flowchart illustrating various scenarios for a cytologist and a pathologist in completing the workflows illustrated in FIGS. 35 and 52-54. FIG. 56 includes a flowchart illustrating various manager / administrator roles and scenarios for the workflows illustrated in FIGS. 35 and 52-54.

[0138]

[0218] FIG. 63 illustrates a block diagram of an exemplary workflow server 106 with backup and archive functionality. The workflow server 106 includes multiple software modules for performing various backup and archive services, including an imager web services module 520, a review station web services module 522, an archive services module 524, and a backup services module 525. The workflow server 106 also includes an Internet Information Services server 526 (e.g., MICROSOFT™ IIS), a security software module 528 (e.g., ASP.NET Core Identification / Authorization), a database server 530 (e.g., a SQL database server) having a database 527 of slide data files, an image repository 532 (e.g., on a RAID storage disk), and a server operating system 534 (e.g., Microsoft Server 2016). The imager web services module 520 is configured for use by the imaging station 190 to send and receive data to and from the database server 530. The review station web services module 522 hosts the review station application and provides services for sending and receiving data to and from the database server 530. The database server 530 stores the SDS 304, including the slide information 306, pyramid / OOI 308, and review information. The archive service module 524 performs archiving of the slide data files, digital images 302, and / or SDS 304. The backup service module 525 performs backup of the database of the slide data files, including the digital image files and review information.

[0139]

[0219] The backup module 525 protects against single disk failures and system failures. The backup module 525 can back up the database and slide image files 302 to a network drive 540 (e.g., using the WINDOWS™ backup feature) or to cloud storage 542 (e.g., using the MICROSOFT AZURE™ backup agent). The purpose of the backup is to protect information stored locally from disasters such as disk failures. If the backup is remote, such as a cloud backup, the backup also protects against catastrophic failures caused by disasters such as fires, floods, and other localized incidents. The backup allows the database and digital image files to be restored as part of recovery from a system failure.

[0140]

[0220] The archive service module 524 is configured to delete local data in the database server 530 and store it for longer periods more efficiently. This may also improve the performance and / or usability of the digital imaging system 100 by reducing its storage requirements. The archived data is stored in external storage, such as a remote network drive 540 or cloud storage 542 (e.g., MICROSOFT AZURE™ cloud storage or AMAZON Web Services™ (AWS) cloud storage). The archive service module 524 is configured to allow a user to specify policies for selecting digital image files 302, slide information 306, pyramid / OOI 308, and / or review information to archive. The archive service module 524 may also archive macro images (low-resolution versions of digital image files 302, if generated by the system 100). For example, a user may set a policy to archive data last accessed beyond a selected period, such as 30 days, 15 days, 60 days, etc. The archive policy may include a set time to perform the archive (e.g., a specified time each day for daily archiving, a date and time for weekly archiving, etc.). When data for a slide 102 is archived, the information in database 527 is not archived, but database 527 is updated to track which slides 102 have been archived. In one embodiment of archive service module 524, pyramid image files (e.g., OOI 308) for archived slides 102 are not archived. Archived files and local copies of pyramid files are deleted from database 527 and image repository 532.

[0141]

[0221] Referring to FIG. 64, a flow chart of a method 600 of archiving slides 102 using the archive service module 524 is shown. In step 602, the archive service module 524 receives and stores an archive policy including an archive schedule. In step 604, at the scheduled time according to the archive schedule, the archive service module 524 identifies slides 102 that meet the archive policy for archiving and places the slides 102 in an archive database table. In step 606, for each slide 102 in the archive database table, the archive service module 524 copies the slide image file 302 and slide information 306 from the database 527 and image repository 532 to the archive external storage 540, 542. If the copy is successful, in step 608, the slide data for each archived slide is deleted from the database 527 and image repository 532. In step 610, the database 527 is updated to indicate that each archived slide 102 was successfully archived. In step 612, an additional archive history database that stores an archive history log may also be updated.

[0142]

[0222] The archive service module 524 is also configured to retrieve archived slide data of the archived slides 102. The archive service module 524 can utilize a retriever web service installed on the workflow server 106. The retriever web service is configured based on the type of archive, such as whether it is a remote network drive or a specific cloud storage. A user selects a slide 102 to be retrieved from the archive using the retriever web service, and the retriever fetches the archived slide data of the selected slide 102, including slide image files 302, slide information 306, macro images, and / or other archive data, from the external storage 540, 542. Upon successful retrieval of the slide data, the system 100 regenerates the pyramid image data including the OOI 308 as described herein. The archive service module 524 updates the archive database table and / or the archive history database regarding the status of the retrieved slide 102. The SDS 304 of each of the retrieved slides 102 can be stored in the database 527 and the image repository 532 of the workflow server 106.

[0143]

[0223] Although certain embodiments have been shown and described, it should be understood that the above description is not intended to limit the scope of these embodiments, and such disclosure is provided for purposes of illustration and example only. Accordingly, various changes and modifications can be made to the disclosed embodiments without departing from the scope of the appended claims. For example, not all of the components described in the embodiments are required, and alternative embodiments include any suitable combination of the components described, and the general shapes and relative sizes of the components may be altered. Furthermore, although the system and method have been described with respect to cytological samples, it can be configured and utilized for any type of sample.

Claims

1. An automated slide imaging system, comprising: an imager; the imager is configured to acquire images of a specimen fixed to a surface of a slide, the specimen including a plurality of objects distributed within a three-dimensional volume, the imager is configured to generate an overall specimen image from the acquired images, the acquired images including a macro-image of the specimen and a plurality of micro-images of the specimen; the objects are depicted in focus within the whole specimen image, regardless of the individual positions of each object within the three-dimensional volume; The imager comprises: a macro vision station having a first slide holder and a macro camera for capturing a macro image of the entire specimen when the slide is seated in the first slide holder; and an imaging station having a slide stage, a stage slide holder mounted on the slide stage, and a micro camera for capturing a plurality of micro images of the specimen when the slide is seated in the stage slide holder; Equipped with The micro camera has an optical axis at a non-orthogonal angle with the slide when the slide is seated in a slide holder of the stage, the micro camera has a field of view at the slide that is much smaller than an entire specimen area of ​​the slide, and the imaging station is configured to automatically move the slide stage and the slide holder of the stage in a scanning pattern that scans across the specimen area to acquire a plurality of micro images of the entire specimen.

1. An automated slide imaging system comprising:

2. The imaging system of claim 1, wherein the macro image includes one or more fiducial markers disposed on the slide surface, and the imager identifies the relative position and boundaries of the specimen on the slide surface based at least in part on the one or more fiducial markers.

3. The imaging system of claim 2, wherein the imager is configured to acquire the micro-image based at least in part on the relative position and boundaries of the specimen on the slide surface identified from the macro-image.

4. The imaging system of claim 3, wherein the slide has a width defining an x-axis and a length defining a y-axis, and the micro-camera is configured such that its field of view on the slide captures the entire x-axis width of the specimen in each micro-image based on the specimen boundary identified on the slide surface.

5. The imaging system of claim 1, wherein the imaging station is configured to acquire micro-images of the specimen in the same z-axis as the three-dimensional volume.

6. In the imaging system according to claim 1, the imager one or more slide holder receptacles, each configured to receive a slide holder including a plurality of slots, each slot configured to hold an individual slide; and a robotic arm assembly, The robot arm assembly includes: Removing a slide from the slide holder slot of the slide holder receptacle; moving and seating the slide in a first slide holder of the macrovision station for acquiring a macro image; Removing the slide from the first slide holder; moving and seating the slide in a slide holder on the stage for acquiring a micro-image; Re-remove the slide from the slide holder of the stage. An imaging system configured as above.

7. The imaging system of claim 6, wherein the robotic arm assembly comprises: transferring the slide to a slide holder that may be the same or different from the slide holder from which the slide was removed; The imaging system further configured to release the slides into respective slots of the same or different slide holders.

8. The imaging system of claim 7, wherein the slot of each of the same or different slide holders is the same slot from which the slide was removed by the robotic arm assembly.

9. An imaging system as described in claim 1, wherein the imager further includes an image processor configured to generate an entire specimen image from the micro-images, the image processor identifying best focus images for each individual object in the micro-images, and the best focus images of the objects are incorporated into the entire specimen image.

10. The imaging system of claim 9, wherein the image processor is configured to identify an object of interest within the specimen and to store an image of the identified object of interest together with an image of the entire specimen.

11. The imaging system of claim 1, wherein the macro image of the specimen includes an image of a barcode on a slide surface, and the imager is configured to obtain information about the specimen from the barcode.

12. The imaging system of claim 1, wherein the specimen is a cytological cell specimen and the object is a cell.

13. The imaging system of claim 1, wherein the specimen is a pathological tissue specimen and the object is a tissue structure.

14. The imaging system of claim 1, wherein the imaging station further comprises an objective lens between the micro camera and the micro slide holder.

15. The imaging system of claim 1, further comprising a review station including a display monitor, a user interface, and a processor functionally coupled to each of the display monitor and user interface, the processor configured to display an entire image of the specimen on the display monitor along with individual images of individual objects within the specimen image.

Images