Digital imaging system and method

Abstract

The invention relates to a digital imaging system and method. An automated system and method of evaluating a sample attached to a substrate, such as a slide, an exemplary system includes a slide imager configured to acquire a plurality of micro-images of a sample attached to a substrate, the sample including a plurality of objects distributed within a three-dimensional volume and to generate an entire sample image of the sample using the micro-images, wherein objects contained in the sample are substantially focused and depicted in the entire sample image regardless of the depth of field of the respective objects within the sample. The entire sample image is stored on a storage medium for subsequent review by a cell lab using a computer-controlled review station including a display and a user interface, wherein the review station user interface is configured such that the cell lab can review and classify the stored entire sample image.

Description

[0001] Relevant information of divisional application

[0002] This application is a divisional application, and the parent case of this divisional application is an invention patent application with the international patent application PCT / US2019 / 055458 filed on October 9, 2019, entering the Chinese national stage, with the patent application number "201980072461.0" and the title "Digital Imaging System and Method". Technical Field

[0003] The present invention generally relates to digital imaging systems and methods, and more particularly, the present invention relates to digital imaging systems and methods for acquiring digital images of samples (such as cytology (cellular) samples and diseased (solid) tissue samples) that can be displayed on a computer monitor and viewed by cytotechnologists and / or cytopathologists on a computer monitor. Background Art

[0004] Cytology is a branch of biology that involves the study of the formation, structure, and function of cells. As applied in a laboratory setting, cytologists, cytotechnologists, and other medical professionals make medical diagnoses of a patient's condition based on a visual inspection of a sample of the patient's cells, which sample is referred to herein as a "cytology" sample. A typical cytology technique is the "Pap smear" examination, in which cells are scraped from a female's cervix and analyzed to detect the presence of abnormal cells (a precursor to the development of cervical cancer). Cytology techniques are also used to detect abnormal cells and diseases in other parts of the human body.

[0005] Cytology techniques are widely employed because the trauma of collecting a cell sample for analysis is generally less than that of traditional surgical pathology procedures, such as a biopsy, in which a dedicated biopsy needle with a spring-loaded translatable stylet, a fixed cannula, and the like is used to excise a solid tissue sample (referred to herein as a "pathological" sample) from a patient. A cell sample can be obtained from a patient by various techniques, which include (for example) scraping or swabbing an area or aspirating body fluid using a needle from the chest cavity, bladder, spinal canal, or other appropriate area. The cell sample obtained is typically placed in a preservative solution and then extracted from the solution and transferred to a glass slide. A fixative is applied to the cell sample to ensure that the cells remain in an appropriate position on the glass slide to facilitate subsequent staining and examination.

[0006] It is generally desirable that the cells on a slide have an appropriate spatial distribution such that individual cells can be examined. A monolayer of cells is typically preferred. Thus, preparing a cell sample from a liquid sample containing many cells (e.g., tens of thousands) generally requires: first separating the cells from each other by mechanical dispersion, liquid shearing, or other techniques so that a thin monolayer of cells can be collected and deposited on a slide. In this way, a cytotechnologist can more easily discern the presence of any abnormal cells in a patient sample. It is also possible to count the cells to ensure that an appropriate number of cells have been evaluated.

[0007] U.S. Pat. Nos. 5,143,627, 5,240,606, 5,269,918, 5,282,978, 6,562,299, 6,572,824, and 7,579,190 disclose specific methods and apparatus for generating a thin monolayer of cells from a liquid sample container and then transferring this thin layer to a "sample slide" that is conducive to visual inspection. The entire text of all the published cases cited herein is incorporated by reference. According to the methods disclosed in these patents, a rotating sample collector inserted into the container is used to disperse patient cells suspended in a preservative solution and stored in the sample container. A controlled vacuum is applied to the sample collector to draw the liquid through its filter until the desired quantity and spatial distribution of cells are collected through the filter. Thereafter, the sample collector is removed from the sample container and the filter portion is pressed against a slide to transfer the collected cells to the slide in substantially the same spatial distribution as collected. Devices manufactured according to one or more of these patents have been successfully commercialized, such as the 2000 processor (processing one sample slide at a time from a patient sample) and 5000 processor (processing batches of sample slides from patient samples). Further reference is made to U.S. Pat. Nos. 7,556,777 and 7,771,662.

[0008] After preparing a sample slide, the sample is typically visually inspected by a cytotechnologist under magnification and with or without various light sources. Additionally or alternatively, an automated slide imaging system is used to facilitate the cytological examination procedure. For example, an automated slide imaging system can capture images of all or substantially all individual cells within a cell sample fixed to a slide and use image processing techniques to perform a preliminary assessment of the cells to direct the cytotechnologist to closely examine the potentially most relevant cells on the slide. Examples of such imaging systems are disclosed in U.S. Patent Nos. 7,587,078, 6,665,060, 7,006,674, 7,369,304, and 7,590,492. Whether by examining the actual sample slide under magnification or examining a magnified image of the sample, the sample is typically classified by a cytotechnologist as "normal" or "abnormal", where abnormal samples are typically assigned to one of the main categories defined by "The Bethesda System for Reporting Cervical / Vaginal Cytologic Diagnosis", which categories include low-grade squamous intraepithelial lesion (LSIL), high-grade squamous intraepithelial lesion (HSIL), squamous cell carcinoma, adenocarcinoma, atypical glandular cells of undetermined significance (AGUS), adenocarcinoma in situ (AIS), and atypical squamous cells (ASC). Additional information regarding the classification of cell samples can be widely used, such as "The Yokohama System for reporting endometrial cytology: Diagnostic Cytopathology" (May 2018, Vol. 46(5), pp. 400 to 412) and "Guidelines for the Reporting of Nongynecologic Cytopathology Specimens, Archives of Pathology & Laboratory Medicine" (November 2009, Vol. 133, No. 11, pp. 1743 to 1756).

[0009] However, there are many drawbacks associated with prior systems and methods for obtaining digital images of biological samples. For example, prior systems and methods suffer from slow acquisition times due to the time required to scan the entire sample. Additionally, prior systems and methods typically only provide a single focal plane across the sample. Biological samples, which include cellular and pathological samples, are actually three-dimensional (i.e., have a depth). Thus, due to the high magnification and focusing aperture required to obtain a digital image of a biological sample, the depth of field of the image is very limited. As a result, portions of the sample that are out of the depth of field in the focal plane will be out of focus or not visible in the image. To obtain focused digital images at multiple different depths of the sample, the focal plane must be adjusted (e.g., by moving the sample or the camera or by adjusting the focusing lens). However, this requires additional scanning of the sample for each focal plane, which further slows the acquisition time. SUMMARY OF THE INVENTION

[0010] Embodiments of an improved automated system for evaluating a sample (e.g., a cellular or pathological sample) attached to a substrate are disclosed and described herein. Generally, the substrate is a microscope slide, and thus, the embodiments are described with respect to using a slide as the substrate, and it should be understood that the automated systems and methods disclosed herein are not limited to using a slide, but may utilize a suitable substrate. The sample can include any type of sample for digital imaging under a microscope, such as a biological or chemical sample, which includes a cell sample, a tissue sample, and the like. As used herein, the term "sample" can refer to the application to the entire sample or a portion thereof, depending on the context.

[0011] In an exemplary embodiment, an automated slide imaging system constructed in accordance with the present invention includes an imager configured to obtain an image of a sample attached to the surface of a slide (the sample including a plurality of objects distributed within a three-dimensional volume) and to generate a whole sample image from the obtained image, wherein the objects are in-focus depicted in the whole sample image regardless of the individual positions of the corresponding objects within the three-dimensional volume, and wherein the three-dimensional sample volume has a length, a width, and a thickness, the thickness defining a z-axis relative to the slide surface, wherein the individual objects of the sample (e.g., individual cells or tissue structures as the case may be) are located at different positions along the z-axis. The obtained image can include a macro image of the sample and a plurality of micro images of the sample, wherein the macro image includes one or more fiducial marks located on the slide surface, and wherein the imager is configured to obtain the micro images at least in part based on the relative position and boundaries of the sample on the slide surface determined from the macro image.

[0012] In an exemplary embodiment, the imager has: a first imaging platform configured to hold the slide; and a first camera configured to acquire the macro image when the slide is held on the first imaging platform; and a second imaging platform configured to hold the slide; and a second camera configured to acquire the micro image of the sample attached to the slide held on the second imaging platform. The imager is configured to automatically move at least one of the second camera and the second imaging platform relative to the other to acquire the micro image, wherein the optical axis of the second camera forms a non-orthogonal angle with the second imaging platform, and wherein the imager is configured to acquire the micro image of the sample at the same z-axis of the three-dimensional sample volume. It is noted that the imaged slide has a thickness, and the micro image may include at least a portion of the slide below the surface. The sample may be covered by a cover glass that is sufficiently transparent to acquire the micro image of the sample through the cover glass, the cover glass having a thickness, wherein each micro image includes at least a portion of the depth of the cover glass. In an exemplary embodiment, the sample slide has a width defining an x-axis and a length defining a y-axis, and the imager is configured to translate the slide along the y-axis relative to the second camera when the second camera acquires the micro image at the corresponding y-axis position, each micro image including the entire x-axis width of the sample based on the determined sample boundary on the slide surface.

[0013] In an exemplary embodiment, the imager has: one or more slide holder sockets, each slide holder socket configured to receive a slide holder including a plurality of slots, each slot configured to hold an individual slide; and a robotic arm assembly configured to (i) engage and remove the slide from a slot of the slide holder in the slide holder socket, (ii) transport the slide and hold the slide on the first imaging platform to acquire the macro image, (iii) engage and remove the slide from the first imaging platform, (iv) transport the slide and hold the slide on the second imaging platform to acquire the micro image, and (v) engage and remove the slide from the second imaging platform. The robotic arm assembly may be further configured to (vi) transport the slide to a slide holder that is the same as or different from the slide holder from which the slide was removed and (vii) release the slide into a slot of the corresponding same or different slide holder, wherein the slot of the corresponding same or different slide holder is the same as the slot from which the robotic arm assembly removed the slide.

[0014] In an exemplary embodiment, the imager includes an image processor configured to generate the entire sample image from the micro-images, wherein the image processor determines corresponding best-focus images of individual objects in the micro-images, and wherein the best-focus images of the objects are incorporated into the entire sample image. The image processor is preferably further configured to identify objects of interest (e.g., individual cells or tissue structures) in the sample and store images of the identified objects of interest and the entire sample image. The macro-image of the sample may include an image of a barcode on the slide surface, in which case the imager is preferably configured to obtain information about the sample from the barcode.

[0015] In an exemplary embodiment, the system further includes a viewing station that includes a display monitor, a user interface, and a processor operatively coupled to the respective display monitor and the user interface, wherein the processor is configured to display the entire sample image and individual images of objects within the sample image on the display monitor.

[0016] Optionally, the viewing station may be configured to allow a system user to select the entire sample image from a list of stored entire sample images using the user interface.

[0017] Optionally, the system may be configured to allow a system user to finally characterize or otherwise forward the entire sample image for secondary viewing using the user interface.

[0018] Optionally, the list of stored entire sample images that have not been finally characterized or otherwise forwarded for secondary viewing may be organized and displayed in a variety of different formats based on input received through the user interface. The system may be further configured to allow an authorized third party to fully or partially populate the list of stored entire sample images for viewing by a specific system user.

[0019] In an exemplary embodiment, the system is configured to allow a system user to add annotations to the entire sample image and / or a data file associated with the entire sample image, where subsequent system users may use the annotations and the entire sample image for viewing. For example, the annotations may be associated with individual objects in the entire sample image and may be in the form of electronic markings made on an image of the entire sample image or a portion of the entire sample image.

[0020] In an exemplary embodiment, the system may be configured to display one or more additional objects sharing one or more characteristics with a corresponding object in response to a system user prompt associated with the corresponding object in the entire sample image, wherein the one or more additional objects (e.g., optionally cells or tissue structures) may be from the entire sample image and / or from a library containing previously classified objects. Without limitation, the system may be configured to allow data regarding the entire sample image or individual objects therein to be input via the user interface and stored in a data file associated with the entire sample image.

[0021] In an exemplary embodiment, the system is configured to display a magnified view of at least a portion of the entire sample image and automatically scan the displayed at least a portion of the entire sample image. In this embodiment, the system preferably automatically scans in a user-selectable scan mode, which includes (but is not limited to) a serpentine mode, a row-by-row mode, and a column-by-column mode. In this embodiment, the system is preferably configured to allow a system user to set the magnification factor when the system displays the scanned at least a portion of the entire sample image via the user interface. In such embodiments, the system is preferably configured to allow a system user to stop and start the scan and set the scan speed at the display position of the scan via the user interface. In such embodiments, the system is configured to allow a system user to pause the scan at each object while the corresponding object is being displayed during the scan.

[0022] In an exemplary embodiment, the system is configured to display a viewing screen on the display monitor, the viewing screen including a main image panel in which the entire sample image is displayed and an object panel in which individual images of objects within the sample image are displayed, wherein a system user may zoom in / out and / or pan the entire sample image within the main image panel via the user interface. Without limitation, the system may be configured to display, within the main image panel, a region of the entire sample image containing the object after the system user selects an individual image of a corresponding object in the object panel via the user interface. Without limitation, the system may be configured to allow a system user to select an object in the displayed entire sample image via the user interface, wherein the system displays an image of the selected object in the image panel selected by the system user.

[0023] According to another aspect of the present invention, there is provided a method of generating a whole sample image of a sample attached to the surface of a slide, the sample including a plurality of objects distributed within a three-dimensional volume, the method comprising: (i) acquiring a macro image of the sample; (ii) acquiring a plurality of micro images of the sample at least in part based on the macro image; and (iii) generating the whole sample image by processing the micro images using an image processor, wherein the objects are substantially in-focus depicted in the whole sample image regardless of the individual positions of the corresponding objects within the three-dimensional volume, the three-dimensional volume having a length, a width and a thickness, the thickness defining a z-axis relative to the slide surface, wherein the corresponding objects (e.g., cells or tissue structures) of the sample are located at different positions along the z-axis.

[0024] Under non-limiting conditions, a first camera can be used to acquire the macro image, and a second camera can be used to acquire the micro images, wherein at least one of the second camera and the slide is automatically moved relative to the other when acquiring the micro images, wherein the optical axis of the second camera forms a non-orthogonal angle with the slide, and wherein the micro images are acquired at the same z-axis of the three-dimensional volume. Then, the whole sample image can be generated by processing the micro images using an image processor to determine the corresponding best-focus images of the individual objects in the micro images and incorporating the corresponding best-focus images of the objects into the whole sample image.

[0025] The macro image preferably captures one or more fiducial marks located on the surface of the slide, and the method further includes determining the relative position and boundaries of the sample on the slide surface at least in part based on the one or more fiducial marks, wherein the micro images are acquired at least in part based on the relative position and the boundaries of the sample on the slide surface determined from the macro image. The micro images can be acquired by translating the slide along the y-axis relative to the second camera when the second camera acquires the micro images at corresponding y-axis positions, each micro image including the entire x-axis width of the sample based on the determined sample boundaries on the slide surface.

[0026] The method can further include identifying objects of interest (e.g., individual cells or tissue structures) in the sample and storing images of the identified objects of interest and the whole sample image.

[0027] The method can further include using a computer-controlled inspection station including a display and a user interface to inspect the whole sample image and images of individual objects therein.

[0028] Other and further features and advantages of the disclosed embodiments of the automated imaging and inspection system are depicted in the drawings and described in detail below. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and other aspects of the embodiments will be described in further detail with reference to the accompanying drawings, in which like reference numerals refer to like elements and the description of like elements will apply to all relevant embodiments described throughout.

[0030] Figure 1 is a block diagram of an automatic digital imaging system according to one embodiment;

[0031] Figure 2 is according to one embodiment Figure 1 of an automatic imaging system, which shows a single imager and inspection station;

[0032] Figure 3 shows a sample slide for an automatic digital imaging system according to one embodiment Figure 1 ;

[0033] Figure 4 is according to one embodiment Figure 2 of a front perspective view of a digital imager of a digital imaging system;

[0034] Figure 5 is according to one embodiment Figure 4 of a side perspective view of a slide carrier of a digital imager;

[0035] Figure 6 is according to one embodiment Figure 5 of a side perspective view of a slide carrier, in which a slide holder filled with slides is mounted in the slide carrier;

[0036] Figure 7 is according to one embodiment Figure 4 of an enlarged front perspective view of a slide carrier compartment of a digital imager;

[0037] Figure 8 is according to one embodiment Figure 5 and 6 of an end view of a slide carrier;

[0038] Figure 9 is according to one embodiment Figure 4 of a rear perspective view of a digital imager;

[0039] Figure 10 is according to one embodiment Figure 4 of a front perspective view of an imager, in which the enclosure panel is removed to show the internal components;

[0040] Figure 11 is according to one embodiment Figure 4 of a schematic diagram of an imager computer of an imager;

[0041] Figure 12 is a schematic diagram of a housing / enclosure assembly of a digital imager according to one embodiment; Figure 4 of a digital imager;

[0042] Figure 13 is a schematic diagram of a slide carrier loading deck of a digital imager according to one embodiment; Figure 4 of a digital imager;

[0043] Figure 14 is an enlarged front view of a slide handler of a digital imager according to one embodiment; Figure 4 of a digital imager;

[0044] Figure 15 is a side view of a slide handler according to one embodiment; Figure 14 of a slide handler;

[0045] Figure 16 is a front perspective view of a slide handler according to one embodiment; Figure 14 of a slide handler;

[0046] Figure 17 is a schematic diagram of a slide handling bracket of a slide handler according to one embodiment; Figure 14 of a slide handler;

[0047] Figure 18 is a schematic diagram of a bracket robot motion controller of a slide handler according to one embodiment; Figure 14 of a slide handler;

[0048] Figure 19 is a front perspective view of a digital imager according to one embodiment, in which the panel, slide handler, and slide carrier deck are removed to show the imaging station; Figure 4 of a digital imager;

[0049] Figure 20 is a front view of a part of a digital imager (and a front cross-sectional view of a shock absorber) according to one embodiment, in which the panel, slide handler, and slide carrier deck are removed to show the imaging station; Figure 4 of a digital imager;

[0050] Figure 21 is a front view of an image acquisition subassembly of an imaging station of a digital imager according to one embodiment; Figure 4 of a digital imager;

[0051] Figure 22 is a front cross-sectional view of an image acquisition subassembly according to one embodiment; Figure 21 of an image acquisition subassembly;

[0052] Figure 23 is a schematic diagram of a digital imager according to one embodiment; Figure 4An enlarged top view of the slide imaging station of a digital imager;

[0053] Figure 24 is according to one embodiment Figure 4 An enlarged perspective view of the slide imaging station of a digital imager;

[0054] Figure 25A Is an explanatory diagram showing the imaging plane of a previous imaging system, where the imaging plane is a nominally parallel plane of a slide (and the sample on the slide);

[0055] Figure 25B Is an explanatory diagram showing Figure 4 The tilted imaging plane of a digital imager according to one embodiment;

[0056] Figure 25C Is according to one embodiment showing Figure 4 The tilted imaging plane of a digital imager for full-depth (Z-dimension) imaging of a sample on a slide and an explanatory diagram showing dimensions for determining an appropriate camera tilt angle;

[0057] Figure 26 Is according to one embodiment Figure 4 A schematic diagram of the imaging station of a digital imager;

[0058] Figure 27 Is according to one embodiment Figure 4 A front view of the illumination subsystem of a digital imager;

[0059] Figure 28 Is according to one embodiment Figure 27 A front cross-sectional view of the illumination subsystem;

[0060] Figure 29 Is according to one embodiment Figure 4 A schematic diagram of the system interface board of a digital imager;

[0061] Figure 30 Is according to one embodiment Figure 4 An enlarged perspective view of the macro field-of-view station of a digital imager;

[0062] Figure 31 Is according to one embodiment Figure 4 A front view of the macro field-of-view station of a digital imager;

[0063] Figure 32 Is according to one embodiment Figure 4 A side cross-sectional view of the macro field-of-view station of a digital imager; and

[0064] Figure 33 Is an explanation according to one embodiment by Figure 4Schematic diagram of the serpentine scanning mode of a digital imager for scanning a sample slide.

[0065] Figure 34 is according to one embodiment Figure 1 Block diagram of the workflow subsystem and digital imaging display system (inspection station) of an automatic digital imaging system;

[0066] Figure 35 is a diagram showing the workflow scenario for viewing digital images of a sample slide according to one embodiment;

[0067] Figure 36 is a partial view of the work order screen displayed by the digital imaging display system during the inspection process according to one embodiment;

[0068] Figure 37 is a partial view of the work order screen displayed by the selection box of the digital image display station for selecting inspection cases according to one embodiment;

[0069] Figure 38 is an overview of the screen displayed by the selection box of the digital image display station for selecting inspection cases according to one embodiment;

[0070] Figure 39 Depicts the ability of the digital image display station to switch between different work order screens according to one embodiment;

[0071] Figure 40 Shows the work order screen in a simple list format displayed by the digital imaging display system according to one embodiment;

[0072] Figure 41 Shows the work order screen in a tile / grid format displayed by the digital imaging display system according to one embodiment;

[0073] Figure 42 Shows the work order screen in a tile / grid format with sorted rows displayed by the digital imaging display system according to one embodiment;

[0074] Figure 43 Shows the work order screen with a detail panel displayed by the digital imaging display system according to one embodiment;

[0075] Figure 44 Shows the work order screen with a graphical status indicator displayed by the digital imaging display system according to one embodiment;

[0076] Figure 45 Shows the inspection screen of a GYN ("gynecology") case (single slide case) displayed by the digital imaging display system according to one embodiment;

[0077] Figure 46 Displays a viewing screen of an NGYN (“non - gynecological”) case (multi - slide case) displayed by a digital imaging display system according to an embodiment;

[0078] Figure 47 Displays another viewing screen of an NGYN case (multi - slide case) displayed by a digital imaging display system according to an embodiment;

[0079] Figure 48 Displays a completion screen of a normal GYN case (single - slide case) displayed by a digital imaging display system according to an embodiment;

[0080] Figure 49 Displays a completion screen of an abnormal GYN case (single - slide case) displayed by a digital imaging display system according to an embodiment;

[0081] Figure 50 Displays a work order screen after a case is completed, as displayed by a digital imaging display system according to an embodiment;

[0082] Figure 51 Displays a work order screen after a case is completed, as displayed by a digital imaging display system according to an embodiment, where the completed case is hidden from the work order;

[0083] Figure 52 Is a diagram showing another workflow scenario for viewing digital images of sample slides according to an embodiment;

[0084] Figure 53 Is a diagram showing yet another workflow scenario for viewing digital images of sample slides according to an embodiment;

[0085] Figure 54 Is a diagram showing yet another workflow scenario for viewing digital images of sample slides according to an embodiment;

[0086] Figure 55 Provides, according to an embodiment, Figure 35 and 52 an overview of the roles and scenarios of the workflow depicted in 54;

[0087] Figures 56A to 56C Is a description of Figure 35 and 52 the various supervisor / manager roles and scenarios of the workflow depicted in 54; and

[0088] Figure 57 Displays another viewing screen showing a digital image of the entire sample area, as displayed by a digital imaging display system according to an embodiment.

[0089] Figure 58 Displays another viewing screen configured according to an embodiment to allow a viewer to add annotations and / or mark / draw on an image;

[0090] Figure 59 Displays another viewing screen configured according to an embodiment to allow a viewer to add recorded audio annotations to a digital image case;

[0091] Figure 60 Displays another viewing screen configured according to an embodiment to allow a first viewer to share the first viewer's screen with a second viewer on a different display system 108 and allow the first viewer to give the second viewer control of the viewing screen 434 on the first viewer's screen;

[0092] Figure 61 Displays another viewing screen configured according to an embodiment to allow a viewer to add bookmarks to a digital image case;

[0093] Figure 62 Illustrates a graphical representation of an image analysis algorithm for digital cytology according to an embodiment;

[0094] Figure 63 Illustrates a block diagram of an exemplary workflow server with backup and archival capabilities according to an embodiment;

[0095] Figure 64 Illustrates a flowchart of a method for archiving a slide using an archival service module according to an embodiment;

[0096] Figure 65 Illustrates an example of a calibration table for performing distortion correction of a digital image according to an embodiment. Detailed Description

[0097] Figure 1 and 2 Illustrates for making a substrate 102 disposed (see Figure 3)An automated digital imaging system 100 for digital imaging of multiple samples on a substrate 102 of a sample. For example, the substrate 102 can be a microscope slide (as depicted in the described embodiments) or any other suitable substrate 102, such as a microplate, a microarray, or other suitable media. The described embodiments of the digital imaging system 100 utilize a microscope slide. Therefore, the substrate 102 will be referred to as a slide, and it should be understood that the digital imaging system 100 is not limited to using a slide but can utilize any suitable substrate 102. Thus, the term "slide" as used herein shall mean any suitable substrate to which a sample is attached, which includes a microplate, a microarray, or other suitable media. As used herein, the term "microscope slide" shall mean a thin, flat piece of glass or plastic used to hold an object for examination under a microscope. The sample can include any type of sample for digital imaging under microscope magnification, such as biological or chemical samples (which include both cell samples and pathological tissue samples), and so on. The case where the sample is a cell sample containing individual cells dispersed within a spatial volume is used to describe the imaging system 100. However, it should be understood that the system can be used for different types of samples, such as pathological tissue samples, in the same manner as it is used for cell samples.

[0098] The automated digital imaging system 100 is designed to acquire high-resolution digital images of an entire sample (or even multiple samples) on the slide 102 and to allow the use of the digital images of the sample instead of the physical slide 102 to view the sample. An optical microscope is not used in the normal viewing workflow of the digital imaging system 100. Instead, digital images of the entire sample on the slide 102 are acquired and made available for viewing on a computer monitor 109 at a viewing station 108 (see Figure 2 ). The digital imaging system 100 may also include an image processing software application 137 (which is installed on the imager computer 105 of the imager 104 and / or the workflow subsystem 106), which embodies image processing algorithms configured to analyze digital images of slide samples and identify objects of interest (OOIs) (i.e., individual cells in the case of a cell sample or individual tissue structures in the case of a pathological solid tissue sample). The identified OOIs can then be used to facilitate a user (such as a cytotechnologist or a cytopathologist) in quickly and accurately viewing the digital images of the slide samples, as described in U.S. Patent Nos. 7,587,078, 6,665,060, 7,006,674, and 7,590,492, cited above. The term "OOI" means an object that has been determined to have characteristics indicating that the object (optionally an individual cell or tissue structure) can be used by a cytotechnologist or a cytopathologist to view for diagnosing or treating a patient, and does not necessarily mean that this object has been determined to be of actual concern or actual importance to the cytotechnologist or the cytopathologist.

[0099] The automated digital imaging system 100 includes one or more imager subsystems 103 (each including an imager 104 and an imager computer 105) and one or more viewing stations 108 operatively connected to a workflow subsystem 106 (also referred to as the "workflow server 106"). Generally, each imager 104 is a desktop system for capturing digital images of slides 102. The imager 104 processes batches of slides in an automated manner. The workflow server 106 is a computer server that primarily acts as a large disk storage system for storing and managing digital images and associated slide data. The workflow server 106 may also include software for image processing and data management and may also provide networking capabilities. The viewing station 108 is a workstation that includes a computer and a monitor for accessing digital images from the workflow server and displaying the digital images for diagnostic review of the images of interest (OOI and / or whole sample images).

[0100] As Figure 1 depicted, the imager subsystem 103 communicates data with the workflow subsystem 106 via a communication network, which may include one or more of a local area network (LAN, such as an Ethernet network), a wide area network (WAN), the Internet (such as a virtual private network (VPN)), or other suitable networks. Similarly, the viewing station 108 communicates data with the workflow subsystem 106 via a communication network, which may include one or more of a local area network (LAN, such as an Ethernet network), a wide area network (WAN), the Internet (such as a virtual private network (VPN)), or other suitable networks.

[0101] In the described embodiment, the imager 104 is designed to operate with microscope slides 102. Refer to Figure 3, showing an exemplary embodiment of a microscope slide 102. The microscope slide 102 is a rectangular glass plate 110 (or other suitable material) having a slide identification region 112, a sample region 114, and fiducial marks 116. The microscope slide 102 can be a standard-sized microscope slide, which is approximately 75 mm × approximately 25 mm or other suitable size. The microscope slide 102 can have beveled edges to facilitate handling and positioning of the slide 102. The sample region 114 can be circular with a diameter of up to approximately 22 mm. It is also possible to image the entire sample region on the microscope slide 102. The slide identification region 112 can be up to approximately 25 mm to approximately 28 mm in length. The slide identification portion 112 can be printed with a barcode, an ID number, and / or other information. The sample region 114 is left as a transparent region of the slide. The fiducial marks 116 can be used by the imager 104 as reference points on the slide 102 to determine the position and / or orientation of the slide 102 and its features relative to the imager 104. A sample 119 containing a plurality of objects distributed within a three-dimensional volume is attached to the slide 102, typically within the sample region 114, but in some cases, the sample can extend outside the sample region 114. The three-dimensional volume has a length (l), a width (w), and a thickness or depth (d). The thickness (d) defines the z-axis relative to the surface of the slide 102. The three-dimensional volume can be substantially uniform in shape (which includes thickness, length, and / or width), or it can be non-uniform. The volume of the sample 119 is uniform, where one or more dimensions of the sample vary by less than 10% or less than 15% or less than 20%. The sample 119 can be any suitable sample, such as a cell sample where the objects are cells, a solid tissue sample where the objects are tissue structures, and so on.

[0102] As Figure 3 depicted, a coverslip 115 can be used to cover the sample 119 in the sample region 114. The sample coverslip 115 is sufficiently transparent to obtain a microimage of the sample through the coverslip 115. In other words, the coverslip 115 does not prevent the imager 104 from using the imaging station 190 and / or the macrofield station 232 to obtain a microimage and / or a macroimage through the coverslip 115. The coverslip 115 is used to hold the sample 119 and protect it from contamination and from contaminating other objects, and also to keep the sample 119 flat and in place. The coverslip 115 has a thickness 117.

[0103] Referring Figure 4 to 5 and Figure 4As shown, imager 104 has a capacity for ten (10) slide carriers 118, but can be configured to hold a suitable number of slide carriers 118. Each slide carrier 118 has two rack holders 202, each of which receives a slide rack 120. Each slide rack 120 can hold a plurality of slides 102, such as twenty (20) slides 102 or forty (40) slides 102 or other suitable numbers. The slide rack 120 can be a standard off-the-shelf rack, such as rack or rack. Thus, each slide carrier 118 holds 40 slides, and the total capacity of imager 104 is 400 slides (10 slide carriers 118 × 40 slides / carrier 118). The slide carrier 118 has a handle 122 for carrying the slide carrier 118 and inserting and removing the slide carrier 118 from imager 104. As Figure 7 and 8 shown, the slide carrier 118 has a T-shaped slot 124 that slidably receives on a T-shaped rail 126 mounted on a slide carrier laminate 128. The slide carrier 118 is installed in imager 104 by aligning the T-shaped slot 124 with the T-shaped rail 126 and sliding the slide carrier 118 onto the slide carrier laminate 128. The slide carrier 118 is removed from imager 104 by pulling the slide carrier 118 straight out of the slide carrier laminate 128.

[0104] Now referring to Figure 2 and 11 , the image subsystem 103 includes an imager 104 and an imager computer 105. The imager 104 and the imager computer 105 communicate data with each other via a network connection, such as an Ethernet connection (e.g., a 10GE optical connection). The imager computer 105 can be any suitable computer, such as a personal computer having one or more computer processors (CPUs) and a graphics processing unit (GPU). For example, Figure 11 shows a computer suitable for the imager computer 105 schematically. The imager computer 105 can have one or more of the following specifications:

[0105] a. Motherboard 130 - dual-processor server class. It should be understood that a dual-processor configuration is suitable for the software desired to run on the imager subsystem 103;

[0106] b. CPU - class or better;

[0107] c. At least 64 Gb of DRAM;

[0108] d. A dedicated GPU 132, such as dual RTX-5000 or better. Ensure sufficient processing speed for image capture and to generate a Slide Data Set (SDS 304, see Figure 34 ) for each slide imaged by imager subsystem 103;

[0109] e. A local hard drive 133 for storing software program 135 which includes image processing software application 137 (described in more detail below);

[0110] f. A local mass data storage device 134 such as a hard drive, solid state drive or the like;

[0111] g. Network and USB input / output (I / O) 136;

[0112] h. A power supply 138;

[0113] i. An uninterruptible power supply (UPS) 140.

[0114] Input / output 136 includes one or more of the following:

[0115] a. Provide imager VGA video, touch screen USB, and computer audio to imager 104. Imager 104 receives VGA video from imager computer 105 and drives imager display monitor 168. A dedicated imager computer USB port provides I / O to the touch screen of imager display monitor 168. Imager computer 105 provides audio to imager 104, where the audio is amplified. Imager speakers are connected to an audio amplifier.

[0116] b. Provide USB ports for manufacturing and service support of keyboards and mice. During normal operation, keyboards and mice are not connected and are not required for normal system operation of the imager. Additional USB ports are provided on the front of the imager for easy user access.

[0117] c. Obtain imager computer power from the main power supply or an external intelligent UPS 140 (if attached) through the imager IEC inlet connector.

[0118] d. Provide a dedicated USB port to communicate with the intelligent UPS. In the event of a main power failure, the UPS will send an event to imager computer 105, and imager computer 105 will initiate controlled power-off so as to maintain computer software, data, and configuration integrity.

[0119] As described above, imager computer 105 communicates data with workflow subsystem 106 via a communication network such as a 1Gb Ethernet wired network connection.

[0120] Go to Figure 4 、 9And 10, the imager 104 is a desktop digital imaging device and serves as the input device of the entire digital imaging system 100. The imager 104 automatically captures digital images of a batch of slides 102 in a slide carrier 118 loaded into the imager 104 in an automatic manner with little or no user intervention. The imager 104 includes a housing assembly 142, a base assembly 148, and an enclosure assembly that provides a frame on which the imager components are mounted and houses the imager components. The housing / enclosure assembly 142 can be configured as a desktop instrument having dimensions equal to or less than 36 inches wide × 28 inches deep × 25 inches high. If no slides 102 or slide carriers 118 are loaded, the imager 104 can have a weight equal to or less than 200 pounds. These are only preferred dimensions and weights and do not necessarily limit the imager 104, unless specified in the claims. Thus, the imager 104 can have other maximum dimensions and weights that are suitable for the intended use.

[0121] The imager 104 includes a housing assembly 142 and an enclosure assembly 146 mounted to the base assembly 148. The imaging station 190 includes a base 144 and a vibration-isolated imaging platen 152 (see Figure 19 ) on which the imager components are mounted. The imaging station 190 is mounted to the base assembly 148. The enclosure assembly 146 includes an enclosure left panel 154, an enclosure right panel 156, an enclosure rear panel 158, an enclosure top panel 159, an enclosure left front window 160, an enclosure right front window 162, an enclosure right panel 164, and a slide carrier shelf door 166. Each enclosure component is mounted to the housing assembly 142 and / or another enclosure component. The enclosure left front window 160 is transparent or translucent such that the internal imager components can be seen through the enclosure left front window 160. The enclosure left front window 160 is pivotally mounted such that it can be opened and closed to access the internal imager components. The enclosure right front window 162 is transparent or translucent such that other internal imager components can be seen through the enclosure right front window 162. The enclosure left panel 154 can have a left window 170 such that the internal imager components can be seen through the left window 170. The enclosure top panel 159 can also have a top window 172 such that the internal imager components can be seen through the top window 172. The enclosure display panel 168 is pivotally mounted such that it can be opened and closed to access the internal imager components. The slide carrier shelf door 166 is pivotally mounted such that it can be opened and closed to access the slide carrier shelf 128 to load and remove the slide carrier 118 from the slide carrier shelf 128 (see Figure 4 and 10 ).

[0122] The enclosure assembly 146 also provides input and output, user access, and environmental control for the imager 104. Figure 12is a block diagram of the enclosure assembly 146. The enclosure assembly 146 provides air management to ensure that sufficient air flow exists within the imager 104 to maintain the internal operating temperature below a maximum value (e.g., less than 40 °C) and to maintain a positive internal pressure (when the slide carrier deck door 166 is closed). The enclosure assembly 146 provides external signal and power connections.

[0123] The imager display monitor 168 can be an LCD touch screen display or other suitable display. The imager display monitor 168 presents the operation and status of the imager 104 and also provides operator control via the touch screen (or other input device). As Figure 12 shown, the imager display monitor 168 is connected to the imager computer 105 via a bulkhead connector of the enclosure assembly 146. The imager computer 105 provides VGA video, touch screen USB, and computer audio to the imager 104. The imager 104 receives VGA video from the imager computer 105 and drives the imager display monitor 168. A dedicated imager computer USB port provides I / O to the touch screen of the imager display monitor 168. The imager computer 105 also provides audio to the imager 104, where the audio is amplified. The imager speaker is connected to the audio amplifier.

[0124] Still referring to Figure 12 , the enclosure assembly 146 also has a window sensor for detecting the open / closed state of the left front window 160 of the enclosure and a window lock for locking the left front window 160 in the closed position during the imaging operation of the imager 104. The enclosure assembly 146 also has a door sensor for detecting the open / closed state of the slide carrier deck door 166 and a door lock for locking the slide carrier deck door 166 in the closed position during the imaging operation of the imager 104.

[0125] Turning now to the internal components of the imager 104, the slide carrier deck 128 is positioned near the bottom of the imager on the substrate 148. The slide carrier deck 128 has a plurality of slide carrier compartments (e.g., ten (10) slide carrier compartments or twenty (20) slide carrier compartments or other suitable number of slide carrier compartments), where each slide carrier compartment is configured to receive a corresponding slide carrier 118. The slide carrier deck 128 is open at the top for the slide handler 176 to pick up the slide 102 contained in the slide carrier 118 loaded in the slide carrier compartments of the slide carrier deck 128. The slide handler 176 removes the slide 102 from the slide carrier 118, moves the slide 102 to various stations of the imager 104, and inserts the slide 102 back into the slide carrier 118.

[0126] Referring to Figure 13, the slide carrier deck 128 has a carrier lock 173 and a carrier presence sensor 175 for each slide carrier compartment 174. The carrier lock 173 can be a solenoid-actuated lock or other suitable electrically actuated locking mechanism, and the carrier presence sensor can be an optical sensor or other sensor suitable for detecting the presence of the slide carrier 118 in the corresponding slide carrier compartment 174. The carrier lock 173 and the carrier presence sensor 175 communicate data with the imager computer 105 via the system interface board 182 (see Figure 29 , the SIB described below).

[0127] See Figure 10 and 14 through 16, the slide handler 176 automatically moves the slides 102 between various stations of the imager 104, including back and forth between the slide carrier 118, the imaging station 190, and the macro field-of-view station 232. The slide handler 176 includes a support platform 180 and a slide handling bracket 178 movably coupled to the support platform 180. The slide handling bracket 178 is a pick-and-place robot that can move the slide gripper 184 in three degrees of motion, namely, lateral motion (X-axis) and vertical motion (Z-axis) (as Figure 14 shown) and forward / backward motion (Y-axis) (as Figure 15 shown)). The slide handling bracket 178 can also rotate the slide gripper 184 about the θ-axis (see Figure 15 ), and close and open the slide gripper 184 to grasp and release the slide 102. The slide handling bracket 178 has a motion mechanism for providing the movement and / or actuation of the slide handling bracket, which includes an X-axis mechanism 254, a Y-axis mechanism 264, a Z-axis mechanism 274, a θ-axis mechanism 290, and a gripper mechanism 185. The X-axis mechanism 254 includes a motor 256, an encoder 258, a home sensor 260, and a drive mechanism 262. The Y-axis mechanism 264 includes a motor 266, an encoder 268, a home sensor 270, and a drive mechanism 272. The Z-axis mechanism 274 includes a motor 276, an encoder 278, a home sensor 280, and a drive mechanism 282. The θ-axis mechanism 290 includes a θ-motor 292, a θ-encoder 294, a θ-home sensor (not shown), and a θ-drive mechanism (not shown). Each motion mechanism includes a motor, an encoder, a home sensor, and a drive mechanism, and is capable of meeting the motion envelope travel, transfer acceleration, speed, and payload requirements. The gripper 184 also has an optical sensor 188 (slide inventory sensor 188) for detecting the presence or absence (inventory) of the slide 102 in the slide carrier 118 and the presence or absence of the slide rack 120 in the slide carrier 118. The gripper 184 can be an existing component designed to grasp the slide 102 from the slide edge, such as by SMC of Los Angeles, CaliforniaTM Electric Gripper TM Manufacture

[0128] As Figure 17 Depicted in, the slide handling support 178 is operatively connected to a gantry robot motion controller 186 that controls the motion of each motion mechanism. The gantry robot motion controller 186 communicates data with the imager computer 105 via a Controller Area Network (CAN), and the CAN sends data and commands to the gantry robot motion controller 186 to control the operation of the slide handling support 178. Figure 18 Schematic diagram showing the gantry robot motion controller 186.

[0129] Now refer to Figure 10 And 19 Through 26, the imager 104 has an imaging station 190 that uses a digital camera 192, a tube lens assembly 194, a mirror 196, and an objective lens 198 to capture digital micro-images of the slide 102 (i.e., the sample 119 on each slide 102). The camera 192 uses a Sony CMOS Pregius second-generation sensor and has a resolution of 4096×2160 pixels. The optical path produces an optical resolution of 0.255 μm / pixel. The field of view (“FOV”) of the camera is much smaller than the entire sample area 114 of the slide 102. Therefore, the imager 104 must take many micro-images to capture the entire sample area 114. For example, the field of view of the camera 192 can be 1044 μm×550 μm or approximately 1 mm×approximately 0.5 mm, while the sample area 114 of the slide 102 can have a diameter of 21 mm to 22 mm. Based on these specifications, the camera 192 requires approximately 885 non-overlapping images to image the entire sample area 114. However, to obtain focused images of all objects within the three-dimensional volume of the sample 119, a step size equal to the image sub-region (which is equal to the depth of field of the imaging station 190) is used, which results in overlapping images and thus captures more frames. For example, as will be explained below, micro-images can be taken every 38.25 μm = approximately 1 / 14 of the frame height. Therefore, the actual number of images captured during the scanning of the entire sample area is approximately 11,000 or greater than 10,000 or greater than 5,000 or between 5000 and 15000. The camera 192 acquires micro-images while the XY slide stage 210 moves the sample 119 at a constant speed. In this example, the camera 192 takes an image every time the stage travels approximately 38 μm in the scanning direction. Therefore, each camera field of view contains 14 images per FOV. The XY slide stage 210 provides relative motion between the field of view of the camera 192 and the slide 102, such that the camera 192 can scan the entire sample area 114 of the slide 102 and take a plurality of discrete digital images that are combined to cover the entire sample area 114.

[0130] As described herein, the image processing software application 137 (see Figure 11 ) can be used to combine discrete micro-images to produce a composite image of the entire sample area 114.

[0131] As Figures 19 to 22 shown in

[0132]

[0133]

[0134] Figure 21 22 Figures 19 to 22 , the imaging station 190 includes a high-resolution digital camera 192, which is part of the image acquisition assembly 191 (which is a microscope camera). The image acquisition assembly 191 can be angled in the range from 0° to 3° or from 0° to 10° or from 0° to 5°. The camera 192 network data communicates with the imager computer 105 via a high-speed network (such as a 10GE Ethernet optical network or other suitable high-speed network capable of transferring high-resolution micro-images from the camera 192 to the imager computer 105 with sufficient throughput during the image scanning procedure).

[0132] The tube lens 194 is disposed in the optical path of the camera 192. The combination of the tube lens 194 and the objective lens 198 sets the optical magnification of the camera 192. The first folding mirror 196 is disposed in the optical path of the camera 192 after the tube lens 194 and is angled at approximately 45° with respect to the camera optical imaging axis. The angle and position of the first folding mirror 196 can change the collinearity of the objective optical axis and the center of the camera 192. The first folding mirror 196 is located in the imaging path and must have sufficient quality (e.g., reflectivity greater than 90% and precision flatness greater than λ / 4), otherwise it can directly affect the image quality by blurring the light at the detector in all or part of the field of view and / or causing distortion.

[0133] Next, the objective lens 198 is located in the optical path after the first folding mirror 196. Due to the first folding mirror 196, the optical axis of the objective lens 198 is angled at approximately 90° with respect to the camera optical axis. The objective lens 198 is mounted to the imaging Z-stage 200. The Z-stage 200 has threads for mounting the objective lens 198. The Z-stage 200 controllably moves the objective lens 198 to adjust the focus of the camera 192 on the slide 102 (i.e., adjusts the focal plane with respect to the slide 102). The Z-stage position can be set by the Z-stage controller. The slide 102 (i.e., the sample 119 on the slide 102) is placed at the front focal plane of the objective lens 198, and the objective lens 198 projects an image of the slide 102 onto the camera 192 via the first folding mirror 196 and the tube lens 194.

[0134] As Figure 21 and 22As shown, all components of the image acquisition subassembly 191, including the digital camera 192, the barrel lens 194, the first folding mirror 196, the objective lens 198, and the Z stage 200, are mounted to the image acquisition base plate 201. The image acquisition base plate 201 has a plurality of imaging subassembly alignment adjusters 203 that allow for positional and / or angular adjustment of the image acquisition subassembly 191 relative to other components of the imaging station 190 and relative positional and / or angular adjustment of the components of the image acquisition subassembly 191.

[0135] Reference Figures 19 to 20 And 23 to 24, the slide 102 is located in the optical path of the camera 192 behind the objective lens 198. The slide 102 is held on the slide holder 202. The slide holder 202 has a slide groove 204 for holding the slide 102 in a horizontal position on the slide holder 202. The slide holder 202 has a first slide alignment arm 206 that is spring-loaded to bias the slide 102 in the Y direction within the slide groove 204. The slide holder 202 has a second slide alignment arm 208 that is spring-loaded to bias the slide 102 in the X direction within the slide groove 204. The first slide alignment arm 206 and the second slide alignment arm 208 hold the slide 102 firmly in place on the slide holder 202 such that the slide 102 does not move during the scanning operation. The imaging station 190 has a slide release carriage 209 that is mounted to the X-Y stage platform 214 fixed relative to the slide holder 202. The slide holder 202 moves relative to the slide release carriage 209 to actuate the first slide alignment arm 206 and the second slide alignment arm 208 to release or engage the first slide alignment arm 206 and the second slide alignment arm 208 to hold or release the slide 102 in the slide holder 202.

[0136] As Figure 20 Best shown in, the optical axis of the objective lens 198 is along the scanning direction (see Figure 33)It is inclined at an inclination angle 205 with respect to the orthogonal line of the plane of the slide 102 (and the slide groove 204). In other words, the resulting optical axes of the camera 192 and the optical device at the surface of the slide (also simply referred to as "the optical axis of the camera 192") are not orthogonal to the plane of the slide 102. As described in more detail herein, the inclination angle 205 allows the imager 104 to obtain a volumetric image of the sample 119 on the slide 102 (i.e., an image extending into the depth of the sample 119). In other words, the micro-images contain focused images of features at different depths of the sample 119 on the slide 102, rather than just a single focal plane as in the case of an image taken at the orthogonal angle of the sample 119. The imaging station 190 can be configured to acquire micro-images in which each micro-image contains at least a portion of the depth of the slide 102 below the surface of the slide 102. If a coverslip 115 is used on the slide 102, the imaging station 190 can be configured to acquire micro-images in which each micro-image contains at least a portion of the depth of the coverslip 115.

[0137] Next, as described herein, these micro-images can be processed by an image processing software application 137 (see Figure 11 ) to stitch the micro-images together and flatten the image into a two-dimensional composite image in which all images of the sample 119 are in focus at all depths of the sample 119. For example, U.S. Patent Publication No. 2009 / 0295963 describes a method of stitching images together to form a single digital image 302 (e.g., see Figure 34 , which shows a block diagram of the digital image 302). As depicted in FIG. 25, the inclination angle 205 suitable for imaging the full depth (Z dimension) of the sample 119 on the slide 102 varies according to the thickness (d) of the sample and the projection field of view (d1) of the camera 192 at the slide 102. The projection field of view (d1) is the in-plane field of view of the camera sensor divided by the combined optical magnification of the camera 192 (e.g., including the magnification of the barrel lens 194, the objective lens 198, and any other magnification in the optical path). As shown in Figure 25C , the inclination angle 205 is arcsin(d / d1). For example, for a sample depth (d) of 10 microns, a camera sensor with an in-plane field of view of 8 mm, and a combined optical magnification of 40x (i.e., d1 = 8 mm / 40 = 0.20 mm), the inclination angle 205 is arcsin[0.01 mm / (8 mm / 40)], which is approximately 2.86°. In Figure 25C Another example depicted, for a sample depth (d) of 24 microns and an effective in-plane field of view (d1) of 0.5 mm (e.g., a sensor field of view of 20 mm and an optical magnification of 40x), the inclination angle 205 is 2.75°. Generally, for typical sample depths, camera sensor sizes, and magnifications, the inclination angle 205 typically ranges from about 2° to about 10°.

[0138] The slide holder 202 has a slide holder base plate 212 mounted to the XY slide stage 210. The slide holder base plate 212 has a slide holder angle adjuster 213 to allow adjustment of the orientation of the slide holder base plate 212 relative to the XY slide stage 210 (and also relative to the optical axis of the objective lens 198).

[0139] The XY slide stage 210 moves the slide holder 202 and the slide 102 in a scanning pattern during the imaging operation to obtain an image covering the entire sample area 114 (or other area of interest) of the slide 102. As Figure 26 shown, the XY slide stage 210 includes a motor, an encoder, and motion limits along each of the X and Y axes. The XY slide stage 210 is connected to an X-Y stage driver 211, which in turn is connected to an X-Y stage controller 215 to control the movement of the XY slide stage 210. The X-Y stage controller 215 communicates data with the imager computer 105 via an Ethernet connection to allow the imager computer 105 to control the operation of the XY slide stage 210.

[0140] The XY slide stage 210 is mounted to an X-Y stage platform 214, which in turn is mounted to the imager base plate 152. The X-Y stage platform 214 has an X-Y stage angle adjuster 218 to allow adjustment of the orientation of the X-Y stage platform 214 relative to the imager base plate 152.

[0141] Thus, the slide holder angle adjuster 213 and the X-Y stage angle adjuster 218 can be adjusted to align the slide holder 202 at an appropriate level with the movement of the XY slide stage 210 or at a desired angle relative to the optical axis of the objective lens 198 (i.e., the optical axis of the camera 192). Typically, the movement of the slide holder 202 and the slide stage 210 will be aligned in parallel to provide a constant focus height for the micro-images during scanning, such that the image acquisition assembly 191 acquires micro-images of the sample 119 at the same height (e.g., a single z-axis focus height) relative to the surface of the slide 102.

[0142] The imager base plate 152 is isolated from the scanner housing 144 via a plurality (in this case, four) of vibration isolation mounts 221. The vibration isolation mounts 221 provide vibration isolation to the imaging station 190.

[0143] The objective lens 198 uses the Z stage 200 to focus relative to the slide 102 held in the slide holder 202, as Figures 19 to 22 shown in and 24. As Figure 26 shown, the Z stage 200 includes a piezoelectric focusing n-point controller 217 and a piezoelectric focusing n-point stage 219 for controlling the Z stage and in turn the focusing of the objective lens 198.

[0144] As Figure 23 and 24 shown in, the microscope calibration assembly 207 (also referred to as the "microscope calibration target") is disposed in the slide holder base plate 212 such that the microscope calibration assembly 207 moves with the movement of the XY slide stage 210. The microscope calibration assembly 207 can be integrated with the slide holder base plate 212 such that it is directly attached to the slide holder base plate 212 or defined by the slide holder base plate 212 and cannot be removed from the slide holder base plate 212. For example, the microscope calibration assembly 207 can be formed or etched into or through the slide holder base plate 212. Alternatively, the microscope calibration assembly 207 can be a component separate from the slide holder base plate 212, which is mounted on the slide holder base plate 212, for example, mounted in a cavity defined by the slide holder base plate 212. The microscope calibration assembly 207 can be configured to perform both position and optical calibration of the microscope camera assembly 191 (also referred to as the image acquisition sub-assembly 191). Alternatively, the microscope calibration assembly 207 can be configured to perform only the optical calibration of the microscope camera sub-assembly 191. Regarding position calibration, the calibration assembly 207 is configured to measure or determine position calibration parameters, which include one or more of the following: the "X" position of the XY slide stage 210, the "Y" position of the XY slide stage 210, the "Z" position of the XY slide stage 210, camera-to-stage alignment, changes in one or both of the "X" and "Y" positions of the XY slide stage 210 during micro-imaging of the slide 102, and / or inspection and changes in the "Z" position of the XY slide stage 210 during micro-imaging of the slide 102. Regarding optical calibration, the calibration assembly 207 is configured to measure or determine optical calibration parameters of the microscope camera sub-assembly 191, which include (for example) one or more gray-scale linearities, magnifications, signal-to-noise ratios, illumination changes, modulation transfer functions (MTFs), checking color pixels or elements of the inspection digital camera 192 (such as CMOS or charge-coupled device (CCD)), uniformity or evenness of illumination, detecting artifacts (such as dust and smudges) that degrade the signal-to-noise ratio. Then, the microscope camera sub-assembly 191 can be calibrated and adjusted as needed based on the position and / or optical calibration parameters measured or determined using the calibration assembly 207. Examples of suitable microscope calibration assemblies 207 are disclosed in U.S. Patent No. 7,848,019B2.

[0145] Refer to Figure 27 and 28, the imaging station 190 also includes an illumination module 222 that provides sufficient uniform light to the slide 102 for the objective lens 198 to collect so that the camera 192 can generate a quality image. The illumination module 222 includes a light source 224 (such as an LED 224), a first lens 226, a second lens 227, a third lens 229, a first aperture 223, a second aperture 225, a second folding mirror 228, a lens barrel lens mount 237, and an adjustable focusing lens 230 required to project the LED light onto the bottom of the slide 102. The illumination module 222 projects uniform light to allow for uniform illumination at the sample focal plane. The term "uniform light" means that the light is substantially uniform in color and brightness. The term "uniform illumination" means that the sample 119 is substantially illuminated with light that is substantially uniform in color and brightness. The illumination module 222 is powered by a main power supply connected through the SIB. Feedback from the camera is used to set the brightness of the illumination module 222 to achieve a target background brightness. The illumination module 222 interfaces with an illumination interface on the SIB 182 (described below). The illumination module 222 has a Y adjustment screw for adjusting the illumination in the Y direction in the XY plane relative to the image acquisition assembly 191 and an X adjustment screw 235 for adjusting the illumination in the X direction in the XY plane relative to the image acquisition assembly 191.

[0146] The illumination module 222 contains a light source 224 (such as an LED), the power, spectral distribution, and radiation profile of which are the starting points for all the illumination uniformity, quality, and quantity required to generate quality micro-images.

[0147] The scanning process for obtaining a micro-image of the slide 102 (which includes focused images of features at different depths of the sample 119 on the slide 102) using the scanning station 190 will now be described. As explained above and depicted in FIG. 25, the imaging station 190 uses the focal plane of the camera 192 that is tilted relative to the surface of the sample 119 on the slide 102 such that the focal plane extends through the depth of the slide 102 to capture the micro-image of the sample 119 on the slide 102. Thus, each micro-image captured by the camera 192 contains an image of the depth of the sample 119 on the slide 102 because it is taken at an angle relative to the surface of the sample 119 (i.e., the angle relative to the plane of the slide 102). As Figure 33As depicted, the XY slide stage 210 moves the slide 102 to scan the entire sample area 114 (or a predetermined area of the entire actual sample 119, e.g., if the actual sample 119 covers an area different from the sample area 114 and the boundaries of the actual sample 119 have been previously determined) with the field of view of the camera 192. The XY slide stage 210 moves the slide 102 along a back-and-forth serpentine path to capture micro-images of the scanned strip of the sample 119 at each pass. To capture the scanned strip, the XY slide stage 210 continuously moves the slide 102 and triggers the camera 192 to capture an image at each trigger point based on the stage encoder position. A serpentine path is used such that the start point of each consecutive scanned strip is close to the end point of the previous scanned strip to minimize the time required to scan the entire sample 119. As the slide 102 moves along the scanned strip, the camera 192 captures micro-images of the sample 119.

[0148] To maintain acceptable focus during the imaging scan, the best focal plane of the slide 102 can be estimated using the measurement of the printed fiducial marks 116 on the slide 102 and the offset defined by the measurement of the best focus offset of the sample 119 relative to the reference plane. Then, the coefficients defining the offset plane are uploaded to the X-Y stage controller 215, which maintains the position of the Z stage 200 during the scan based on the coefficients. During the imaging scan, the scan quality is monitored by various methods, which include focus quality metrics, stage tracking error, and image quality metrics. Image quality metrics can include one or more of the following: sharpness, noise, dynamic range, tone reproduction, contrast, color accuracy, distortion, vignetting, etc., which can be measured by digital image algorithms known to those skilled in the art.

[0149] As Figure 25A shown, in previous imaging systems, the imaging plane was nominally parallel (i.e., parallel within manufacturing tolerances) to the plane of the slide (and the sample 119 on the slide). A single best focal plane is captured for each adjacent field of view taken at intervals approximately the length of the field of view of the camera (the micro-images can have some overlap to facilitate stitching the images). Then, the micro-images are stitched together to produce a composite image of the entire sample 119. It should be noted that due to the different heights of objects above the slide within the three-dimensional volume of the sample 119, a single imaging process cannot be used to focus and capture all objects.

[0150] In contrast, as Figure 25BAs shown, the image capture method of imager 104 utilizes an imaging plane that is tilted with respect to sample 119 on slide 102. Because the imaging plane is tilted with respect to sample 119 on slide 102, different sub-regions of the camera pixel array capture images of the sample at different heights. Thus, object A is captured in focus by the camera at imaging position #1, but in the same camera frame, object B is near the right edge of the frame and completely out of focus. Instead of stepping the camera position by the length of the field of view in the scan direction between images, a step size of an image sub-region equal to the depth of field of the matching imaging optics is used. The image sequences overlap. When using the camera at position #1 to focus on object A, object B is captured in focus when the camera reaches position #4. In this way, all objects within a height equal to the difference between the left and right edges of the imager focal plane are captured in focus in a certain image.

[0151] Reference Figure 25C , in an example, the camera pixel array can be divided into 14 sub-regions representing different focal planes across a 24-μm depth of the sample. After the same sample region (tile) is captured in each camera sub-region, the sub-regions are vertically recombined to produce a cross-focus stack of the tiles. Then, the stack is converted into a single-plane merged focused image by selecting the pixels of the plane with the best focus quality by comparing each pixel with adjacent pixels in the same plane and using an algorithm (such as the algorithm taught in U.S. Patent No. 7,769,219) for determining the pixel focus with a focus metric. Examples of determining the best focused pixels and other objects are also described in U.S. Patent No. 7,736,304 incorporated above. Techniques for determining the best focus of slide fiducial marks are taught in U.S. Patent No. 8,116,550. The number of sub-regions is only an example, and digital imaging system 100 can use any suitable number of sub-regions for image processing.

[0152] The following will describe an exemplary method for processing an image by the GPU 132 to obtain a merged compressed image of the entire sample 119 on the slide 102. Image processing involves many steps of taking the original image and converting it into a final merged compressed image. The color camera 192 may utilize a Bayer mask (a color filter array for arranging RGB color filters on a grid of light sensors). In this case, the first step in the image processing algorithm is to demosaic and convert the image into an RGB image (a bitmapped image that holds the red / green / blue values for each pixel). Next, the uniformity and distortion of the image are corrected by a pre-mapping calibration table. In one example, the calibration table is an array of the size of the camera image. The table contains table values for each pixel position in the resulting corrected image. Each table value indicates a 2×2 pixel sub-region of the source image and the weighting factors applied to the source pixels and summed to obtain the resulting pixel value. This allows the corrected image to represent (by interpolation) the sub-pixel shifts required to correct the small optical distortions measured during the calibration process. Figure 65 An example of a pre-mapping calibration table depicting the case where the position distortion correction is 1 / 2 of the top left pixel.

[0153] After capturing all sub-region images of a tile (e.g., 14 sub-regions), a composite image is calculated. The difference between each pixel and a specific pixel in its neighboring region is calculated. Then, this difference is weighted based on the pixel values to determine the relative metric of the specific plane of the specific pixel to produce a tile of plane values (e.g., 0 to 13). Then, a moving average is applied to the tile of plane values to provide a better transition between objects in the final image. Then, pixels of the composite image are selected based on the associated plane values. Then, the composite images are tiled together to produce a scan strip. After assembling the complete scan strip, it is spliced to the previous scan strip. After stitching together the images of the entire sample 119, a JPEG, JPEG2000, or other suitable compression algorithm is used to compress the full image at a ratio of approximately 20:1. Then, a pyramid of the image is generated, and this is transmitted to the workflow subsystem 106 together with the slide metadata, where the pyramid and slide metadata can be accessed by the viewing station 108 for viewing. Slide metadata refers to non-image information associated with the entire sample image, such as a slide identifier (which can be a barcode read from the slide), imaging time and date, etc. Slide metadata can also contain a list of the positions of the OOIs 308 identified by the image analysis algorithm. An example of a "pyramid" is a set of lower-resolution images obtained from the original entire full-sample image by repeatedly reducing the size of the original entire full-sample image by 50% until it reaches a default minimum size. The purpose of this pyramid is to support more efficient rendering on the viewing screen. When the viewer views a reduced lower-resolution variant of the image, data can be supplied by accessing the appropriate level of the pyramid image data, which involves less data transfer than sending the full-resolution image and then reducing its size at the viewing station 108.

[0154] Reference Figures 30 to 32, the imager 104 also has a macro view station 232 for obtaining identification information of each slide 102 and for obtaining a whole slide (e.g., obtaining information outside the sample area 114) and / or an unenlarged macro image of the sample area 114. The macro view station 232 includes a first slide holder 234 at an imaging position in the imaging area of a macro camera 231 (e.g., a digital camera with a CCD or other imaging sensor) of the macro view station 232 and a second slide holder 236 at a queue position outside the imaging area of the macro view station 232. A lower illumination module 238 having a PCB and LEDs is positioned below the first slide holder 234 to illuminate the bottom of the slide 102 positioned in the first slide holder 234. A diffuser 244 may be positioned between the lower illumination module 238 and the first slide holder 234. An upper illumination module 240 having a PCB and LEDs is positioned below the first slide holder 234 to illuminate the top of the slide 102 positioned in the first slide holder 234. The macro view station 232 has a macro camera 231 and a second folding mirror 242 for reflecting the optical axis of the macro camera 231 onto the slide 102 in the first slide holder 234. The macro camera 231 is mounted using a camera adjuster 246 to allow adjustment of the orientation of the macro camera 231.

[0155] The macro view station 232 captures a macro image of the slide 102 (which includes barcode identification and / or other information on the slide 102) and an image of the whole slide 102 and / or the sample area 114 and transmits the macro image to the imager computer 105. The macro image includes one or more of fiducial markers 116 and the sample area 114 and / or the sample 119. The imager computer 105 is configured to determine the relative position and boundaries of the sample area 114 and / or the sample 119 on the surface of the slide 102 based at least in part on one or more fiducial markers 116. The relative position and boundaries of the sample area 114 and / or the sample 119 are provided to the imaging station 190. The imaging station 190 (e.g., the image acquisition subassembly 191) is configured to acquire micro images based at least in part on the relative position and boundaries of the sample area 114 and / or the sample 119 on the slide 102. In other words, the imaging station 190 uses the relative position and boundaries of the sample area 114 and / or the sample 119 to determine the scanning position of the camera 192 when acquiring micro images, so as not to waste time and storage on overscanning past the edges of the sample area 114 and / or the sample 119. The macro view station 232 also interfaces with the SIB 182, which in turn interfaces with the imager computer 105 to control the operation of the macro view station. The macro camera 231 communicates data with the imager computer 105 via a communication network (e.g., a 100baseT Ethernet network).

[0156] Figure 29Schematic diagram illustrating SIB 182. The SIB integrates and redistributes external signals between the imager computer 105 and the imager 104 to the appropriate components and subassemblies of the imager 104 and the appropriate inputs of the imager computer 105.

[0157] The complete operation of the image subsystem 103 to image the slide 102 and generate a single composite image of the entire sample 119 on each slide 102 will now be described. The operator manually loads each slide carrier 118 with a slide 102 to be scanned. If the slide 102 is provided in a slide rack 120, this only involves inserting the slide rack 120 into the slide carrier 118. If the slide 102 is not in a slide rack 120, the operator first loads the slide 102 into a slide rack 120 and then inserts the slide rack 120 into the slide carrier 118. The slide carrier 118 can be used as a repository for the slide 102 before and after imaging and for transporting and loading the slide 102 into the imager 104.

[0158] The slide 102 in the slide carrier 118 is manually inspected to verify that the slide 102 is properly oriented in the slide carrier 118, with the printed side of the slide identification portion 112 all facing away (towards the back of the slide carrier 118). When the imager 104 is in an idle or paused mode, the operator opens the slide carrier deck door 166 and manually inserts each slide carrier 118 loaded with up to 40 slides into the corresponding one of the slide carrier compartments 174. A carrier presence sensor 175 in each corresponding slide carrier compartment 174 detects whether a slide carrier 118 is installed and thus signals the imager computer 105. When the imager 104 removes the slide 102 from the active slide carrier 118 and images the slide 102, the carrier lock 173 of the slide carrier 118 currently being actively imaged by the imager 104 is actuated to lock the active slide carrier 118 in place. The other slide carriers 118 not being actively imaged by the imager 104 are not locked (are considered non-active).

[0159] When a new slide carrier 118 is inserted into the slide carrier compartment 174 and its carrier presence sensor 175 signals the presence of the slide carrier 118, the imager 104 schedules the inventory of this slide carrier 118. The imager 104 uses the optical slide inventory sensor 188 to perform an inventory of each slide carrier 118 mounted in the imager 104. The slide handling bracket 178 causes the sensor 188 to scan each slide carrier 118 and determine which possible slide slot in the slide rack 120 has a slide 102 inserted therein. This slide inventory information is provided to the imager computer 105 and is used when the corresponding slide carrier 118 becomes active and ready to be scanned. Thus, the slide handling bracket 178 only traverses to the slot in the slide carrier 118 in which there is a slide 102.

[0160] When the corresponding one of the slide carriers 118 is scheduled for imaging, the slide handling bracket 178 moves to the first slide 102 in the slide carrier 118, actuates the gripper 184 to grasp the slide 102, and moves the slide 102 to the macro view station 232 and places the first slide 102 on the first slide holder 234 in the imaging position of the macro view station 232. The macro view station 232 uses the macro camera 231 to take a plurality of macro images of the first slide 102, generates a full sample image (i.e., a macro image that includes a single image of the entire sample 119), and reads printed identification information (such as a bar code). This information becomes part of the slide data set (SDS) record for the first slide 102. As described herein, the macro image also contains one or more of the fiducial markers 116 and the sample area 114 and / or the sample 119. The imager computer 105 uses the macro image to determine the relative position and boundaries of the sample area 114 and / or the sample 119 on the surface of the slide 102 based at least in part on one or more fiducial markers 116. The relative position and boundaries of the sample area 114 and / or the sample 119 are provided to the imaging station 190.

[0161] When the imaging station 190 is ready to receive a new slide 102, the slide handling bracket 178 moves the first slide from the first slide holder 234 in the imaging position of the macro-view station 232 to the imaging station 190 and places the first slide 102 on the slide holder 202 of the imaging station 190. During loading of the first slide 102 onto the slide holder 202, the XY slide table 210 can be positioned such that the slide release carriage 209 actuates the first slide alignment arm 206 and the second slide alignment arm 208 to the release position to allow the first slide 102 to be placed on the slide holder 202. After the first slide 102 is placed on the slide holder 202, the XY slide table 210 moves the slide holder 202 such that the first slide alignment arm 206 and the second slide alignment arm 208 are biased to their respective engagement positions to thereby bias and hold the first slide 102 in the proper position in the slide holder 202 during scanning.

[0162] After moving the first slide 102 from the first slide holder 234 in the imaging position of the macro-view station 232 to the slide holder 202 of the imaging station 190, the slide handling bracket 178 can be used to move the next slide 102 (i.e., the second slide 102) in the slide carrier 118 to the first slide holder 234 in the imaging position of the macro-view station 232. While the imaging station 190 obtains a micro-image of the sample 119, the macro-view station 232 can perform the process of obtaining a macro-image as described for the first slide 102 in parallel (i.e., simultaneously), as will be described below. The second slide can be loaded onto the first slide holder 234 of the macro-view station 232 and macro-imaged while the first slide 102 is being imaged at the imaging station 190. After the macro-view station 232 completes the macro-imaging of the second slide 102, the second slide 102 typically must wait in the macro-view station 232 until the imaging station 190 completes the micro-imaging of the first slide 102 and the slide handling bracket 178 has moved the first slide 102 from the slide holder 202 to the second slide holder 236 in the queue position, as will be further described below.

[0163] Next, imaging station 190 scans the first slide 102 to capture multiple micro-images of the sample 119 on the first slide 102. As described above, while the digital camera 192 captures the multiple micro-images, the first slide 102 is continuously moved along a serpentine path by the XY slide stage 210 to move the first slide 102 under the objective lens 198. The imager 104 coordinates the movement of the XY slide stage 210 and the Z-axis movement (focus) to image the slide 102. When scanning the first slide 102, the image data from the camera 192 is streamed to the imager computer 105. The micro-images are recombined into segmented image data from the camera 192 and streamed to the imager computer 105. Imaging station 190 acquires the micro-images at least in part based on the relative positions and boundaries of the sample area 114 and / or the sample 119 on the slide 102, such that it minimizes the wasted time and storage caused by scanning and imaging outside the edges of the sample area 114 and / or the sample 119.

[0164] After imaging the first slide 102, the slide handling bracket 178 moves the first slide 102 to the second slide holder 236 in the queue position. The queue position is a queue point for completing the imaging operation. This function is used to optimize the throughput of the slide handling bracket 178 by placing the slide 102 that has completed the imaging operation close to the slide holder 202 of the imaging station 190 instead of putting the first slide 102 back into the slide carrier 118. The first slide 102 is held in the queue position while the slide handling bracket 178 moves the next slide 102 (e.g., the second slide 102) to be micro-imaged by the imaging station 190 from the first slide holder 234 in the imaging position of the macro-view station 232 to the slide holder 202 of the imaging station 190. If not used for the queue position, the first slide 102 needs to be moved back into the slide carrier 118 before the next slide 102 can be placed in the slide holder 202 of the imaging station 190, which will slow down the process because the slide carrier 118 is further away from the slide holder 202 than the second slide holder 236. After moving the next slide 102 to be micro-imaged by the imaging station 190 to the slide holder 202, the slide handling bracket 178 can be used to move the first slide 102 from the second slide holder 236 back into the slide carrier 118 (i.e., when the imaging station 190 images the next slide 102). After moving the second slide 102 onto the imaging station 190, the slide handling bracket 178 moves the third slide 102 from the active slide carrier 118 to the macro-view station 232 and places it on the first slide holder 234, and the macro-view station 232 images the third slide 102. This process is repeated for all the slides 102 in the active slide carrier 118. Thus, using the queue position of the second slide holder 202 increases the total throughput of the imager 104.

[0165] The imaging station computer 105 receives the camera micro-images from the imaging station 190 and reassembles them, which are reassembled into a focal plane image (as described earlier) and combined into a single best focal plane (as described earlier). This stitching / aligning with adjacent scan bands is all performed in the GPU. The resulting image is streamed from the GPU to the CPU and then compressed and stored on disk.

[0166] The imager computer 105 also uses an image analysis algorithm (described below) to scan the image data to identify objects of interest (OOIs) and generate OOI location data. The OOI location data identifies the locations of the OOIs in the image. This data also becomes part of the SDS 304. Each SDS 304 is processed to generate an optimized data set (“pyramid”) for image viewing at the review station 108. The pyramid data (a subset of the compressed SDS) is designed to allow faster slide data access for recording the OOIs. The pyramid data is also added to the SDS. After image processing and pyramid generation are complete, the compressed SDS is sent to the workflow subsystem 106 and moved to the active storage partition on the NAS disk array dedicated only to active data (see Figure 1 ).

[0167] The image analysis algorithm can use any suitable process to identify the OOIs. As an example, the image analysis algorithm, for instance, analyzes the digital image 302 by scanning the digital image 302 and performing primary and secondary segmentation. The primary and secondary segmentation measures, determines, and / or extracts various characteristics of each individual object and clustered objects in the digital image 302. Specific characteristics can be utilized because they are known to be associated with classified objects (such as classified as normal, abnormal, diseased, healthy, pre-cancerous, cancerous, etc.). For example, for a digital image 302 of a cell sample, the characteristics can include cell size, nucleus-to-cytoplasm area ratio, nuclear corrected integrated optical density, cytoplasmic vacuolization, darkness, etc., which can be used to classify the cells as pre-cancerous, cancerous, normal, and / or abnormal. The primary and secondary segmentation is described in U.S. Patent Publication No. 2004 / 0253616. Then, the algorithm calculates an object score for each object based on the measured values of the corresponding characteristics. Based on this score, the algorithm identifies or selects the objects and clustered objects that are considered OOIs 308 (e.g., see Figure 34 ).

[0168] The imager computer 105 can also analyze the determined characteristics of each OOI 308 and use an OOI matching algorithm to identify similar OOIs 308 on the same digital image 302 (i.e., the same sample). The OOI matching algorithm compares the corresponding determined characteristics of the OOIs 308 to identify similar OOIs 308 that have determined characteristics (e.g., one or more characteristics within a corresponding predetermined range of each other). The OOI matching algorithm can also be used to determine whether the OOIs 308 on the digital image 302 are similar to the OOIs 308 in a library of OOIs 308. The library of OOIs 308 can be stored in a database of a laboratory information system (LIS) 408 (see Figure 1 ) or in a database stored on another data storage device or system that is in network communication with the imager computer 105. The database of OOIs 308 contains a library of OOI images and / or digital images 304 of previously classified and characterized objects (e.g., cells). Each object in the library of OOI images and / or digital images 304 has previously determined characteristics that can be compared to the corresponding characteristics of the OOIs 308 on the digital image 302 analyzed by the imager computer 105. The objects in the library images can also be previously classified, e.g., classified as abnormal, normal, pre-cancerous, cancerous, malignant, premalignant, benign, etc.

[0169] Figure 62 A graphical representation of an image analysis algorithm 500 for digital cytology is depicted. In step 502, the digital image 302 is segmented into frames. In step 504, the frames are processed by frame processors running in parallel threads. The frame processors segment the frames (e.g.) by using a watershed method applied to the G channel of the RGB. Additionally, the frame processors calculate the features (characteristics) of the objects and execute a convolutional neural network (CNN) interface (e.g., using a modified MobileNet with a 192×192 window, color) that runs in parallel on the CPU and GPU (graphics processing unit) of the imager computer 105. In step 506, the slide is processed, and in step 508, the OOIs are selected and grouped based on the determined characteristics.

[0170] The automated digital imaging system 100 may include a machine learning diagnostic module configured to characterize and / or diagnose an object on a digital image 302 using machine learning. The machine learning diagnostic module may be installed and run on the imager computer 105, the review station computer 111, or other suitable computer systems. The machine learning diagnostic module performs a process of characterizing an object on the digital image 302 and may also diagnose a patient based on the characterization. As depicted in an exemplary process, the process includes generating a training set of characterized / diagnosed objects. The training set is generated by imaging a large number of sample slides 102 (e.g., hundreds, thousands, tens of thousands, etc. of slides 102) to obtain a digital image 304 of each sample slide 102. Next, the OOI is identified on each digital slide 102 as described herein. The OOIs are independently classified by type by a plurality of clinicians (e.g., cytotechnologists and / or pathologists for cell samples) (e.g., 3, 4, 5, or more clinicians). For example, an object may be classified as HSIL, LSIL, normal, abnormal, diseased, healthy, pre-cancerous, cancerous, etc. "HSIL" is an abbreviation for high-grade squamous intraepithelial lesion, and "LSIL" means low-grade squamous intraepithelial lesion. A lesion is an area of abnormal tissue, and high-grade versus low-grade refers to its likelihood of developing into cancer. Next, the machine learning diagnostic module determines a plurality of characteristics of each OOI (e.g.) by using the image analysis algorithms described herein or similar algorithms. For example, with respect to a digital image 304 of a cell sample, the characteristics may include cell size, nucleus to cytoplasm area ratio, nuclear corrected integrated optical density, cytoplasmic vacuolization, darkness, etc., which may be used to classify the cell as pre-cancerous, cancerous, normal, and / or abnormal.

[0171] Next, the machine learning diagnostic module uses a pattern recognition algorithm to determine the characterization / diagnosis relationship between the characteristics identified by the image analysis algorithm and the characterization / diagnosis determined by the clinician. The characterization / diagnosis relationship may be an algorithm, a function, and an equation, or other suitable relationships.

[0172] Next, the machine learning diagnostic module may use the characterization / diagnosis relationship to characterize / diagnose an object on a new sample slide 102. The imaging station 103 generates a digital image 302 of the slide 102. The OOI on the digital image 302 is identified using the image analysis algorithm. Next, the machine learning diagnostic module determines a plurality of characteristics of each OOI (e.g.) by using the image analysis algorithm. Finally, the image analysis algorithm uses the characterization / diagnosis relationship to characterize / diagnose each OOI. The characterization / diagnosis of each OOI is also added to the SDS 304.

[0173] The machine learning diagnostic module can continue to learn and optimize the characterization / diagnosis relationship by adding the digital image 304 from the new sample slide 102 to the training set and then performing a process using a pattern recognition algorithm to generate the characterization / diagnosis relationship.

[0174] The imager computer 105 can also include a slide statistics module for identifying slide statistics (such as the number of objects on the slide (e.g., the number of cells), the number of boundaries, etc.). The imager computer 105 can use a process similar to determining the OOI to determine the slide statistics. Similar to determining the OOI, the imager computer scans the digital image 302 and identifies individual objects, clustered objects, and boundaries to count the number of objects, clustered objects, and boundaries. The slide statistics are added to the SDS 304.

[0175] The image subsystem 103 completes this process until it has captured images and processed the images of all the slides 102 in each slide carrier 118 loaded into the imager 104. After imaging is complete, the slide carrier 118 is removed from the slide carrier laminate 128. Then, the slide 102 can be removed from the slide carrier 118 and physically stored or discarded when no longer needed according to laboratory procedures.

[0176] As described above, the processed digital images (including their corresponding SDSs) are transferred to the workflow subsystem 106 and stored in the workflow subsystem 106. The workflow subsystem 106 stores the digital images in an active storage device, where the digital images can be accessed by the review station 108. The workflow subsystem 106 includes a workflow subsystem management application 248 (see Figure 1 ), which is configured to manage the workflow for reviewers (such as cytotechnologists and / or cytopathologists) to review the digital images. For example, after a reviewer logs in to the review station 108, the workflow subsystem 106 can send a case worksheet including the digital images of the reviewer to the review station 108. The review station 108 displays the worksheet to the reviewer. Then, the reviewer can manage the worksheet, select cases to process, review the digital images in the selected cases, complete the cases, and / or track the completed cases and the cases in progress.

[0177] After selecting the case to be reviewed, the review station 108 accesses the selected case from the workflow subsystem 106, and the workflow subsystem transfers the digital image and SDS to the review station 108. The review station 108 displays the digital image on various review screens. The user interface of the review station 108 allows the reviewer to select different digital images (e.g., a case may contain multiple digital images, or a case may contain a single digital image). The review screen displays the selected digital image and also displays an image of the OOI. The OOI may be displayed as a thumbnail, such that multiple OOIs can be displayed on the review screen at one time and allows the reviewer to select an OOI to view a full-screen display of the OOI. After the reviewer reviews the digital image of the case, the review station 108 displays a completion screen that allows the reviewer to disposition the case (e.g., mark the case as "completed" or "reviewed by a second reviewer" (e.g., a cytopathologist) or "complete later" (if the reviewer has not completed a full review of the case)).

[0178] After completing the case, optionally including a second review (e.g., by a cytopathologist), the case is entered into the Laboratory Information System (LIS) 250 (see Figure 1 ). The workflow subsystem 106 may have an LIS communication interface 252 for sending the completed case to the LIS, or it may be manually entered into the LIS. Subsequently, the clinician can access the results of the case via the LIS 250.

[0179] Reference Figures 34 to 62 , embodiments of a system and method for displaying digital images generated by an automated digital imaging system 100 will be described in further detail. The digital display system and method allow a reviewer of a biological sample (e.g., cells, tissues, and other targets) to more efficiently and accurately review digital images of the sample and identify, characterize, and / or diagnose attributes within the sample. Thus, the digital display system and method provide certain improvements in the way computers operate and impart certain functions to the display system of biological samples, which address various problems in the computerization of digital microscopy of biological samples.

[0180] Reference Figure 34 , the workflow subsystem 106 of the digital imaging system 100 and the digital imaging display system 108 (also referred to as the "review station 108") are described in further detail. As described above, the imager computer 105 uses image processing algorithms (e.g., software application 135 installed on the imager 104 and / or the workflow subsystem 106), which analyze the digital image of the slide sample 119 and identify the object of interest (OOI) 308 (see Figure 36 ). Subsequently, the identified OOI 308 can be used to facilitate the rapid and accurate review of the digital image of the slide sample 119 by a user (e.g., a cytotechnologist or a cytopathologist).

[0181] As described above, the workflow server 106 is a computer server that primarily serves as a large disk storage system for storing and managing digital images and associated slide data stored in the SDS 304 data record of each slide 102. The workflow server 106 may also include software for image processing and data management and may also provide network capabilities. The SDS 304 includes the stitched digital image 302, slide information 306 captured by the imager 104 for each slide 102, and OOI location data 308. Each SDS 304 is processed by the imager computer 305 to generate an optimized added data set ("pyramid") for image viewing at the digital display system 108. As described herein, the pyramid data (a subset of the SDS) is designed to allow for faster slide data access for recording OOIs. The pyramid data may also include information for improving the display of the entire sample digital image 302 by providing coordinates and display information for translating and scaling regions of the digital image 302 that do not have OOIs (especially in non-gynecological slides). The pyramid data is also added to the SDS. After image processing and pyramid generation are complete, the compressed SDS 304 is sent to the workflow subsystem 106 and moved to the active storage partition 107 on the NAS disk array dedicated only to active data (see Figure 1 )

[0182] The digital display system 108 is a workstation that includes a computer 111 and a monitor 109 for accessing digital images from the workflow server and displaying the digital images for diagnostic review of the images (OOI 308 and / or the entire sample image 302). As Figure 1 depicted, the imager subsystem 103 communicates data with the workflow subsystem 106 via a communication network, which may include one or more of a local area network (LAN, such as an Ethernet network), a wide area network (WAN), the Internet (such as a virtual private network (VPN)), or other suitable networks. Similarly, the viewing station 108 communicates data with the workflow subsystem 106 via the communication network 113, which may include one or more of a local area network (LAN, such as an Ethernet network), a wide area network (WAN), the Internet (such as a virtual private network (VPN)), or other suitable networks.

[0183] The digital imaging display system 108 is configured to access the SDS 304 of each slide 102 from the workflow subsystem to allow a viewer (such as a cytotechnologist and / or a cytopathologist) to view the digital image of the slide 102 on the monitor 109. Refer to Figure 34, each digital imaging display system 108 includes a computer 111 and a computer monitor 109. The computer 111 can be any suitable computer having a microprocessor (CPU), memory, storage device, and network adapter. The computer 111 may also have a graphics processing unit (GPU) for increasing the speed of processing large digital image files. The digital imaging display system 108 may also include input devices such as a keyboard, mouse, touchpad, and the like. The monitor 109 can be a touch screen monitor such that the touch screen is an input device. As described herein, the digital imaging display system 108 communicates data with the workflow subsystem 106 via the communication network 113.

[0184] The monitor 109 can be a special configured color monitor for displaying the digital image 302 of the sample slide 102. For example, the monitor 109 may have a specified resolution and / or color calibration for the display of the sample slide 102 in pathology or other specified applications. For example, in cytopathology, colors are used to identify different types of cells, and specific types of color dyes and / or reagents are used to prepare the sample for viewing by an examiner who marks the sample. For example, U.S. Patent No. 6,661,501 describes various dyes and methods of producing dyes, methods of staining cells for cell or tissue analysis to contrast the nuclear and cytoplasmic parts of the cells, and systems and methods for irradiating cell samples, which are suitable for preparing cell samples for the digital imaging system 100. The effects of the dyes and / or reagents must be detectable in the digital image displayed on the monitor 109 so that the examiner can properly view the slide 102. In other words, the digital imaging system 100 can be configured such that the digital image of the slide 102 viewed on the monitor 109 looks the same or substantially the same as its appearance on the actual slide 102 when viewed using a microscope-based system.

[0185] Reference Figure 35 , an exemplary workflow path 310 for viewing the digital image 302 of the sample slide 102 using the digital imaging display system 108 is shown. The examiner accesses a case 312 from a plurality of cases 312 stored in a case management system (a component of the workflow subsystem management module 314 of the workflow subsystem 106). Each case 312 includes or is associated with one or more SDSs 304 each having one or more digital images 302. A first examiner 316 (such as a cytotechnologist) pulls the case 312 onto the digital imaging display system 108.

[0186] As Figures 36 to 44 shown, the display system 108 can display a work order for the case 312 to be viewed, and the first examiner 316 can select to view the case from the work order. As Figure 36As shown, the display system 108 generates and displays a navigation menu 318 on the monitor 109 of the display system 108. Figure 36 The displayed navigation menu 318 can be collapsed to provide more space on the monitor 109 for displaying other information. After selecting any option in the navigation menu 318, the display system 108 will display a list 320 of the cases 312 within the menu option. As Figure 37 shown, the list 320 includes checkboxes 322 from which the viewer 316 can select to view next. The list 320 also includes a "select all" box for selecting all the cases 312 in the list 320.

[0187] Figure 38 Shows an overview of the different screens displayed by the display system 108 during use of the display system 108, which includes different screen options and different screen layouts depending on the type of the case 312. At Figure 38 the far left, the display system 108 displays a login screen 324. After the user logs in, the display system 108 displays one of the work order screens 326, 328, or 330. The different work order screens 326, 328, 330 have different display formats. The work order screen 326 displays the work order in a simplified list format with different columns each listing different information of the cases 312, as Figure 40 shown in more detail. The work order screen 328 displays the work order in a tile / grid format with a two-dimensional array of tiles, as Figure 41 shown in more detail. The work order screen 330 displays the work order in a billboard format, where the cases 312 are displayed as tiles in separate columns according to the status of each case, as Figure 42 shown in more detail. After the user selects a case 312 from the work order screen, the display system 108 displays the selected case in a viewing screen 332 for gynecological cases (GYN) or a viewing screen 334 for non-gynecological cases (NGYN). Gynecological cases typically contain only a single slide 102, so the viewing screen 332 is formatted to display the digital image of the single slide 102 on the screen. In contrast, non-gynecological cases typically contain multiple slides 102, so the viewing screen 334 is formatted to simultaneously display the digital images of multiple slides 102 on the screen. Thus, the GYN cases referred to in the description and the drawings can be considered to refer to cases with a single slide 102, and NGYN can be considered to refer to cases with multiple slides 102.

[0188] Still referring to Figure 38, after the user completes the inspection of inspection screens 332, 334, the display system 108 displays an inspection completion screen 336 or 338, depending on whether it is a GYN or NGYN case. After the user completes the inspection completion screens 336, 338, the display system 108 displays a work order completion screen 340 or a work order save screen 342. The work order completion screen 340 displays a list of cases 312 completed by the user, and the work order save screen 342 displays a list of in-progress cases 312 (which means cases that have started but not been completed).

[0189] Figure 39 The display system 108 can be configured to switch between different work order screens 326, 328, 330. For example, the work order screens 326, 328, 330 can include work order screen commands 344 that the user can select using a cursor or other input commands using an input device to instruct the display system 108 to display the desired work order screen.

[0190] Go to Figure 40 , showing the work order screen 326 in detail. The work order screen 326 shows the work orders of cases 312 to the user in a simplified list, where each case 312 is in a corresponding row, and the information about each case is in different columns of the corresponding row. The "My Work Orders" title includes a case count 351 that shows the number of cases 312 in the work order screen 326. The work order screen 326 includes the following information columns for each case 312: slide count indicator, access number, case type, status, and due date. The slide count indicator shows that case 312 has a single slide indicated by a slide icon 352 or multiple slides indicated by a folder icon 354 that shows the number of slides in case 312. The work order screen 326 also includes a filtering function 346 that causes the display system 108 to filter the cases 312 in the displayed list. The work order screen 326 also displays a completed inspection counter 348 that indicates the number of inspections completed by the user on the current day. The work order screen 326 also has a search box 350 that allows the user to enter a search query, in which case the display system 108 performs a search for cases 312 for the search query. The display system 108 can be configured to search the user's work orders, the entire database of cases 312 in the active storage device 107, or other desired databases of cases 312.

[0191] Figure 41 Showing the work order screen 328 in more detail. The work order screen 328 shows the work orders of cases 312 to the user in a tile / grid format. Each tile contains the same or similar to Figure 40The information of the columns in the work order screen 326, which includes a slide count indicator, an access number, a case type, a status, and a due date. The work order screen 328 also includes a case count 351, a filtering function 346, a completed review counter 348, and a search box 350 that are the same as those in the work order screen 326.

[0192] Figure 42 Show the work order screen 330 in more detail. The work order 330 shows the work order of the case 312 to the user in a billboard format, where the cases 312 are displayed as tiles in separate columns according to the status of each case. The tiles are basically the same as those in the work order screen 326. The tiles are arranged in three columns: a waiting for initial review column 356, an in-review column 358, and an initial review completed column 360. Each tile contains information that is the same as or similar to the columns in the work order screen 326, which includes a slide count indicator, an access number, a case type, a status, and a due date. The cases 312 in the initial review completed row 360 are shown with reduced contrast (e.g., grayed out) from the waiting for initial review column 356 and the in-review column 358 to reduce distraction from tasks yet to be completed while still allowing the user to see their progress. The work order screen 330 also includes a case count 351, a filtering function 346, a completed review counter 348, and a search box 350 that are the same as those in the work order screens 326 and 328.

[0193] Figure 43 Depict a detail panel 361 that can be displayed by the display system 108 in any of the work order screens 326, 328, 330. Although only this feature of the work order screen 326 is shown, it can be used similarly in the work order screens 328 and 330. The work order screen 326 includes a detail button 362. When the user selects the case 312 in the work order (e.g., by clicking on the case 312), the display system 108 highlights the case in the list, as Figure 45 shown. Then, when the user clicks the detail button 362, the display system 108 displays the detail panel 360, which contains additional details of the selected case 312, including (e.g.) a progress indicator 364, an imaging date 366, the name of the reviewer assigned to the case 312, and a medical record 368. The width of the detail panel 360 can be adjusted to make it wider or narrower, which also makes the main list narrower or wider respectively.

[0194] Figure 44 Illustrate using a graphical status indicator 370 to indicate the status of the case 312 in any of the work order screens 326, 328, 330. In Figure 44In it, the graphical status indicator 370 of the work order screen 326 is a colored line on the left side of each row with a specific status (such as the status of "in progress"). The tiles of the work order screens 328 and 330 may similarly include colored lines or colored borders to provide status indicators that are easy to see.

[0195] Return reference Figure 35 , after the reviewer 316 selects the case 312 from the work order screens 326, 328, and 330 for review, the reviewer 316 selects the case 312 from the work order screens 326, 328, and 330. After receiving the selection of the case 312, the display system 108 accesses the SDS 304 of the selected case 312 from the active storage device 107 of the workflow subsystem 106 and loads the case 312 onto the display system 108. After loading the SDS 304 of the selected case 312, the display system 108 displays the review screen of the case 312. The review screen may vary depending on whether the selected case 312 is a GYN case or an NGYN case.

[0196] Figure 45 Show the exemplary GYN review screen 332. The GYN review screen 332 has a case identification panel 374 across the top of the screen. The case identification panel 374 includes an access number, the type of the case (such as GYN or NGYN), the review deadline, the imaging date, patient information (such as name, date of birth, age, etc.) and the reviewer identifier. The GYN review screen 332 has an OOI panel 376, a main image panel 378, and an image panel 380 selected by the reviewer.

[0197] The display system 108 displays the digital image 302 of the selected case 312 in the main image panel 378. The user can zoom in and pan the digital image 302 in the main image panel 378 using the commands displayed on the GYN review screen 332 and / or the optional commands of the input device of the display system 108. The display system 108 also displays the area of the selected OOI image 382 as will be described below.

[0198] The display system 108 may have a function of scanning the entire digital image, where the system scans and displays the entire digital image 302 in the main image panel 378. The function of scanning the entire digital image may be an optional function with selectable buttons on the viewing screen 332. The function of scanning the entire digital image may allow the user to set the zoom level for automatically scanning the digital image 302 using, for example, zoom commands or selectable commands displayed on the viewing screen 332. The function of scanning the entire digital image scans and displays the entire digital image 302 in the main image panel 378 according to the selected zoom level. The scanning mode may be a serpentine mode similar to the scanning mode for imaging the slide 102 or a progressive scan or a column-by-column scan in the same direction (e.g., from left to right) or other suitable scanning modes. The function of scanning the entire digital image may also include user-selectable commands that allow the user to stop (i.e., pause) and start scanning, set the stop time, set the scanning speed, and / or set the number of regions of the slide where it will stop (e.g., 2000 most important regions, 20 most important regions, etc.). The function of scanning the entire digital image may also have a user-selectable option to stop at each object (e.g., each cell or other object) on the digital image 302. The stop-at-each-object option may allow the user to set the stop time or until the user selects an optional option to continue scanning.

[0199] The OOI panel 376 includes a plurality of OOI images 382 of regions of a digital image 302 corresponding to the OOI location data 308 for each OOI in the SDS 304. The OOI images 382 are reduced-size images (downscaled) of the digital image 302. When the user selects an OOI image 382, the region of the digital image 302 of the selected OOI image 382 is displayed in the main image panel 378 at a scaled size (i.e., magnified compared to the OOI image 382). The user can scale and pan the region of the selected OOI image 382 in a manner similar to that described for the digital image 302, because the main image panel 378 still displays the digital image 302, but it has been scaled and moved to start displaying the region of the OOI image 382. The OOI images may also include display-like commands 384 on the OOI image 382, such as a “plus sign”. When the user selects the display-like command 384 of the selected OOI image 382, the display system 108 displays additional OOI images 382 having characteristics similar to those of the selected OOI image 382. For example, the display system 108 may display 3 to 6 additional OOI images 382 that are included in the SDS similar to the selected OOI image 382. The additional OOI images 382 include objects that share one or more characteristics with the objects in the selected OOI image 382, such as cells. One or more additional OOI images 382 may be obtained from the same entire sample digital image 302 or from an image library that contains previously classified objects that share one or more characteristics with the objects in the selected OOI image 382.

[0200] The image panel 380 selected by the viewer is a region of the GYN viewing screen 332 where the viewer 316 can place one or more OOI images 382 or any other images / positions on the digital image 302 that the viewer adds as OOIs or that the viewer selects for special attention. For example, an OOI image 382 may be selected for viewing by a second viewer 317, such as a pathologist.

[0201] Figure 46 The exemplary NGYN viewing screen 334 is shown. As explained herein, the main difference between a GYN case and an NGYN case 312 is that: the GYN case includes a digital image 302 of a single slide 102, while the NGYN case 312 includes digital images 302 of each of a plurality of slides associated with the case. The NGYN viewing screen 334 has many of the same features as the GYN viewing screen 332, such as the main image panel 378 and the viewer-selected image panel 380.

[0202] In addition, the NGYN review screen 334 has a slide list 388 that shows a list of each slide 102 for which the NGNY case 312 has a digital image 302. Each slide 102 in the NGYN case 312 is represented by a slide icon 387 in the slide list 388 that has an identifier (such as a letter or number). When the reviewer selects a slide 102 in the slide list 388, the display system 108 displays the digital image 302 of the selected slide 102 in the main image panel 378. The selected slide 102 can be highlighted in the slide list 388 to indicate that it is the currently displayed slide 102. The display system 108 allows the reviewer to interact with the digital image 302 in the same manner as the GNY case 312, as described above. The reviewer can select each slide 102 in the slide list 388, and the display system 108 repeats the display process for each slide 102. The NGYN review screen 334 can also include a slide count 392 that indicates the number of slides in the reviewed NGYN case 312 and the total number of slides in the NGYN case 312.

[0203] Figure 47 Another example of the NGYN review screen 394 is shown, which is the same as the NGYN review screen 334 except that the slide list 388 is wider and the slide icons 387 are larger to better show each of the individual slides 102.

[0204] Any review screen (which includes the GYN review screen 332 and the NGYN review screens 334, 394) can also display other data from the SDS of the digital images 302 shown on the review screen. The review screen can display certain characteristics of one or more OOIs. The review screen can include selectable options (similar to the “+ sign” described above) in each OOI image 382 that, when selected, show the certain characteristics of the OOI image 382. The certain characteristics can be displayed in a pop-up window. The review screen can also display machine learning characterization / diagnosis associated with the OOI image 382. The review screen can include selectable options in each OOI image 382 that can be selected to (for example) show the characterization / diagnosis of the OOI image 382 in a pop-up window. The review screen can also display slide statistics from the SDS 304. The slide statistics can be displayed in response to a selectable option to display the slide statistics in a pop-up window, or the slide statistics can be automatically displayed in the case identification panel 374 or other locations on the review screen.

[0205] Figures 58 to 61Displays a viewing screen with additional features (such as adding annotations to the OOI image 382 and / or the digital image 302, sharing the screen, and bookmarking the OOI image 382 and / or the digital image 302). It should be understood that any one or more of these additional features and the graphical user interface can be included in any viewing screen shown in the figures and described herein.

[0206] Figure 58 Illustrates an exemplary GYN viewing screen 420, which is configured to allow a viewer to add annotations and / or mark / draw on the OOI image 382 and / or the digital image 302. The viewing screen 420 is similar to Figure 45 the viewing screen 332 and has most of the same features. As Figure 58 shown in, the viewing screen 420 includes an annotation panel 422. The viewing screen 420 has an add annotation command 424. When the viewer selects the add annotation command 424, the viewing screen 420 allows the viewer to select the OOI image 382 in the OOI panel 376 or an annotation area 432 on the OOI image 382 or the digital image 302 displayed in the main image panel 378. Then, the viewer can add a text annotation 426 in the annotation panel 422. In this way, the viewer can make annotations for specific objects on the OOI image 382 or the digital image 302. The annotation panel 422 displays an annotation ID 428 (such as a number or letter), a viewer ID 430 (such as the name of the viewer or other identifier of the viewer), and the annotation text 426. The viewer can add additional annotations by selecting the add annotation command 424 and repeating the process. Annotations (which include the annotation text 426, the annotation area 432, the annotation ID 428, and the viewer ID 430) are stored in the SDS 304 of the corresponding slide 102. The annotation panel 422 allows other viewers to view the annotations of previous viewers and add additional annotations using the same process. The annotations of other viewers have different annotation IDs 428 and / or viewer IDs 430. The viewer can add overall slide-level annotations by skipping the annotation area selection and only including the text annotation 426 within the annotation panel 422. Alternatively, the viewing screen 420 can include a separate overall slide-level annotation panel similar to the annotation panel 422.

[0207] Figure 59 Illustrates an exemplary GYN viewing screen 450, which is configured to allow a viewer to add recorded audio annotations (such as dictated annotations) to case 312 of the digital image 302 and / or mark / draw on the OOI image 382 and / or the digital image 302. The viewing screen 450 is similar to Figure 45 the viewing screen 332 and has most of the same features. As Figure 59As shown, the review screen 450 includes a dictate annotation command 452. When the reviewer selects the dictate annotation command 452, the review screen 450 allows the reviewer to record a dictate annotation. For example, the review screen 450 displays a dictate annotation window 454. The dictate annotation window 454 includes a dictate control bar 456. The dictate control bar 456 has a record button 458, a play button 458, a pause button 460, and a navigation slider 462. The record button 458 starts recording an audio annotation. The play button 458 plays the audio annotation. The pause button 460 pauses the recording or playback. The navigation slider 462 allows the reviewer to advance or rewind within the recorded annotation. The dictate annotation command 452 may also allow the reviewer to mark an annotation area 432, similar to the review screen 420. Thus, the reviewer can make an annotation for a specific object on the OOI image 382 or the digital image 302. The dictate annotation window 454 displays an annotation ID 428 (e.g., a number or letter) for each audio annotation, a reviewer ID 430 (e.g., the reviewer's name or other identifier of the reviewer), and the dictate control bar 456. The reviewer can add additional dictate annotations by selecting the dictate annotation command 452 and repeating the process. The audio annotation (which includes the recorded audio, the annotation area 432, the annotation ID 428, and the reviewer ID 430) is stored in the SDS 304 of the corresponding slide 102. The dictate annotation window 454 allows other reviewers to play the audio annotations of previous reviewers and add additional audio annotations using the same process. The annotations of other reviewers have different annotation IDs 428 and reviewer IDs 430.

[0208] Figure 60Disclosed is an exemplary GYN examination screen 434, which is configured to allow a first examiner to share the first examiner's screen with a second examiner on a different display system 108 and allow the first examiner to give the second examiner the right to control the examination screen 434 on the first examiner's screen. In other words, the second examiner's display system 108 displays the same examination screen 434 as that displayed on the first examiner's display system 108, and the second examiner uses the second examiner's display system 108 to control the examination screen 434 on the first examiner's screen. The examination screen 434 has a share screen command 436. When the first examiner selects the share screen command 436, the examination screen 434 displays a share screen window 438, such as a pop-up window. The share screen window 438 requests the share screen ID of the second examiner. The first examiner enters the share screen ID of the second examiner (such as name, number, code, etc.) and clicks enter to transmit the share screen ID to the workflow subsystem 106. The workflow subsystem 106 verifies the share screen ID and then shares the examination screen 434 from the first examiner's display system 108 with the second examiner's display system 108. The examination screen 434 also has a share control command 440, which allows the first examiner to transfer the control right of the examination screen 434 to the second examiner. The share screen command 436 can be a toggle command that allows the first user to share and unshare the examination screen 434 by selecting the command 436. Similarly, the share control command 440 can be a toggle command that allows the first examiner to share and unshare the control right of the examination screen 434. Selecting the share screen command 436 can also enable a chat window 442, which can be displayed in any open space on the examination screen 434 or in a movable pop-up window. The chat window 442 allows the first examiner and the second examiner to chat in text in the chat window 442. In addition, the examination screen 434 can enable an audio or audio / video communication connection between the first examiner and the second examiner via the workflow subsystem to allow audio or audio / video communication between the examiners. In addition, the examination screen 434 can be configured to allow additional examiners to share the first examiner's examination screen 434 by repeating the share command to the additional examiners.

[0209] Figure 61 Disclosed is an exemplary GYN examination screen 444, which is configured to allow an examiner to add a bookmark (for example) to case 312 of digital image 302 to generate a reference library that focuses on case 312 and digital image 302. The examination screen 444 is similar to Figure 45 examination screen 332 and has most of the same features. As Figure 61As shown, inspection screen 444 has a bookmark add command 446. When the inspector selects the bookmark add command 446, inspection screen 444 displays a bookmark annotation window 448 that allows the inspector to enter an annotation. For example, the inspector can enter annotation text 426 regarding why the inspector added case 312 to the watch list. The bookmark add command 446 can also allow the inspector to mark an annotation area 432, similar to inspection screen 420. The bookmark annotation window can include an annotation ID 428 and an inspector ID 430, also similar to inspection screen 420. Additionally, the bookmark record is also recorded in the SDS 304 of the digital image 302, which includes the annotation text 426, the annotation ID 428, the inspector ID, and the annotation area 432. The workflow subsystem 106 designates the use of the bookmarked case 312 in a reference library (such as the reference library of watched cases).

[0210] After the inspector selects and inspects slide 102 in the slide list 388, the display system 108 provides a visual indicator 390 to indicate that the corresponding slide 102 has been inspected. For example, Figure 48 the slide list 388 shown contains a dot 390 above each slide icon 387 in the slide list 388. The dot 390 indicates whether the corresponding slide 102 has been inspected by the inspector.

[0211] Figure 57 Shows another example of an inspection screen 414, which is similar to Figure 45 inspection screen 332 and has most of the same features. The main difference is that inspection screen 414 shows the entire sample area of the digital image 302 in the main image panel 378. In other words, the image in the main image panel 378 is fully zoomed out to show the entire digital image 302. Inspection screen 414 also has a zoom slider 416 for adjusting the zoom of the image in the main image panel 378. Any of the other inspection screens 332, 334, and 394 can also display the fully zoomed-out entire digital image 302 in the main image panel 378 and can also include a zoom slider 416.

[0212] After inspector 316 inspects case 312, inspector 316 selects a complete inspection command 396 on inspection screens 332, 334, 394 (see Figures 45 to 47 ). After receiving the complete inspection command 396, the display system 108 displays a completion screen. Figure 48An example of the completion screen 336 of the GYN case 312 when the viewer 316 determines that the sample is normal (as a result of the examination) is shown. The completion screen 336 may include features identical to the GYN examination screen 332 and a completion frame 399 displayed on the main image panel 378. The viewer 316 selects the normal completion command 400. After receiving the normal completion command, the display system 108 sends a normal completion signal to the workflow subsystem 106 to update the determined case 312 as normal. Then, as shown in the workflow diagram of Figure 35 , the selected case 312 is flagged for archiving in the completed cases or routed to quality control. The percentage of cases 312 determined to be normally completed by the first viewer 316 is sent to quality control for inspection by another viewer. The quality control inspection can determine whether the viewer 316 misrecords one or more cases 312 as normal.

[0213] The completion frame 399 also includes a later completion command 401 when the viewer 316 does not complete the examination of the selected case 312. After receiving the later completion command 401, the display system 108 marks the case 312 as in progress and leaves the case in the viewer's work order as an in-progress case.

[0214] Figure 49 An example of the completion screen 336 of the GYN case 312 when the viewer 316 determines that the sample is abnormal (as a result of the examination) is shown. The completion screen 336 may include features identical to the GYN examination screen 332 and a completion frame 399 displayed on the main image panel 378. If an abnormality is determined in the sample slide 102, then the first viewer 316 selects the send-to-pathologist command 404. Then, the workflow subsystem 106 schedules the pathologist 317 to examine the case 312, as shown in Figure 35 and 52 in the workflow diagram to 54. Then, as shown in Figure 35 , the pathologist can log in to the display system 108 and the display system 108 displays a work order screen, an examination screen, and a completion screen to allow the pathologist to select, examine, and arrange the case in substantially the same manner as the first viewer 316.

[0215] After the viewer completes the examination of the case, the viewer can return to the work order screen. Figure 50 An example of the work order screen 340 is shown, which shows the detail panel 361 of the case 312 marked as abnormal by the first viewer 316 and assigned to the pathologist for the pathologist to examine. By default, the completed case 312 remains in the viewer's work order until the viewer selects to hide the completed case 312. Figure 50The work order screen 340 shows a Hide Case command 406 for hiding selected completed cases from the work order screen 340. The default can be modified to use rules to automatically hide completed cases. Figure 51 Shows a work order screen 340 in which completed cases 312 have been hidden.

[0216] The annotations, review results, and status of each case 312 are stored in a slide data file in the workflow server 106 (such as a database server 530( Figure 63 ))). The slide data file can contain digital images 302, SDS 304, slide information 306, and / or pyramids / OOIs 308, or it can be different data files.

[0217] Next, as Figure 35 shown, the layout of each case 312 can be entered into a laboratory information system (LIS) 408, which can be accessed by a clinician 412 on a clinician computer 410 that communicates with the LIS 408 data.

[0218] Figure 35 And 52 Figures 55 and 56 show several variations of the workflow for reviewing cases 312 by a first reviewer 316 (such as a cytotechnologist) and a second reviewer 317 (such as a pathologist) using the digital imaging display system 108 and the workflow subsystem 106. In the Figure 35 And 52 workflow depicted in Figures 53 and 54, the terms "extract" and "push" describe whether the reviewer extracts case 312 from the imaging cases waiting to be reviewed or whether the case is pushed to the reviewer by a supervisor or an automated process (such as a rule-based algorithm, using rules such as case type, case results, etc.). Figure 55 Contains flowcharts showing various scenarios of the workflow for a cytotechnologist and a pathologist to complete the Figure 35 And 52 workflow depicted in Figures 53 and 54. Figure 56 contains flowcharts showing various supervisor / management roles and scenarios of the Figure 35 And 52 workflow depicted in Figures 53 and 54.

[0219] Figure 63 A block diagram illustrating an exemplary workflow server 106 with backup and archival capabilities. The workflow server 106 includes multiple software modules for performing various backup and archival services, which include an imager network service module 520, a review station network service module 522, an archival service module 524, and a backup service module 525. The workflow server 106 also includes an Internet Information Services server 526 (such as MICROSOFT TMIIS), security software module 528 (such as ASP.NET Core identification / authentication), a database server 530 with a database 527 of slide data files (such as an SQL database server), an image repository 532 (such as on a RAID storage disk), and a server operating system 534 (such as Microsoft Server 2016). The imager network service module 520 is configured to be used by the imaging station 190 to send and retrieve data using the database server 530. The viewer network service module 522 hosts the viewer application and provides services to send and retrieve data using the database server 530. The database server 530 stores the SDS 304 containing slide information 306, pyramid / OOI 308, and viewing information. The archive service module 524 performs the archiving of slide data files, digital images 302, and / or SDS 304. The backup service module 525 performs the backup of the database of slide data files (which includes digital image files and viewing information).

[0220] The backup module 525 prevents single disk failures and system failures. The backup module 525 can back up the database and the slide image files 302 to a network drive 540 (such as using the WINDOWS TM built-in backup component) or a cloud storage device 542 (such as using MICROSOFT AZURE TM backup agent). The purpose of the backup is to protect the locally stored information from disasters such as disk failures. If the backup is remote (such as a cloud backup), then the backup also prevents catastrophic failures caused by, for example, fire, flood, or other local accidents. The backup allows for the restoration of the database and digital image files as part of the repair from a system failure.

[0221] The archive service module 524 is configured to remove local data in the database server 530 and store it more efficiently for a long time. This can also improve the performance and / or availability of the digital imaging system 100 by reducing its storage requirements. The archived data is stored on an external storage device (such as a remote network drive 540 or a cloud storage device 542 (such as MICROSOFTAZURE TM cloud storage device or AMAZON Web Services TM(AWS) cloud storage device). The archival service module 524 is configured to allow a user to specify a policy for selecting which digital image files 302, slide information 306, pyramid / OOI 308, and / or view information to archive. The archival service module 524 can also archive macro images (lower resolution variants of the digital image files 302 when generated by the system 100). For example, a user can set a policy to archive data that has been accessed most recently over a selected period (e.g., 30 days, 15 days, 60 days, etc.). The archival policy can also include a set time for performing the archive, such as a specified time for daily archiving or the time and date for weekly archiving, etc. When archiving data for a slide 102, the information in the database 527 is not archived, but the database 527 is updated to track which slides 102 have been archived. In one embodiment of the archival service module 524, the pyramid image file (e.g., OOI 308) of the archived slide 102 is not archived. Local copies of the archived files and pyramid files are deleted from the database 527 and the image repository 532.

[0222] Reference Figure 64 , a flowchart depicting a method 600 for archiving a slide 102 using the archival service module 524. In step 602, the archival service module 524 receives and stores an archival policy that includes an archival schedule. In step 604, at each scheduled time of the archival schedule, the archival service module 524 identifies slides 102 that meet the archival policy and places the slides 102 in an archival database table. In step 606, for each slide 102 in the archival database table, the archival service module 524 copies the slide image file 302 and the slide information 306 from the database 527 and the image repository 532 to the archival external storage devices 540, 542. After successful copying, in step 608, the slide data for each archived slide is deleted from the database 527 and the image repository 532. In step 610, the database 527 is updated to indicate that each archived slide 102 has been successfully archived. In step 612, an additional archival history database that stores an archival history log can also be updated.

[0223] The archival service module 524 is also configured to retrieve the archival slide data of the archival slide 102. The archival service module 524 can utilize a retriever web service installed on the workflow server 106. The retriever web service is configured based on the type of archival, whether it is a remote network drive or a specific cloud storage device. The user uses the retriever web service to select the slide 102 to be retrieved from the archive, and the retriever fetches the archival slide data of the selected slide 102 from the external storage devices 540, 542, which includes the slide image file 302, the slide information 306, the macro image, and / or other archival data. After successfully retrieving the slide data, the system 100 regenerates the pyramid image data containing the OOI 308, as described herein. The archival service module 524 updates the archival database table and / or the archival history database regarding the status of the retrieved slide 102. The SDS 304 of each retrieved slide 102 can be stored in the database 527 and the image repository 532 of the workflow server 106.

Claims

1. An automatic slide imaging system, comprising: an imager configured to obtain an image of a sample attached to a surface of a slide and generate a whole sample image from the obtained image, the sample including a plurality of objects distributed within a three-dimensional volume having a length, a width, and a thickness, the thickness defining a z-axis relative to the slide surface, wherein respective objects of the sample are located at different positions along the z-axis; the imager having a first camera and a second camera, the first camera configured to obtain a macro image of the sample, the second camera having an optical axis non-orthogonal to the slide and configured to obtain a plurality of micro images of the sample at the same z-axis of the three-dimensional volume; the imager having an image processor configured to generate the whole sample image from the micro images by: (a) dividing the micro images into a plurality of sub-regions representing different focal planes of each tile captured by a digital camera across the depth of the sample; (b) vertically recombining the plurality of micro images of the sub-regions along the z-axis to generate a cross-focus stack of the tiles; (c) converting the cross-focus stack of each tile into a single-plane merged focused image by comparing each pixel with adjacent pixels in the same plane and using an algorithm for determining the focus of a pixel with a focus metric, calculating the difference between each pixel in the merged focused image and its adjacent pixels, weighting this difference based on the pixel values to determine a relative metric of the specific plane of a specific pixel to generate a tile of plane values, applying a moving average to the tile of plane values to provide a better transition between objects in the final image, and selecting the merged focused image based on the associated plane values; (d) tiling the merged focused images to generate corresponding scan bands; and (e) stitching the scan bands together to form the whole sample image; wherein the object focus is depicted in the whole sample image regardless of the individual positions of the respective objects within the three-dimensional volume.

2. The imaging system according to claim 1, wherein the macro image includes one or more fiducial marks located on the slide surface, and wherein the imager is configured to determine the relative position and boundaries of the sample on the slide surface at least in part based on the one or more fiducial marks.

3. The imaging system according to claim 2, wherein the imager is configured to obtain the micro images at least in part based on the relative position and boundaries of the sample on the slide surface determined from the macro image.

4. The imaging system according to claim 2 or 3, wherein the imager includes a first imaging platform and the first camera, the first imaging platform configured to carry the slide, the first camera configured to obtain the macro image when the slide is carried on the first imaging platform, and A second imaging platform and the second camera, the second imaging platform being configured to carry the slide, and the second camera being configured to acquire the micro-images of the sample attached to the slide carried on the second imaging platform.

5. The imaging system according to claim 4, wherein the imager is configured to automatically move at least one of the second camera and the second imaging platform relative to the other to acquire the micro-images.

6. The imaging system according to claim 4, wherein the optical axis of the second camera forms a non-orthogonal angle with the second imaging platform.

7. The imaging system according to claim 4, the slide having a thickness, wherein each of the micro-images includes at least a portion of the slide below the surface, and the second camera is configured to acquire the plurality of micro-images of the sample through the thickness of the slide.

8. The imaging system according to claim 4, wherein the sample is covered by a coverslip, the coverslip being transparent enough to acquire the micro-images of the sample through the coverslip, the coverslip having a thickness, wherein the second camera is configured to include at least a portion of the depth of the coverslip in the acquired micro-images.

9. The imaging system according to claim 4, the slide having a width defining an x-axis and a length defining a y-axis, wherein the imager is configured to translate the slide along the y-axis relative to the second camera when the second camera acquires the micro-images at corresponding y-axis positions, each micro-image including the entire x-axis width of the sample that determines the sample boundary on the slide surface.

10. The imaging system according to claim 4, the imager comprising one or more slide holder sockets, each slide holder socket being configured to receive a slide holder including a plurality of slots, each slot being configured to hold an individual slide, and a robotic arm assembly configured to engage and remove the slide from the slot of the slide holder in the slide holder socket, transport the slide and carry the slide on the first imaging platform to acquire the macro-image, re-engage and remove the slide from the first imaging platform, transport the slide and carry the slide on the second imaging platform to acquire the micro-images, and re-engage and remove the slide from the second imaging platform.

11. The imaging system according to claim 10, wherein the robotic arm assembly is further configured to transport the slide to a slide holder that is the same as or different from the slide holder from which the slide is removed, and release the slide into the slot of the corresponding same or different slide holder.

12. The imaging system according to claim 11, wherein the slot of the corresponding same or different slide holder is the same as the slot from which the robotic arm assembly removes the slide.

13. The imaging system according to claim 4, wherein the image processor includes an image computer having one or more computer processors and a graphics processor, the image processor being configured to generate the entire sample image from the micro-images, wherein the image processor determines corresponding best-focus images of individual objects in the micro-images, and wherein the best-focus images of the objects are incorporated into the entire sample image.

14. The imaging system according to claim 13, wherein the image processor is configured to identify objects of interest in the sample and store images of the identified objects of interest and the entire sample image.

15. The imaging system according to any one of claims 1 to 3, wherein the macro-image of the sample includes an image of a barcode on the surface of the slide, and wherein the imager is configured to obtain information about the sample from the barcode.

16. The imaging system according to any one of claims 1 to 3, wherein the sample is a cytological cell sample and the object is a cell.

17. The imaging system according to any one of claims 1 to 3, wherein the sample is a pathological tissue sample and the object is a tissue structure.

18. The imaging system according to any one of claims 1 to 3, further comprising an inspection station including a display monitor, a user interface, and a processor operably coupled to the respective display monitor and user interface, wherein the processor is configured to display the entire sample image and individual images of objects within the sample image on the display monitor.

Images