Digital imaging system and method

Abstract

To provide an automated system and a method for evaluating a specimen attached to a substrate such as a microscope slide.SOLUTION: An automatic digital imaging system 100 includes an imager 104 for acquiring a plurality of micro images of a specimen mounted on a substrate. The specimen includes a plurality of objects dispersed in a three-dimensional volume, and generates a whole specimen image of the specimen using the micro images. The object included in the specimen is rendered substantially in focus in the whole specimen image, regardless of a z-axis depth of each object in the specimen. The whole specimen image is saved in storage media for later review by a cytologist using a computer-controlled review station 108 that includes a display and a user interface. The cytologist reviews and classifies the saved entire specimen image by the user interface of the review station.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] Cross - Reference to Related Applications

[0001] This patent application claims priority to U.S. Provisional Application No. 62 / 755,151, titled "Digital Imaging System and Method," filed on November 2, 2018, and U.S. Provisional Application No. 62 / 755,291, titled "Digital Image Display System and Method," filed on November 2, 2018, which are hereby incorporated by reference in their entirety.

[0002]

[0002] The present disclosure generally relates to digital imaging systems and methods, and more specifically to digital imaging systems and methods for acquiring digital images of specimens such as cytological (cellular) specimens and pathological (solid) tissue specimens that can be displayed and viewed on a computer monitor by cytologists and / or cytopathologists.

Background Art

[0003]

[0003] Cytology is a branch of biology that deals with the study of the formation, structure, and function of cells. As applied in a laboratory setting, cytologists, cytopathologists, and other medical professionals make medical diagnoses of a patient's condition based on a visual examination of a sample of the patient's cells. Such samples are referred to herein as "cytological" specimens. A typical cytological technique is the "Pap smear," in which cells are scraped from a woman's cervix and analyzed to detect the presence of abnormal cells that are a precursor to the development of cervical cancer. Cytological techniques are also used to detect abnormal cells and diseases in other parts of the human body.

[0004]

[0004] Cytological techniques are widely used because, generally, the collection of cell samples for analysis is less invasive than conventional surgical pathological procedures such as biopsies. In a biopsy, a solid tissue sample (referred to herein as a “pathological” specimen) is removed from the patient using a special biopsy needle having a spring-loaded, translatable stylet or a fixation cannula. Cell samples are obtained from the patient by a variety of techniques, including, for example, scraping or swiping an area, or using a needle to aspirate bodily fluids from the pleural cavity, bladder, spinal canal, or other suitable area. The obtained cell samples are usually placed in an antiseptic solution, then extracted from the solution, and transferred to a glass slide. A fixative is applied to the cell sample to keep the cells in place on the glass slide, facilitating subsequent staining and examination.

[0005]

[0005] It is generally desirable that the cells on the slide have an appropriate spatial distribution so that individual cells can be examined. Typically, a single layer of cells is preferred. Therefore, to prepare a cytological specimen from a liquid sample containing many cells (e.g., tens of thousands), it is usually necessary to first separate the cells from each other by mechanical dispersion, liquid shearing, or other techniques so that a thin monolayer of cells is collected and placed on the slide. This method allows the cytologist to more easily identify the presence of abnormal cells in the patient sample. The cells can also be counted to confirm that an appropriate number of cells have been evaluated.

[0006]

[0006] Specific methods and apparatus for generating a thin monolayer of cells from a liquid sample container and then transferring this thin layer to a “specimen slide” suitable for visual inspection are disclosed in U.S. Patents 5,143,627, 5,240,606, 5,269,918, 5,282,978, 6,562,299, 6,572,824 and 7,579,190. All publications cited herein are incorporated in their entirety by reference. According to one method disclosed in these patents, patient cells suspended in a preservation solution and stored in a sample container are dispersed using a spin sample collector inserted into the container. A controlled vacuum is applied to the sample collector to aspirate the liquid through a screen filter until a desired quantity and spatial distribution of cells are collected against the filter. Subsequently, the sample collector is removed from the sample container, and the filter portion is pressed against a glass slide to transfer the collected cells to the slide in substantially the same spatial distribution as they were collected. Apparatus manufactured in accordance with one or more teachings of these patents have been commercially successful, such as the ThinPrep® 2000 processor (where specimen slides are processed one at a time from patient samples) and the ThinPrep® 5000 processor (where specimen slides are processed in batches from patient samples), manufactured and sold by Hologic, Inc. in Marlborough, Massachusetts. See also U.S. Patents 7,556,777 and 7,771,662.

[0007]

[0007] Once the specimen slides are ready, the specimens can be visually examined by a cytologist, usually under magnification, with or without various illumination sources. Additionally or alternatively, automated slide imaging systems are used to assist the cytological examination process. For example, an automated slide imaging system may capture images of all or substantially all individual cells within a cytological specimen fixed to a slide and perform a preliminary assessment of cells using image processing techniques so that the cytologist can see the potentially most relevant cells on the slide and perform a close examination. Examples of such imaging systems are disclosed in U.S. Patents 7,587,078, 6,665,060, 7,006,674, 7,369,304 and 7,590,492. Depending on whether the actual specimen slides are examined under magnification or magnified images of the specimen are examined, the specimen is typically classified by the cytologist as either “normal” or “abnormal,” and abnormal samples are usually classified into one of the main categories defined in the Bethesda System for Reporting Cytological Diagnoses of the Cervix / Vagina. This category includes low-grade squamous intraepithelial lesions (LSIL), high-grade squamous intraepithelial lesions (HSIL), squamous cell carcinoma, adenocarcinoma, atypical glandular cells (AGUS), adenocarcinoma in situ (AIS), and atypical squamous cells (ASC). Additional information on the classification of cell specimens is widely available in "Yokohama System for Reporting Endometrial Cytology," Diagnostic Cytopathology, May 2018, Vol. 46(5), pp. 400-412, and "Reporting and Guidelines for Non-Gynecological Cytopathological Specimens," Pathology and Experimental Medicine Archives, November 2009, Vol. 133, No. 11, pp. 1743-1756.

[0008]

[0008] However, conventional systems and methods for acquiring digital images of biological specimens have many drawbacks. One is that conventional systems and methods have the problem of slow acquisition time because it takes time to scan the entire specimen. Furthermore, conventional systems and methods usually provide only a single focal plane for the entire specimen. Biological specimens, whether cytological or pathological, are actually three-dimensional (i.e., have depth). Therefore, the depth of field of the image is very limited due to the high magnification and focal aperture required to acquire digital images of biological specimens. Consequently, parts of the specimen that are outside the depth of field of the focal plane are out of focus or not shown in the image. To acquire a digital image in focus at multiple different depths of the specimen, it is necessary to adjust the focal plane by moving the specimen or camera or adjusting the focal lens. However, this requires an additional scan of the specimen for each focal plane, which further slows down the acquisition time. [Overview of the Initiative]

[0009]

[0009] Embodiments of improved automated systems for evaluating specimens (e.g., cytological or pathological specimens) mounted on a substrate are disclosed and described herein. Typically the substrate is a microscope slide, and therefore embodiments using slides as substrates are described, but it should be understood that the automated systems and methods disclosed herein are not limited to those using slides, and any suitable substrate can be used. Specimens may include any type of specimen that is digitally imaged under microscopic magnification, such as biological or chemical specimens including cytological specimens, tissue specimens, etc. Where used herein, the term “specimen” may apply to the whole specimen or a part of the specimen, depending on the context.

[0010]

[0010] An exemplary automated slide imaging system constructed according to the disclosed invention comprises an imager configured to acquire an image of a specimen attached to the surface of a slide, wherein the specimen comprises a plurality of objects dispersed within a three-dimensional volume, and the imager is configured to generate an overall specimen image from the acquired image, wherein the objects are depicted in focus within the overall specimen image, regardless of the individual position of each object within the three-dimensional volume, the three-dimensional specimen volume having length, width, and thickness, the thickness defining the z-axis with respect to the slide surface, and wherein each object of the specimen (e.g., possibly an individual cell or tissue structure) is positioned at a different location along the z-axis. The acquired image comprises a macro image of the specimen and a plurality of micro images of the specimen, wherein the macro image comprises one or more reference markers placed on the slide surface, and the imager is configured to acquire the micro images at least partially based on the relative positions and boundaries of the specimen on the slide surface determined from the macro image.

[0011]

[0011] In exemplary embodiments, the imager comprises a first imaging platform configured to seat the slide, a first camera configured to acquire a macro image when the slide is seated on the first imaging platform, a second imaging platform configured to seat the slide, and a second camera configured to acquire a micro image of a specimen attached to a slide mounted on the second imaging platform. The imager is configured to automatically move at least one of the second camera and the second imaging platform relative to the other in order to acquire a micro image, wherein the optical axis of the second camera forms a non-orthogonal angle with the second imaging platform, and the imager is configured to acquire a micro image of the specimen along the same z-axis of the three-dimensional specimen volume. In particular, the slide to be imaged has thickness, and the micro image may include at least a portion of the slide below the surface. The specimen is covered with a coverslip that is transparent enough to acquire a micro image of the specimen through the coverslip, the coverslip has thickness, and each micro image includes at least a portion of the depth of the coverslip. In an exemplary embodiment, the specimen slide has a width defining the x-axis and a length defining the y-axis, and the imager is configured to translate the slide along the y-axis relative to the second camera as the second camera acquires a microimage at each y-axis position, and each microimage includes the entire x-axis width of the specimen based on the specimen boundaries determined on the surface of the slide.

[0012]

[0012] In an exemplary embodiment, the imager has one or more slide holder receptacles, each slide holder receptacle configured to accept a slide holder having a plurality of slots, each slot configured to hold an individual slide, and a robot arm assembly is configured to (i) remove a slide from the slot of the slide holder of the slide holder receptacle, (ii) move and seat the slide on the first imaging platform to acquire a macro image, (iii) remove the slide again from the first imaging platform, (iv) move and seat the slide on the second imaging platform to acquire a micro image, and (v) remove the slide again from the second imaging platform. The robot arm assembly is further configured to (vi) transfer the slide to the same or different slide holder from which the slide was removed, and (vii) release the slide into the slot of each of the same or different slide holders, where the slot of each of the same or different slide holders is the same slot from which the slide was removed by the robot arm assembly.

[0013]

[0013] In exemplary embodiments, the imager includes an image processor configured to generate a whole specimen image from a microimage, the image processor identifying a best-focus image for each individual object in the microimage, and the best-focus images of the objects are incorporated into the whole specimen image. The image processor is preferably further configured to identify objects of interest in the specimen (e.g., individual cells or tissue structures) and to store images of the identified objects of interest together with the whole specimen image. The macroimage of the specimen includes an image of a barcode on the slide surface, in which case the imager is preferably configured to obtain information about the specimen from the barcode.

[0014]

[0014] In an exemplary embodiment, the system further comprises a review station including a display monitor, a user interface, and a processor functionally coupled to the respective display monitor and user interface, wherein the processor is configured to display an overall image of the specimen along with individual images of individual objects within the specimen image on the display monitor.

[0015]

[0015] The review station may be configured, but not limited to, to allow the system user to select a whole specimen image from a list of saved whole specimen images using a user interface.

[0016]

[0016] The system may be configured, but not limited to, to allow the user of the system to finally characterize the entire sample image using a user interface, or to transfer it for secondary review.

[0017]

[0017] A list of stored whole specimen images that have not been ultimately characterized or otherwise not forwarded for secondary review may be organized and displayed in several different formats based on input received through the user interface. The system may further be configured to allow an authorized third party to input a list of stored whole specimen images, either entirely or partially, for review by a particular system user.

[0018]

[0018] In an exemplary embodiment, the system is configured to allow a system user to add annotations to the whole specimen image and / or to a data file associated with the whole specimen image, which are available for subsequent system users to review along with the whole specimen image. For example, the annotations may be associated with individual objects in the whole specimen image and may take the form of electronic markings created on the whole specimen image or on images of a portion of the whole specimen image.

[0019]

[0019] In an exemplary embodiment, the system is configured to display one or more additional objects that share one or more characteristics with each object in the whole specimen image in response to a system user prompt associated with each object, where the one or more additional objects (e.g., cells or tissue structures, where applicable) may be from the whole specimen image and / or from a library containing previously classified objects. The system may be configured, but not limited to, to allow data about the whole specimen image or individual objects within it to be entered via a user interface and stored in a data file associated with the whole specimen image.

[0020]

[0020] In exemplary embodiments, the system is configured to display a magnified view of at least a portion of the whole specimen image and to automatically scan at least a portion of the displayed whole specimen image. In such embodiments, the system preferably automatically scans with a user-selected scan pattern, which includes, but is not limited to, a meandering pattern, a row-by-row pattern, and a column-by-column pattern. In such embodiments, the system is preferably configured so that a system user can set, via a user interface, the magnification level at which the system displays the scanned at least a portion of the whole specimen image. In such embodiments, the system is preferably configured so that a system user can, via a user interface, stop and start scanning at the displayed position of the scan, and set the scan speed. In such embodiments, the system is configured so that a system user can pause the scan at each object as each object becomes visible during the scan.

[0021]

[0021] In an exemplary embodiment, the system is configured to display a review screen on a display monitor, which includes a main image panel displaying an overall image of a specimen and an object panel containing separate images of individual objects within the specimen image, and the system user can zoom in / zoom out and / or pan the overall image of the specimen in the main image panel via the user interface. The system may be configured, but not limited to, to display a region of the overall image of the specimen containing each of the individual objects in the object panel when the system user selects an individual image of an object in the object panel via the user interface. The system may be configured, but not limited to, to allow the system user to select an object in the displayed overall image of the specimen via the user interface, where the system displays an image of the selected object in a system user selection image panel.

[0022]

[0022] According to further aspects of the present invention disclosed herein, a method is provided for generating a whole-specimen image of a specimen attached to the surface of a slide, wherein the specimen comprises a plurality of objects dispersed within a three-dimensional volume, the method comprising (i) taking a macro-image of the specimen, (ii) taking a plurality of micro-images of the specimen at least partially based on the macro-image, and (iii) generating a whole-specimen image by processing the micro-images using an image processor, wherein the objects are depicted substantially in focus on the whole-specimen image regardless of the individual position of each object within the three-dimensional volume, the three-dimensional volume having length, width, and thickness, the thickness defining the z-axis with respect to the slide surface, and each object of the specimen (e.g., a cell or tissue structure) being positioned at a different location along the z-axis.

[0023]

[0023] The macro image is acquired using a first camera, the micro image is acquired using a second camera, at least one of the second camera and the slide is automatically moved relative to the other when acquiring the micro image, the optical axis of the second camera forms a non-orthogonal angle with the slide, and the micro image is acquired on the same z-axis as the three-dimensional volume. Next, the micro image can be processed using an image processor to determine the best-focus image of each individual object in the micro image, and the best-focus images of each object can be incorporated into the whole-sample image to generate a whole-sample image.

[0024]

[0024] The macro image preferably captures one or more reference markers placed on the slide surface, and the method further includes the step of identifying the relative position and boundaries of a specimen on the slide surface, at least partially based on the one or more reference markers, where the micro image is acquired at least partially based on the relative position and boundaries of the specimen on the slide surface identified from the macro image. The micro image can be acquired by translating the slide along the y-axis relative to the second camera as the second camera acquires a micro image at each y-axis position, and each micro image includes the entire x-axis width based on the boundaries of the specimen identified on the slide surface.

[0025]

[0025] This method may further include the steps of identifying objects of interest within a specimen (e.g., individual cells or tissue structures) and saving images of the identified objects of interest together with an image of the entire specimen.

[0026]

[0026] This method may further include the step of reviewing an overall image of a specimen, including images of the individual objects therein, using a computer-controlled review station that includes a display and a user interface.

[0027]

[0027] Other and further features and advantages of the disclosed embodiments of the automated imaging and review system are shown in the accompanying drawings and described in the detailed description of the drawings.

Brief Description of the Drawings

[0028]

[0028] The foregoing and other aspects of the embodiments will be described in more detail with reference to the accompanying drawings, where like reference numerals refer to like elements and the description of like elements applies to all embodiments described whenever relevant. [Figure 1]

[0029] FIG. 1 is a block diagram of an automatic digital imaging system according to an embodiment. [Figure 2]

[0030] FIG. 2 is a block diagram of the automatic digital imaging system of FIG. 1 according to an embodiment, showing a single imager and review station. [Figure 3]

[0031] FIG. 3 shows a specimen slide for use in the automatic digital imaging system of FIG. 1 according to an embodiment. [Figure 4]

[0032] FIG. 4 is a front perspective view of the digital imager of the digital imaging system of FIG. 2 according to an embodiment. [Figure 5]

[0033] FIG. 5 is a side perspective view of the slide carrier of the digital imager of FIG. 4 according to an embodiment. [Figure 6]

[0034] FIG. 6 is a side perspective view of the slide carrier of FIG. 5 with a slide rack loaded with slides attached to the slide carrier according to an embodiment. [Figure 7]

[0035] FIG. 7 is an enlarged front perspective view of the slide carrier bay of the digital imager of FIG. 4 according to an embodiment. [Figure 8]

[0036] FIG. 8 is an end view of the slide carrier of FIGS. 5 and 6 according to an embodiment. [Figure 9]

[0037] Figure 9 is a rear perspective view of the digital imager shown in Figure 4, according to one embodiment. [Figure 10]

[0038] Figure 10 is a front perspective view of the digital imager shown in Figure 4, according to one embodiment, with the enclosure panel removed to show the internal components. [Figure 11]

[0039] Figure 11 is a schematic diagram of the imager computer of the digital imager shown in Figure 4, according to one embodiment. [Figure 12]

[0040] Figure 12 is a schematic diagram of the chassis / enclosure assembly of the digital imager shown in Figure 4, according to one embodiment. [Figure 13]

[0041] Figure 13 is a schematic diagram of the slide carrier loading deck of the digital imager shown in Figure 4, according to one embodiment. [Figure 14]

[0042] Figure 14 is an enlarged front view of the slide handler of the digital imager shown in Figure 4, according to one embodiment. [Figure 15]

[0043] Figure 15 is a side view of the slide handler shown in Figure 14, according to one embodiment. [Figure 16]

[0044] Figure 16 is a front perspective view of the slide handler shown in Figure 14, according to one embodiment. [Figure 17]

[0045] Figure 17 is a schematic diagram of the slide handling gantry of the slide handler shown in Figure 14, according to one embodiment. [Figure 18]

[0046] Figure 18 is a schematic diagram of a gantry robot motion controller for the slide handler shown in Figure 14, according to one embodiment. [Figure 19]

[0047] Figure 19 is a front perspective view of the digital imager of Figure 4 according to one embodiment, with the panel, slide handler, and slide carrier deck removed to show the imaging station. [Figure 20]

[0048] Figure 20 is a partial front view (and a front cross-sectional view of the vibration absorber) of the digital imager shown in Figure 4, according to one embodiment, with the panel, slide handler, and slide carrier deck removed to show the imaging station. [Figure 21]

[0049] Figure 21 is a front view of an image acquisition subassembly of an imaging station for the digital imager shown in Figure 4, according to one embodiment. [Figure 22]

[0050] Figure 22 is a front cross-sectional view of the image acquisition subassembly shown in Figure 21, according to one embodiment. [Figure 23]

[0051] Figure 23 is an enlarged top view of the slide imaging station of the digital imager shown in Figure 4, according to one embodiment. [Figure 24]

[0052] Figure 24 is an enlarged perspective view of the slide imaging station of the digital imager shown in Figure 4, according to one embodiment. [Figure 25A]

[0053] Figure 25A shows the imaging plane of a conventional imaging system, where the imaging plane is a plane nominally parallel to the slide (and the specimen on the slide). [Figure 25B]

[0054] Figure 25B shows the inclined imaging surface of the digital imager shown in Figure 4, according to one embodiment. [Figure 25C]

[0055] Figure 25C shows the inclined imaging surface of the digital imager of Figure 4 for imaging the entire depth (Z dimension) of a specimen on a slide, according to one embodiment, and shows dimensions for determining the appropriate camera tilt angle. [Figure 26]

[0056] Figure 26 is a schematic diagram of the imaging station of the digital imager shown in Figure 4, according to one embodiment. [Figure 27]

[0057] Figure 27 is a front view of the lighting subsystem of the digital imager shown in Figure 4, according to one embodiment. [Figure 28]

[0058] Figure 28 is a front cross-sectional view of the lighting subsystem shown in Figure 27, according to one embodiment. [Figure 29]

[0059] Figure 29 is a schematic diagram of the system interface board of the digital imager shown in Figure 4, according to one embodiment. [Figure 30]

[0060] Figure 30 is an enlarged perspective view of the macrovision station of the digital imager shown in Figure 4, according to one embodiment. [Figure 31]

[0061] Figure 31 is a front view of the macrovision station of the digital imager shown in Figure 4, according to one embodiment. [Figure 32]

[0062] Figure 32 is a side cross-sectional view of the macrovision station of the digital imager shown in Figure 4, according to one embodiment. [Figure 33]

[0063] Figure 33 is a schematic diagram showing a meandering scan pattern used by the digital imager in Figure 4 to scan a specimen slide, according to one embodiment. [Figure 34]

[0064] Figure 34 is a block diagram of the workflow subsystem and digital imaging display system (review station) of the automated digital imaging system shown in Figure 1, according to one embodiment. [Figure 35]

[0065] Figure 35 shows a workflow scenario for reviewing digital images of specimen slides according to one embodiment. [Figure 36]

[0066] Figure 36 is a partial view of a worklist screen displayed by a digital image display system during the review process, according to one embodiment. [Figure 37]

[0067] Figure 37 is a partial view of a worklist screen displayed by a digital image display station selection box for selecting a case to review, according to one embodiment. [Figure 38]

[0068] Figure 38 is an overview of a screen displayed by a digital image display station selection box for selecting a case to review, according to one embodiment. [Figure 39]

[0069] Figure 39 shows the capability of a digital image display station to switch between different worklist screens according to one embodiment. [Figure 40]

[0070] Figure 40 shows a worklist screen having a simple list format displayed on a digital image display system according to one embodiment. [Figure 41]

[0071] Figure 41 shows a worklist screen having a tile / grid format displayed on a digital image display system according to one embodiment. [Figure 42]

[0072] Figure 42 shows a worklist screen having a sorted column tile / grid format displayed on a digital image display system according to one embodiment. [Figure 43]

[0073] Figure 43 shows a worklist screen with a detail panel displayed on a digital image display system according to one embodiment. [Figure 44]

[0074] Figure 44 shows a worklist screen with a graphical status indicator displayed on a digital image display system according to one embodiment. [Figure 45]

[0075] Figure 45 shows a review screen of a GYN ("Gynecology") case (in the case of a single slide) displayed on a digital image display system according to one embodiment. [Figure 46]

[0076] Figure 46 shows a review screen of an NGYN ("non-gynecological") case (in the case of multiple slides) displayed on a digital image display system according to one embodiment. [Figure 47]

[0077] Figure 47 shows another review screen of an NGYN case (in the case of multiple slides) displayed on a digital image display system according to one embodiment. [Figure 48]

[0078] Figure 48 shows the completion screen of a typical GYN case (single slide) displayed on a digital image display system according to one embodiment. [Figure 49]

[0079] Figure 49 shows the completion screen for an abnormal GYN case (in the case of a single slide) displayed on a digital image display system according to one embodiment. [Figure 50]

[0080] Figure 50 shows a worklist screen displayed on a digital image display system after a case has been completed, according to one embodiment. [Figure 51]

[0081] Figure 51 shows a worklist screen displayed on a digital image display system after a case has been completed, according to one embodiment, where the completed case is hidden from the worklist. [Figure 52]

[0082] Figure 52 shows another workflow scenario for reviewing digital images of specimen slides according to one embodiment. [Figure 53]

[0083] Figure 53 shows yet another workflow scenario for reviewing digital images of specimen slides according to one embodiment. [Figure 54]

[0084] Figure 54 shows yet another workflow scenario for reviewing digital images of specimen slides according to one embodiment. [Figure 55]

[0085] Figure 55 shows an overview of the roles and scenarios in the workflow shown in Figures 35 and 52-54, according to one embodiment. [Figure 56]

[0086] Figures 56A-56C illustrate the various administrator / management levels related to the roles and scenarios of the workflows shown in Figures 35 and 52-54. [Figure 57]

[0087] Figure 57 shows another review screen displaying a digital image of the entire specimen area shown on the digital image display system, according to one embodiment. [Figure 58]

[0088] Figure 58 shows another review screen according to one embodiment, configured to allow reviewers to add comments and mark / draw on images. [Figure 59]

[0089] Figure 59 shows another review screen configured according to one embodiment, which allows the reviewer to add recorded audio comments to the case of a digital image. [Figure 60]

[0090] Figure 60 shows another review screen according to one embodiment, configured such that a first reviewer shares their screen with a second reviewer on a different display system 108, and the first reviewer gives control to the second reviewer so that the review screen 434 on the first reviewer's screen can be controlled. [Figure 61]

[0091] Figure 61 shows another review screen configured according to one embodiment, which allows the reviewer to bookmark cases of digital images. [Figure 62]

[0092] Figure 62 shows a graphical representation of an image analysis algorithm for digital cytology according to one embodiment. [Figure 63]

[0093] Figure 63 shows a block diagram of an exemplary workflow server with backup and archiving functions according to one embodiment. [Figure 64]

[0094] Figure 64 shows a flowchart of a method for archiving slides using an archive service module according to one embodiment. [Figure 65]

[0095] Figure 65 shows an example of a calibration table for performing distortion correction of digital images according to one embodiment. [Modes for carrying out the invention]

[0029]

[0096] Figures 1 and 2 show an automated digital imaging system 100 for digitally imaging multiple specimens on a substrate 102 (see Figure 3), on which the specimens are placed. For example, the substrate 102 is a microscope slide as shown in the embodiments described, but may be any other suitable substrate 102, such as a microplate, microarray, or other suitable medium. The embodiments of the digital imaging system 100 described utilize a microscope slide, and therefore the substrate 102 is referred to as a slide, but it should be understood that the digital imaging system 100 is not limited to using a slide and may utilize any suitable substrate 102. Accordingly, the term “slide” as used herein means any suitable substrate on which a specimen is fixed, including a microplate, microarray, or other suitable medium. As used herein, the term “microscope slide” means a thin, flat piece of glass or plastic used to hold an object for examination under a microscope. Specimens may include any type of specimen to be digitally imaged under microscopic magnification, such as biological or chemical specimens, including both cytological and pathological tissue specimens. For the purpose of illustrating this imaging system 100, the specimen used is a cytological specimen containing individual cells dispersed within a certain dimensional volume. However, it should be understood that this system can be used in the same way for other types of specimens, such as pathological tissue specimens, as it is for cytological specimens.

[0030]

[0097] The automated digital imaging system 100 is designed to acquire high-resolution digital images of the entire specimen (or multiple specimens) on slide 102, enabling the review of the specimen using the digital image of the specimen rather than the physical slide glass 102. An optical microscope is not used in the normal review workflow of the digital imaging system 100. Instead, a digital image of the entire specimen on slide 102 is acquired and made available for viewing on the computer monitor 109 of the review station 108 (see Figure 2). The digital imaging system 100 may also include an image processing software application 137 (installed on the imager computer 105 and / or workflow subsystem 106 of the imager 104) that embodies an image processing algorithm configured to analyze the digital image of the slide specimen and identify objects of interest (OOIs), i.e., individual cells in the case of cytological specimens, or individual tissue structures in the case of pathological solid tissue specimens. Identified OOIs are then used to assist users (e.g., cytologists or cytopathologists) in quickly and accurately reviewing digital images of slide specimens, as described in U.S. Patents 7,587,078, 6,665,060, 7,006,674 and 7,590,492 above. The term "OOI" means that an object is determined to have characteristics that indicate that the object (individual cells or tissue structures, in some cases) may be useful in the diagnosis or treatment of a patient, and does not necessarily mean that such an object is determined to be of genuine interest or actual importance to a cytologist or cytopathologist.

[0031]

[0098] The automated digital imaging system 100 includes one or more imager subsystems 103 (each including an imager 104 and an imager computer 105) and one or more review stations 108, functionally connected to a workflow subsystem 106 (also called the “workflow server 106”). Generally, each imager 104 is a benchtop system for capturing digital images of slides 102. The imager 104 processes batches of slides in an automated manner. The workflow server 106 is a computer server that primarily functions as a high-capacity disk storage system for storing and managing digital images and associated slide data. The workflow server 106 also includes software for image processing and data management and may also provide networking capabilities. The review station 108 is a workstation that includes a computer and monitor for accessing digital images from the workflow server and displaying the digital images for diagnostic review of the images (OOI and / or whole specimen images).

[0032]

[0099] As shown in Figure 1, the imager subsystem 103 communicates data with the workflow subsystem 106 via a communication network that may include one or more of the following: a local area network (LAN, e.g., Ethernet), a wide area network (WAN), the internet (e.g., a virtual private network (VPN)), or other suitable networks. Similarly, the review station 108 communicates data with the workflow subsystem 106 via a communication network that may include one or more of the following: a local area network (LAN, e.g., Ethernet), a wide area network (WAN), the internet (e.g., a virtual private network (VPN)), or other suitable networks.

[0033]

[0100] The imager 104 in the described embodiment is designed to work with a microscope slide 102. Referring to Figure 3, an exemplary embodiment of the microscope slide 102 is shown. The microscope slide 102 is a rectangular glass plate 110 (or other suitable material) having a slide identification area 112, a specimen area 114, and a reference mark 116. The microscope slide 102 may be a standard-sized microscope slide, approximately 75 mm × 25 mm, or other suitable size. The microscope slide 102 may have beveled corners to facilitate handling and positioning of the slide 102. The specimen area 114 may be a circle with a diameter of up to approximately 22 mm. The entire specimen area on the microscope slide 102 can also be imaged. The slide identification area 112 may be up to approximately 25–28 mm in length. A barcode, ID number, and / or other information may be printed on the slide identification area 112. The specimen area 114 is left as a transparent area of ​​the slide glass. Reference marks 116 can be used by the imager 104 as reference points on slide 102 to identify the position and / or orientation of slide 102, and its characteristics to the imager 104. Specimen 119, which contains multiple objects dispersed within a three-dimensional volume, is usually fixed within the specimen area 114 of slide 102, but in some cases, the specimen may extend outside the specimen area 114. The three-dimensional volume has length (l), width (w), and thickness or depth (d). The thickness (d) defines the z-axis with respect to the surface of slide 102. The three-dimensional volume may or may not have substantially uniform shape, including thickness, length, and / or width. The volume of specimen 119 is uniform, i.e., one or more dimensions of the specimen vary by less than 10%, less than 15%, or less than 20%. Specimen 119 may be any suitable specimen, such as a cytological specimen where the objects are cells, or a solid tissue sample where the objects are tissue structures.

[0034]

[0101] As shown in Figure 3, a coverslip 115 can be used to cover the specimen 119 within the specimen area 114. This specimen coverslip 115 is transparent enough to acquire a micro-image of the specimen through the coverslip 115. In other words, the coverslip 115 does not interfere with the imager 104 when acquiring micro-images and / or macro-images through the coverslip 115 using the imaging station 190 and / or macrovision station 232. The coverslip 115 serves to preserve the specimen 119, protect it from contamination of itself and others, and also holds the specimen 119 flat in place. The coverslip 115 has a thickness of 117.

[0035]

[0102] Referring to Figures 4 and 5, slide carriers 118 are used to load multiple slides 102 into the imager 104. As shown in Figure 4, the imager 104 has a capacity of 10 slide carriers 118, but may be configured to hold any appropriate number of slide carriers 118. Each slide carrier 118 has two rack holders 202, each of which accepts a slide rack 120. Each slide rack 120 may hold 20 slides 102, or 40 slides 102, or any other appropriate number of slides 102. The slide racks 120 may be standard off-the-shelf racks such as Sakura® racks or Leica® racks. Thus, each slide carrier 118 holds 40 slides, and the total capacity of the imager 104 is 400 slides (10 slide carriers 118 × 40 slides per carrier 118). The slide carriers 118 have handles 122 for carrying the slide carriers 118 and for inserting and removing the slide carriers 118 into and from the imager 104. As shown in Figures 7 and 8, the slide carrier 118 has a T-slot 124 that slides onto a T-rail 126 mounted on the slide carrier deck 128. The slide carrier 118 is installed on the imager 104 by aligning the T-slot 124 with the T-rail 126 and sliding the slide carrier 118 onto the slide carrier deck 128. The slide carrier 118 is removed from the imager 104 by pulling the slide carrier 118 straight off the slide carrier deck 128.

[0036]

[0103] Referring here to Figures 2 and 11, the imaging subsystem 103 includes an imager 104 and an imager computer 105. The imager 104 and the imager computer 105 communicate with each other via a network connection such as an Ethernet connection (e.g., a 10GE optical connection). The imager computer 105 can be any suitable computer, such as a personal computer having one or more computer processors (CPUs) and graphics processors (GPUs). For example, one computer suitable for the imager computer 105 is schematically shown in Figure 11. This imager computer 105 may have one or more of the following specifications:

[0104] a. Motherboard 130 - Dual-processor server class. This dual-processor configuration is understood to be suitable for software intended to run on imager subsystem 103;

[0105] b. CPU - Intel® Xeon® class or higher;

[0106] c. At least 64 Gb of DRAM;

[0107] d. Two dedicated GPUs, such as Nvidia® RTX-5000 or higher. 132. Ensure sufficient processing speed for image capture and generating slide datasets (SDS304, see Figure 34) for each slide captured by the imager subsystem 103;

[0108] e. A local hard drive 133 for storing a software program 135, including an image processing software application 137 (detailed below);

[0109] f. Local mass data storage such as hard drives and solid-state drives 134;

[0110] g. Network and USB input / output (I / O) 136;

[0111] h.Power supply 138;

[0112] i. Uninterruptible power supply (UPS) 140;

[0037]

[0113] Input / Output 136 includes one or more of the following:

[0114] a. Imager VGA video, touchscreen USB, and computer audio are provided to imager 104. Imager 104 receives VGA video from imager computer 105 and drives imager display monitor 168. A dedicated USB port on the imager computer provides the touchscreen I / O for imager display monitor 168. Imager computer 105 provides audio to imager 104, which is amplified there. Imager speakers are connected to the audio amplifier.

[0115] b. USB ports are provided for the manufacturing and service support of keyboards and mice. Under normal operation, the keyboard and mouse are not connected and are not required for normal system operation of the imager. An additional USB port is located on the front of the imager for easy user access.

[0116] c. The imager's computer power is obtained from the mains power supply or from an external smart UPS140 (if connected) via the imager IEC entry connector.

[0117] d. A dedicated USB port is provided for communication with the smart UPS. In the event of a mains power failure, the UPS communicates the event to the imager computer 105, which then initiates a controlled power-down to maintain the integrity of the computer software, data, and configuration.

[0038]

[0118] As described above, the imager computer 105 communicates data with the workflow subsystem 106 via a communication network such as a 1Gb Ethernet wired network connection.

[0039]

[0119] Turning to Figures 4, 9, and 10, the imager 104 is a benchtop digital imaging device that functions as an input device for the entire digital imaging system 100. The imager 104 captures digital images of batches of slides 102 loaded into the imager 104 in a slide carrier 118 in an automated manner with little to no user intervention. The imager 104 comprises a chassis assembly 142, a base assembly 148, and an enclosure assembly, which provide a framework for mounting and housing the imager components. The chassis / enclosure assembly 142 can be configured to be a benchtop device with dimensions of no more than 36 inches wide × 28 inches deep × 25 inches high. The imager 104 may weigh no more than 200 pounds when unloaded with slides 102 or slide carrier 118. These are merely preferred dimensions and weights and do not necessarily limit the imager 104 unless specified in the claims. Therefore, the imager 104 may have other maximum dimensions and weights that match its intended use.

[0040]

[0120] The imager 104 includes a chassis assembly 142 mounted to a base assembly 148 and an enclosure assembly 146. The imaging station 190 includes a base 144 and a vibration-damping imager base plate 152 (see Figure 19) to which the imager components are mounted. The imaging station 190 is mounted to the base assembly 148. The enclosure assembly 146 includes an enclosure left panel 154, an enclosure right panel 156, an enclosure rear panel 158, an enclosure top panel 159, an enclosure left front window 160, an enclosure right front window 162, an enclosure right panel 164, and a sliding carrier deck door 166. Each of the enclosure components is mounted to the chassis assembly 142 and / or another enclosure component. The enclosure left front window 160 is transparent or translucent so that the internal imager components are visible through the enclosure left front window 160. The enclosure left front window 160 is pivotally mounted so that it can be opened and closed to provide access to the internal imager components. The enclosure right front window 162 is transparent or translucent so that other internal imager components can be seen through the enclosure right front window 162. The enclosure left panel 154 has a left window 170, through which the internal imager components can be seen. The enclosure top panel 159 may also have a top window 172 so that the internal imager components can be seen through the top window 172. The enclosure display panel 168 is pivotally mounted so that it can be opened and closed to provide access to the internal imager components. The slide carrier deck door 166 is pivotally mounted so that it can be opened and closed to provide access to the slide carrier deck 128 for loading and removing the slide carrier deck 128 (see Figures 4 and 10).

[0041]

[0121] The enclosure assembly 146 also provides inputs and outputs for the imager 104, user access, and environmental control. Figure 12 is a block diagram of the enclosure assembly 146. The enclosure assembly 146 provides air management to ensure sufficient airflow inside the imager 104, maintain the internal operating temperature below its maximum value (e.g., below 40°C), and maintain internal positive pressure (when the slide carrier deck door 166 is closed). The enclosure assembly 146 provides external signal and power connections.

[0042]

[0122] The imager display monitor 168 may be an LCD touchscreen display or other suitable display. The imager display monitor 168 indicates the operation and status of the imager 104 and also provides operator control via the touchscreen (or other input device). As shown in Figure 12, the imager display monitor 168 is connected to the imager computer 105 via a bulkhead connector on the enclosure assembly 146. The imager computer 105 provides VGA video, touchscreen USB, and computer audio to the imager 104. The imager 104 receives VGA video from the imager computer 105 and drives the imager display monitor 168. A dedicated USB port on the imager computer provides I / O to the touchscreen of the imager display monitor 168. The imager computer 105 also provides audio to the imager 104, which is amplified there. The imager speaker is connected to the audio amplifier.

[0043]

[0123] Continuing to refer to Figure 12, the enclosure assembly 146 also includes a window sensor for detecting the open / closed state of the left front window 160 of the enclosure, and a window lock for locking the left front window 160 of the enclosure in the closed position during imaging operations of the imager 104. The enclosure assembly 146 also includes a door sensor for detecting the open / closed state of the slide carrier deck door 166, and a door lock for locking the slide carrier deck door 166 in the closed position during imaging operations of the imager 104.

[0044]

[0124] Next, looking at the internal components of the imager 104, the slide carrier deck 128 is located near the bottom of the imager on the base 148. The slide carrier deck 128 has multiple slide carrier bays, such as 10 slide carrier bays, 20 slide carrier bays, or any other appropriate number of slide carrier bays, and each slide carrier bay is configured to receive each slide carrier 118. The slide carrier deck 128 is open at the top, providing access to the slides 102 contained in the slide carriers 118 loaded into the slide carrier deck 128 by the slide handler 176. The slide handler 176 removes the slides 102 from the slide carrier 118, moves the slides 102 to various stations in the imager 104, and inserts the slides 102 back into the slide carrier 118.

[0045]

[0125] Referring to Figure 13, the slide carrier deck 128 has a carrier lock 173 and a carrier presence sensor 175 for each slide carrier bay 174. The carrier lock 173 may be a solenoid actuated lock or other suitable power locking mechanism, and the carrier presence sensor may be an optical sensor or other suitable sensor for detecting the presence of slide carriers 118 in each slide carrier bay 174. The carrier lock 173 and carrier presence sensor 175 communicate data with the imager computer 105 via the system interface board 182 (see Figure 29, SIB, described later).

[0046]

[0126] In Figures 10 and 14-16, the slide handler 176 automatically moves the slide 102 between various stations of the imager 104, including between the slide carrier 118, the imaging station 190, and the macrovision station 232. The slide handler 176 includes a support platform 180 and a slide handling gantry 178 movably coupled to the support platform 180. The slide handling gantry 178 is a pick-and-place robot capable of moving the slide gripper 184 with three degrees of freedom: lateral movement (X-axis), vertical movement (Z-axis) (as shown in Figure 14), and forward / backward movement (Y-axis) (as shown in Figure 15). The slide handling gantry 178 can also rotate the slide gripper 184 around the theta axis (see Figure 15), opening and closing the slide gripper 184 to grip and release the slide 102. The slide handling gantry 178 has motion mechanisms for providing movement and / or operation of the slide handling gantry, including an X-axis mechanism 254, a Y-axis mechanism 264, a Z-axis mechanism 274, a theta-axis mechanism 290, and a gripper mechanism 185. The X-axis mechanism 254 includes a motor 256, an encoder 258, a home sensor 260, and a drive mechanism 262. The Y-axis mechanism 264 includes a motor 266, an encoder 268, a home sensor 270, and a drive mechanism 272. The Z-axis mechanism 274 includes a motor 276, an encoder 278, a home sensor 280, and a drive mechanism 282. The theta-axis mechanism 290 includes a theta motor 292, a theta encoder 294, a theta home sensor (not shown), and a theta drive mechanism (not shown). Each motion mechanism includes a motor, encoder, home sensor, and drive mechanism and can meet the requirements of movement, traversal acceleration, velocity, and payload of the motion envelope. The gripper 184 also has an optical sensor 188 (slide stock sensor 188) used to detect the presence or absence (stock) of slides 102 in the slide carrier 118, and the presence or absence of slide racks 120 in the slide carrier 118.The gripper 184 may be a ready-made part designed to grip the slide 102 from the edge of the slide, such as one manufactured by SMC® Electric Gripper® in Los Angeles, California.

[0047]

[0127] As shown in Figure 17, the slide handling gantry 178 is functionally connected to a gantry robot motion controller 186 that controls the motion of each motion mechanism. The gantry robot motion controller 186 communicates with the imager computer 105 via a Controller Area Network (CAN), and the Controller Area Network (CAN) sends data and commands to the gantry robot motion controller 186 to control the operation of the slide handling gantry 178. Figure 18 shows a schematic diagram of the gantry robot motion controller 186.

[0048]

[0128] Referring here to Figures 10 and 19-26, the imager 104 has an imaging station 190 for capturing digital microimages of slides 102 (i.e., specimens 119 on each slide 102) using a digital camera 192, a tube lens assembly 194, a mirror 196, and an objective lens 198. The camera 192 uses a Sony CMOS Pregius GenII sensor and has a resolution of 4096 × 2160 pixels. The optical path provides an optical resolution of 0.255 μm / pixel. Because the camera's field of view ("FOV") is much smaller than the entire specimen area 114 of slide 102, the imager 104 needs to take many microimages to capture the entire specimen area 114. For example, the field of view of the camera 192 may be 1044 μm × 550 μm, or about 1 mm × 0.5 mm, while the specimen area 114 of slide 102 may have a diameter of 21-22 mm. Based on these specifications, camera 192 requires approximately 885 non-overlapping images to image the entire specimen area 114. However, to acquire a focused image of all objects within the three-dimensional volume of specimen 119, a step size equal to the image sub-region equal to the depth of field of the imaging station 190 is used, resulting in overlapping images and thus capturing many more frames. For example, as described later, a micro-image is taken every 38.25 μm = ~1 / 14 of the frame height, so the actual number of images captured during a scan of the entire specimen area is approximately 11,000, or more than 10,000, or more than 5,000, or between 5,000 and 15,000. Camera 192 acquires micro-images while the XY slide stage 210 moves specimen 119 at a constant speed. In this example, camera 192 takes an image approximately every 38 μm of stage movement in the scanning direction, and therefore every camera field of view contains 14 images per FOV. The XY slide stage 210 provides relative motion between the camera 192's field of view and the slide 102, so that the camera 192 can scan the entire specimen area 114 of the slide 102 and combine to take multiple separate digital images that cover the entire specimen area 114.

[0049]

[0129] As described herein, individual microimages can be combined using image processing software application 137 (see Figure 11) to create a composite image of the entire specimen region 114.

[0050]

[0130] As shown in Figures 19-22, the imaging station 190 includes a high-resolution digital camera 192, which is part of the image acquisition assembly 191 (which is a microscope camera). The image acquisition assembly 191 can be angled in the range of 0° to 3°, 0° to 10°, or 0° to 5°. The camera 192 communicates with the imager computer 105 via a high-speed network such as a 10GE Ethernet optical network, or another suitable high-speed network capable of transmitting high-resolution microimages from the camera 192 to the imager computer 105 with sufficient throughput during the image scanning process.

[0051]

[0131] A tube lens 194 is positioned in the optical path of the camera 192. The tube lens 192, in combination with the objective lens 198, sets the optical magnification of the camera 192. A first folding mirror 196 is positioned in the optical path of the camera 192 after the tube lens 194 and is angled approximately 45° with respect to the optical imaging axis of the camera. The angle and position of the first folding mirror 196 can change the collinearity between the optical axis of the objective lens and the center of the camera 192. The first folding mirror 196 is located in the imaging path and must have sufficient quality (e.g., reflectivity greater than 90% and precision flatness greater than λ / 4), otherwise it will directly affect image quality by blurring light at the detector in all or part of the field of view and / or cause distortion.

[0052]

[0132] The objective lens 198 is located after the optical path of the first folding mirror 196. The optical axis of the objective lens 198 is at an angle of approximately 90° from the optical axis of the camera by the first folding mirror 196. The objective lens 198 is mounted on the imaging Z-stage 200. The Z-stage 200 has threads for mounting the objective lens 198. The Z-stage 200 can controllably move the objective lens 198 to adjust the focus of the camera 192 on the slide 102 (i.e., adjust the focal plane relative to the slide 102). The position of the Z-stage can be set via the Z-stage controller. The slide 102 (i.e., the specimen 119 on the slide 102) is positioned in the front focal plane of the objective lens 198, and the objective lens 198 projects the image of the slide 102 onto the camera 192 via the first folding mirror 196 and the tube lens 194.

[0053]

[0133] As shown in Figures 21 and 22, all components of the image acquisition subassembly 191, including the digital camera 192, tube lens 194, first folding mirror 196, objective lens 198, and Z-stage 200, are mounted on the image acquisition base plate 201. The image acquisition base plate 201 has several image subassembly alignment adjusters 203, which allow for positional and / or angular adjustment of the image acquisition subassembly 191 relative to other components of the imaging station 190, as well as relative positional and / or angular adjustment of the components of the image acquisition subassembly 191.

[0054]

[0134] In Figures 19-20 and 23-24, slide 102 is located behind the objective lens 198 in the optical path of camera 192. Slide 102 is held on a slide holder 202. The slide holder 202 has a slide recess 204 for holding slide 102 in a horizontal position on the slide holder 202. The slide holder 202 has a first slide justifier arm 206, which is spring-loaded to deflect slide 102 in the Y direction within the slide recess 204. The slide holder 202 has a second slide justifier arm 208, which is spring-loaded to deflect slide 102 in the X direction within the slide recess 204. The first and second slide justifier arms 206 and 208 firmly hold slide 102 in the correct position on the slide holder 202 so that slide 102 does not move during scanning. The imaging station 190 has a slide release bracket 209 mounted on an XY stage platform 214 which is stationary relative to the slide holder 202. The slide holder 202 is moved relative to the slide release bracket 209 to operate the first and second slide justifier arms 206, 208 to release or engage the first and second slide justifier arms 206, 208, thereby holding or releasing the slide 102 within the slide holder 202.

[0055]

[0135] As best shown in Figure 20, the optical axis of the objective lens 198 is tilted at an angle of 205 with respect to the orthogonality of the plane of the slide 102 (and slide recess 204) in the scanning direction (see Figure 33). In other words, the resulting optical axis of the camera 192 and the optical system on the surface of the slide (also simply called the “optical axis of the camera 192”) are non-orthogonal to the plane of the slide 102. As will be described in more detail herein, the angle of 205 allows the imager 104 to acquire a volume image of the specimen 119 on the slide 102 (i.e., an image extending to the depth of the specimen 119). In other words, the microimage includes focused images of features at different depths of the specimen 119 on the slide 102, rather than just a single focal plane as in the case of an image taken at an orthogonal angle of the specimen 119. The imaging station 190 may be configured so that each microimage acquires a microimage that includes at least a portion of the depth of the slide 102 below the surface of the slide 102. When a coverslip 115 is used on slide 102, the imaging station 190 may be configured to acquire a microimage in which each microimage includes at least a portion of the depth of the coverslip 115.

[0056]

[0136] As described herein, these microimages can be processed with image processing software application 137 (see Figure 11) to stitch the microimages together and flatten the images to obtain a two-dimensional composite image of specimen 119 in focus at all depths of specimen 119. For example, U.S. Patent Application Publication 2009 / 0295963 describes a method for stitching images together to form a single digital image 302 (see, for example, Figure 34 showing a block diagram of the digital image 302). As shown in Figure 25, a suitable tilt angle 205 for imaging the entire depth (Z dimension) of specimen 119 on slide 102 is a function of the specimen thickness (d) and the projected field of view (d1) of camera 192 on slide 102. The projected field of view (d1) is the in-plane field of view of the camera sensor divided by the combined optical magnification of camera 192 (e.g., including the tube lens 194, objective lens 198, and other magnifications in the optical path). As shown in Figure 25C, the tilt angle 205 is the arcsine (d / d1). For example, if the specimen depth (d) is 10 micrometers, the in-plane field of view is 8 millimeters, and the combined optical magnification is 40x (i.e., d1 = 8 mm / 40 = 0.20 mm), then the tilt angle 205 is the arcsine [0.01 mm / (8 mm / 40)], which is approximately 2.86 degrees. In another example shown in Figure 25C, if the specimen depth (d) is 24 micrometers and the effective in-plane field of view (d1) is 0.5 millimeters (e.g., sensor field of view 20 millimeters, optical magnification 40x), then the tilt angle 205 is 2.75 degrees. In general, for typical specimen depths, camera sensor dimensions, and magnifications, the tilt angle 205 is typically between approximately 2 and 10 degrees.

[0057]

[0137] The slide holder 202 has a slide holder base plate 212 attached to the XY slide stage 210. The slide holder base plate 212 has a slide holder angle adjustment section 213 so that the orientation of the slide holder base plate 212 can be adjusted relative to the XY slide stage 210 (and also relative to the optical axis of the objective lens 198).

[0058]

[0138] The XY slide stage 210 moves the slide holder 202 and slide 102 in a scanning pattern during the imaging procedure to acquire an image covering the entire sample area 114 (or other area of ​​interest) of slide 102. As shown in Figure 26, the XY slide stage 210 includes motors, encoders, and movement limits in the X and Y axes, respectively. The XY slide stage 210 is connected to an XY stage driver 211, which is connected to an XY stage controller 215 for controlling the movement of the XY slide stage 210. The XY stage controller 215 communicates data with the imager computer 105 via an Ethernet network connection, allowing the imager computer 105 to control the operation of the XY slide stage 210.

[0059]

[0139] The XY slide stage 210 is mounted on the XY stage platform 214, which is mounted on the imager base plate 152. The XY stage platform 214 has an XY stage angle adjustment unit 218, which allows adjustment of the orientation of the XY stage platform 214 relative to the imager base plate 152.

[0060]

[0140] Therefore, the slide holder angle adjustment section 213 and the XY stage angle adjustment section 218 can be adjusted to appropriately align the movement of the slide holder 202 and the XY slide stage 210 horizontally or at a desired angle with respect to the optical axis of the objective lens 198 (i.e., the optical axis of the camera 192). Typically, the movement of the slide holder 202 and the slide stage 210 are aligned to be parallel in order to provide a constant focal height of the microimage during scanning, so that the image acquisition assembly 191 acquires a microimage of the specimen 119 at the same height (e.g., a single z-axis focal height) relative to the surface of the slide 102.

[0061]

[0141] The imager baseplate 152 is isolated from the scanner chassis 144 via multiple (in this case, four) vibration isolation mounts 221. The vibration isolation mounts 221 provide vibration isolation to the imaging station 190.

[0062]

[0142] The objective lens 198 is focused on the slide 102 held in the slide holder 202 using the Z-stage 200, as shown in Figures 19-22 and 24. As shown in Figure 26, the Z-stage 200 includes a piezo focus n-point controller 217 and a piezo focus n-point stage 219 for controlling the Z-stage and thus controlling the focus of the objective lens 198.

[0063]

[0143] As shown in Figures 23 and 24, the microscope calibration component 207 (also called the “microscope calibration target”) is positioned within the slide holder base plate 212 so that the microscope calibration component 207 moves with the XY slide stage 210. The microscope calibration component 207 may be directly mounted to the slide holder base plate 212 or may be integrated with the slide holder base plate 212 so as to be defined by the slide holder base plate 212 and not removable from it. For example, the microscope calibration component 207 may be formed or etched within or through the slide holder base plate 212. Alternatively, the microscope calibration component 207 may be a separate component from the slide holder base plate 212, such as being mounted in a cavity defined by the slide holder base plate 212. The microscope calibration component 207 may be configured to perform both positional and optical calibration of the microscope camera assembly 191 (also called the image acquisition subassembly 191). Alternatively, the microscope calibration component 207 may be configured to perform only the optical calibration of the microscope camera subassembly 191. For positional calibration, the calibration component 207 is configured to measure or determine positional calibration parameters including one or more of the following: the "X" position of the XY slide stage 210, the "Y" position of the XY slide stage 210, the "Z" position of the XY slide stage 210, camera-stage alignment, changes in one or both of the "X" and "Y" positions of the XY slide stage 210 during microimaging of slide 102, and / or review and changes in the "Z" position of the XY slide stage 210 during microimaging of slide 102.In the case of optical calibration, the calibration component 207 is configured to measure or determine optical calibration parameters of the microscope camera subassembly 191, including, for example, one or more grayscale linearity, magnification, signal-to-noise ratio, illumination variation, modulation transmission function (MTF), checking for stationary pixels or elements of the digital camera 192 (e.g., CMOS or charge-coupled device (CDD)), illumination uniformity or homogeneity, and detection of artifacts such as dust or dirt that reduce the signal-to-noise ratio. The microscope camera subassembly 191 is then calibrated and adjusted as necessary based on the positional and / or optical calibration parameters measured or determined using the calibration component 207. An example of a suitable microscope calibration component 207 is disclosed in U.S. Patent No. 7,848,019B2.

[0064]

[0144] Referring to Figures 27 and 28, the imaging station 190 also includes an illumination module 222 to provide slide 102 with enough uniform light for the objective lens 198 to collect so that the camera 192 can produce a high-quality image. The illumination module 222 includes a light source 224 (e.g., an LED 224), a first lens 226, a second lens 227, a third lens 229, a first aperture 223, a second aperture 225, a second folding mirror 228, a tube lens mount 237, and an adjustable focusing lens 230 necessary to project the LED light onto the bottom of slide 102. The illumination module 222 projects uniform light to achieve uniform illumination at the focal plane of the specimen. The term “uniform light” means light that is substantially uniform in color and brightness. The term “uniform illumination” means that the specimen 119 is substantially illuminated with light of substantially uniform color and brightness. The lighting module 222 is powered by a mains power supply wired via the SIB. The light level of the lighting module 222 is set by feedback from the camera to achieve the target background brightness. The lighting module 222 interfaces with the lighting interface on the SIB 182 (described later). The lighting module 222 has a Y adjustment screw for adjusting the illumination in the Y direction in the XY plane relative to the image acquisition assembly 191, and an X adjustment screw 235 for adjusting the illumination in the X direction relative to the image acquisition assembly 191 in the XY plane.

[0065]

[0145] The lighting module 222 includes a light source 224 (e.g., an LED), whose power, spectral distribution, and radiance profile serve as the starting point for all the uniformity, quality, and quantity of illumination required to produce high-quality micro-images.

[0066]

[0146] Next, the scanning process for acquiring a micro-image of slide 102, including focused images of features of specimen 119 at different depths, using the scan station 190, is described. As mentioned above and shown in Figure 25, the imaging station 190 captures a micro-image of specimen 119 on slide 102 by tilting the focal plane of the camera 192 relative to the surface of specimen 119 on slide 102 such that the focal plane extends through the depth of slide 102. Thus, each micro-image captured by the camera 192 is taken at an angle to the surface of specimen 119 (i.e., at an angle to the surface of slide 102), and therefore includes an image of the depth of specimen 119 on slide 102. As shown in Figure 33, slide 102 is moved by the XY slide stage 210, and the field of view of the camera 192 is scanned over the entire specimen area 114 (or a predetermined area of ​​the entire actual specimen 119, for example, if the actual specimen 119 covers a different area from the specimen area 114 and the boundaries of the actual specimen 119 are predetermined). The XY slide stage 210 moves slide 102 along a meandering path, capturing micro-images of the swath of specimen 119 at each path. To capture the swath, the XY slide stage 210 moves slide 102 continuously, and the camera 192 is triggered to capture images when it reaches each trigger point according to the position of the stage encoder. The meandering path, in which the start of each consecutive swath is close to the end of the previous swath, minimizes the time required to scan the entire specimen 119. As slide 102 moves along the swath, the camera 192 captures micro-images of specimen 119.

[0067]

[0147] To maintain an acceptable focus during the imaging scan, the best focal plane of slide 102 can be estimated using the measurement of the reference mark 116 printed on slide 102 and the offset defined by the measurement of the best focal offset of sample 119 from the reference plane. The coefficients defining the offset plane are then uploaded to the XY stage controller 215, which maintains the position of the Z stage 200 according to these coefficients during the scan. During the imaging scan, scan quality is monitored in various ways, including focus quality metrics, stage tracking error, and image quality metrics. Image quality metrics may include one or more of the following, which can be measured by digital image algorithms known to those skilled in the art: sharpness, noise, dynamic range, tone reproduction, contrast, color accuracy, distortion, and vignette.

[0068]

[0148] As shown in Figure 25A, in conventional imaging systems, the imaging plane is nominally parallel to the plane of the slide (and to specimen 119 on the slide) (i.e., parallel within manufacturing tolerances). For each adjacent field of view, taken at intervals approximately the length of the camera's field of view, a single best focal plane is captured (micro-images may overlap slightly to aid in image stitching). The micro-images are then stitched together to create a composite image of the entire specimen 119. Note that because the heights of the objects on the slide glass within the three-dimensional volume of specimen 119 vary, not all objects will be captured in focus in a single imaging pass.

[0069]

[0149] In contrast, as shown in Figure 25B, the image capture method of imager 104 utilizes an imaging plane tilted relative to the specimen 119 on slide 102. Because the imaging plane is tilted relative to the specimen 119 on slide 102, different subregions of the camera pixel array capture images of specimens at different heights. Therefore, at imaging position #1, object A is captured in focus, but in the same camera frame, object B is near the right edge of the frame and is considerably out of focus. Instead of stepping the camera position by the length of the field of view in the scan direction between images, a step size equal to the image subregion corresponding to the depth of field of the imaging optical system is used. The series of images overlap. When the camera is at position #1, object A is in focus, but when the camera reaches position #4, object B is captured in focus. In this way, all objects within a height range equal to the difference between the left and right edges of the imager's focal plane are captured in focus in a given image.

[0070]

[0150] Referring to Figure 25C, in one example, the camera pixel array may be divided into 14 sub-regions representing different focal planes over a sample depth of 24 μm. Once the same sample region (tile) is captured in each camera sub-region, those sub-regions are reconstructed vertically to create a tile focus stack. The stack is then transformed into a single-plane merged focus image by selecting the plane pixel with the best focus quality using an algorithm for determining pixel focus with a focus metric such as that taught in U.S. Patent No. 7,769,219, comparing each pixel with adjacent pixels in the same plane. Examples of identifying the pixel or other object with the best focus quality are also described in U.S. Patent No. 7,736,304. A technique for determining the best focus of a slide reference mark is taught in U.S. Patent No. 8,116,550. The number of sub-regions is merely an example, and the digital imaging system 100 can use any appropriate number of sub-regions for image processing.

[0071]

[0151] The following describes an exemplary method of processing images using the GPU 132 to obtain a merged image of the entire sample 119 on slide 102. Image processing involves many steps to obtain raw images and convert them into a final merged compressed image. The color camera 192 can utilize a Bayer mask (a color filter array for arranging RGB color filters on a grid of photosensors). In such cases, the first step of the image processing algorithm is to debayer the image and convert it into an RGB image (a bitmap image holding the red / green / blue values ​​for each pixel). The image is then corrected for uniformity and distortion using a pre-mapped calibration table. In one example, the calibration table is an array of camera image sizes. This table contains entries for each pixel position of the corrected image result. Each entry represents a 2x2 pixel subregion of the source image and weight coefficients applied to those source pixels and summed to obtain the resulting pixel values. This allows the corrected image to represent (by interpolation) the subpixel shifts necessary to correct for small optical distortions measured during the calibration process. Figure 65 shows an example of a pre-mapped calibration table when positional distortion correction is 1 / 2 pixel in all directions (up, down, left, and right).

[0072]

[0152] Once all sub-region images of a tile (e.g., 14 sub-regions) are captured, the merged image is calculated. The difference between each pixel and a specific pixel in its adjacent region is calculated. This difference is then weighted based on the pixel's value to determine the relative merit of that particular plane to that particular pixel, and tiles with plane value values ​​(e.g., 0-13) are generated. Next, a moving average is applied to the plane value tiles to improve transitions between objects in the final image. Then, pixels in the merged image are selected based on their associated plane values. The merged images are tiled to generate swaths. Once a complete swath is formed, it is stitched together with the previous swath. Once the entire image of sample 119 is stitched together, the entire image is compressed to a ratio of approximately 20:1 using JPEG, JPEG2000, or another suitable compression algorithm. A pyramid is then created for the image, which, along with the slide metadata, is transferred to the workflow subsystem 106, where the review station 108 can access it for review. Slide metadata refers to non-image information associated with the entire specimen image, such as the slide identifier (a barcode read from the slide) and the date and time of acquisition. Slide metadata may include a list of OOI308 locations identified by image analysis algorithms. An example of a "pyramid" is a set of low-resolution images obtained by repeatedly reducing the size of the original complete specimen image by 50% until it reaches a predetermined minimum size. The purpose of this pyramid is to support more efficient rendering on the review screen. If the reviewer is viewing a zoomed-out, low-resolution version of the image, data can be provided by accessing the appropriate level of pyramid image data, which requires less data transfer than sending the full-resolution image and then reducing its size at the review station 108.

[0073]

[0153] As shown in Figures 30-32, the imager 104 also has a macrovision station 232 for acquiring identification information for each slide 102 and for acquiring unmagnified macro images of the entire slide and / or the sample area 114 (e.g., for acquiring information outside the sample area 114). The macrovision station 232 includes a first slide holder 234 located in an imaging position within the imaging area of ​​the macro camera 231 of the macrovision station 232 (e.g., a digital camera having a CCD or other imaging sensor), and a second slide holder 236 located in a queuing position outside the imaging area of ​​the macrovision station 232. A bottom illumination module 238 having a PCB and LEDs is positioned below the first slide holder 234 to illuminate the bottom of the slides 102 placed in the first slide holder 234. A diffuser 244 may be positioned between the bottom illumination module 238 and the first slide holder 234. An upper illumination module 240 having a PCB and LEDs is positioned below the first slide holder 234 to illuminate the top of the slide 102 positioned in the first slide holder 234. The macrovision station 232 has a macro camera 231 and a second folding mirror 242 that reflects the optical axis of the macro camera 231 onto the slide 102 in the first slide holder 234. The macro camera 231 is mounted using a camera adjustment unit 246 to allow adjustment of the orientation of the macro camera 231.

[0074]

[0154] The macrovision station 232 acquires a macro image of slide 102, including barcode identification information and / or other information on slide 102, and an image of the entire slide 102 and / or the sample area 114, and transmits the macro image to the imager computer 105. The macro image includes one or more reference marks 116 and the sample area 114 and / or the sample 119. The imager computer 105 is configured to determine the relative position and boundaries of the sample area 114 and / or the sample 119 on the surface of slide 102, at least partially based on one or more reference marks 116. The relative position and boundaries of the sample area 114 and / or the sample 119 are provided to the imaging station 190. The imaging station 190 (e.g., image acquisition subassembly 191) is configured to collect a micro image, at least partially based on the relative position and boundaries of the sample area 114 and / or the sample 119 on slide 102. In other words, the imaging station 190 uses the relative position and boundaries of the boundary sample area 114 and / or sample 119 to determine the scan position of the camera 192 when acquiring a micro image, thereby preventing overscanning beyond the edges of the sample area 114 and / or sample 119, which would waste time and storage. The macrovision station 232 also interfaces with the SIB 182, which interfaces with the imager computer 105 to control the operation of the macrovision station. The macro camera 231 communicates data with the imager computer 105 via a communication network such as a 100baseT Ethernet network.

[0075]

[0155] A schematic diagram of the SIB182 is shown in Figure 29. The SIB integrates external signals transmitted between the imager computer 105 and the imager 104 and redistributes them to the appropriate components and subassemblies of the imager 104 and the appropriate inputs of the imager computer 105.

[0076]

[0156] Next, the complete operation of the imager subsystem 103 for imaging slides 102 and creating a single merged image of the entire specimen 119 on each slide 102 will be described. The operator manually loads the slides 102 to be scanned into each of the slide carriers 118. If the slides 102 are provided in the slide rack 120, this simply requires inserting the slide rack 120 into the slide carrier 118. If the slides 102 are not in the slide rack 120, the operator first loads the slides 102 into the slide rack 120, and then inserts the slide rack 120 into the slide carrier 118. The slide carrier 118 can be used as a storage location for the slides 102 before and after imaging, and also for transporting and loading the slides 102 into the imager 104.

[0077]

[0157] The slide 102 in the slide carrier 118 is manually inspected to ensure that the slide 102 is properly oriented within the slide carrier 118 and that all printed surfaces of the slide identification portion 112 are facing the opposite direction (towards the back of the slide carrier 118). With the imager 104 in idle or paused mode, the operator opens the slide carrier deck door 166 and manually inserts the slide carriers 118, each loaded with up to 40 slides, into their respective slide carrier bays 174. A carrier presence sensor 175 in each slide carrier bay 174 detects whether a slide carrier 118 is installed and sends a signal to the imager computer 105 accordingly. The carrier lock 173 of the slide carrier 118 currently being actively imaged by the imager 104 is activated to lock the active slide carrier 118 in place when the imager 104 removes the slide 102 from the active slide carrier 118 for imaging. Other slide carriers 118 that are not being actively imaged by the imager 104 are not locked (considered inactive).

[0078]

[0158] When a new slide carrier 118 is inserted into the slide carrier bay 174 and its carrier presence sensor 175 signals that the slide carrier 118 is present, the imager 104 schedules an inventory of such slide carriers 118. The imager 104 performs an inventory on each of the slide carriers 118 installed in the imager 104 using an optical slide inventory sensor 188. The slide handling gantry 178 scans the sensor 188 on each of the slide carriers 118 to determine which of the possible slide slots in the slide rack 120 contains the slide 102. This slide inventory information is provided to the imager computer 105 and used when each slide carrier 118 becomes active and ready for scanning, so the slide handling gantry 178 only moves to the slots of slide carriers 118 where the slide 102 is located.

[0079]

[0159] When each of the slide carriers 118 is scheduled to image, the slide handling gantry 178 moves to the first slide 102 in the slide carrier 118, activates the gripper 184 to grasp it, and moves the slide 102 to the macrovision station 232. The first slide 102 is then placed on the first slide holder 234 at the imaging position of the macrovision station 232. The macrovision station 232 uses the macro camera 231 to acquire multiple macro images of the first slide 102, creating a complete specimen image (i.e., a macro image containing a single image of the entire specimen 119) and reading printed identification information (e.g., a barcode). This information becomes part of the slide data set (SDS) record of the first slide 102. As described herein, the macro image also includes the reference mark 116 and one or more of the specimen area 114 and / or specimen 119. The imager computer 105 uses macro images to determine the relative positions and boundaries of the sample area 114 and / or sample 119 on the surface of the slide 102, at least partially based on one or more reference marks 116. The relative positions and boundaries of the sample area 114 and / or sample 119 are provided to the imaging station 190.

[0080]

[0160] When the imaging station 190 is ready to receive the new slide 102, the slide handling gantry 178 moves the first slide from the first slide holder 234 at the imaging position of the macrovision station 232 to the imaging station 190 and places the first slide 102 on the slide holder 202 of the imaging station 190. While the first slide 102 is being loaded into the slide holder 202, the XY slide stage 210 may be positioned so that the first slide 102 is placed on the slide holder 202 by having the slide release bracket 209 actuate the first and second slide justifier arms 206, 208 to the release position. Once the first slide 102 is positioned on the slide holder 202, the XY slide stage 210 moves the slide holder 202 so that the first and second slide justifier arms 206, 208 are biased to their respective engagement positions, thereby biasing and holding the first slide 102 in a predetermined position within the slide holder 202 during scanning.

[0081]

[0161] After the first slide 102 is moved from the first slide holder 234 at the imaging position of the macrovision station 232 to the slide holder 202 of the imaging station 190, the slide handling gantry 178 can move the next slide 102 (i.e., the second slide 102) in the slide carrier 118 to the first slide holder 234 at the imaging position of the macrovision station 232. The macrovision station 232 can perform the process of acquiring a macro image in parallel with (i.e., simultaneously with) the acquisition of a micro image of the specimen 119 by the imaging station 190, as described later, in the same manner as described for the first slide 102. Loading the second slide into the first slide holder 234 of the macrovision station 232 and macro imaging of the second slide 102 can be performed while the imaging station 190 is imaging the first slide 102. After the macrovision station 232 completes macro imaging of the second slide 102, the second slide 102 must typically remain at the macrovision station 232 until the imaging station 190 completes micro imaging of the first slide 102 and the slide handling gantry 178 slides the first slide 102 from the slide holder 202 to the second slide holder 236 in the queue position (details below).

[0082]

[0162] Next, the imaging station 190 scans the first slide 102 to capture multiple micro-images of the specimen 119 on the first slide 102. As described above, while the digital camera 192 takes multiple micro-images, the first slide 102 is continuously moved in a meandering path by the XY slide stage 210, moving the first slide 102 under the objective lens 198. The imager 104 adjusts the movement of the XY slide stage 210 and the movement of the Z axis (focus) to image the slide 102. While the first slide 102 is being scanned, image data from the camera 192 is streamed to the imager computer 105. The micro-images are reconstructed into segmented image data from the camera 192 and streamed to the imager computer 105. The imaging station 190 acquires microimages based at least partially on the relative position and boundaries of the specimen area 114 and / or specimen 119 on the slide 102, thus minimizing wasted time and storage caused by scanning and imaging outside the specimen area 114 and / or specimen 119.

[0083]

[0163] After the first slide 102 is imaged, the slide handling gantry 178 moves the first slide 102 to the second slide holder 236, which is in a cue position. This cue position is the cue point when the imaging operation is complete. This function is used to optimize the throughput of the slide handling gantry 178 by positioning the slide 102, which has completed the imaging process, near the slide holder 202 of the imaging station 190, rather than returning the first slide 102 to the slide carrier 118. The first slide 102 remains in the cue position, while the slide handling gantry 178 moves the next slide 102 (e.g., the second slide 102), which will be micro-imaged by the imaging station 190, from the first slide holder 234 at the imaging position of the macrovision station 232 to the slide holder 202 of the imaging station 190. If not in a queue position, the first slide 102 needs to be returned to the slide carrier 118 before the next slide 102 can be placed in the slide holder 202 of the imaging station 190, but this process is slower because the slide carrier 118 is further from the slide holder 202 than the second slide holder 236. After the next slide 102 to be micro-imaged by the imaging station 190 has been moved to the slide holder 202, the slide handling gantry 178 is available to return the first slide 102 from the second slide holder 236 into the slide carrier 118 (i.e., while the imaging station 190 is imaging the next slide 102). Once the second slide 102 has been moved onto the imaging station 190, the slide handling gantry 178 moves the third slide 102 from the active slide carrier 118 to the macrovision station 232, places it on the first slide holder 234, and the macrovision station 232 images the third slide 102. This process is repeated for all slides 102 in the active slide carrier 118. In this way, the use of the queue position of the second slide holder 202 increases the overall throughput of the imager 104.

[0084]

[0164] The imager computer 105 receives camera microimages from the imaging station 190, reconstructs them, and these are reconstructed into focal plane images as described above and joined together into a single best-focus plane as described above. All of this is done on the GPU, and adjacent swaths are also joined / aligned. The resulting image is streamed from the GPU to the CPU, compressed, and saved to disk.

[0085]

[0165] The imager computer 105 also uses image analysis algorithms (described later) to scan the image data, identify objects of interest (OOIs), and generate OOI location data. OOI location data identifies the location of OOIs within the image. This data also becomes part of the SDS 304. Each SDS 304 is processed to generate an optimized dataset ("pyramid") used for image display at the review station 108. The pyramid data (a subset of the compressed SDS) is designed to speed up access to slide data for recorded OOIs. The pyramid data is also added to the SDS. Once image processing and pyramid generation are complete, the compressed SDS is sent to the workflow subsystem 106 and moved to the active storage partition on the NAS disk array dedicated to active data (see Figure 1).

[0086]

[0166] Image analysis algorithms can identify OOIs using any appropriate process. For example, an image analysis algorithm analyzes a digital image 302 by scanning it and performing primary and secondary segmentation. Primary and secondary segmentation measures, identifies, and / or extracts various characteristics for each individual object and clustered object within the digital image 302. Since specific characteristics are known to be relevant to object classification, they can be used, for example, to classify objects as normal, abnormal, diseased, healthy, precancerous, or cancerous. For example, in a digital image 302 of a cytological sample, characteristics may include cell size, nuclear-to-cytoplasmic area ratio, nuclear-corrected integrated optical density, cytoplasmic vacuolation, and darkness, which can be used to classify cells as precancerous, cancerous, normal, and / or abnormal. Primary and secondary segmentation are described in U.S. Patent Application Publication No. 2004 / 0253616. The algorithm then calculates an object score for each object based on the measured values ​​of each characteristic. Based on this score, the algorithm identifies or selects objects that are considered OOI308 and clustered objects (see, for example, Figure 34).

[0087]

[0167] The imager computer 105 may analyze the determined characteristics of each OOI 308 and use an OOI matching algorithm to identify similar OOI 308 on the same digital image 302 (i.e., the same sample). The OOI matching algorithm compares the identified characteristics of each OOI 308 to identify OOI 308 that have similar identified characteristics, such as having one or more characteristics that are within a predetermined range of each other. The OOI matching algorithm may also be used to determine whether an OOI 308 on a digital image 302 is similar to an OOI 308 in a library of OOI 308s. The library of OOI 308s may be stored in a database of the laboratory information system (LIS) 408 (see Figure 1) or in a database stored in another data storage device or system that communicates with the imager computer 105 via a network. The database of OOI 308s includes a library of OOI images and / or digital images 302 of previously classified and characterized objects (e.g., cells). Each object in the library of OOI images and / or digital images 302 has previously identified characteristics that can be compared to the respective characteristics of the OOI 308 on the digital image 302 being analyzed by the imager computer 105. Objects in the library images may be pre-classified as abnormal, normal, precancerous, cancerous, malignant, premalignant, benign, etc.

[0088]

[0168] Figure 62 shows a graphical representation of the image analysis algorithm 500 for digital cytology. In step 502, the digital image 302 is divided into frames. In step 504, the frames are processed by a frame processor running in parallel threads. The frame processor segments the frames, for example, by using the watershed method applied to the G channel of RGB. The frame processor then calculates the features (characteristics) of the objects and performs convolutional neural network (CNN) inference (e.g., using a 192x192 window and color-corrected MobileNet) running in parallel on the CPU and GPU (graphics processing unit) of the imager computer 105. In step 506, the slides are processed, and in step 508, OOIs are selected and grouped based on the identified characteristics.

[0089]

[0169] The automated digital imaging system 100 may include a machine learning diagnostic module configured to utilize machine learning to characterize and / or diagnose objects on digital images 302. The machine learning diagnostic module may be installed and run on an imager computer 105, a review station computer 111, or other suitable computer system. The machine learning diagnostic module can perform the process of characterizing objects on digital images 302 and also diagnose a patient based on the characterization. For example, as shown in one exemplary process, the process includes creating a training set of characterized / diagnosed objects. This training set is created by imaging a large number of specimen slides 102, such as hundreds, thousands, or tens of thousands of slides 102, and obtaining a digital image 302 of each specimen slide 102. Next, OOIs are identified in each of the digital slides 102 as described herein. The OOIs are independently classified to be categorized by multiple clinicians, such as three, four, five, or more clinicians (e.g., a cytologist and / or pathologist in the case of cytological specimens). For example, objects may be classified as HSIL, LSIL, normal, abnormal, diseased, healthy, precancerous, cancerous, etc. "HSIL" is an acronym for high-grade squamous intraepithelial lesion, and "LSIL" means low-grade squamous intraepithelial lesion. A lesion is an area of ​​abnormal tissue, and high-grade and low-grade refer to its potential to progress to cancer. Next, a machine learning diagnostic module determines several characteristics of each OOI, for example, by using the image analysis algorithms described herein or similar algorithms. For example, in the case of a digital image 302 of a cytological sample, characteristics include cell size, nuclear-to-cytoplasmic area ratio, nuclear-corrected integrated optical density, cytoplasmic vacuolation, darkness, etc., which can be used to classify cells as precancerous, cancerous, normal, and / or abnormal.

[0090]

[0170] Next, the machine learning diagnostic module uses a pattern recognition algorithm to determine the characterization / diagnosis relationship between the characteristics identified by the image analysis algorithm and the characterization / diagnosis determined by the clinician. The characterization / diagnosis relationship can be an algorithm, a function, an equation, or any other appropriate relationship.

[0091]

[0171] Next, the machine learning diagnostic module can characterize / diagnose objects on a new sample slide 102 using a feature / diagnosis relationship. The imaging station 103 generates a digital image 302 of slide 102. An image analysis algorithm is used to identify OOIs on the digital image 302. The machine learning diagnostic module then determines multiple properties of each OOI, for example, by using the image analysis algorithm. Finally, the image analysis algorithm characterizes / diagnoses each OOI using a feature / diagnosis relationship. The characterization / diagnosis of each OOI is also added to the SDS 304.

[0092]

[0172] The machine learning diagnostic module can continue to learn and optimize featureization / diagnosis relationships by adding 302 digital images from new sample slides 102 to the training set and then performing the process of generating featureization / diagnosis relationships using a pattern recognition algorithm.

[0093]

[0173] The imager computer 105 may also include a slide statistics module for identifying slide statistics such as the number of objects on the slide (e.g., number of cells) and the number of boundaries. The imager computer 105 can utilize a process similar to that used to determine OOIs to determine slide statistics. Similar to determining OOIs, the imager computer scans the digital image 302, identifies individual objects, clustered objects, and boundaries, and counts the number of objects, clustered objects, and boundaries. The slide statistics are added to the SDS 304.

[0094]

[0174] The imaging subsystem 103 takes images of all slides 102 in each slide carrier 118 loaded into the imager 104, processes the images, and then completes the process. Once imaging is complete, the slide carrier 118 is removed from the slide carrier deck 128. The slides 102 are then removed from the slide carrier 118 and can be physically stored according to laboratory procedures or discarded if they are no longer needed.

[0095]

[0175] As described above, the processed digital images, including each SDS, are sent to the workflow subsystem 106 and stored there. The workflow subsystem 106 stores the digital images in active storage, which can be accessed by the review station 108. The workflow subsystem 106 includes a workflow subsystem management application 248 (see Figure 1) configured to manage workflows for reviewing digital images by reviewers such as cytologists and / or cytopathologists. For example, when a reviewer logs into the review station 108, the workflow subsystem 106 sends a worklist of cases containing digital images for the reviewer to the review station 108. The review station 108 displays the worklist to the reviewer. The reviewer can then manage the worklist, select cases to work on, review the digital images of the selected cases, complete the cases, and / or track completed and ongoing cases.

[0096]

[0176] When a case to be reviewed is selected, the review station 108 accesses the selected case from the workflow subsystem 106, and the workflow subsystem sends the digital images and SDS to the review station 108. The review station 108 displays the digital images on various review screens. The user interface of the review station 108 allows the reviewer to select different digital images (for example, one case may contain multiple digital images, or one case may contain a single digital image). The review screen displays the selected digital images and images of the OOIs. The OOIs may be displayed as thumbnails to show multiple OOIs at once on the review screen, or the reviewer may be able to select one OOI to view its full size. Once the reviewer has reviewed the digital images of the case, the review station 108 displays a completion screen that allows the reviewer to dispose of the case, such as marking the case as "Completed," "Review by a Second Reviewer (e.g., a Cell Pathologist)," or "Complete Later" (if the reviewer has not yet completed a full review of the case).

[0097]

[0177] Once a case is completed, including a second review (by a cytopathologist, etc.) if necessary, the case is entered into the Laboratory Information System (LIS) 250 (see Figure 1). The workflow subsystem 106 may have an LIS communication interface 252 for sending completed cases to the LIS, or it may be entered into the LIS manually. Clinicians can then access the case results via the LIS 250.

[0098]

[0178] Referring to Figures 34-62, embodiments of the system and method for displaying digital images generated by the automated digital imaging system 100 are described in further detail. The digital display system and method enable reviewers of biological specimens, such as cells, tissues, and other subjects, to review digital images of specimens more efficiently and accurately, and to identify, characterize, and / or diagnose attributes within the specimens. Thus, the digital display system and method provides specific improvements to the way computers operate and gives specific functionality to the display system of biological specimens, solving various problems in the computerization of digital microscopy of biological specimens.

[0099]

[0179] Referring to Figure 34, the workflow subsystem 106 and the digital imaging display system 108 (also called “review station 108”) of the digital imaging system 100 are shown in more detail. As described above, the imager computer 105 uses an image processing algorithm (e.g., a software application 135 installed on the imager 104 and / or workflow subsystem 106) to analyze the digital image of the slide specimen 119 and identify objects of interest (OOIs) 308 (see Figure 36). The identified OOIs 308 can then be used to assist the user (e.g., a cytologist or cytopathologist) in quickly and accurately reviewing the digital image of the slide specimen 119.

[0100]

[0180] As described above, the workflow server 106 is a computer server that primarily functions as a high-capacity disk storage system for storing and managing digital images and associated slide data stored in the SDS 304 data record for each slide 102. The workflow server 106 also includes software for image processing and data management and may provide networking capabilities. The SDS 304 includes a stitched digital image 302, slide information 306 captured by the imager 104 for each slide 102, and OOI location data 308. Each SDS 304 is processed by the imager computer 305 to generate an optimized and augmented dataset ("pyramid") used for image display on the digital display system 108. As described herein, the pyramid data (a subset of the SDS) is designed to speed up access to slide data for recorded OOIs. The pyramid data may also include information to improve the display of the digital image 302 of the entire specimen by providing coordinate and display information for panning and zooming into areas of the digital image 302 that do not have OOIs (particularly non-gynecological slides). The pyramid data is also added to the SDS. Once image processing and pyramid generation are complete, the compressed SDS304 is sent to the workflow subsystem 106 and moved to the active storage partition 107 on the NAS disk array dedicated to active data (see Figure 1).

[0101]

[0181] The digital display system 108 is a workstation including a computer 111 and a monitor 109 for accessing digital images from the workflow server and displaying the digital images for diagnostic review of the images (OOI 308 and / or whole specimen image 302). As shown in Figure 1, the imager subsystem 103 communicates data with the workflow subsystem 106 via a communication network which may include one or more of the following: a local area network (LAN, e.g., Ethernet), a wide area network (WAN), the internet (e.g., a virtual private network (VPN)), or other suitable networks. Similarly, the review station 108 communicates data with the workflow subsystem 106 via a communication network 113 which may include one or more of the following: a local area network (LAN, e.g., Ethernet), a wide area network (WAN), the internet (e.g., a virtual private network (VPN)), or other suitable networks.

[0102]

[0182] The digital image display system 108 is configured to access the SDS 304 for each slide 102 from the workflow subsystem so that a reviewer (e.g., a cytologist and / or cellular pathologist) can review the digital image of the slide 102 on the monitor 109. Referring to Figure 34, each digital image display system 108 includes a computer 111 and a computer monitor 109. The computer 111 can be any suitable computer having a microprocessor (CPU), memory, storage, and network adapter. The computer 111 may have a graphics processing unit (GPU) to improve the speed of processing large digital image files. The digital image display system 108 may also include input devices such as a keyboard, mouse, and touchpad. The monitor 109 may be a touchscreen monitor such that the touchscreen is the input device. As described herein, the digital image display system 108 communicates data with the workflow subsystem 106 via a communication network 113.

[0103]

[0183] Monitor 109 may be a color monitor specifically configured to display a digital image 302 of the specimen slide 102. For example, Monitor 109 may have a specific resolution and / or color calibration for displaying the specimen slide 102 for use in pathology or other specific applications. For example, in cytopathology, color may be used to identify different types of cells, and specific types of colored stains and / or reagents may be used to mark the specimens and prepare them for review by a reviewer. For example, U.S. Patent No. 6,661,501 describes various stains and methods for producing them, methods for staining cells for cytological or histological analysis to contrast the nuclear portion of cells with the cytoplasmic portion, and systems and methods for illuminating cytological samples, which are suitable for preparing cytological specimens for use in the digital imaging system 100. For the reviewer to properly review the slide 102, the effect of the stain and / or reagent must be detectable on the digital image when displayed on Monitor 109. In other words, the digital imaging system 100 may be configured such that the digital image of slide 102 viewed on the monitor 109 looks the same as, or substantially the same as, the actual appearance of slide 102 when viewed using a microscope-based system.

[0104]

[0184] Referring to Figure 35, an exemplary workflow path 310 for reviewing digital images 302 of a specimen slide 102 using a digital image display system 108 is shown. The reviewer accesses a case 312 from a group of cases 312 stored in a case management system (a component of the workflow subsystem management module 314 of the workflow subsystem 106). Each case 312 contains or is associated with one or more SDS 304, each having one or more digital images 302. A first reviewer 316 (e.g., a cytologist) pulls a case 312 onto the digital image display system 108.

[0105]

[0185] As shown in Figures 36-44, the display system 108 can display a worklist of cases 312 to be reviewed, from which the first reviewer 316 can select for review. As shown in Figure 36, the display system 108 generates a navigation menu 318, which is displayed on the monitor 108 of the display system 109. Figure 36 shows that the navigation menu 318 can be collapsed to make space on the monitor 109 for displaying other information. When one of the selection items in the navigation menu 318 is selected, the display system 108 displays a list 320 of cases 312 in that menu selection item. As shown in Figure 37, the list 320 includes checkboxes 322 from which the reviewer 316 can select which cases to review next. The list 320 also includes a "check all" box to select all cases 312 in the list 320.

[0106]

[0186] Figure 38 provides an overview of the different screens displayed by the display system 108 during use, including different screen options and different screen layouts depending on the type of case 312. At the far left of Figure 38, the display system 108 displays the login screen 324. After the user logs in, the display system 108 displays one of the worklist screens 326, 328, or 330. The different worklist screens 326, 328, and 330 have different display formats. The worklist screen 326 displays the worklist in a simplified list format with different columns for different information for each listed case 312, as shown in more detail in Figure 40. The worklist screen 328 displays the worklist in a tile / grid format with a two-dimensional array of tiles, as shown in more detail in Figure 41. The worklist screen 330 displays the worklist in a Kanban format, where the cases 312 are displayed as tiles in separate columns according to the state of each case, as shown in more detail in Figure 42. After the user selects case 312 from one of the worklist screens, the display system 108 displays the selected case on review screen 332 if it is a gynecological case (GYN), or on review screen 334 if it is a non-gynecological case (NGYN). Since gynecological cases typically contain only a single slide 102, review screen 332 is formatted to display a screen for reviewing the digital image of a single slide 102. In contrast, non-gynecological cases typically contain multiple slides 102, so review screen 334 is formatted to display a screen for reviewing the digital images of multiple slides 102 simultaneously. Therefore, references to GYN cases throughout the description and drawings may be considered to refer to cases with a single slide 102, and NGYN cases may be considered to refer to cases with multiple slides 102.

[0107]

[0187] Referring to Figure 38, after the user completes the review on review screens 332 and 334, the display system 108 displays either the review completion screen 336 or 338, depending on whether it is a GYN case or an NGGYN case. After the user exits the review completion screens 336 and 338, the display system 108 displays either the worklist completion screen 340 or the worklist save screen 342. The worklist completion screen 340 displays a list of cases 312 completed by the user, while the worklist save screen 342 displays a list of cases 312 in progress, which means cases that have been started but have not yet been completed.

[0108]

[0188] Figure 39 shows that the display system 108 may be configured to toggle between different worklist screens 326, 328, and 330. For example, the worklist screens 326, 328, and 330 may include worklist screen commands 344, which the user can use an input device to select with a cursor or other input commands to instruct the display system 108 to display the desired worklist screen.

[0109]

[0189] Turning to Figure 40, details of the worklist screen 326 are shown. The worklist screen 326 displays the user's worklist of cases 312 in a simplified list, with each case 312 on its own row, and information about each case is shown in different columns on each row. The title "MY WORKLIST" includes a case count 351 indicating the number of cases 312 in the worklist screen 330. For each case 312, the worklist screen 326 includes fields for slide number indicator, accession number, case type, status, and due date. The slide number indicator indicates whether the case 312 has a single slide, indicated by a slide icon 352, or multiple slides, indicated by a file folder icon 354 indicating the number of slides in the case 312. The worklist screen 326 also includes a filter function 346 in the display system 108 to filter the cases 312 in the displayed list. The worklist screen 326 also displays a completed review counter 348 indicating the number of reviews the user has completed that day. The worklist screen 326 also has a search box 350 where the user can enter a search query, in which case the display system 108 performs a search of case 312 for the search query. The display system 108 may be configured to search the user's worklist, the entire database of case 312 in active storage 107, or any other desired database of case 312.

[0110]

[0190] Figure 41 shows details of the worklist screen 328. The worklist screen 328 displays the user's case 312 worklist in a tile / grid format. Each tile contains the same or similar information as the columns in worklist 326, including a slide number indicator, accession number, case type, status, and due date. The worklist screen 328 also includes a case count 351, a filter function 346, a review completion counter 348, and a search box 350, similar to the worklist screen 326.

[0111]

[0191] Figure 42 shows the worklist screen 330 in more detail. The worklist 330 displays the user's case 312 worklist in Kanban format, with each case 312 displayed as a tile in a separate column according to its status. The tiles are substantially the same as those on the worklist screen 326. The tiles are arranged in three columns: column 356 for cases awaiting primary review, column 358 for cases under review, and column 360 for cases completed with primary review. Each tile contains the same or similar information as the columns on the worklist 326, including a slide count indicator, accession number, case type, status, and due date. Case 312 in the completed primary review column 360 is displayed with lower contrast (e.g., grayed out) than Case 312 in the cases awaiting primary review column 356 and under review column 358, allowing the user to check their progress without being distracted from tasks that need to be completed. The worklist screen 330 also includes a case count 351, a filter function 346, a completed review counter 348, and a search box 350, similar to the worklist screens 326 and 328.

[0112]

[0192] Figure 43 shows a detail panel 361 that can be displayed on the display system 108 in any of the worklist screens 326, 328, or 330. This function is shown only for the worklist screen 326, but can be used similarly in the worklist screens 328 and 330. The worklist screen 326 includes a detail button 362. When a user selects one of the cases 312 in the worklist, such as by clicking on case 312, the display system 108 highlights that case in the list, as shown in Figure 45. Next, when the user clicks the detail button 362, the display system 108 displays a detail panel 360 that contains additional details of the selected case 312, including, for example, a progress indicator 364, the imaging date 366, the name of the reviewer in charge of case 312, and the case history 368. The width of the detail panel 360 can be adjusted to widen or narrow it, which in turn narrows or widens the main list accordingly.

[0113]

[0193] Figure 44 shows the use of a graphical status indicator 370 to display the status of case 312 in any of the worklist screens 326, 328, or 330. For worklist screen 326, the graphical status indicator 370 in Figure 44 is a colored line along the left side of each column that has a specific status, such as "In Progress." Similarly, the tiles in worklist screens 328 and 330 may also include colored lines or colored borders to provide a convenient and easily visible status indicator.

[0114]

[0194] Returning to Figure 35, once reviewer 316 has determined which case 312 to review from the worklist screens 326, 328, and 330, reviewer 316 selects the case 312 from the worklist screens 326, 328, and 330. Upon receiving the selection of case 312, the display system 108 accesses the SDS 304 of the selected case 312 from the active storage 107 of the workflow subsystem 106 and loads the case 312 into the display system 108. After loading the SDS 304 of the selected case 312, the display system 108 displays the review screen for case 312. The review screen may differ depending on whether the selected case 312 is a GYN case or an NGGN case.

[0115]

[0195] Figure 45 shows an exemplary GYN review screen 332. The GYN review screen 332 has a case identification panel 374 at the top of the screen. The case identification panel 374 includes accession number, case type (e.g., GYN or NGYN), review deadline, imaging date, patient information (e.g., name, date of birth, age, etc.), and reviewer identification information. The GYN review screen 332 also has an OOI panel 376, a main image panel 378, and a reviewer selection image panel 380.

[0116]

[0196] The display system 108 displays the digital image 302 of the selected case 312 on the main image panel 378. The user can zoom and pan around the digital image 302 in the main image panel 378 using commands displayed on the GYN review screen 332 and / or commands selectable using the input devices of the display system 108. The display system 108 also displays a region of the OOI image 382 when selected as described below.

[0117]

[0197] The display system 108 may have a function to scan the entire digital image, which the system scans the entire digital image 302 and displays it on the main image panel 378. The scan the entire digital image function may be a selectable function having a selectable button on the review screen 332. The scan the entire digital image function allows the user to set a zoom level at which the digital image 302 is automatically scanned, for example, by using a zoom command or a selectable command displayed on the review screen 332. The scan the entire digital image function scans the entire digital image 302 at the selected zoom level and displays it on the main image panel 378. The scan pattern may be a meandering pattern similar to the scan pattern for capturing slide 102, a row-by-row scan in the same direction (e.g., left to right), a column-by-column scan, or another appropriate scan pattern. The scan the entire digital image function may also include user-selectable commands that allow the user to stop (i.e., pause) and start the scan, set the pause time, set the scan speed, and / or set the number of areas on the slide (e.g., 2000 critical areas, 20 critical areas, etc.). The function for scanning the entire digital image may also have a user-selectable option to pause at all objects on the digital image 302 (e.g., each cell or other object). Using the option to pause at all objects, the user can pause for a set pause time, or until they select a user-selectable option to continue scanning.

[0118]

[0198] The OOI panel 376 contains multiple OOI images 382 of regions of the digital image 302 corresponding to the OOI location data 308 of each OOI in the SDS 304. The OOI images 382 are reduced-size (zoomed-out) versions of the digital image 302. When a user selects one of the OOI images 382, ​​the region of the digital image 302 of the selected OOI image 382 is displayed in the main image panel 378 at an enlarged size (i.e., enlarged compared to the OOI image 382). The main image panel 378 still displays the digital image 302, although it has been zoomed and moved to initially display the region of the OOI image 382, ​​so the user can zoom and pan around the region of the selected OOI image 382 in the same way as described for the digital image 302. The OOI images may also contain a show similar command 384, such as a "+" symbol on the OOI image 382. When the user selects a similarity search command 384 for the selected OOI image 382, ​​the display system 108 displays additional OOI images 382 that have similar characteristics to the selected OOI image 382. For example, the display system 108 may display 3 to 6 additional OOI images 382 included in the SDS as similar to the selected OOI image 382. The additional OOI images 382 include objects such as cells that share one or more characteristics with the objects in the selected OOI image 382. One or more additional OOI images 382 may be obtained from the same whole-specimen digital image 302, or from a library of images containing previously classified objects that share one or more characteristics with the objects in the selected OOI image 382.

[0119]

[0199] The reviewer selection image panel 380 is an area of ​​the GYN review screen 332 where the reviewer 316 can place one or more OOI images 382, ​​other images added by the reviewer as OOIs, or any other images / locations on the digital image 302 that the reviewer has selected as of particular interest. For example, the OOI image 382 may be selected for review by a second reviewer 317, such as a pathologist.

[0120]

[0200] Figure 46 shows an exemplary NDYN review screen 334. As described herein, the main difference between a GYN case and an NDYN case 312 is that a GYN case includes a digital image 302 of a single slide 102, whereas an NDYN case 312 includes one digital image 302 for each of the multiple slides related to the case. The NDYN review screen 334 has many of the same features as the GYN review screen 332, including the main image panel 378 and the reviewer selection image panel 380.

[0121]

[0201] Furthermore, the NGYN review screen 334 has a slide list 388 that shows a list of slides 102 in the NGYN case 312, each having a digital image 302. Each slide 102 in the NGYN case 312 is represented by a slide icon 387 in the slide list 388, which has an identifier such as a letter or number. When the reviewer selects a slide 102 in the slide list 388, the display system 108 displays the digital image 302 of the selected slide 102 in the main image panel 378. The selected slide 102 may be highlighted in the slide list 388 to indicate that it is the slide 102 currently being displayed. The display system 108 allows the reviewer to interact with the digital image 302 in the same way as the NGYN case 312 as described above. The reviewer can select each of the slides 102 in the slide list 388, and the display system 108 repeats the display process for each of the slides 102. The NGYN review screen 334 may also include a slide count 392 indicating the number of slides in the reviewed NGGYN case 312 and the total number of slides in the NGGYN case 312.

[0122]

[0202] Figure 47 shows another example of the NGIN review screen 394, which is similar to the NGIN review screen 334, but with a wider slide list 388 and larger slide icons 387 to better display each of the individual slides 102.

[0123]

[0203] Any review screen, including GYN review screen 332 and NGGYN review screens 334, 394, may also display other data from the SDS of the digital image 302 displayed on the review screen. The review screen may display identified characteristics for one or more OOIs. The review screen may include a selectable option (similar to the "+" symbol above) for each OOI image 382 to display identified characteristics for that OOI image 382 when selected. The identified characteristics may be displayed in a pop-up window. The review screen may also display machine learning characterization / diagnosis associated with the OOI image 382. The review screen may include a selectable option for each OOI image 382 that can be selected to display the characterization / diagnosis of the OOI image 382 in a pop-up window or the like. The review screen may display slide statistics from the SDS 304. The display of slide statistics may be in response to a selectable option to display slide statistics in a pop-up window, or the slide statistics may be automatically displayed on the case identification panel 374 or elsewhere on the review screen.

[0124]

[0204] Figures 58–61 show review screens with additional features such as adding comments to OOI image 382 and / or digital image 302, screen sharing, and bookmarking OOI image 382 and / or digital image 302. It should be understood that one or more of these additional features and graphical user interfaces may be included in any of the illustrated and described review screens.

[0125]

[0205] Figure 58 shows an exemplary GYN review screen 420 configured to allow a reviewer to add comments and / or mark / draw on the OOI image 382 and / or digital image 102. The review screen 420 is similar to the review screen 332 in Figure 47 and has almost the same features as the review screen 332 in Figure 47. As shown in Figure 58, the review screen 420 includes a comment panel 422. The review screen 420 has an add comment command 424. When the reviewer selects the add comment command 424, the reviewer can select a comment area 432 of the OOI image 382 in the OOI panel 376 or on the OOI image 382 or digital image 302 displayed in the main image panel 378 in the review screen 420. The reviewer can then add a text comment 426 to the comment panel 422. In this way, the reviewer can comment on a specific object on the OOI image 382 or digital image 302. The comment panel 422 displays the comment ID 428 (e.g., a number or letter), the reviewer ID 430 (e.g., the reviewer's name or other identifier), and the comment text 426. Reviewers can add comments by selecting the add comment command 424 and repeating the process. The comment, including the comment text 426, the comment area 432, the comment ID 428, and the reviewer ID 430, is stored in the SDS 304 of each slide 102. The comment panel 422 allows other reviewers to view previous reviewers' comments and add their own using the same process. Other reviewers' comments will have different comment IDs 428 and / or reviewer IDs 430. Reviewers can add overall slide-level comments by skipping the comment area selection and simply including the text comment 426 within the comment panel 422. Alternatively, the review screen 420 may include a separate overall slide-level comment panel similar to the comment panel 422.

[0126]

[0206] Figure 59 shows an exemplary GYN review screen 450 configured to allow a reviewer to add recorded audio comments (e.g., dictated comments) to case 312 of digital image 102 and / or mark / draw on OOI image 382 and / or digital image 102. Review screen 450 is similar to review screen 332 in Figure 47 and has almost the same features as review screen 332 in Figure 47. As shown in Figure 59, review screen 450 includes a dictated comment command 452. When the reviewer selects the dictated comment command 452, the review screen 450 becomes available for the reviewer to record dictated comments. For example, review screen 450 displays a dictated comment window 454. The dictated comment window 454 has a dictation control bar 456. The dictation control bar 456 has a record button 458, a play button 458, a pause button 460, and a navigation slide 462. The record button 458 starts recording the audio comments. The play button 458 plays the audio comment. The pause button 460 pauses either recording or playback. The navigation slide 462 allows the reviewer to move forward or backward within the recorded comment. The dictation comment command 452 may also allow the reviewer to mark a comment area 432, similar to the review screen 420. Thus, the reviewer can comment on a specific object on the OOI image 382 or the digital image 302. The dictation comment window 454 displays, for each audio comment, a comment ID 428 (e.g., a number or letter), a reviewer ID 430 (e.g., the reviewer's name or other identifier), and a dictation control bar 456. The reviewer can add dictation comments by selecting the dictation comment command 452 and repeating the process. The audio comment, including the recorded audio, comment area 432, comment ID 428, and reviewer ID 430, is stored in the SDS 304 of each slide 102. The dictation comment window 454 allows other reviewers to play back the previous reviewer's audio comments and add their own using the same process.The comments from other reviewers have different IDs: comment ID 428 and reviewer ID 430.

[0127]

[0207] Figure 60 shows an exemplary GYN review screen 434 configured to allow a first reviewer to share their screen with a second reviewer on a different display system 108, and to allow the first reviewer to give the second reviewer control of the review screen 434 on the first reviewer's screen. In other words, the second reviewer's display system 108 displays the same review screen 434 as displayed on the first reviewer's display system 108, and the second reviewer can use their display system 108 to control the review screen 434 on the first reviewer's screen. The review screen 434 has a screen sharing command 436. When the first reviewer selects the screen sharing command 436, the review screen 434 displays a screen sharing window 438, such as a pop-up window. The screen sharing window 438 requests the second reviewer's screen sharing ID. The first reviewer enters the second reviewer's screen sharing ID (e.g., name, number, code, etc.) and presses Enter to send the screen sharing ID to the workflow subsystem 106. The workflow subsystem 106 verifies the screen sharing ID and then shares the review screen 434 from the first reviewer's display system 108 with the second reviewer's display system 108. The review screen 434 also has a share control command 440 that allows the first reviewer to pass control of the review screen 434 to the second reviewer. The screen sharing command 436 may be a toggle command that allows the first reviewer to share and unshare the review screen 434 by selecting command 436. Similarly, the share control command 440 may be a toggle command that allows the first reviewer to share and unshare control of the review screen 434. Additionally, by selecting the screen sharing command 436, the chat window 442 can be enabled, which can be displayed in any open space on the review screen 434 or as a movable pop-up window. In the chat window 442, the first reviewer and the second reviewer can text chat.Furthermore, the review screen 434 can enable audio or audio / video communication between the first reviewer and the second reviewer via the workflow subsystem, thereby facilitating audio or audio / video communication between reviewers. Additionally, the review screen 434 can be configured to allow additional reviewers to share the first reviewer's review screen 434 by repeatedly issuing a share command to those additional reviewers.

[0128]

[0208] Figure 61 shows an exemplary GYN review screen 444, which is configured to allow, for example, a reviewer to bookmark case 312 of digital image 302 to create a reference library of interesting cases 312 and digital images 302. Review screen 444 is similar to review screen 332 in Figure 47 and has almost the same features as review screen 332 in Figure 47. As shown in Figure 61, review screen 444 has a bookmark command 446. When the reviewer selects the bookmark command 446, review screen 444 displays a bookmark comment window 448 that allows the reviewer to enter comments. For example, the reviewer can enter comment text 426 about why they bookmarked case 312 as of interest. The bookmark command 446 may also allow the reviewer to mark a comment area 432, similar to review screen 420. The bookmark comment window may include a comment ID 428 and a reviewer ID 430, similar to review screen 420. Furthermore, a bookmark record is also recorded in the SDS304 of the digital image 302, including comment text 426, comment ID 428, reviewer ID, and comment area 432. The workflow subsystem 106 assigns the bookmarked case 312 to a reference library, such as a reference library of interesting cases.

[0129]

[0209] After a reviewer selects and reviews slide 102 in slide list 388, the display system 108 provides a visual indicator 390 to show that each slide 102 has been reviewed. For example, the slide list 388 shown in Figure 48 has a dot 390 above each slide icon 387 in slide list 388. This dot 390 indicates whether each slide 102 has been reviewed by the reviewer.

[0130]

[0210] Figure 57 shows yet another example of review screen 414, which is similar to review screen 332 in Figure 47 and has almost the same features as review screen 332 in Figure 47. The main difference is that review screen 414 displays the entire sample area of ​​the digital image 302 in the main image panel 378. In other words, the image in the main image panel 378 is fully zoomed out, and the entire digital image 302 is displayed. Review screen 414 also has a zoom slider 416 to adjust the zoom of the image in the main image panel 378. Any of the other review screens 332, 334, and 394 may also display the entire digital image 302 fully zoomed out in the main image panel 378 and may also include a zoom slider 416.

[0131]

[0211] After reviewer 316 reviews case 312, reviewer 316 selects the review complete command 396 on review screens 332, 334, and 394 (see Figures 45-47). Upon receiving the review complete command 396, the display system 108 displays a completion screen. Figure 48 shows an example of the completion screen 336 for GYN case 312 when reviewer 316 determines that the sample is normal as a result of the review. The completion screen 336 includes the same functionality as the GYN review screen 332 and may also include a completion frame 399 displayed on the main image panel 378. Reviewer 316 selects the "Complete as Normal" command 400. Upon receiving this normal completion command, the display system 108 sends a normal completion signal to the workflow subsystem 106, updating the case 312 to be determined to be normal. As shown in the workflow diagram of Figure 35, the selected case 312 is then flagged to be filed as a completed case or routed to quality control. The percentage of reviewed cases completed as successful by the first reviewer 316 is sent to quality control and reviewed by another reviewer as a quality control check. The quality control check may determine whether reviewer 316 incorrectly completed one or more cases 312 as successful.

[0132]

[0212] The completion box 399 also includes a “Complete Later” command 401 if reviewer 316 does not complete the review of the selected case 312. Upon receiving the Complete Later command 401, the display system 108 marks case 312 as in progress and leaves the case as an in-progress case in the reviewer’s worklist.

[0133]

[0213] Figure 49 shows an example of the completion screen 336 for GYN case 312 when, as a result of the review, reviewer 316 determines that the specimen is abnormal. The completion screen 336 includes the same functionality as the GYN review screen 332 and may also include a completion frame 399 displayed on the main image panel 378. If the specimen slide 102 is determined to be abnormal, the first reviewer 316 selects the send command 404 to the pathologist. Then, as shown in the workflow diagrams in Figures 35 and 52-54, the workflow subsystem 106 schedules case 312 for review by pathologist 317. Next, as shown in Figure 35, the pathologist logs into the display system 108, which displays the worklist screen, review screen, and completion screen, allowing the pathologist to select, review, and complete the case in essentially the same way as the first reviewer 316.

[0134]

[0214] Once a reviewer has finished reviewing a case, they can return to the worklist screen. Figure 50 shows an example of the worklist screen 340 displaying the details panel 361 of case 312, which was completed as abnormal by the first reviewer 316 and assigned to a pathologist for pathologist review. By default, completed cases 312 remain in the reviewer's worklist until the reviewer chooses to hide the completed case 312. The worklist screen 340 in Figure 50 displays the hide case command 406 to hide the selected completed case from the worklist screen 340. The default can be changed to apply a rule that automatically hides completed cases. Figure 51 shows the worklist screen 340 with completed case 312 hidden.

[0135]

[0215] The comments, review results, and status for each case 312 are stored in a slide data file within the workflow server 106 (e.g., database server 530, Figure 63). The slide data file may contain digital images 302, SDS 304, slide information 306, and / or pyramids / OOI 308, or it may be a different data file.

[0136]

[0216] As shown in Figure 35, the disposal of each case 312 may then be entered into a laboratory information system (LIS) 408 accessible to the clinician 412 on the clinician computer 410, which communicates data with the LIS 408.

[0137]

[0217] Figures 35 and 52-56 show several variations of the workflow for reviewing case 312 by a first reviewer 316 (e.g., a cytologist) and a second reviewer 317 (e.g., a pathologist) using the digital image display system 108 and the workflow subsystem 106. In the workflows shown in Figures 35 and 52-54, the terms “Pull” and “Push” indicate whether the reviewer pulls case 312 from the imaged cases awaiting review, or whether the case is assigned to the reviewer (pushed) by a manager or automated process (e.g., a rule-based algorithm with rules such as case type and case outcome). Figure 55 includes flowcharts illustrating various scenarios for the cytologist and pathologist when completing the workflows shown in Figures 35 and 52-54. Figure 56 includes flowcharts illustrating various manager / administrator roles and scenarios for the workflows shown in Figures 35 and 52-54.

[0138]

[0218] Figure 63 shows a block diagram of an exemplary workflow server 106 with backup and archiving capabilities. The workflow server 106 includes multiple software modules for performing various backup and archiving services, including an imager web service module 520, a review station web service module 522, an archive service module 524, and a backup service module 525. The workflow server 106 also includes an Internet Information Services server 526 (e.g., Microsoft® IIS), a security software module 528 (e.g., ASP.NET Core Identification / Authorization), a database server 530 (e.g., a SQL database server) with a database 527 for slide data files, an image repository 532 (e.g., on a RAID storage disk), and a server operating system 534 (e.g., Microsoft Server 2016). The imager web service module 520 is configured for use by the imaging station 190 to send and receive data to and from the database server 530. The review station web service module 522 hosts the review station application and provides services for sending and receiving data to and from the database server 530. The database server 530 stores the SDS 304, which contains slide information 306, pyramid / OOI 308, and review information. The archive service module 524 performs archiving of the slide data files, digital images 302, and / or SDS 304. The backup service module 525 performs backups of the database of slide data files, including digital image files and review information.

[0139]

[0219] Backup module 525 protects against single disk failures and system failures. Backup module 525 can back up databases and slide image files 302 to a network drive 540 (e.g., using the Windows® backup function) or to cloud storage 542 (e.g., using the Microsoft Azure® backup agent). The purpose of backup is to protect locally stored information from disasters such as disk failures. If the backup is remote, such as a cloud backup, the backup is also protected from catastrophic failures caused by disasters such as fire, flood, or other localized incidents. Backups allow for the restoration of databases and digital image files as part of recovery from system failures.

[0140]

[0220] The archive service module 524 is configured to delete local data in the database server 530 and store it more efficiently for longer periods. This can also improve the performance and / or usability of the digital imaging system 100 by reducing its storage requirements. The archived data is stored in external storage such as a remote network drive 540 or cloud storage 542 (e.g., Microsoft Azure® cloud storage or Amazon Web Services®, AWS cloud storage). The archive service module 524 is configured to allow the user to specify policies for selecting digital image files 302, slide information 306, pyramids / OOIs 308, and / or review information to archive. The archive service module 524 can also archive macro images (low-resolution versions of digital image files 302 if generated by the system 100). For example, the user may set a policy to archive data that was last accessed beyond a selected period, such as 30 days, 15 days, or 60 days. The archive policy may also include a set time for performing the archive (e.g., a specified time each day for daily archiving, or a date and time for weekly archiving). When the data for slide 102 is archived, the information in database 527 is not archived, but database 527 is updated to track which slides 102 have been archived. In one embodiment of the archiving service module 524, the pyramid image file (e.g., OOI308) of the archived slide 102 is not archived. Local copies of the archived file and pyramid file are removed from database 527 and image repository 532.

[0141]

[0221] Referring to Figure 64, a flowchart of method 600 for archiving slide 102 using the archive service module 524 is shown. In step 602, the archive service module 524 receives and stores an archive policy, including an archive schedule. In step 604, at the time scheduled according to the archive schedule, the archive service module 524 identifies slide 102 that satisfies the archive policy for archiving and places slide 102 in the archive database table. In step 606, for each slide 102 in the archive database table, the archive service module 524 copies the slide image file 302 and slide information 306 from the database 527 and image repository 532 to the external archive storage 540, 542. If the copy is successful, in step 608, the slide data for each archived slide is deleted from the database 527 and image repository 532. In step 610, the database 527 is updated to indicate that each archived slide 102 was successfully archived. In step 612, an additional archive history database that stores the archive history log may also be updated.

[0142]

[0222] The archive service module 524 is also configured to retrieve archived slide data for archived slide 102. The archive service module 524 can utilize the retriever web service installed on the workflow server 106. The retriever web service is configured based on the type of archive, such as whether it is a remote network drive or a specific cloud storage. The user selects slide 102 to be retrieved from the archive using the retriever web service, and the retriever fetches the archived slide data for the selected slide 102 from external storage 540, 542, including slide image files 302, slide information 306, macro images, and / or other archive data. Upon successful retrieval of the slide data, the system 100 regenerates the pyramid image data, including the OOI 308, as described herein. The archive service module 524 updates the archive database table and / or archive history database with respect to the status of the retrieved slide 102. Each SDS 304 of the retrieved slide 102 may be stored in the database 527 and image repository 532 on the workflow server 106.

[0143]

[0223] While specific embodiments have been illustrated and described, it should be understood that the above description is not intended to limit the scope of these embodiments, and such disclosures are provided for illustrative and illustrative purposes only. Therefore, various changes and modifications can be made to the disclosed embodiments without departing from the scope of the appended claims. For example, not all components described in the embodiments are essential, and alternative embodiments may include any suitable combination of the described components, and the general shape and relative size of the components may be modified. Furthermore, while the systems and methods have been described for cytological samples, they can be configured and utilized for any type of sample.

Claims

1. An automatic slide imaging system, wherein the system includes an imager, The imager is configured to acquire an image of a specimen fixed to the surface of a slide, the specimen comprising a plurality of objects distributed within a three-dimensional volume, and the imager is configured to generate an image of the entire specimen from the acquired image, the acquired image comprising a macro image of the specimen and a plurality of micro images of the specimen. The aforementioned imager is: A macrovision station comprising a first slide holder and a macro camera for acquiring a macro image of an entire specimen while the slide is seated in the first slide holder, wherein the macro image includes one or more reference markers positioned on the surface of the slide, and the imager is configured to determine the relative position and boundaries of the specimen on the surface of the slide, at least partially based on the one or more reference markers; An imaging station comprising a slide stage, a slide holder of the stage mounted on the slide stage, and a microcamera for acquiring a plurality of microimages of the specimen when the slide is seated in the slide holder of the stage, wherein the microcamera has an optical axis at a non-orthogonal angle to the slide when the slide is seated in the slide holder of the stage, the microcamera has a field of view on the slide that is much smaller than the entire specimen area of ​​the slide, the imaging station is configured to automatically move the slide stage and the slide holder of the stage in a scanning pattern that scans the entire specimen area to acquire a plurality of microimages of the entire specimen, and the imager is configured to acquire the microimages at least partially based on the relative position and boundaries of the specimen on the slide surface identified from the macroimages; and An image processor configured to generate a whole specimen image from a plurality of microimages, wherein the image processor is configured to identify a best-focus image for each individual object in the plurality of microimages, and the best-focus images of the objects are incorporated into a merged image of a single plane including the whole specimen image, such that the objects are depicted in focus within the whole specimen image regardless of their individual positions within the three-dimensional volume. An automatic slide imaging system characterized by comprising the following features.

2. The imaging system according to claim 1, wherein the slide has a width defining the x-axis and a length defining the y-axis, and the microcamera is configured such that the field of view on the slide captures the entire x-axis width of the specimen in each microimage, based on the boundary of the specimen identified on the surface of the slide.

3. The imaging system according to claim 1, characterized in that the imaging station is configured to acquire a micro-image of the specimen in the same z-axis as the three-dimensional volume.

4. In the imaging system according to claim 1, the imager is One or more slide holder receptacles, each slide holder receptacle configured to accept a slide holder having multiple slots, and each slot configured to hold an individual slide, Further equipped with a robotic arm assembly, The robot arm assembly is Remove the slide from the slot of the slide holder of the slide holder receptacle. To acquire a macro image, move the slide to the first slide holder of the macrovision station and seat it. Remove the slide again from the first slide holder, To acquire a micro-image, move the slide to the slide holder on the stage and seat it. Remove the slide again from the slide holder of the aforementioned stage. An imaging system characterized by being configured in such a way.

5. In the imaging system according to claim 4, the robot arm assembly is The slide is transferred to the same or a different slide holder from which the slide was removed. An imaging system further configured to release the slide into the slots of the same or different slide holders.

6. The imaging system according to claim 5, characterized in that each of the same or different slide holder slots is the same slot from which the slide was removed by the robot arm assembly.

7. The imaging system according to claim 1, wherein the image processor is configured to identify an object of interest within the specimen and to store an image of the identified object of interest together with an image of the entire specimen.

8. The imaging system according to claim 1, wherein the macro image of the specimen includes an image of a barcode on the surface of the slide, and the imager is configured to acquire information about the specimen from the barcode.

9. The imaging system according to claim 1, characterized in that the specimen is a cytological cell specimen and the object is a cell.

10. The imaging system according to claim 1, characterized in that the specimen is a pathological tissue specimen and the object is a tissue structure.

11. The imaging system according to claim 1, wherein the imaging station further comprises an objective lens between the micro camera and the slide holder of the stage.

12. The imaging system according to claim 1, further comprising a review station including a display monitor, a user interface, and a processor functionally coupled to the respective display monitor and user interface, wherein the processor is configured to display an image of the entire specimen along with individual images of individual objects within the specimen image on the display monitor.

Images