Digital imaging system and method

Abstract

A method and system for providing multi-planar viewing of a digital image of an object without requiring viewing multiple images taken at multiple image planes. A pseudo-focus image is generated from a flattened fused image of a three-dimensional object. The fused image is generated by fusing multiple images taken at different focal planes of the object to form a through-focus image stack. The fused image includes best-focus pixels originating from the different respective image planes. A depth map is also generated from the multiple images, relating each best-focus pixel to a pixel depth value. A plane of interest (POI) is selected from the image planes. A pseudo-focus image is then generated by using the fused image and the pixel depth map to generate a blur focus value for each best-focus pixel based on the focus offset of each pixel from the POI.

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates generally to digital imaging systems and methods, and more particularly to digital imaging systems and methods for generating a pseudo-focus image of a digital image of an object. [Background technology]

[0002] Capturing a digital image using a camera, such as a digital camera with an imaging sensor (e.g., a charge-coupled device (CCD) or CMOS image sensor), typically generates a two-dimensional image of a three-dimensional scene. When the camera's field of view is capturing images of objects placed at various distances from the camera, only regions of the scene that are at a focusing distance (the "focal length") from the camera and within a particular focusing range of focal lengths will be sharply focused. The focusing range, i.e., the distance between the nearest and farthest regions of the field of view that are sharply focused, is called the "depth of field" (DOF). Generally, the region of a camera's field of view that is in focus can be adjusted by (1) adjusting the camera's focal length by adjusting the focal length of the camera lens, (2) moving the camera relative to the object within the field of view, and / or (3) moving the object relative to the camera. The DOF can also be adjusted to alter the region of the field of view that is in focus by adjusting the camera aperture. In optical systems, camera aperture is typically considered in terms of "numerical aperture" (NA), which is effectively a measure of how quickly light converges to the focal point of the camera lens. In air, NA is defined as the radius of the entrance or exit pupil at the first principal plane divided by the focal length. Generally speaking, decreasing the camera NA will increase the DOF, while increasing the camera NA will decrease the DOF.

[0003] In some cases, it is desirable to capture images of an area of ​​an object in focus at different focal lengths. For example, a user may want multiple images starting from an initial location closest to the camera, with images taken at successively greater distances from the initial location, and ending with the image farthest from the camera, or vice versa. Each of the images would then have a different area or depth of the object in focus.

[0004] This process can be particularly useful in imaging biological or chemical samples under microscopic magnification. Due to the high magnification and numerical aperture required to acquire microscopic digital images of biological and chemical samples, the DOF of the image is very limited. Often, the DOF of a microscope digital imaging camera is in the 1-3 μm range or even less. For example, a microscope digital imaging camera with a combined optical magnification of 40x may have a typical DOF of approximately 2 μm, while the depth or thickness of a biological or chemical sample on a specimen slide or the like may be 20 μm or more. Thus, an imaging system with these typical specifications would produce an image in which only 2 μm of the total sample depth of 20 μm is in focus. Therefore, acquiring a focused image of the entire depth of the sample would require adjusting the focal length and / or the relative distance between the specimen and the camera multiple times (approximately 10 times) and taking 10 different images, one at each focal depth. Each image would then show the portion of the object that is within the DOF at the respective focal depth.

[0005] Multiple images taken at different focal depths form a "through-focus" image stack of multiple image planes (10 image planes in the above example), each representing an image plane at a different focal depth of the sample. The through-focus image stack can be fused into a single two-dimensional fused image by selecting the most in-focus pixels from the different image planes for every pixel in the fused image and fusing such pixels into the fused image. This produces a single two-dimensional image in which all of the objects in the image appear to be in focus regardless of the respective image planes in which the objects originate.

[0006] The ability to acquire and display microscopic digital images of biological or chemical samples under microscopic magnification (e.g., 40x or greater optical magnification) has recently been disclosed for use in cytology and pathology. Instead of a cytologist or pathologist using a microscope to examine and analyze a biological sample on a slide, a digital image of the sample is captured, and the cytologist or pathologist accesses the digital image to examine and analyze the sample. Cytology is the branch of biology that deals with the study of cell formation, structure, and function. When applied in a laboratory setting, cytologists, cytotechnologists, and other medical professionals make medical diagnoses of a patient's condition based on visual examination of a sample of the patient's cells, i.e., such a sample is referred to herein as a "cytological" sample. A typical cytological technique is the "Pap smear" test, in which cells are scraped from a woman's cervix and analyzed to detect the presence of abnormal cells, i.e., precursors to the development of cervical cancer. Cytological techniques are also used to detect abnormal cells and disease in other parts of the human body.

[0007] Cytological techniques are widely adopted because collecting cell samples for analysis is generally less invasive than traditional surgical pathology procedures, such as biopsies, whereby a solid tissue sample, referred to herein as a "pathology" sample, is excised from a patient using a specialized biopsy needle with a spring-loaded translatable stylet, fixed cannula, or the like. Cell samples can be obtained from a patient by a variety of techniques, including, for example, by scraping or swabbing an area, or by using a needle to aspirate bodily fluids from the chest cavity, bladder, spinal canal, or other suitable area. The obtained cell sample is typically placed in a preservative solution, subsequently extracted from the solution, and transferred to a glass slide. A fixative is applied to the cell sample to ensure that the cells remain in place on the slide to facilitate subsequent staining and examination.

[0008] Various automated slide imaging systems for capturing images of cytological samples fixed on slides have been previously disclosed. Examples of such imaging systems are disclosed in U.S. Patent Nos. 7,587,078, 6,665,060, 7,006,674, 7,369,304, and 7,590,492. In addition, the automated slide imaging system can perform a preliminary assessment of cells using image processing techniques to guide the cytotechnologist to potentially the most relevant cells on the slide for closer inspection. The imaging system also includes a review station where a reviewer can review the processed digital images. Whether by inspection of the actual specimen slide under magnification or by inspection of a magnified image of the specimen, specimens are typically classified by the cytotechnologist as either "normal" or "abnormal," with abnormal samples typically falling within one of the major categories defined by the Bethesda System for Reporting Cervical / Vaginal Cytology, which include low-grade squamous intraepithelial lesion (LSIL), high-grade squamous intraepithelial lesion (HSIL), squamous cell carcinoma, adenocarcinoma, atypical glandular cells of undetermined significance (AGUS), adenocarcinoma in situ (AIS), and atypical squamous cells (ASC). Additional information regarding cytological specimen classification is widely available, such as Yokohama System for reporting endometrial cytology: Diagnostic Cytopathology (May 2018); Vol. 46(5), pp. 400-412, and Guidelines for The Reporting of Nongynecologic Cytopathology Specimens, Archives of Pathology & Laboratory Medicine: (November 2009), Vol. 133, No. 11, pp. 1743-1756.

[0009] However, there are several drawbacks associated with conventional systems and methods for generating and displaying fused images of objects, particularly fused images of biological or chemical samples in the field of digital cytology. For example, the fused image "flattens" all of the objects in the fused image, making it impossible to grasp the relative depth of the objects or different parts of the objects in the image. For example, if an abnormally shaped object is present in the fused image, it may not be possible to determine whether the object is a single, oddly shaped cell or two overlapping cells at different depths in the sample (i.e., two objects in different image planes in the original image stack). As a result, the objects cannot be properly analyzed and characterized, which may cause cellular abnormalities in the sample that indicate a disease state to be overlooked. Therefore, cytotechnologists are unable to make accurate classifications or diagnoses.

[0010] One possible solution would be to make multiple images taken at different image planes available to the reviewer. However, this would significantly increase the amount of data that needs to be stored for each fused image. Furthermore, this increased amount of image data is particularly problematic when the images are stored remotely from the reviewer's location and must be transmitted to a review station where the reviewer can review and analyze the images.

[0011] Therefore, a need exists for improved methods and systems for providing multi-planar viewing of digital images of objects, such as cytological samples, without requiring the storage, transfer, and / or retrieval of multiple images taken at different image planes. [Prior art documents] [Patent documents]

[0012] [Patent Document 1] U.S. Patent No. 7,587,078 Summary of the Invention [Means for solving the problem]

[0013] Disclosed and described herein are embodiments of improved methods and systems for providing a multiplanar view of a digital image of an object that does not require access to multiple images of the object captured at multiple image planes. While the methods and systems are particularly useful in the context of imaging biological and chemical samples, such as digital cytology applications, the systems and methods are not limited to such use and may be used in any application for providing a multiplanar view of a digital image of an object. Thus, the embodiments disclosed herein are described for general application, with more specific examples included for use in digital cytology.

[0014] One exemplary embodiment is directed to a method for generating pseudo-focus images to provide a sense of depth in a digital image of an object as disclosed herein. In other words, the method provides for generating pseudo-focus images of each of multiple image planes of the digital image. The method utilizes a previously generated fused image of an object having a three-dimensional volume. For example, the object can be a biological sample attached to a specimen slide that is being imaged by an automated slide imaging system, such as the automated slide imaging systems disclosed and / or referenced herein. The slide imaging system can capture multiple digital images at different depths of the object, forming a through-focus image stack. A fused image is then generated that includes best-focus pixels selected from the multiple digital images, such that the best-focus pixels originate from different respective image planes at different respective depths of the object. Thus, each image plane is one of multiple image planes that stack to form at least a partial depth of the object. The method also utilizes a previously generated pixel depth map that associates each best-focus pixel with a respective image plane corresponding to the depth value at which the best-focus pixel occurred within the object. As two examples, the depth map may be embedded within the digital image file of the fused image, or it may be a separate file from the digital image file.

[0015] Next, a first plane of interest (POI) is selected from the multiple image planes. For example, the first POI can be the top image plane, the bottom image plane, or any image plane therebetween. For example, the first plane of interest can be selected in response to user input while viewing the fused image on a review station or other image display system, such as a computing device. On the review system, a user can view the fused image and use controls, such as sliders, dials, etc., to scroll up and down through the multiple image planes and input the first POI.

[0016] A first pixel focus offset is then calculated for each best-focus pixel, the first pixel offset being equal to the difference between the depth value of the first POI and the respective depth value for each best-focus pixel. In other words, the depth map is utilized to determine a depth value for each best-focus pixel, and the first pixel offset is determined by considering the difference between such depth value and the depth value of the first POI. As an example, consider a fused image generated from 14 images taken at 14 sequential focus depths of an object, such that image planes can be numbered 0-13, with image plane 0 being the bottom image plane of the multiple image planes and image plane 13 being the top image plane. For example, assume image plane 7 (i.e., approximately the middle of the image stack) is selected as the first POI. The first pixel offset for each best-focus pixel is then the difference between the depth value of image plane 7 (the first POI) and the depth value of each respective best-focus pixel.

[0017] Based on the respective first pixel focus offsets, a first blur focus value is then generated for each best-focus pixel. In an additional aspect, the blur focus value for each best-focus pixel may be generated using a Gaussian blur function. To simulate blur that would be present in an image captured by the imaging system, the Gaussian blur function may utilize a Gaussian blur radius that is based on the depth of field and / or the effective numerical aperture of the camera system of the imaging system.

[0018] A first pseudo-focus image of the final fused image for the first POI is then generated by combining the first blur focus values ​​for each best-focus pixel. In another aspect of the method, the first pseudo-focus image can then be displayed on a display device such as an LCD display, an LED display, an OLED display, and a computer monitor.

[0019] In yet another aspect, the method may further include generating a pseudo-focus image at another or second POI. Selection may be responsive to user input, similar to the first POI. Then, similar to calculating the first pixel focus offset, a second pixel focus offset is calculated for each best-focus pixel, the second pixel focus offset being equal to the difference between the depth value of the second POI and the respective depth value for each best-focus pixel. For example, assume that the user selects image plane 8 (second POI) in the above example. The second pixel offset for each best-focus pixel is the difference between the depth value of image plane 8 (second POI) and the depth value of each respective best-focus pixel (obtained from the depth map).

[0020] Based on each second pixel focus offset, a second blurred focus value is generated for each best-focus pixel, similar to generating the first blurred focus value, by using a Gaussian blur function or the like. A second pseudo-focus image of the final fused image for the second POI is generated by combining the second blurred focus values ​​for each best-focus pixel. In yet another aspect, the second pseudo-focus image can be displayed on a display device.

[0021] Generating and displaying pseudo focus images of any POI can be repeated for selection of any POI. Indeed, a user can scroll up and down through the image plane, and the method can generate and / or display pseudo focus images for the selected image plane as the user scrolls through the image plane in substantially real time. The term "substantially real time" means that there is a negligible delay that can be noticed by the user between the selection of a POI and the display of the pseudo focus image for such selected POI. In another aspect, once a POI is selected and each pseudo focus image is generated, any one or more of the pseudo focus images for the selected POI can be stored in a buffer or otherwise. Then, if the user selects a previously selected POI, the method displays the stored pseudo focus image for such POI. Buffering can smooth the processing time for displaying pseudo focus images in substantially real time.

[0022] In another aspect of the method, some of the steps may be performed across the entire fused image, making the process faster and / or more efficient. In one aspect, calculating the first pixel focus offset for each best-focus pixel may be determined for all of the best-focus pixels by generating a pixel focus offset array. The pixel focus offset array includes, for each best-focus pixel across the entire fused image, a pixel focus offset equal to the difference between the depth value of the first POI and the respective depth value for each best-focus pixel. The pixel focus offset array defines a range of pixel focus offsets within which multiple focus offset values ​​are defined.

[0023] Further, generating a first blur focus value for each of the best-focus pixels may be performed by applying a blur function to the entire fused image for each pixel focus offset within a range of pixel focus offsets, which may be done by calculating, for each best-focus pixel, a pixel focus offset equal to the difference between the depth value of the first POI and the respective depth value for each best-focus pixel, generating a pixel focus offset array, the pixel focus offset array defining a first range of pixel focus offsets within which a plurality of focus offset values ​​are defined.

[0024] A plurality of focus offset images are generated corresponding to each first focus offset within the pixel focus offset array. Each focus offset image is generated by (1) generating a blur focus value for each of the focus offset values ​​and (2) applying the respective blur focus value to the fused image for each focus offset value. For example, in the above example where the first POI is image plane 7, the range of first focus offsets is 0 to 7 (i.e., the focus offset for each image plane is the absolute value of the difference between the image plane value and the first POI; thus, image plane 0 has a first focus offset of 7, image plane 1 has a first focus offset of 6, ..., image plane 7 has a first focus offset of 0, image plane 8 has a first focus offset of 1, etc.). The process generates a focus offset image for each offset from 0 to 7, where each focus offset image contains the blur focus value for the respective focus offset for all best-focus pixels. For example, focus offset image 7 with a focus offset of 7 is an image with all of the pixels in the fused image at a blur focus value of 7, focus offset image 6 for a first focus offset of 6 is an image with all of the pixels in the fused image at a blur focus value of 6 for focus offsets from 0 to 7, etc.

[0025] In yet another aspect, generating a pseudo-focus image for a particular POI, such as the first POI, may also be accomplished in a convolutional manner by generating a focus mask for each image plane. Each focus mask includes all of the best-focus pixels associated with the respective image plane. In other words, focus mask 0 consists of all of the best-focus pixels in image plane 0, focus mask 1 consists of all of the best-focus pixels in image plane 1, and so on. A pseudo-focus image is then generated for each image plane of the multiple image planes by summing the product of (1) the focus mask for the respective image plane and (2) a focus offset image having focus offset values ​​corresponding to the respective image plane. In the above example with 14 image planes 0-13, the pseudo-focus image is the sum of the following:

[0026] The product of focus mask 0 (for image plane 0 with a focus offset value of 7) and focus offset image 7 +

[0027] The product of focus mask 1 (for an image plane with a focus offset value of 6) and focus offset image 6 +

[0028] ···+

[0029] The product of focus mask 7 (for image plane 7 with focus offset value 0) and focus offset image 0 +

[0030] The product of the focus mask 8 (for image plane 8 with a focus offset value of 1) and the focus offset image 1 +

[0031] ...Total for all 14 image planes 0-13.

[0032] Additionally, a convolution process can be used to generate each pseudo-focus image for any selection of POI. Indeed, the process for generating each successive pseudo-focus image can be further streamlined by buffering focus offset images for each focus offset value as they are generated for each selected POI. In the example above, focus offset images for focus offset values ​​0 through 7 are generated and buffered when image plane 7 is selected as the POI. For example, consider a second POI selected as image plane 10. The range of focus offset values ​​for the second POI, image plane 10, is 0 through 10 (image plane 0 has a focus offset value of 10, image plane 1 has a focus offset value of 9, image plane 2 has a focus offset value of 8, image plane 3 has a focus offset value of 7, and so on for image planes 4 through 13). Thus, focus offset images have previously been generated for focus offset values ​​0-7, and additional focus offset images only need to be generated for focus offset values ​​8-10 to generate pseudo-focus images for the second POI / image plane 10. After generating focus offset images for focus offset values ​​8-10, a pseudo-focus image is generated for each image plane of the multiple image planes by summing the products of (1) the focus mask for the respective image plane and (2) the focus offset image with the focus offset value corresponding to the respective image plane. The newly generated focus offset images for focus offset values ​​8-10 may also be buffered for use when needed to generate pseudo-focus images for a subsequently selected POI.

[0033] In yet another aspect of the method, the fused image can be an RGB digital image file and the pixel depth map can be stored in a channel of the RGB digital image file. Alternatively, the fused image can be stored as a digital image file and the depth map can be stored in a file separate from the digital image file.

[0034] In yet another aspect, a method for generating a pseudo-focus image may also include capturing multiple images of an object, processing the images to generate a fused image, and generating a pixel depth map. First, multiple images of the object are captured using a digital camera system. The multiple images are focused through a range of depths of the object so that the images together form a through-focus set of images comprising multiple image planes at different depth values ​​of the object, such that the final fused image has multiple pixels from different image planes for each pixel. Next, a best-focus pixel from the through-focus set of images is determined for each pixel in the final fused image. The final fused image of the multiple images is generated by combining the best-focus pixel for each pixel in the final fused image. A pixel depth map is generated by determining and associating depth values ​​for each best-focus pixel in the final fused image.

[0035] In another aspect, multiple images can be captured using a camera system with an optical axis that is non-orthogonal to the surface of the object or the surface to which the object is attached, such as a specimen slide or imaging platform.

[0036] Another embodiment disclosed herein is directed to a second method for generating a pseudo-focus image of a digital image to provide a sense of depth to the digital image. Similar to the method described above, a fused digital image of an object is accessed. Again, the object may be a biological sample attached to a specimen slide being imaged by an automated slide imaging system, such as the automated slide imaging systems disclosed and / or referenced herein. Multiple digital images are taken at different depths of the object so that the images form a through-focus image stack. The fused image includes best-focus pixels selected from the multiple digital images such that best-focus pixels originate from different respective image planes at different respective depths of the object, each image plane being one of multiple image planes that stack to form at least a partial depth of the object. The method also utilizes a previously generated pixel depth map that associates each best-focus pixel with a respective image plane corresponding to the depth value at which each best-focus pixel occurred within the object. As two examples, the depth map may be embedded within the digital image file of the fused image, or it may be a separate file from the digital image file.

[0037] Next, a first plane of interest (POI) is selected from the multiple image planes. This step is the same as in the first method embodiment described above. The method then utilizes an overall fused image approach to generate a pseudo-focus image for the selected POI in a process the same as or similar to the process described as an aspect of the first method embodiment. Multiple focus offset images are generated, with each focus offset image corresponding to each focus offset value within the pixel focus offset array, by (1) generating a blur focus value for each focus offset value and (2) applying the respective blur focus value to the fused image for each focus offset value. This is the same as or similar to the process described above that results in a focus offset image for each focus offset value.

[0038] A focus mask for each image plane is also generated, with each focus mask including all of the best-focus pixels associated with the respective image plane. For example, if there are 14 image planes for fused images 0-13, as in the example above, a focus mask for each of the 14 image planes is generated, with focus mask 0 consisting of the best-focus pixels in image plane 0, focus mask 1 consisting of the best-focus pixels in image plane 1, and so on, equivalently for all 14 image planes.

[0039] A pseudo-focus image is then generated as a union of the focus offset values ​​for each respective pixel for the focus offset value at which each respective pixel occurred. This is accomplished by associating each of the focus masks with its appropriate focus offset image, and then each association is summed to form the pseudo-focus image. In other words, a first pseudo-focus image is generated for each image plane of the multiple image planes by summing the product of (1) the focus mask for the respective image plane and (2) the focus offset image with the focus offset value corresponding to the respective image plane.

[0040] In another aspect of the method, the first pseudo-focus image may then be displayed on a display device such as an LCD display, an LED display, an OLED display, and a computer monitor.

[0041] In yet another aspect, the method may further include generating a pseudo-focus image at another or second POI. Selection may be responsive to user input, similar to the first POI. A second or complementary pixel focus offset array is generated for each best-focus pixel by calculating a pixel focus offset equal to the difference between the depth value of the second POI and the respective depth value for each best-focus pixel, the second pixel focus offset array defining a second range of pixel focus offsets within which a plurality of focus offset values ​​are defined. Additional focus offset images corresponding to each focus offset value within the second range of pixel focus offsets are generated only for those focus offset values ​​not generated for the first POI, each additional focus offset image being generated by (1) generating a blur focus value for each of the focus offset values ​​and (2) applying the respective blur focus value to the fused image for each focus offset value. Continuing with the above example in which the second POI is selected as image plane 10. The range of focus offset values ​​for the second POI, image plane 10, is 0-10 (image plane 0 has a focus offset value of 10, image plane 1 has a focus offset value of 9, image plane 2 has a focus offset value of 8, image plane 3 has a focus offset value of 7, and similarly for image planes 4-13). Thus, focus offset images have previously been generated for focus offset values ​​0-7, and additional focus offset images for focus offset values ​​8-10 only need to be generated to generate pseudo-focus images for the second POI / image plane 10. After generating the additional focus offset images, a second pseudo-focus image is generated for each image plane of the multiple image planes by summing the products of (1) the focus mask for the respective image plane and (2) the focus offset image with the focus offset value corresponding to the respective image plane. The newly generated focus offset images for focus offset values ​​8-10 may also be buffered for use when needed to generate pseudo-focus images for a subsequently selected POI.The second pseudo-focused image can then be displayed on a display device such as an LCD display, an LED display, an OLED display, and a computer monitor.

[0042] In another aspect, the process of generating and displaying a pseudo focus image of any POI can be repeated for the selection of any number of POIs. The user can scroll up or down through the image plane, and the method can generate and / or display a pseudo focus image for the selected image plane by repeating the above process for generating a second pseudo focus image as the user scrolls through the image plane in substantially real time. In another aspect, once a POI is selected and each pseudo focus image is generated, any one or more of the pseudo focus images for the selected POI can be stored in a buffer or otherwise. Then, if the user selects a previously selected POI, the method displays the stored pseudo focus image for such POI. Buffering can smooth the processing time for displaying the pseudo focus images in substantially real time.

[0043] In another aspect, a second method may also capture multiple images of an object the same as or similar to the method described above, process the images to generate a fused image, and generate a pixel depth map.

[0044] In yet another aspect of the second method, the focus offset image for each focus offset value may be generated using a two-dimensional Gaussian blur function. In another aspect, the Gaussian blur function may utilize a Gaussian blur radius based on the depth of field and / or the effective numerical aperture of the camera system of the imaging system to simulate the blur that would be present in an image captured by the imaging system.

[0045] In yet another aspect of the second method, the fused image may be an RGB digital image file and the depth map may be stored in a channel of the RGB digital image file, or alternatively, the fused image may be stored as a digital image file and the depth map may be stored in a file separate from the digital image file.

[0046] Another embodiment described herein is directed to an image review system for generating pseudo-focus images to provide a sense of depth to a digital image of an object according to any of the methods disclosed herein. The review system includes a computing system and a display device. The computing system includes one or more software applications, one or more microprocessors (CPUs), memory, and a network adapter. The computing system may also have a graphics processing unit (GPU) to improve speed when handling large digital image files. The computing system may also include input devices such as a keyboard, mouse, or touchpad. The display device is operably coupled to the computing system and configured to display images generated by the computing system. The display device may be any suitable display, such as an LCD display, LED display, OLED display, or other computer monitor. The display device may be a touchscreen monitor, where the touchscreen is also an input device. One or more software applications are configured to program the computing system to perform at least one of the methods for generating pseudo-focus images to provide a sense of depth to a digital image of an object as disclosed herein. By way of example, the review station may be configured to access and view the fused digital image of the biological sample. For example, the review system may be a cytology review station, a pathology review station, etc.

[0047] In another aspect of the image review system, the computing system may be in network communication via a network adapter with a remote storage system in which the fused images and depth map files are stored. For example, the remote storage system may be part of an automated imaging system, such as an automated slide imaging system for capturing images of samples on slides. The remote storage system may also be a cloud-based storage system. In this case, the computing system may access the fused images and depth files from the remote storage system via a communications network. The communications network may include one or more of a local area network (LAN, e.g., Ethernet), a wide area network (WAN), the Internet (e.g., a virtual private network (VPN)), and / or other suitable networks.

[0048] Thus, a user may utilize a computing system to access and view the fused image file and associated depth map, select various POIs within the fused image, and view pseudo-focus images for the selected POIs.

[0049] Other and further features and advantages of the disclosed embodiments of methods and systems for providing multi-planar viewing of a digital image of an object without requiring access to multiple images of the object captured in multiple image planes are depicted in the accompanying drawings and explained in the following detailed description thereof. The present invention further provides, for example, the following: (Item 1) 1. A method for generating pseudo-focus images for providing a sense of depth in a digital image of an object having a three-dimensional volume, the method comprising: (1) accessing a fused image of an object having a three-dimensional volume, the fused image comprising images generated from best-focus pixels selected from a plurality of images of the object forming a through-focus image stack, the best-focus pixels originating from different respective image planes at different respective depths of the object, each image plane being one of a plurality of image planes stacked to form at least a partial depth of the object; and (2) a pixel depth map associating each best-focus pixel with a respective image plane corresponding to a depth value at which the best-focus pixel occurred within the object; selecting a first plane of interest (POI) from the plurality of image planes; calculating, for each best focus pixel, a first pixel focus offset equal to the difference between the depth value of the first POI and the respective depth value for each best focus pixel; generating a first blur focus value for each best focus pixel based on the respective first pixel focus offset; generating a first pseudo-focus image of the fused image for the first POI constructed from the first blur focus values ​​for each best-focus pixel; A method comprising: (Item 2) Item 10. The method of item 1, further comprising displaying the first pseudo-focused image on a display device. (Item 3) Item 3. The method of item 2, wherein the display device is selected from the group consisting of an LCD display, an LED display, an OLED display, and a computer monitor. (Item 4) selecting a second POI from the plurality of image planes, the second POI being different from the first POI; calculating, for each best focus pixel, a second pixel focus offset equal to the difference between the depth value of the second POI and a respective depth value for each best focus pixel; generating a second blur focus value for each best focus pixel based on the respective second pixel focus offset; generating a second pseudo-focus image of the fused image for the second POI constructed from the second blur focus values ​​for each best-focus pixel; Item 1, the method of claim 1 further comprising: (Item 5) Item 5. The method of item 4, further comprising displaying the second pseudo-focused image on a display device. (Item 6) 2. The method of claim 1, wherein the multiple images of the object used to generate the fused image are captured using a camera having an optical axis perpendicular to the surface of the slide to which the object is attached. (Item 7) Item 10. The method of claim 1, wherein the first blur focus value for each best focus pixel is generated using a two-dimensional Gaussian blur function. (Item 8) Item 8. The method of item 7, wherein the two-dimensional Gaussian blur function utilizes a Gaussian blur radius that is based on the depth of field of a camera system used to acquire the multiple images of the object. (Item 9) Item 9. The method of item 8, wherein the Gaussian blur radius is also based on the effective numerical aperture of the camera system. (Item 10) generating the first blur focus value for each best focus pixel based on the respective pixel focus offset; Item 10. The method of item 9, comprising generating, for each best-focus pixel, a pixel focus offset array having a pixel focus offset equal to the difference between the depth value of the first POI and a respective depth value for each best-focus pixel, wherein the pixel focus offset array defines a range of pixel focus offsets within which a plurality of focus offset values ​​are defined. (Item 11) generating a first blur focus value for each best focus pixel based on the respective pixel focus offset; Item 11. The method of item 10, comprising generating a plurality of focus offset images corresponding to each first focus offset within the range of the pixel focus offset array by: (1) generating a blur focus value for each of the focus offset values; and (2) applying, for each focus offset value, the respective blur focus value to the fused image. (Item 12) generating a first pseudo-focus image of the fused image for the first POI constructed from the first blur focus values ​​for each best-focus pixel; generating a focus mask for each image plane, each focus mask including all of the best focus pixels associated with the respective image plane; for each image plane of the plurality of image planes, summing the product of (1) the focus mask for the respective image plane and (2) the focus offset image having the focus offset value corresponding to the respective image plane; Item 12. The method according to item 11, comprising: (Item 13) Item 10. The method of item 1, wherein the fused image is an RGB digital image file and the pixel depth map is stored in a channel of the RGB digital image file. (Item 14) Item 10. The method of item 1, wherein the fused image is stored as a digital image file and the pixel depth map is stored in a file separate from the digital image file. (Item 15) 1. A method for generating pseudo-focus images for providing a sense of depth in a digital image of an object having a three-dimensional volume, the method comprising: capturing multiple images of an object using a digital camera system, the images being in focus through a range of depths of the object, together forming a through-focus set of images comprising multiple image planes at different depth values ​​of the object such that there are multiple pixels from different image planes for each pixel in a fused image; determining, for each pixel in the fused image, a pixel of best focus from the through-focus set of images; generating, for each pixel in the fused image, the fused image of the plurality of images constructed from the pixel of best focus; generating a pixel depth map composed of depth values ​​for each best-focus pixel in the fused image; selecting a first plane of interest (POI) from the plurality of image planes; calculating, for each best focus pixel, a first pixel focus offset equal to the difference between the depth value of the POI and the respective depth value for each best focus pixel; generating a first blur focus value for each best focus pixel based on the respective pixel focus offset; generating a first pseudo-focus image of the fused image for the POI constructed from the first blur focus values ​​for each best-focus pixel; A method comprising: (Item 16) Item 16. The method of item 15, further comprising displaying the first pseudo-focused image on a display device. (Item 17) Item 17. The method of item 16, wherein the display device is selected from the group consisting of an LCD display, an LED display, an OLED display, and a computer monitor. (Item 18) selecting a second POI from the plurality of image planes, the second POI being different from the first POI; calculating, for each best focus pixel, a second pixel focus offset equal to the difference between the depth value of the second POI and the respective depth value for each best focus pixel; generating a second blur focus value for each best focus pixel based on the respective second pixel focus offset; generating a second pseudo-focus image of the fused image for the second POI constructed from the second blur focus values ​​for each best-focus pixel; Item 16. The method of item 15, further comprising: (Item 19) Item 19. The method of item 18, further comprising displaying the second pseudo-focused image on a display device. (Item 20) Item 16. The method of item 15, wherein the multiple images of the object used to generate the fused image are captured using a camera system having an optical axis that is non-orthogonal to a major plane of a specimen slide to which the object is attached. (Item 21) Item 16. The method of item 15, wherein the first blur focus value for each best focus pixel is generated using a two-dimensional Gaussian blur function. (Item 22) Item 22. The method of item 21, wherein the two-dimensional Gaussian blur function utilizes a Gaussian blur radius that is based on the depth of field of a camera system used to acquire the multiple images of the object. (Item 23) Item 23. The method of item 22, wherein the Gaussian blur radius is also based on the effective numerical aperture of the camera system. (Item 24) generating a first blur focus value for each best focus pixel based on the respective pixel focus offset; Item 24. The method of item 23, comprising generating a pixel focus offset array for each best focus pixel having a pixel focus offset equal to the difference between the depth value of the first POI and the respective depth value for each best focus pixel, the pixel focus offset array defining a range of pixel focus offsets within which a plurality of focus offset values ​​are defined. (Item 25) generating a blur focus value for each best focus pixel based on the respective pixel focus offset; Item 25. The method of item 24, comprising generating a plurality of focus offset images corresponding to each focus offset value within the range of the pixel focus offset array by: (1) generating a first blur focus value for each of the focus offset values; and (2) applying, for each focus offset value, the respective blur focus value to the fused image. (Item 26) generating, for each best-focus pixel, a first pseudo-focus image of the fused image for the first POI constructed from the blurred focus values; generating a focus mask for each image plane, each focus mask including all of the best focus pixels associated with the respective image plane; for each image plane of the plurality of image planes, summing the product of (1) the focus mask for the respective image plane and (2) the focus offset image having the focus offset value corresponding to the respective image plane; Item 26. The method according to Item 25, comprising: (Item 27) Item 16. The method of item 15, wherein the fused image is formatted as an RGB digital image file and the pixel depth map is stored in a channel of the RGB digital image file. (Item 28) Item 16. The method of item 15, wherein the fused image is stored as a digital image file and the pixel depth map is stored in a file separate from the digital image file. (Item 29) 1. A method for providing multiple planar views of a digital image of an object having a three-dimensional volume, the multiple planar views providing a sense of depth in the digital image, the method comprising: acquiring a fused image of an object having a three-dimensional volume, the fused image comprising pixels generated from a plurality of images of the object having best focus pixels occurring at different respective depths within the object, a pixel depth map associating each best focus pixel with a respective image plane corresponding to a depth value at which each best focus pixel occurs within the object, each image plane being one of a plurality of image planes stacked to form at least a partial depth of the object; selecting a first plane of interest (POI) from the plurality of image planes; generating a pixel focus offset array by calculating, for each best focus pixel, a pixel focus offset equal to the difference between the depth value of the first POI and the respective depth value for each best focus pixel, wherein the pixel focus offset array defines a first range of pixel focus offsets within which a plurality of focus offset values ​​are defined; (1) generating a blur focus value for each of the focus offset values; and (2) generating a plurality of focus offset images corresponding to each focus offset value within the range of the pixel focus offset array by applying, for each focus offset value, the respective blur focus value to the fused image. generating a focus mask for each image plane, each focus mask including all of the best focus pixels associated with the respective image plane; generating, for each image plane of the plurality of image planes, a first pseudo-focus image of the fused image for the first POI by summing the product of (1) the focus mask for the respective image plane and (2) the focus offset image having the focus offset value corresponding to the respective image plane; selecting a second POI from the plurality of image planes, the second POI being different from the first POI; generating a second pixel focus offset array by calculating, for each best focus pixel, a pixel focus offset equal to the difference between the depth value of the second POI and the respective depth value for each best focus pixel, wherein the second pixel focus offset array defines a second range of pixel focus offsets in which a plurality of focus offset values ​​are defined; generating additional focus offset images corresponding to each focus offset value within the second range of pixel focus offsets only for those focus offset values ​​not generated for the first POI, each additional focus offset image being generated by (1) generating a blur focus value for each of the focus offset values, and (2) applying, for each focus offset value, the respective blur focus value to the fused image; generating, for each image plane of the plurality of image planes, a second pseudo-focus image of the fused image for the second POI by summing the product of (1) the focus mask for the respective image plane and (2) the focus offset image having the focus offset value corresponding to the respective image plane; A method comprising: (Item 30) displaying the first pseudo-focused image on a display device; displaying the second pseudo-focused image on the display device; 30. The method of claim 29, further comprising: (Item 31) Item 31. The method of item 30, wherein the display device is selected from the group consisting of an LCD display, an LED display, an OLED display, and a computer monitor. (Item 32) storing each of the generated focus offset images in a buffer; utilizing the stored focus offset images in generating a series of pseudo-focus images of the fused image in successive image planes as the selected POI is scrolled up and down through a plurality of the image planes by a user; displaying, on a display, the sequence of pseudo-focus images of the fused image in successive image planes as the selected POI is scrolled up or down through a plurality of the image planes; 30. The method of claim 29, further comprising: (Item 33) Item 33. The method of item 32, wherein the sequence of pseudo-focus images of the fused image is displayed substantially in real time as the selected POI is scrolled up or down by a user. (Item 34) 30. The method of claim 29, wherein the multiple images of the object used to generate the fused image are captured using a camera system having an optical axis that is non-orthogonal to the slide surface to which the object is attached. (Item 35) 30. The method of claim 29, wherein the focus offset image for each focus offset value is generated using a two-dimensional Gaussian blur function. (Item 36) Item 36. The method of item 35, wherein the two-dimensional Gaussian blur function utilizes a Gaussian blur radius that is based on the depth of field of a camera system used to acquire the multiple images of the object. (Item 37) Item 37. The method of item 36, wherein the Gaussian blur radius is also based on the effective numerical aperture of the camera system. (Item 38) 30. The method of claim 29, wherein the fused image is formatted as an RGB digital image file and the pixel depth map is stored in a channel of the RGB digital image file. (Item 39) 30. The method of claim 29, wherein the fused image is stored as a digital image file and the pixel depth map is stored in a file separate from the digital image file. [Brief explanation of the drawings]

[0050] The foregoing and other aspects of the embodiments will be described in further detail with reference to the accompanying drawings, in which like reference numerals refer to like elements and descriptions of like elements are intended to be applicable with respect to any described embodiment where relevant.

[0051] [Figure 1] FIG. 1 is a block diagram of a digital imaging system including an image review station according to one embodiment.

[0052] [Figure 2] FIG. 2 illustrates an example of multiple imaging planes of images taken at different focal depths of an object, forming a through-focus image stack and a fused image of best-focus pixels generated from the multiple images.

[0053] [Figure 3] FIG. 3 shows the tilted imaging plane of a digital imager and a series of tilted images taken by the digital imager, where the optical axis of the digital image camera is non-orthogonal to the major plane of the sample slide on which the sample is attached.

[0054] [Figure 4] FIG. 4 shows a diagram of the tilted imaging plane of a digital imager and the vertically reassembled image plane in a through-focus image stack of tilted images taken by the tilted camera.

[0055] [Figure 5] FIG. 5 is a diagram of a pyramidal image of a fused image including successively reduced resolution images of the fused image.

[0056] [Figure 6]FIG. 6 illustrates an example of a fused image and its associated graphical pixel depth map.

[0057] [Figure 7] FIG. 7 illustrates a graphical example of focus offset images generated for exemplary fused images for a range of focus offsets based on a selected plane of interest.

[0058] [Figure 8] FIG. 8 illustrates a graphical example of focus masks for multiple image planes of the exemplary fused image of FIG.

[0059] [Figure 9] FIG. 9 is a table showing, for each image plane of the example fused image of FIG. 7, the focus offset and the product of the focus mask and focus offset image for each respective image plane.

[0060] [Figure 10] FIG. 10 illustrates a graphical example of the organization of the final pseudo-image by summing the products of the focus masks for each image plane and the associated focus offset images for the exemplary fused image of FIG.

[0061] [Figure 11] FIG. 11 is a line drawing that reproduces FIG. 6 and illustrates an example of a fused image and its associated graphical pixel depth map.

[0062] [Figure 12] FIG. 12 is a line drawing that replicates FIG. 7 and illustrates a graphical example of focus offset images generated for exemplary fused images for a range of focus offsets based on a selected plane of interest.

[0063] [Figure 13] FIG. 13 is a line drawing that reproduces FIG. 8 and illustrates a graphical example of focus masks for multiple image planes of the exemplary fused image of FIG.

[0064] [Figure 14] FIG. 14 is a line drawing that replicates FIG. 10 and illustrates a graphical example of the organization of the final pseudo-image by summing the products of the focus masks for each image plane and the associated focus offset images for the exemplary fused image of FIG. 12. DETAILED DESCRIPTION OF THE INVENTION

[0065] FIG. 1 illustrates a block diagram of one embodiment of an image review station 100 that provides multiplanar viewing of a digital image of an object without requiring access to multiple images of the object captured in multiple image planes. The object has a three-dimensional volume that includes depth along an axis orthogonal to the object's primary plane. This is accomplished by an innovative process for generating pseudo-focused images of fused images to provide a sense of depth in the digital image. The systems and methods described herein can be utilized for multiplanar viewing of digital images of any object where images have been captured in multiple image planes relative to the object's depth. While the systems and methods are still particularly useful in the context of images of biological and chemical samples, such as for digital cytology applications, the systems and methods are not limited to such use. Thus, the system and method embodiments described in detail herein are disclosed in the context of digital cytology, where the imaged objects in the digital images are biological and chemical samples. Nevertheless, the systems and methods disclosed herein are not limited to use with images of biological and chemical samples and can be used to provide multiplanar views of digital images of any type of object.

[0066] The image review station 100 includes a computing system 102 and a display device 104 operably coupled to the computing system 102. The computing system 102 includes one or more microprocessors 106, which may include one or more computer processors (CPUs) 106a and one or more graphics processors (GPUs) 106b. The computing system further includes memory 108, one or more storage devices 110 (e.g., a local hard drive for storing software applications 112 and data), one or more network adapters 116, such as a LAN, WAN, or USB network adapter, and other input / output ports 118 (I / O ports, such as USB ports). The software applications 112 include an image processing software application 112a (described in more detail below). The computing system also includes one or more input devices 114, such as a keyboard, mouse, or touchpad, to allow a user to interact with the computing system 102. The display device 104 may be a touchscreen monitor, where a touchscreen is the input device 114.

[0067] As depicted in FIG. 1 , the image review station 100 can be part of a digital imaging system 120 that includes an imager 122, an image processor 124, and a remote storage system 126. The imager 122 has a digital camera 128 for acquiring digital images of an object. For example, the imager 122 can be an automated digital imaging system for automatically acquiring multiple digital images of samples on each substrate with the sample disposed thereon. The substrate can be a microscope sample slide or any other suitable substrate, such as a microplate, microarray, or other suitable medium. The automated digital imaging system has a substrate handling device (e.g., a robot) for handling a large load of substrates with samples and digitally imaging each of the sample substrates. A suitable automated digital imaging system is disclosed in U.S. Patent Application No. 17 / 282,380. The contents of all publications cited herein, including but not limited to patents and patent applications, are incorporated by reference in their entirety for all purposes.

[0068] The imager 122 is configured to acquire multiple digital images of the object, the digital images being focused through a range of depths of the object, whereby the multiple digital images of the object together form a through-focus group of images or an "image stack." Turning to FIG. 2 , an example is shown of multiple imaging planes (multiple imaging planes 132 forming a through-focus image stack 134) generated from multiple images taken at different focal depths of the object 130 (also referred to as sample 130). In the illustrated example of FIG. 2 , there are 14 image planes, namely, image plane 132 a (bottom image plane) through image plane 132 n (top image plane). As shown in FIG. 2 , each image plane 132 a-132 n is at a different respective depth of the object 130, from the bottom image plane 132 a upward through the object 130 to the top image plane 132 n. The multiple digital images are acquired by the digital camera 128 of the imager 122. An example system and method for acquiring multiple digital images is also described in US patent application Ser. No. 17 / 282,380.

[0069] For microscopic imaging, the field of view of the digital camera 128 is typically much smaller than the area of ​​the sample 130 to be imaged. Therefore, the digital camera 128 and the sample 130 are moved relative to one another to scan the field of view of the camera 128 over the entire sample area (or a predetermined subarea of ​​the entire sample 130). To acquire digital images at different focal depths in multiple image planes 132, the focal depth of the camera 128 can be adjusted at each scan location to acquire images at different image planes, or alternatively, as shown in FIGS. 3 and 4, the optical axis of the digital camera 128 can be non-orthogonal to the plane of the sample 130. As shown in FIG. 3, the digital camera 128 utilizes an imager focal plane that is tilted relative to the sample 130 on a sample slide 131 having a sample slide cover 133. Because the imager focal plane is tilted relative to the sample 130 on the slide 131, different subregions of the camera pixel array capture images of the sample 130 at different depths within the sample 130. Thus, object A is captured in focus by camera 128 at imaging position #1, but in that same camera frame, object B is near the right edge of the frame and is far out of focus. Instead of stepping the camera position by the length of the field of view in the scan direction between images, a step size equal to the image subregion that matches the depth of field of the imaging optics is used. The digital images in the sequence may overlap. Object A is in focus with the camera at position #1, but object B is captured in focus when the camera reaches position #4. In this way, all objects within a height equal to the difference between the left and right edges of the imager focal plane are captured in focus in one or more of the digital images in the sequence.

[0070] The image processor 124 is configured to process the multiple images acquired by the imager 122, including generating a fused image 140 of the through-focus image stack and generating a pixel depth map that associates each best-focus pixel with the depth value at which the best-focus pixel occurred in the image stack. The computing system 136 may be the same as or similar to the computing system 102. The computing system 136 includes an image processing software application 138 that programs the computing system to access and process the digital images acquired by the imager 122. The image processor 124 processes the multiple digital images to form a fused image 140, which includes best-focus pixels from different image planes 132 at different respective depths for each pixel of the fused image 140. Methods of processing multiple digital images to form a fused image are also described in U.S. Patent Application No. 17 / 282,380.

[0071] The following describes one exemplary method implemented by image processor 124 to process multiple digital images to generate fused image 140 and its associated pixel depth map. Digital camera 128 may utilize a Bayer mask (a color filter array for placing RGB color filters on a grid of light sensors). In such cases, the first step in the image processing algorithm is to de-Bayer the image and convert it to an RGB image (a bitmap image that retains red / green / blue values ​​for each pixel). The digital image is then uniformity and distortion corrected, such as through a pre-mapped calibration table, to correct for optical distortions in the digital image. The table contains an entry for each pixel location in the resulting corrected image. Each entry indicates a 2x2 pixel subregion of the source image and the weighting factors to be applied to those source pixels and summed to obtain the resulting pixel value. This allows the corrected image to represent (by interpolation) the sub-pixel shift required to correct for the small optical distortions measured during the calibration process.

[0072] 4, in the case where the imager 122 is configured with the optical axis of the digital camera 128 non-orthogonal to the plane of the sample 130 (i.e., the imager focal plane is tilted with respect to the plane of the sample 130), the camera pixel array is divided into multiple image planes 132 (e.g., 14 image planes 132), which represent different focal planes across the sample depth. As the same sample area (tile) is captured in each image plane 132, the image planes 132 are vertically reassembled to create a through-focus image stack 134 of that tile.

[0073] Once all of the subregion images for a tile (e.g., 14 subregions) have been captured, the best-focus pixel from each different image plane is selected for each pixel in the fused image to generate the fused image. In one exemplary process, the difference between each pixel and a pixel in its neighboring region is calculated. This difference is then weighted based on the pixel's value to determine the relative merit of that particular plane for that particular pixel, resulting in a tile of plane values ​​(e.g., 0-13). A moving average is then applied to the tile of plane values ​​to provide better transitions between objects in the final image. The best-focus pixel in the fused image is then selected based on the associated plane value. The fused image tiles are then tiled together to generate a swath. Once a complete swath is assembled, it is stitched onto the previous swath. Once the images of the entire sample 130 are stitched together, the entire fused image 140 can be compressed. For example, the fused image 140 may be compressed at a ratio of approximately 20:1 using JPEG, JPEG 2000, or another suitable compression algorithm, and the fused image 140 may be stored as a TIF or JPG file. As depicted in FIG. 5, a pyramid image 142 may also be generated from the fused image 140. The pyramid image 142 aids in image rendering and display on the display device 104 while at lower magnifications. The pyramid image 142 subsamples the fused image 140 at progressively lower resolutions for display purposes until the minimum required image is generated. For example, the pyramid image may include four levels of successively reduced resolution, from ½ resolution down to 1 / 16 resolution.

[0074] Image processor 124 also generates a pixel depth map, which associates each best-focus pixel with a depth value (e.g., the image plane 132 at which the best pixel focus occurred in through-focus image stack 134). The pixel depth map associates each best-focus pixel with a depth value, such as by associating each best-focus pixel with the image plane 132 at which the best pixel focus occurred in through-focus image stack 134. 11 1, line drawing) illustrates an example of a fused image 140 and its associated pixel depth map 146, in which color (grayscale used in this example) represents the origin image plane 132 for each best-focus pixel in the fused image.

[0075] There are two main options for storing the fused image 140 and the pixel depth map. In one option, the fused image 140 is stored as a pyramidal shaped standard image file with multiple channels, such as a TIF or JPG file (e.g., an RGBA file where the "A" channel is typically the opacity of the image), and the pixel depth map is stored in one of the channels of the digital image file (e.g., the "A" channel of an RGBA image file). In a second option, the pyramidal fused image 140 can be stored as a digital image file (e.g., a TIF or JPG file), and the depth map can be stored in a file separate from the digital image file. In cases where only 14 or fewer image planes 132 are used in the fused image 140, a 4-bit-per-pixel image can be used, which can lead to reduced storage requirements, especially if image compression can be used.

[0076] Each pyramidal fused image 140 and its associated pixel depth map are then stored on a remote storage device / server 126. The remote storage device / server 126 comprises a data storage device accessible by the image review station 100 via a communications network 144. The communications network 144 may include one or more of a LAN, a WAN, the Internet (e.g., a virtual private network (VPN)), and / or other suitable networks. As some examples, the remote storage device / server 126 may be a private data server or a third-party cloud storage system.

[0077] Referring back to FIG. 1 , image review station 100 is configured to access fused image 140 and the respective pixel depth map and pyramid image, display fused image 140 on display device 104 (including generating and displaying a pseudo-focus image of fused image 140), and provide a sense of depth of objects in fused image 140. As described herein, fused image 140 includes all of the best-focus pixels, and therefore all of the fused image is substantially in focus, thereby effectively “flattening” the image into a single plane. As a result, it may be difficult or impossible to grasp the relative depth of objects, or to determine the extent or boundaries of objects, or to distinguish separate objects (e.g., when images of objects overlap). Thus, in the context of cytology, it may not be possible to determine whether an odd-shaped object is a single odd-shaped cell or two overlapping cells at different depths in the sample (i.e., two objects in different image planes of the original image stack). Thus, cytologists may be unable to properly analyze and characterize such cells, which may cause them to overlook cellular abnormalities in the sample that indicate a disease state, thereby preventing the cytotechnologist from making an accurate diagnosis. Instead of storing all of the multiple digital images at different image planes for different depths of the object of interest 130, the image review station 100 can use the perfectly focused fused image 140 and the pixel depth map to generate a pseudo-focus image that simulates the blurring of objects in the fused image that are not within the depth of field of the currently viewed selected image plane 132 (i.e., the POI). Thus, the image processing software application 112a is configured to access the fused image 140 and the associated pixel depth map and pyramid image, and program the image review station 100 to generate and display a pseudo-focus image of the fused image 140 for the selected image plane.

[0078] The image review station 100 generates a pseudo-focus image of the fused image by applying a blur function to simulate the blur observed in viewing a field of view with a depth that extends beyond the DOF. The DOF is the range of focus between the nearest and farthest regions of the field of view that are sharply focused, essentially eliminating any difference in focus quality to the viewer. Quantitatively, the DOF is the approximate mean wavelength of light (approximately 580 nm) divided by the square of the numerical aperture (NA).

[0079]

number

[0080] Numerical aperture is effectively the degree to which light converges to a focal point. In air, it is defined as the radius of the entrance or exit pupil at the first principal plane divided by the focal length. Effective NA (NAeff) is used when the illumination NA is smaller than the imaging NA at the sample. It is an average of the two and works well to determine the expected amount of optical blur due to defocus. The equation for calculating Naeff is:

[0081]

number

[0082] The image processing performed by the image review station 100 to generate a pseudo-focus image is based on several fundamental optical effects. One method for simulating optical blur is to convolve a 2D Gaussian function over the image, which is effectively what aberration-free optical blur does. This is not the same as actual optical blur in a microscope (because there are both aberrations and partial coherence effects that are not accounted for in the method). However, to a user, such a blurred image will generally appear natural.

[0083] It is desirable that the amount of simulated optical blur in the pseudo-image be as close as possible to the actual amount of optical blur, and therefore the optical parameters of the imager 122 are used to calculate the Gaussian blur radius, which is affected by the DOF of the optical system, the distance from best focus, and the NAeff.

[0084] In the imager 122 disclosed herein, NAeff is calculated from the imaging NA and illumination NA. For example, for a typical imaging NA of 0.75 and a typical illumination NA of 0.3, NAeff is calculated as 0.525. At 580 nm, it is 0.580 μm / 0.525 2 = 2.10 μm, resulting in a DOF of 10 μm.

[0085] To determine the Gaussian blur radius, NAeff is again used. NAeff defines the cone angle for light outside the DOF, and therefore the blur radius is defined by the following formula:

[0086]

number

[0087] The Gaussian blur function is then:

[0088]

number

[0089] To calculate the Focus Offset in the above formula, one must start with a selected first POI from the multiple image planes 132 of the fused image 14 that the user selects to apply the blur function to. For example, the first POI may be selected in response to user input using the input device 114 while viewing the fused image 140 on the image review station 100. The input device 114 may adjust a slider, dial, etc., to scroll up or down through the multiple image planes 132 to input the first POI.

[0090] Using the example of FIG. 2, where there are 14 image planes 132a-132n numbered image plane 0-image plane 13, we start with an example where the user selects an "intermediate" image plane. This would be image plane 7 (image plane 132h). The pixel depth map contains image plane data for each pixel in the fused image 0-13. Therefore, for each pixel, there is the difference between the POI and the depth map value. This is the pixel's plane offset. The plane offset can be converted to a focus offset by multiplying it by the Z offset between the planes (which for this example is 1.85 μm). Therefore, the focus offset for each pixel is given by the following equation, in terms of the offset at the pixel location (px, py):

[0091]

number

[0092] Once the focus offset is determined for a given pixel, a 2D Gaussian blur function is applied to blend that fused image pixel with neighboring pixels to generate a blurred pixel value. The entire pseudo-focus image of the fused image is generated by essentially convolving the 2D Gaussian blur function over the entire fused image, where the Gaussian blur kernel is a function of pixel position (i.e., the pixel position gives the focus offset, the focus offset gives the blur radius, and the blur radius gives the kernel parameters).

[0093] As different POIs among the multiple image planes are selected by the user, the image review station 100 repeats this process, generating pseudo-focus images for each POI and displaying the pseudo-focus images on the display device 104.

[0094] The application of the blur function to generate the pseudo-focus image is essentially the convolution of a Gaussian function over the original fused image with the radius as input. Most image processing packages use a fast Fourier transform (FFT) to generate the blur, which is substantially faster than standard convolution. In this case, a different blur function must be applied for each pixel. When implementing the method using traditional convolution, the kernel may actually be modified pixel-by-pixel according to the depth map. However, in practice, it is faster to simply apply the blur function to the original image for each possible focus offset, generate a focus offset image corresponding to each possible focus offset, and then select pixels from the appropriate offset image to generate the merged pseudo-focus image. This is done by generating a pixel focus offset array by calculating, for each best-focus pixel, a pixel focus offset equal to the difference between the depth value of the first POI and the respective depth value for each best-focus pixel. The pixel focus offset array defines a first range of pixel focus offsets within which multiple focus offset values ​​are defined. For example, in the example of FIG. 2, with image plane 7 (image plane 132h) as the selected POI, the pixel focus offset ranges from 0 to 7 (i.e., the focus offset for each image plane is the absolute value of the difference between the image plane value and the first POI: thus, image plane 0 has a first focus offset of 7, image plane 1 has a first focus offset of 6, ... image plane 7 has a first focus offset of 0, image plane 8 has a first focus offset of 1, etc.).

[0095] A plurality of focus offset images are generated, each corresponding to a respective focus offset within the pixel focus offset array. Each focus offset image is generated by (1) generating a blur focus value for each of the focus offset values ​​and (2) applying the respective blur focus value to the fused image for each focus offset value. In the example of FIG. 2, with image plane 7 (image plane 132h) as the selected POI, this process generates a focus offset image for each focus offset from 0-7, each focus offset image including a blur focus value for each focus offset for each best-focus pixel. FIG. 7 (and / or FIG. 12, line drawing) illustrates a graphical example of focus offset images 150 for focus offsets 0-7. As shown in FIG. 7 (and / or FIG. 12, line drawings), focus offset image 150a for a focus offset of 7 is an image where all of the pixels in the fused image are at a blur focus value of 7, focus offset image 150b for a focus offset of 6 is an image where all of the pixels in the fused image are at a blur focus value of 6, equivalent for focus offsets of 0 to 7, etc. As can be seen in FIG. 7 (and / or FIG. 12, line drawings), both image plane 1 and image plane 13 have the same focus offset of 6, both image plane 2 and image plane 12 have the same focus offset of 5, both image plane 3 and image plane 11 have the same focus offset of 4, both image plane 4 and image plane 10 have the same focus offset of 3, both image plane 5 and image plane 9 have the same focus offset of 2, and both image plane 6 and image plane 8 have the same focus offset of 1, so for focus offsets of 1 to 6, there are two different planes 132 associated with the same focus offset image.

[0096] Image review station 100 also generates a focus mask 152 for each image plane 132. Each focus mask 152a-152n includes all of the best-focus pixels associated with the respective image plane. For example, focus mask 0 (152a) consists of all of the best-focus pixels in image plane 0, focus mask 1 (152b) consists of all of the best-focus pixels in image plane 1, etc. Figure 8 (and / or Figure 13, line drawing) illustrates a graphical example of focus masks 152 for image planes 0-13 for the example of Figure 2.

[0097] A pseudo-focus image of the fused image for the selected POI is then generated by summing, for each image plane 11 of the multiple image planes 132, the product of (1) a focus mask 152 for the respective image plane and (2) a focus offset image 150 having a focus offset value corresponding to the respective image plane 132. For the example of FIG. 2 and image plane 7 (image plane 132h) as the selected POI, the table in FIG. 9 shows, for each image plane 132, the focus offset and the product of the focus mask and focus offset image for each respective image plane. Thus, in this example, focus mask 0 is associated with focus offset image 7, focus mask 1 is associated with focus offset image 6, focus mask 2 is associated with focus offset image 5, focus mask 3 is associated with focus offset image 4, focus mask 4 is associated with focus offset image 3, focus mask 5 is associated with focus offset image 2, focus mask 6 is associated with focus offset image 1, focus mask 7 is associated with focus offset image 0 (fused image, i.e., no blur), focus mask 8 is associated with focus offset image 1, focus mask 9 is associated with focus offset image 2, focus mask 10 is associated with focus offset image 3, focus mask 11 is associated with focus offset image 4, focus mask 12 is associated with focus offset image 5, and focus mask 13 is associated with focus offset image 6. Thus, the pseudo-focus image for the POI that is image plane 7 is the sum of the third column in the table of FIG. 9.

[0098] 10 (and / or FIG. 14, line drawing) shows a graphical example of the final pseudo-image organization by summing the products of the focus masks 152 for each image plane 132 and the associated focus-offset images 150. For context, the sum of all of the focus masks 152 is an all-white image, while the sum of each focus mask 152 multiplied by its associated focus-offset image 150 is the final pseudo-focus image 156.

[0099] One of the most processing-intensive tasks in generating a pseudo-focus image is generating the blurred images, i.e., generating each of the focus offset images. In contrast, the generation of the focus mask and the sum of the pseudo-focus images is relatively very quick. Because the generation of the pseudo-focus image for any of the image planes 132 selected as a POI utilizes the same focus offset images, these same focus offset images can be stored in a buffer for use in generating and displaying pseudo-focus images for any selected POI as the user scrolls up and down through the image planes 132 while the displayed field of view remains the same. In practice, it is likely that the user will repeatedly move the POI (i.e., the image plane 132 that is in focus within the pseudo-focus image) and examine the image differences; therefore, the buffer can significantly smooth processing time. This can help enable the image review station 100 to display pseudo-focus images for successively selected POIs as the user scrolls through the image planes in substantially real time. Furthermore, once a POI is selected and a respective pseudo focus image is generated, any one or more of the pseudo focus images for the selected POI may be stored, such as in a buffer, or otherwise. Then, when the user selects a previously selected POI, the method displays the stored pseudo focus image for such POI. Buffering may smooth the processing time for displaying the pseudo focus images in substantially real time.

[0100] Therefore, to generate pseudo-focus images of the fused image 140 at another POI, the image review station 100 need only generate additional focus offset images for those focus offsets outside the range of focus offsets for which focus offset images were previously generated. The additional focus offset images are generated in the same manner as the previously generated focus offset images. Continuing with the example of Figure 2 above, for example, image plane 10 is selected as the next POI. The range of focus offset values ​​for this next POI as image plane 10 is 0 to 10 (image plane 0 has a focus offset value of 10, image plane 1 has a focus offset value of 9, image plane 2 has a focus offset value of 8, image plane 3 has a focus offset value of 7, image plane 4 has a focus offset of 6, image plane 5 has a focus offset of 5, image plane 6 has a focus offset of 4, image plane 7 has a focus offset of 3, image plane 8 has a focus offset of 2, image plane 9 has a focus offset of 1, image plane 10 has a focus offset of 0, image plane 11 has a focus offset of 1, image plane 12 has a focus offset of 2, and image plane 13 has a focus offset of 3). Therefore, focus offset images have previously been generated for focus offset values ​​0 to 7, and additional focus offset images for focus offset values ​​8 to 10 only need to be generated to generate a pseudo-focus image for image plane 10. After generating focus offset images for focus offset values ​​8-10, pseudo focus images are generated in the same manner as described above for image plane 7, with, of course, appropriate association of each focus offset image with a focus mask for image plane 10 as the POI, where these newly generated focus offset images for focus offset values ​​8-10 may also be buffered for use when needed to generate pseudo focus images for a subsequently selected POI.

[0101] The process of generating and displaying a pseudo-focus image of the fused image can be repeated for any selection of POI.

[0102] While particular embodiments have been shown and described, it should be understood that the above description is not intended to limit the scope of these embodiments, and that such disclosure is provided for purposes of explanation and illustration only. Accordingly, various changes and modifications may be made to the disclosed embodiments without departing from the scope of the following claims. For example, not all of the components described in the embodiments are required; alternative embodiments may include any suitable combination of the described components, and the general shape and relative sizes of the components may be modified. Furthermore, although system and method embodiments are described with reference to cytological samples as digital image objects, they may be configured and utilized with any type of object.

Claims

1. 1. A method for generating a pseudo-focused image of a digital image of an object having a three-dimensional volume, the method comprising: (1) accessing a fused image of an object having a three-dimensional volume, the fused image comprising images generated from best-focus pixels selected from a plurality of images of the object forming a through-focus image stack, the best-focus pixels originating from different respective image planes at different respective depths of the object, each image plane being one of a plurality of image planes stacked to form at least a partial depth of the object; and (2) a pixel depth map associating each best-focus pixel with a respective image plane corresponding to a depth value at which the best-focus pixel occurred within the object; selecting a first plane of interest (POI) from the plurality of image planes; calculating, for each best focus pixel, a first pixel focus offset equal to the difference between the depth value of the first POI and the respective depth value for each best focus pixel; generating a first blur focus value for each best focus pixel based on the respective first pixel focus offset; generating a first pseudo-focus image of the fused image for the first POI constructed from the first blur focus values ​​for each best-focus pixel; Including, generating the first blur focus value for each best focus pixel based on the respective pixel focus offset; generating, for each best focus pixel, a pixel focus offset array having a pixel focus offset equal to the difference between the depth value of the first POI and the respective depth value for each best focus pixel, the pixel focus offset array defining a pixel focus offset range within which a plurality of focus offset values ​​are defined; generating the first blur focus value for each best focus pixel based on the respective pixel focus offset; generating a plurality of focus offset images corresponding to each first focus offset within the range of the pixel focus offset array by (1) generating a blur focus value for each of the focus offset values; and (2) applying, for each focus offset value, the respective blur focus value to the fused image; generating the first pseudo-focus image of the fused image for the first POI constructed from the first blur focus values ​​for each best-focus pixel includes: generating a focus mask for each image plane, each focus mask including all of the best focus pixels associated with the respective image plane; for each image plane of the plurality of image planes, summing the product of (1) the focus mask for the respective image plane and (2) the focus offset image having the focus offset value corresponding to the respective image plane; A method comprising:

2. Displaying the first pseudo-focused image on a display device. The method of claim 1 further comprising:

3. The method of claim 2, wherein the display device is selected from the group consisting of an LCD display, an LED display, an OLED display, and a computer monitor.

4. selecting a second point of interest (POI) from the plurality of image planes, the second point of interest being different from the first point of interest; calculating, for each best focus pixel, a second pixel focus offset equal to the difference between the depth value of the second POI and the respective depth value for each best focus pixel; generating a second blur focus value for each best focus pixel based on the respective second pixel focus offset; generating a second pseudo-focus image of the fused image for the second POI constructed from the second blur focus values ​​for each best-focus pixel; The method of claim 1 further comprising:

5. Displaying the second pseudo-focused image on a display device. The method of claim 4 further comprising:

6. The method described in claim 1, wherein the multiple images of the object used to generate the fused image are captured using a camera having an optical axis perpendicular to the slide surface to which the object is attached.

7. The method of claim 1 , wherein the first blur focus value for each best focus pixel is generated using a two-dimensional Gaussian blur function.

8. The method of claim 7 , wherein the two-dimensional Gaussian blur function utilizes a Gaussian blur radius that is based on the depth of field of a camera system used to acquire the multiple images of the object.

9. The method of claim 8 , wherein the Gaussian blur radius is also based on an effective numerical aperture of the camera system.

10. The method described in claim 1, wherein the fused image is an RGB digital image file and the pixel depth map is stored in a channel of the RGB digital image file.

11. The method of claim 1, wherein the fused image is stored as a digital image file and the pixel depth map is stored in a file separate from the digital image file.

Images