Image processing and segmentation of a set of Z-stack images of a 3D biological sample

Abstract

Methods are provided for projecting a depth-spanning stack of limited depth-of-field images of a sample into a single image of the sample that can provide focused image information about the three-dimensional content of the image. These methods include applying a filter to the stack of images to identify pixels in each image taken in focus. These focused pixels are then combined to provide a single image of the sample. Filtering such an image stack can also enable the determination of a depth map or other shape information about the sample's contents. Such depth information can also be used to inform segmentation of the image of the sample at multiple different depths, for example, by further dividing identified regions corresponding to the sample's contents.

Description

[Technical field]

[0001] This application relates to image processing and segmentation of a set of Z-stack images of a three-dimensional biological sample. [Background technology]

[0002] Various biological experiments involve the analysis of a large number of samples, each of which may be associated with some parameter or other information generated through the measurement or evaluation of the sample. Such samples may include cells or other biological contents, each differing with respect to the growth medium (e.g., hormones, cytokines, drugs, or other substances in the growth medium), source (e.g., cultured from natural tissue, biopsied, or possibly transplanted), incubation conditions (e.g., temperature, pH, light level or spectrum, ionizing radiation), or some other controlled conditions to observe the response of the cells or other biological contents to the applied conditions. This may be done, for example, to evaluate the response of the sample to a putative treatment, to elucidate some in vivo action, or to investigate some other question of interest.

[0003] Evaluating such samples may include using a microscope, a fluorescent imager, or other means to image the sample microscopically. In practice, imaging a sample over a large depth of field may be hindered by the limitations of the optical or other devices used to image the sample. For example, the objective lens or other elements used to image the sample may be limited in terms of depth of focus, making it difficult or impossible to focus and simultaneously image the entire sample that spans a volume along the optical axis of the device used to image the sample that is greater than the depth of focus of the imaging device. Such "three-dimensional" samples may include various structures of interest, such as organoids, tumor spheroids, or other three-dimensional multicellular structures, as opposed to samples that are spread across a glass slide and / or sectioned into slices that span a volume that can fit within the depth of field of the imaging device. Summary of the Invention [Means for solving the problem]

[0004] One aspect of the disclosure relates to a method for generating a projection image of a three-dimensional sample, the method including the steps of: (i) acquiring a set of images of the sample, each image of the set of images corresponding to a respective focal plane within the sample; (ii) applying a filter to each image of the set of images to determine a respective depth value for each pixel of an output image of the sample, a given depth value representing a depth within the sample at which contents of the sample may be imaged in focus; and (iii) determining an image value for each pixel of the output image based on the depth value of the pixel of the output image. Determining the image value for a particular pixel of the output image includes the steps of: (1) identifying one image of the set of images that corresponds to the depth value of the particular pixel; and (2) determining the image value for the particular pixel based on a pixel of the identified image that has a location within the identified image that corresponds to the particular pixel.

[0005] Another aspect of the present disclosure relates to a method for generating a projection image of a three-dimensional sample, the method including the steps of: (i) acquiring a set of images of the sample, where each image of the set of images corresponds to a respective focal plane within the sample; (ii) applying a filter to each image of the set of images to determine a respective depth value for each pixel of a depth map, where the depth values ​​represent depths within the sample at which contents of the sample may be imaged in focus; and (iii) determining an image value for each pixel of an output image based on the depth value of a corresponding pixel of the depth map.

[0006] Yet another aspect of the present disclosure relates to a method for segmenting an image of a sample, the method including the steps of: (i) acquiring an image of the sample; (ii) acquiring a depth map of contents of the sample; (iii) generating a first segmentation map of the sample based on the image; and (iv) generating a second segmentation map of the sample based on the depth map by further dividing at least one region of the first segmentation map.

[0007] Yet another aspect of the present disclosure relates to a computer-readable medium configured to store at least computer-readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform computer operations to perform one or more of the methods described herein. Such a computer-readable medium may be a non-transitory computer-readable medium.

[0008] Yet another aspect of the present disclosure relates to a system that includes: (i) one or more processors; and (ii) a non-transitory computer-readable medium configured to store at least computer-readable instructions that, when executed by the one or more processors, cause the system to perform one or more of the methods described herein.

[0009] These and other aspects, advantages and alternatives will become apparent to those skilled in the art upon reading the following detailed description, with reference, as appropriate, to the accompanying drawings. Moreover, it should be understood that the description provided in this Abstract, and elsewhere in this document, is intended to illustrate the claimed subject matter by way of example and not by way of limitation. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a schematic cross-sectional view of a sample imaged by an imaging system. [Figure 2A] FIG. 1 shows a set of sample example images. [Figure 2B] FIG. 2B illustrates a projection image generated based on the example image set of FIG. 2A. [Figure 3A] 1A-1C show an example projection image and an example depth map of a sample. [Figure 3B] FIG. 3B illustrates a set of sample example images used to generate the depth map of FIG. 3A. [Figure 3C] FIG. 13 is a diagram of an exemplary set of texture values ​​determined over a range of sample depths. [Figure 4] FIG. 2 illustrates a set of example images of a sample used to generate a projection image of the sample. [Figure 5A] FIG. 13 is an example image of a sample. [Figure 5B] FIG. 5B is an exemplary first segmentation map of the sample represented by the image of FIG. 5A. [Figure 5C] FIG. 5C illustrates a particular segment of the first segmentation map of FIG. 5B overlaid on an exemplary depth map. [Figure 5D] FIG. 5C is a diagram of an exemplary second segmentation map determined from the first segmentation map of FIG. 5B. [Figure 6] FIG. 1 is a perspective view of elements of an exemplary system. [Figure 7] FIG. 1 is a schematic diagram of elements of an exemplary system. [Figure 8] 1 is a flow chart of an exemplary method. [Figure 9] 1 is a flow chart of an exemplary method. [Figure 10] 1 is a flow chart of an exemplary method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Examples of methods and systems are provided herein. It should be understood that the words "exemplary," "example," and "illustrative" are used herein to mean "serving as an example, instance, or aid in explanation." Any embodiment or feature described herein as "exemplary," "example," and "illustrative" should not necessarily be construed as being desirable or advantageous over other embodiments or features. Moreover, the exemplary embodiments described herein are not intended to be limiting. It will be readily understood that certain aspects of the disclosed systems and methods can be arranged and combined in a wide variety of different configurations.

[0012] I. Exemplary Sample Imaging Microscopic imaging of a sample can be beneficial in a variety of applications. For example, the sample can include cultured human cells (e.g., cancer cells, normal cells), and imaging the sample can facilitate determining the effectiveness of a drug on cells to be removed within the sample (e.g., the effectiveness of a chemotherapy drug on removing cancer cells), determining the toxicity of a substance (e.g., the toxicity of a chemotherapy drug to non-cancerous cells), or determining some other information about the contents of the sample and / or about the effect of a substance on the contents of the sample. Imaging the sample can include using bright field microscopy, fluorescence microscopy, structured illumination, confocal microscopy, or some other imaging technique to generate image data about the contents of the sample.

[0013] Brightfield microscopy has the advantage that it can be performed without the addition of dyes, fluorophores, or other labels (e.g., by adding a label and / or by transfecting the sample's contents to express a label) that may alter the "natural" behavior of the sample's contents (such imaging without adding a label is sometimes referred to as "label-free" imaging). Alternatively, fluorescent dyes or other labels may be added to facilitate imaging of specific tissue structures and / or physiological processes. For example, Annexin V green fluorescent dye-based reagents may be added to image location, rate, or other information about cell death within the sample. In another example, NucLight Red indicators may be added to image information about cell proliferation within the sample. In some examples, cells within the sample may be genetically engineered to express a fluorescent protein or other label for a process of interest. For example, patient-derived neural induced pluripotent stem cells (iPSCs) may be used to express Aβ to facilitate evaluation of putative Alzheimer's treatments. 1-42 The cells can be genetically engineered to express -GFP, or any other fluorescently labeled protein associated with Alzheimer's disease.

[0014] Many objective lenses or other optical elements of an imaging device used to image a sample may be constrained with respect to depth of field. That is, the optical device may be limited to focusing and imaging light received from a relatively small and / or substantially flat volume of the sample. Light received from outside this volume will be received out of focus (by a charge-coupled device or other light receiving element of the imaging device). This constraint may be related to the cost or overall quality of the imaging device, a desire to reduce chromatic, axial, spherical or other aberrations in the imaging device, a desire to reduce the volume, size, weight or number of parts of the imaging device, a desire to improve the performance of the device in some way by limiting its use to imaging small and / or substantially flat volumes or by some other factor. Such a constraint in the depth of field of an imaging device may be compensated for in various ways.

[0015] In some examples, the sample to be imaged may be small in volume and / or substantially flat in nature, or may be modified to be so. However, many samples of interest include individual objects that may span a volume that does not fit within the narrow depth of field of the imaging device, and / or may contain useful information about the distribution, interconnection, or other spatial information of multiple objects in a direction perpendicular to the imaging plane of the imaging device, over a distance within the sample that exceeds the narrow depth of field of the imaging device. Such "three-dimensional" samples may include organoids, tumor spheroids, or some other three-dimensional multicellular structures of interest. Furthermore, such structures may be cultured or, in some cases, arranged in a medium (e.g., a dome or other shaped volume of extracellular medium) that spans a volume that does not fit within the narrow depth of field of the imaging device. For example, the sample may be part of a multi-spheroid assay, where tumor spheroids (or some other variety of spheroids) are arranged in a format that is layered and embedded within a volume of Matrigel.

[0016] Organoids (e.g., pancreatic organoids, liver organoids, intestinal organoids) and tumor spheroids are of particular interest because their three-dimensional structure more closely mimics the "natural" three-dimensional environment of the cells in which they are cultured. Accordingly, the response of organoids, tumor spheroids, or other such three-dimensional multicellular structures to drugs or other applied experimental conditions may more closely mimic the response to a corresponding sample in the human body or some other environment of interest. Organoids may be cultured from a patient's own cells to predict a particular patient's response to a range of different potential treatments. For example, iPSCs may be extracted from a patient and used to culture neural organoids, breast cancer organoids, or organoids that approximate any other healthy or unhealthy (e.g., cancerous) tissue of interest to facilitate evaluation of the response of such tissues to potential treatments.

[0017] Various techniques may be used to use limited depth of field imaging devices to image the contents of such samples. In some examples, a part or all of the sample may be spread on a microscope slide or other flat member to facilitate the imaging of the sample. However, sample spreading methods or other sample pretreatment methods to make the sample compatible with limited depth of field imaging devices may not be applicable to the sample of interest, may destroy the sample, may cause loss or distortion of information about the sample (e.g., distortion of the sample due to interaction with the microtome, freezing, sticking, being removed from the sample container, spreading, etc.), or may be inapplicable to the sample of interest. Furthermore, such sample pretreatment methods eliminate the possibility of imaging the same sample (e.g., the same sample of cultured organoids or tumor spheroids) at multiple time points, for example, to analyze the long-term effects of drugs or other experimental conditions over time.

[0018] Additionally or alternatively, an imaging device having a shallower depth of field could be substituted for an imaging device having a deeper depth of field, but such improved imaging devices may be too costly, too large to fit within the incubator or other environment of interest, or undesirable for some other consideration.

[0019] The above limitations of limited depth of field imagers may be alleviated, in whole or in part, by using the methods described herein. Applying these methods to the operation of such limited depth of field imagers allows for the generation of image information that may alternatively be obtained from an imager having a depth of field equal to or exceeding the volume that spans the sample in a direction parallel to the optical axis of the imager. The image processing methods described herein include generating multiple images of a "three-dimensional" sample (which may be referred to as a "stack" of images, each image corresponding to a different depth within the sample). Each image may be taken at a different depth within the sample that is in focus by using a motor or other actuator to control the location of the imager relative to the sample. This may include moving the camera, moving the sample container, or moving both the sample container and the camera.

[0020] FIG. 1 shows an exemplary system 100 that can be used to generate such a set of images. The system 100 includes an imaging device 110 (e.g., a camera, an objective lens, and / or other optical or electrical elements) and a light source 120 that can be used to image a sample contained in a sample container 130. The sample container 130 can be one well of a multi-well incubation or culture plate (e.g., a 96-well culture plate). The sample container 130 contains several objects 105 (e.g., organoids, tumor spheroids, or other three-dimensional biological objects) contained in a sample medium 135 (e.g., a dome of extracellular medium used to culture organoids before transferring them to a well of a 96-well multi-well culture plate or some other multi-well culture plate before applying a variety of different putative treatments or other experimental conditions, or a multi-spheroid assay that includes a layered and embedded array of tumor spheroids in a volume of Matrigel).

[0021] The arrangement of elements (e.g., camera 110 and light source 120) in FIG. 1 is intended as a non-limiting example. The light source used to illuminate the sample may be located on the opposite side of the sample from the camera or other imager, on the same side of the sample as the imager, on the side of the sample relative to the imager, or at some other location relative to the sample and imager. Furthermore, the light source may be incorporated within the imager and / or may share one or more optical elements or optical paths with the imager (e.g., the objective lens of a confocal imaging system may be shared between a laser or other element used to illuminate the sample and a charge-coupled device or other element used to image the light received from the sample). In some examples, multiple light sources may be provided in multiple locations to facilitate operations of imaging the sample according to different modes (e.g., fluorescence imaging, bright field imaging, reflected / scattered visible light imaging, structured illumination imaging, phase contrast imaging), for example.

[0022] The camera 110 and / or the light source 120 may be configured such that the camera 110 can image light received from a small, substantially flat volume within the sample container 130. Such a configuration may be selected instead of a large depth of field configuration to reduce cost, improve reliability, reduce size or mass, or provide some other benefit. For example, such a configuration may be selected to reduce the size and / or weight of the camera to facilitate mounting the camera 110, the light source 120, and / or other elements of the imaging device on a moving gantry within the incubator to enable long-term automated imaging of multiple samples within the incubator at multiple different time points. Region 140a represents, in cross section, a first exemplary extent of a volume that may be focused and imaged by such a limited depth of field imaging device. Region 140b represents, in cross section, another example of such a region that differs from the first region 140a in terms of depth within the sample container 130, but spans substantially the same area in a direction perpendicular to the optical axis of the camera 110.

[0023] The location of the camera 110 relative to the sample vessel 130 may be set at different times to facilitate imaging different volumes or depths within the sample vessel 130. This may include using a moving gantry or other means to move the camera 110 relative to the sample vessel 130 (e.g., by moving the camera 110 and / or moving the sample vessel 130) to select a flat area within the sample vessel 130 for imaging in focus. For example, during a first period of time (e.g., the period of time shown in FIG. 1), the camera 110 may be operated to image a first volume 140a in focus. The camera 110 may then be moved downward relative to the sample vessel 130 and subsequently operated to image a second volume 140b in focus. Such a procedure may be repeated for multiple different volumes and corresponding depths within the sample vessel 130 to generate a set or "stack" of images representative of the contents of the sample vessel 130. Such a procedure can facilitate the generation of focused image information for objects 105 (e.g., organoids, tumor spheroids, or other three-dimensional biological objects) that have a characteristic dimension greater than the depth of field of the camera 110 and / or are located at points extending along a vertical dimension within the sample container 130 at a distance greater than the depth of field of the camera 110.

[0024] Such a process for generating a set of "stacked" images may be performed automatically. For example, the camera and associated actuators may be operated to generate a set of such images of the sample at one of a number of specified time points (e.g., once every hour, once every 24 hours, etc., to facilitate analyzing the response of the sample contents to applied experimental conditions over time). Additionally or alternatively, the samples may be placed in a multi-well sample vessel, and a set of images may be generated for each of the samples in the vessel by actuating the camera in two dimensions to select a particular sample (e.g., by manipulating an actuator of a gantry containing the camera) and in three dimensions to image different depths within the particular sample.

[0025] FIG. 2A illustrates an exemplary set (or "stack") 200 of images of a sample. Each image of the set 200 of images corresponds to a respective focal plane within the sample. Accordingly, each image may represent a different content of the sample in focus depending on the location of the content relative to the focal plane corresponding to the image. In a first example, the top image 210a of the set 200 of images corresponds to a first focal plane where no sample content is located. Accordingly, all of the image information of the top image 210a is out of focus. In another example, the second image 210b of the set 200 of images corresponds to a second, different focal plane where some of the sample content is located. Accordingly, some of the image information of the second image 210a (i.e., the three-dimensional object to the left and bottom of the second image) is in focus, while other portions of the image information of the second image 210b are out of focus.

[0026] Such a set of images 200 can contain sufficient focused image information to generate a projection image of the sample that provides an improved view of the distribution of contents within the sample. For example, some or all of the contents of the sample can be represented in focus in the projection image. The projection image can approximate the image of the sample as if it were generated using a wider depth of field. FIG. 2B shows an example of a projection image 250 generated from a set of images including the set of images 200 shown in FIG. 2A. Such a projection image can facilitate improved imaging and analysis of the contents of the sample. For example, the projection image can be segmented to identify individual cells, organoids, tumor spheroids, or other three-dimensional contents of the projection image. Such image segmentation can then be used to automatically determine the number, size, identity, morphology, or other information of the cells, organoids, tumor spheroids, or some other analysis of the contents of the sample. Additionally or alternatively, such projection images can facilitate subjective analysis of the contents of the sample by a pathologist, researcher, epidemiologist, or some other person.

[0027] A projection image may be generated for a sample from a set of images corresponding to respective different focal planes within the sample in a variety of ways. In some examples, this may include using the set of images to determine a depth at which the contents of the sample are located and / or in focus. Pixels or other image information from the image or set of images corresponding to the determined depth may then be used (or "projected") to generate pixels of a corresponding projection image. Additionally or alternatively, such depth information may be used to generate a depth map for the contents of the sample (e.g., to facilitate analysis of the three-dimensional shape and / or arrangement of the contents of the sample) and / or to improve segmentation of the projection image (or of some other image of the sample).

[0028] II. Example Depth Map Generation A set of narrow depth-of-field images (e.g., bright-field images) of a sample, each corresponding to a different depth spanning a range of depths within the sample, may contain sufficient information to determine information about the depth of the sample's contents. A set of such "stacked" images may also contain sufficient image information that may be combined with the depth information to generate a projection image of the sample that represents the sample's contents in focus across a range of depths (e.g., as a sort of simulated wide depth-of-field image of the sample). The methods described herein facilitate generating such depth information (e.g., as a depth map) and such projection images from a set of narrow field-of-field images (e.g., bright-field images, fluorescent images).

[0029] These methods include detecting edges, textures, or other high spatial frequency content in each image of the set of images. The presence of such high spatial frequency content at a particular location in a particular image may indicate that the sample's content at the particular location was imaged in focus. Accordingly, the sample's content at the particular location may be assumed to be located at a depth within the sample that corresponds to the depth of the particular image. Thus, a depth map or other depth information may be generated for the sample by identifying "in focus" regions within each image of the set of images of the sample. These identified regions may then be combined across the set of images to generate a single depth map and / or projection image of the sample.

[0030] Identifying "in focus" regions in each image of the set of sample images may include applying a filter or transform to each of the images. For example, a Canny edge detector or other edge detection filter or algorithm may be applied to each image in the set of images to generate a respective "edge image" representing where edges are located in each of the images. In another example, a texture filter may be applied to each image in the set of images to generate a respective "texture image" representing where areas of increased high spatial frequency information (or "texture") are located in each image.

[0031] "Texture" may be determined in a variety of ways. For example, a texture value for a particular pixel of an image may be determined by determining the entropy, numerical range, standard deviation, variance, coefficient or variation, or some other measure of variability of a set of pixels in a neighborhood of the particular pixel. Such a neighborhood may be a square or other shaped region of nearby pixels, e.g., a 5×5 square region of pixels centered on the particular pixel for which a texture value is being determined for the pixel. The pixels in the "neighborhood" region may be weighted equally or may be used in a weighting method to determine the texture value (e.g., by giving higher weights to pixels closer to the particular pixel for which a texture value is being determined than to pixels further away when determining the entropy, standard deviation, etc.). Such texture values ​​may be determined for all pixels in the image, for a subset of pixels in the image (e.g., for every other pixel), or for some other set of locations in the image.

[0032] As noted above, regions of a particular narrow depth of field image having higher texture values ​​(or higher values ​​of some other characteristic related to the high spatial frequency content of the image) are more likely to have been imaged in focus. Thus, the content of the sample corresponding to a particular high texture location of a particular image of the sample is likely to be located at a depth within the sample that corresponds to the depth of the focal plane of the particular image. Texture information for a particular location in a set of images (e.g., corresponding to the location of a particular pixel index in each of the images) may be compared across all of the images to determine a single depth value for the particular location. Such a determination may be used to generate a full depth map for the sample and / or to generate pixels of a projected output image of the sample (e.g., by selecting a pixel of one of the set of images that corresponds to the determined depth for use in generating the pixel of the projected image).

[0033] To illustrate this process, FIG. 3A shows an image 301 of a sample (e.g., a projection image generated as described herein). Image 301 is shown next to a depth map 300 of the sample. Each object in image 301 may include one or more organoids, tumor spheroids, or other three-dimensional multicellular objects. Depth map 300 indicates that some of the seemingly single objects shown in image 301 may represent multiple overlapping objects at different depths within the sample.

[0034] 3B shows a depth map 300 of a sample and a set of narrow depth-of-field images 310a, 310b, 310c, 310d of the sample corresponding to different focal planes at different depths within the sample. The depth map 300 and the images of the set of images have the same resolution and pixel locations for ease of explanation. Those skilled in the art will appreciate that the methods herein may be adapted to environments in which the depth map of the set of images, the projected images, and / or the input images have different resolutions and / or pixel locations.

[0035] A depth value for a particular pixel 305 of the depth map 300 may be determined based on texture values ​​(or other values ​​representing the local magnitude of high spatial frequency image content) of corresponding pixels 315a, 315b, 315c, 315d in each image of the set of images 310a, 310b, 310c, 310d. As shown in FIG. 3B, the images of the set of images and the depth map may have the same size and resolution, in which case the pixel correspondence may be one-to-one. This is intended as a non-limiting example of determining a depth value for a particular pixel of the depth map based on texture information for each of the input images at a location in the input images that corresponds to the location of the particular pixel in the depth map.

[0036] The depth value may be selected in various ways based on a set of texture (or other high spatial frequency image content) values. For example, the highest texture value (or the lowest texture value, in cases where lower texture values ​​correspond to a greater amount of high spatial frequency image content) may be determined, and the depth is determined according to the depth of the input image that corresponds to the highest texture value. This may be done based on the assumption that the highest texture value is likely to correspond, for a particular pixel location, to the image that is most in focus at that particular location, and therefore the content of the sample is likely to be located at the corresponding depth within the sample.

[0037] Additional or alternative methods may be applied to determine a depth value based on a set of texture values. For example, depth may be determined based on the depth or other characteristics of a peak detected within the set of texture values. FIG. 3C shows a plot of a set of texture values ​​320 determined for a particular pixel of an output depth map as a function of the depth of an input image used to generate the texture value. A peak 325 is present in the texture data, which may be detected and the depth ("d1" in FIG. 3C) determined from this peak. Peak detection may be used to determine a depth value based on the assumption that for a particular pixel location, the peak in the texture value as a function of depth is likely to correspond to sample content that is in focus at that particular location, and therefore the sample content is likely to be located at the corresponding depth within the sample.

[0038] The depth values ​​determined as described above may be used as is as pixels of a depth map to generate pixels of a projection image, to improve segmentation of one or more images of the sample, or to facilitate some other application. Alternatively, some level of spatial preprocessing may be applied to the determined depth values ​​prior to such application (e.g., spatial preprocessing may be applied to a set of depth values ​​determined as described above to generate a depth map). For example, a two-dimensional low-pass filter or other type of linear filter may be applied to the depth values ​​prior to applying them within an application. Additionally or alternatively, non-linear preprocessing methods may be applied. For example, an edge-preserving low-pass spatial filter may be applied. As another example, depth values ​​that are outliers relative to their neighbors (e.g., exceeding the mean value for their neighbors by a specified amount, e.g., a multiple of multiple standard deviations of their neighbors) may be removed, may be set to a specific amount (e.g., the mean, median, or other central measure of their neighbors), may be filtered using more aggressive filter parameters, or may be preprocessed in some other way to reduce the impact of such outliers on subsequent processing.

[0039] III. Example Image Compositions Depth information for the sample (e.g., depth map, individual depth values) may be used to project pixels or other image information from a set of images of the sample onto a single projection image of the sample (e.g., as in exemplary projection images 250 and 301). The depth information may be used to determine which image of the set of images to draw from when setting intensity, color, or other image information for each pixel of the projection image. As noted above, if the set of input images varies relative to a focal plane within the sample, the depth information may be used to select which image of the images to project from when generating pixels of the projection image such that the projection image appears perfectly in focus or possibly improved relative to the set of input images.

[0040] 4 shows a projected image 400 of a sample and a set of narrow depth-of-field images 410a, 410b, 410c, 410d of the sample corresponding to different focal planes at different depths within the sample. The projected image 400 and the images in the set of images have the same resolution and pixel locations for ease of explanation. Those skilled in the art will appreciate that the methods herein may be adapted to environments in which the depth maps, projected images, and / or input images of the set of images have different resolutions and / or pixel locations.

[0041] An image value (e.g., one or more of luminance, chrominance, red, green, and / or blue channel color values) for a particular pixel 405 of the projected image 400 may be determined based on a depth value determined for the particular pixel 405. Such depth values ​​may be obtained from a sample depth map or may be determined on a pixel-by-pixel basis. The depth values ​​may be determined as described above (e.g., by identifying the depth of one image of the set of images having the highest texture value at a location corresponding to each pixel of the projected image) or may be determined using some other method, e.g., using a depth sensor of a phase difference image, some other depth detection means.

[0042] Determining an image value for the particular pixel 405 may include copying an image value of a corresponding pixel of an image that matches the depth value for the particular pixel 405. For example, in FIG. 4, the second image 410b has a depth that corresponds to the depth value determined for the particular pixel 405. In response, the image value (e.g., luminance, intensity, etc.) of the corresponding pixel 415b of the second image 410b is copied or projected to become the image value for the particular pixel 405.

[0043] This one-to-one projection of image values ​​is intended as a non-limiting example of determining an image value for a particular pixel of the projected image based on a depth value and one or more narrow depth-of-field images corresponding to that depth value. Additional pixels may be used (e.g., combined in a weighted combination) to generate an image value for the particular pixel 405. In some examples, the image value of the particular pixel 405 and / or neighboring pixels may be determined in whole or in part based on the number of pixels neighboring the corresponding pixel 415b of the second image 410b. Additionally or alternatively, image information from multiple images corresponding to depth values ​​within a defined range of the depth value of the particular pixel 405 may be used to generate an image value for the particular pixel 405. For example, an image value of a particular pixel 405 may be determined based on a combination of a corresponding pixel 415b of the second image 410b and corresponding pixels 415a, 415c of the first image 410a and the third image 410c, where the first image 410a and the third image 410c correspond to depths within a vicinity (e.g., a defined range) of the depth value of the particular pixel 405.

[0044] IV. Exemplary Image Segmentation It may be useful in various contexts to automatically identify the extent, location, size, identity, and / or other information about the individual contents of organoids, tumor spheroids, cells, particles, or other three-dimensional multicellular objects within an image of a biological sample or other environment of interest. Such a process may be referred to as "image segmentation." Image segmentation may be used to automatically perform various analyses on the contents of an image, for example, to determine the number, type, volume / size, spatial distribution, shape, growth rate, or other characteristics of cells, organoids, or tumor spheroids within a sample.

[0045] A variety of methods are available in the art for performing segmentation on microscopic images of biological samples. Such methods may include one or more of thresholding, clustering, edge detection, region growing, artificial neural network or other machine learning algorithms, or any other technique or combination of techniques for presumptively identifying different adjacent regions in an image. Each region identified using such methods may correspond to a respective organoid, cancer spheroid, cell, or component or part thereof. Such methods may be applied to one or more narrow depth of field images of the sample. Segmentation may be improved by segmenting the projection image of the sample determined as described above. Because such projection image represents more of the focused sample contents than any individual narrow depth of field image used to generate the projection image.

[0046] FIG. 5A shows an exemplary image 500 of a sample. The image may be a projection image generated as described elsewhere herein or may be acquired in some other manner. As shown in FIG. 5B, the image 500 may be segmented to generate a first segmented map 510 of the sample. The regions include an exemplary first region 515a, a second region 515b, and a third region 515c. Each region may represent a respective different object, objects, and / or portions of objects (e.g., organoids, tumor spheroids, cells) within the sample.

[0047] The accuracy of the identified regions may be limited by the image information available in the input image 500. For example, boundaries in the input image between separate overlapping or adjacent objects may be unclear or, in some cases, suboptimal in the sense that they prevent the segmentation method from identifying the separate objects as respective distinct regions in the segmentation map. FIG. 5C shows a sample depth map 520 of the input image 500. The depth map 520 may be generated based on a set of narrow depth-of-field images using the methods described herein (e.g., based on texture information present in the set of images) or may be generated via some other method (e.g., using a depth sensor). The depth map 520 indicates that at least the first region 515a and the second region 515b of the first segmentation map 510 are likely to represent multiple distinguishable objects based on the depth difference between different adjacent regions in the areas 525a, 525b of the depth map 520 corresponding to the first region 515a and the second region 515b, respectively.

[0048] Accordingly, depth information provided by the depth map 520 or from some other source may be applied to improve the segmentation of the image. This may include using the depth information from the depth map to further divide one or more regions of the first segmented image to generate an improved second segmented image. For example, depth information corresponding to a particular region of the first segmented map 510 may be analyzed to determine whether it represents two or more potential discrete populations of depth values. If the depth information represents the populations, the particular region may be further divided according to the location of the depth values ​​within each of the discrete populations.

[0049] 5C and 5D show an analysis of a first region 515a and a second region 515b of the first segmentation map 510 to determine whether to further divide those regions when generating an improved second segmentation map 530. For the first region 515a, depth values ​​(e.g., pixels of the depth map 520) corresponding to the location and extent of the first region 515a (shown in FIG. 5C as a first patch of depth values ​​525a) may be analyzed to identify regions into which the first region 515a may be further divided.

[0050] This may include performing clustering, region growing, or other analysis on the population of depth values ​​in the first patch 525a to identify two or more regions in the first patch 525a. The first region 515a may then be further divided into regions 535a, 535b, and 535c in the second segmentation map 530. This division may include generating additional divisions based on the location of depth map pixels in the depth map 520 corresponding to each of the identified clusters. Such a process may include performing filtering, region growing, edge preserving, or other processes to ensure that the determined division results in contiguous and / or fairly smooth regions 535a, 535b, and 535c following the division of the parent region 515a. A similar process may be applied to divide the second region 515b of the first segmentation map 510 into corresponding regions 535d, 535e of the second segmentation map 530. It should be noted that such an analysis may also result in no segmentation being performed since the third region 515c of the first segmentation map 510 is maintained as a single region 535f in the second segmentation map 530.

[0051] V. Example Applications The systems and methods described herein can be used to facilitate a variety of biological applications in sample imaging. This can include imaging at multiple time points over hours, days, weeks, or some other duration, or using an automated imaging system to image multiple samples (e.g., 96 samples contained in each well of a 96-well sample plate) placed in an incubator to avoid the need to perturb the samples by removing them from the incubator. The imaged samples can contain 3D cultures of human or tumor cells, organoids, tumor spheroids, or other cells. The cells can be natural or the result of some experimental process (e.g., the result of exposure to a carcinogen to generate cancer cells). Additionally, the cells can be labeled with fluorescent dyes or genetically modified to express fluorescent proteins, other reporter substances, or to provide some other biological insight (e.g., to evaluate the effect of genetic modification on the cells).

[0052] Samples may be prepared and imaged to perform drug discovery and / or to evaluate the toxicology of drugs or other substances in the fields of immunology, oncology, neuroscience, cell therapy, or other research fields. In examples where the sample contains and / or allows the development of organoids, imaging the sample can facilitate the study of organ development, the development and / or quantification of disease models, and / or the development of regenerative medicine.

[0053] The described system and method support various imaging modalities. For example, label-free bright field images can be taken of organoids, multi-spheroids embedded in extracellular medium, or other three-dimensional samples that can be placed in the wells of a 96-well assay. Fluorescence imaging can be used to image fluorescent reporters that can serve to label cells and / or can be reporters of cell function or of some other property of interest as part of a fluorescence assay. For example, Annexin V green fluorescent dye-based reagents can be imaged to evaluate location, rate, or other information about cell death in a sample, NucLight Red indicators can be imaged to evaluate information about cell proliferation in a sample, or some other fluorescent reporter or reporter system (e.g., multicolor FUCCI assay) can be fluorescently imaged to evaluate information about cell division, function, identity, or differentiation. This can be done, for example, to perform a cell health assay of a sample containing an organoid, multi-spheroid, or some other three-dimensional object of interest.

[0054] When multiple different samples are imaged in a multi-well plate (e.g., a 96-well plate), the samples can vary from well to well with respect to cellular content (e.g., cell type, tumor cell type), amount or identity of additional substances (e.g., administration of additional drugs, variants of various additional drugs specified as part of a drug discovery assay), type of genetic modification, or some other varying experimental condition. The automated imaging and image processing techniques described herein can then be applied to assess efficacy and / or toxicity of the applied substances / treatments, or to determine some other experimental data of interest.

[0055] VI. Exemplary Systems A variety of systems may be used (e.g., programmed) to perform the various embodiments described herein. Such systems may include desktop computers, laptop computers, tablets, or other single-user workstations. Additionally or alternatively, the embodiments described herein may be performed by a server, a cloud computing environment, or other multi-user system.

[0056] Such a system can analyze data received from other systems, for example, from a remote data storage device on a server, from a remote cell counter or other instrument, or from some other source. Additionally or alternatively, a system configured to perform the embodiments described herein may include and / or be coupled to an automated incubator, a sample imaging system, or some other instrument capable of generating experimental data for analysis. For example, such an instrument may include an incubator containing a multi-well sample vessel. Samples in such multi-well sample vessels may differ with respect to the genome of the sample, the source of the sample, the growth medium applied to the sample, the pharmaceutical or biological agent applied to the sample, or some other condition applied to the sample.

[0057] Samples in such devices may be experimentally evaluated in a variety of ways. Samples may be imaged (e.g., using visible, infrared, and / or ultraviolet light). Such imaging may include fluorescent imaging of the contents of the sample, e.g., imaging fluorescent dyes or reporters that are added to the sample and / or produced by cells of the sample (e.g., after insertion of genes encoding fluorescent dye molecules). Automated gantries may be installed within such incubators to facilitate imaging of, or measurement and analysis of, the various samples within each well of the sample container.

[0058] As an example, an automated imaging system may be used to automatically acquire images (e.g., bright field images, fluorescent images) of multiple biological samples in respective wells of a sample container during multiple different scanning periods over time. During each of the scanning periods, a set of images of each sample may be taken by the automated imaging system, e.g., a set of different images with respect to focal planes within the sample. The images may then be analyzed to determine a depth map, a projection image, or some information about the sample, e.g., according to the methods described herein.

[0059] The use of such an automated imaging system can not only significantly reduce the labor costs of imaging biological samples, but can also improve the uniformity of images generated with respect to timing, positioning, and image parameters when compared to manual imaging. Furthermore, such an automated imaging system can be configured to operate within an incubator, eliminating the need to remove samples from the incubator for imaging. Accordingly, the growth environment for the sample can be more consistently maintained. In addition, if the automated imaging system serves to move the microscope or other imaging device relative to the sample container (e.g., rather than moving the sample container imaged by a static imaging device), movement-related sample perturbations can be reduced. This can improve sample growth and development and reduce movement-related deterioration.

[0060] Such automated imaging systems can operate to acquire one or more images between scans that are separated by 24 hours or more, 3 days or more, 30 days or more, or some longer period. The scans can be specified to occur at a specified rate, for example, once a day, more than twice a day, more than three times a day, or more than four times a day. The scans can be specified to occur at least two, at least three, or some greater number of scans within a 24-hour period. In some examples, data from one or more scans can be analyzed (e.g., according to methods described herein) and used to determine the timing of additional scans (e.g., to increase the rate, duration, image capture rate, or some other characteristic of the scans to detect the occurrence of distinct events predicted to occur within the sample).

[0061] The use of such automated imaging systems can facilitate imaging the same biological sample at multiple time points over a long period of time. Accordingly, the development and / or behavior of individual cells and / or networks of cells (e.g., organoids, tumor spheroids) can be analyzed over time. For example, sets of cells, parts of cells, or other objects can be identified within scans taken during different, widely separated time periods within a single sample. These sets of identified objects can then be compared between scans to identify the same objects across scans. Thus, the behavior of individual organoids, tumor spheroids, cells, or parts of cells can be tracked and analyzed over hours, days, weeks, or months.

[0062] FIG. 6 illustrates elements of such an automated imaging system 600. The automated imaging system 600 includes a frame 610 to which other elements of the automated imaging system 600 are mounted. The frame 610 may be configured (e.g., sized) to fit within an incubator. The automated imaging system 600 includes sample containers 620 removably mounted within a sample container tray 630 coupled to the frame 610. The sample container tray 630 may be removable and / or may include removable inserts to facilitate holding a variety of different sample containers (e.g., various industry standard sample containers). The system 600 additionally includes an actuation gantry 650 configured to position the imaging device 640 relative to the sample containers 620 such that the imaging device 640 is operable to acquire images of the contents of individual wells of the sample containers 620 (e.g., exemplary well 625).

[0063] The imaging device 640 can include a microscope, a fluorescence imager, a two-photon imaging system, a phase-contrast imaging system, one or more light sources, one or more optical filters, and / or other elements configured to facilitate imaging of a sample contained within the sample vessel 620. In some examples, the imaging device 640 includes elements disposed on either side of the sample vessel 620 (e.g., a source of coherent, polarized, monochromatic, or otherwise defined illumination light to facilitate phase-contrast imaging of a biological sample). In such examples, the elements on either side of the sample vessel 620 may be coupled to different gantries, to the same gantry, and / or the elements on one side of the sample vessel 620 cannot move relative to the sample vessel 620.

[0064] The actuation gantry 650 is coupled to the frame 610 and the imaging device 640 and is configured to control the location of the device 640 in at least two directions relative to the sample container 620 to facilitate imaging of a plurality of different samples in the sample container 620. The actuation gantry 650 may also be configured to control the location of the imaging device 640 in a third direction toward and away from the sample container 620 to facilitate controlling the focal length of an image acquired using the imaging device 640 and / or to control the depth of material in the sample container 620 that may be imaged using the imaging device 640. Additionally or alternatively, the imaging device 640 may include one or more actuators to control the focal length of the imaging device 640. The imaging device 640 may include one or more motors, piezoelectric elements, liquid lenses, or other actuators to facilitate control of the focus setting of the imaging device 640. For example, the imaging device 640 may include an actuator configured to control the distance between the imaging device 640 and the sample being imaged. This may be done to ensure that images are taken in focus and / or to allow images to be taken such that various different focal planes within the sample are represented in each of the different images.

[0065] The actuation gantry 650 may include elements configured to facilitate detection of the absolute and / or relative location of the imaging device 640 with respect to the sample vessels 620 (e.g., with respect to a particular well of the sample vessels 620). For example, the actuation gantry 650 may include encoders, limit switches, and / or other location sensing elements. Additionally or alternatively, the imaging device 640 or other elements of the system may be configured to detect fiducial marks or other features of the sample vessels 620 and / or sample vessel tray 630 to determine the absolute and / or relative location of the imaging device 640 with respect to the sample vessels 620.

[0066] Computational functions (e.g., functions for operating actuation gantry 650 and / or imaging device 640 to image samples in sample containers 620 for a defined period of time and / or to perform any other method described herein) may be performed by one or more computing systems. Such computing systems may be integrated within a labware system (e.g., 600), may be associated with such a system (e.g., by being connected via a direct wired or wireless connection, via a local network, and / or via a secure connection over the Internet), and / or may take some other form (e.g., a cloud computing system in communication with an automated imaging system and / or having access to a storage of images of biological samples).

[0067] 7 illustrates an example of such a computing system 700 that may be used to implement the methods described herein. The exemplary computing system 700 includes a communications interface 702, a user interface 704, a processor 706, one or more sensors 707 (e.g., a photodetector, a camera, a depth sensor, a microscope, or some other laboratory instrumentation), and a data storage device 708, all of which are communicatively linked together by a system bus 710.

[0068] The communication interface 702 may function to enable the computing system 700 to communicate with other devices, access networks, and / or transport networks using analog or digital modulation of electrical, magnetic, electromagnetic, optical, or other signals. Thus, the communication interface may facilitate circuit-switched and / or packet-switched communications, such as plain old telephone service (POST) communications and / or Internet Protocol (IP) or other packet communications. For example, the communication interface 702 may include a chipset and an antenna arranged for wireless communications with a radio access network or access point. Also, the communication interface 702 may take the form of or include a wireline interface, such as an Ethernet, Universal Serial Bus (USB), or High-Definition Multimedia Interface (HDMI) port. The communication interface may also take the form of or include a wireless interface, such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide area wireless interface (e.g., WiMAX or 3GPP Long Term Evolution (LTE)). However, other forms of physical layer interfaces and other types of standard or proprietary communication protocols may be used over communication interface 702. Additionally, communication interface 702 may include multiple physical communication interfaces (e.g., a Wi-Fi interface, a Bluetooth interface, and a wide area wireless interface).

[0069] In some embodiments, the communications interface 702 may function to enable the computing system 700 to communicate with other devices, remote servers, access networks, and / or transport networks. For example, the communications interface 702 may function to transmit and / or receive a display of an image of a biological sample (e.g., a set of bright field images or other types of images that differ relative to a focal plane imaged within the sample), or some other information.

[0070] The user interface 704 of such a computing system 700 may function to enable the computing system 700 to interact with a user, e.g., receive input from a user and / or provide output to a user. Thus, the user interface 704 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, etc. The user interface 704 may also include one or more output components, such as, for example, a display screen that may be combined with a presence-sensitive panel. The display screen may be based on CRT, LCD, and / or LED technology, or other technologies now known or later developed. The user interface 704 may also be configured to generate audible output via a speaker, a speaker jack, an audio output port, an audio output device, an earphone, and / or other similar devices.

[0071] In some embodiments, the user interface 704 may include a display that serves to present video or other images to a user (e.g., video of images generated during a particular scan of a particular biological sample). In addition, the user interface 704 may include one or more buttons, switches, knobs, and / or dials that facilitate configuration and operation of the computing device. Some or all of these buttons, switches, knobs, and / or dials may be capable of being implemented as features on a touch-sensitive or presence-sensitive panel. The user interface 704 may enable a user to define the types of samples contained within the automated imaging system, define a schedule for imaging or other evaluation of samples, define parameters of image segmentation, event analysis, and / or any other analysis performed by the system 700, or enter any other commands or parameters for the operation of the automated laboratory system and / or for the analysis of data generated thereby.

[0072] The processor 706 may include one or more general-purpose processors, e.g., microprocessors, and / or one or more special-purpose processors, e.g., digital signal processors (DSPs), graphic processing units (GPUs), floating point units (FPUs), network processors, tensor processing units (TPUs), or application specific integrated circuits (ASICs). In some examples, the special-purpose processors may be capable of image processing, image alignment, statistical analysis, filtering, or noise reduction, among other applications or functions. The data storage 708 may include one or more volatile and / or non-volatile storage components, such as magnetic, optical, flash, or organic storage devices, and may be fully or partially integrated with the processor 706. The data storage 708 may include removable and / or non-removable components.

[0073] The processor 706 may be capable of executing program instructions 718 (e.g., compiled or uncompiled program logic and / or machine code) stored in the data storage device 708 to perform various functions described herein. Thus, the data storage device 708 may include a non-transitory computer-readable medium having stored thereon program instructions that, when executed by the computing device 700, cause the computing device 700 to perform any of the methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Execution of the program instructions 718 by the processor 706 may result in the processor 706 using the data 712.

[0074] By way of example, program instructions 718 may include an operating system 722 (e.g., an operating system kernel, device drivers, and / or other modules) and one or more application programs 720 (e.g., filtering functions, data processing functions, statistical analysis functions, image processing functions, depth determination functions, image segmentation functions) installed on computing device 700. Data 712 may include microscopy images or other data, including sets of images of individual samples, depth information for the samples, and / or segmentation information for the samples.

[0075] The application programs 720 may communicate with the operating system 722 via one or more application programming interfaces (APIs). These APIs may, for example, facilitate the application programs 720 receiving information via the communications interface 702, receiving and / or displaying information on the user interface 704, etc.

[0076] The application programs 720 may take the form of "apps" that may be downloadable to the computing device 700 via one or more online application stores or application markets (e.g., via communications interface 702). However, the application programs may also be installed on the computing device 700 in other manners, such as via a web browser or via a physical interface of the computing device 700 (e.g., a USB port).

[0077] In some examples, parts of the methods described herein may be performed by different devices according to the application. For example, different devices of the system may have different amounts of computational resources (e.g., memory, processor cycles) and different information bandwidths for communication between the devices. For example, a first device may be an embedded processor that can operate an actuation gantry, an imager, or other elements to generate information about a biological sample at and / or during multiple different time periods. A second device may then receive information (e.g., image information, depth information) from the first device (e.g., via the Internet, via a dedicated wired link) and perform the processing and analysis methods described herein on the received data. Different parts of the methods described herein may be apportioned according to such considerations.

[0078] VII. Exemplary Methods FIG. 8 is a flow chart of a method 800 for generating a projection image of a three-dimensional sample. The method 800 includes acquiring a set of images of the sample, each image of the set of images corresponding to a respective focal plane within the sample (810). The method 800 additionally includes applying a filter to each image of the set of images to determine a respective depth value for each pixel of an output image of the sample, the given depth value representing a depth within the sample at which contents of the sample may be imaged in focus (820). The method 800 additionally includes determining an image value for each pixel of the output image based on the depth value of the pixel of the output image, the determining an image value for a particular pixel of the output image including (i) identifying an image of the set of images that corresponds to the depth value of the particular pixel, and (ii) determining an image value for the particular pixel based on a pixel of the identified image having a location within the identified image that corresponds to the particular pixel (830). The method 800 may include additional elements or features.

[0079] 9 is a flow chart of a method 900 for generating a projection image of a three-dimensional sample. The method 900 includes obtaining a set of images of the sample, each image of the set of images corresponding to a respective focal plane within the sample (910). The method 900 additionally includes applying a filter to each image of the set of images to determine a respective depth value for each pixel of the depth map, the given depth value representing a depth within the sample at which the contents of the sample may be imaged in focus (920). The method 900 further includes determining an image value for each pixel of the output image based on the depth value of the corresponding pixel of the depth map (930). The method 900 may include additional elements or features.

[0080] 10 is a flow chart of a method 1000 for segmenting an image of a sample. The method 1000 includes acquiring an image of the sample (1010). The method 1000 additionally includes acquiring a depth map of the contents of the sample (1020). The method 1000 further includes generating a first segmentation map of the sample based on the image (1030). The method 1000 further includes generating a second segmentation map of the sample by further dividing at least one region of the first segmentation map based on the depth map (1040). The method 1000 may include additional elements or features.

[0081] VIII. Conclusion The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying drawings. In the figures, generally, similar symbols identify similar components unless the context indicates otherwise. The exemplary embodiments described in the detailed description, figures, and claims are not meant to be limiting. Other embodiments may be utilized and other changes may be made without departing from the scope of the subject matter presented herein. It will be readily understood that the aspects of the disclosure as generally described and illustrated in the figures herein may be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are expressly contemplated herein.

[0082] With respect to any or all of the message flow diagrams, scenarios, and flowcharts in the figures and described herein, each step, block, and / or communication may represent the processing of information and / or the transmission of information according to the exemplary embodiments. Alternative embodiments are included within the scope of these exemplary embodiments. For example, in these alternative embodiments, functions described as steps, blocks, transmissions, communications, requests, responses, and / or messages may be performed in a different order than that shown or described, including substantially simultaneously or in reverse order, depending on the functionality involved. Furthermore, more or fewer steps, blocks, and / or functions may be used with any of the message flow diagrams, scenarios, and flowcharts described herein, and these message flow diagrams, scenarios, and flowcharts may be combined with each other, either in part or in whole.

[0083] The steps or blocks representing the processing of information may correspond to circuits that may be configured to perform certain logical functions of the methods or techniques described herein. Alternatively or additionally, the steps or blocks representing the processing of information may correspond to modules, segments, or portions of program code (including associated data). The program code may include one or more instructions executable by a processor to implement certain logical functions or actions in the method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device including a disk drive, hard drive, or other storage medium.

[0084] The computer readable medium may also include non-transitory computer readable media, such as computer readable media that store data for a short period of time, such as a register memory, a processor cache, and / or a random access memory (RAM). The computer readable medium may also include non-transitory computer readable media that store program code and / or data for a longer period of time, such as a secondary storage device or a persistent long-term storage device, such as a read-only memory (ROM), an optical or magnetic disk, and / or a compact disk read-only memory (CD-ROM). The computer readable medium may also be any other volatile or non-volatile storage system. The computer readable medium may be considered, for example, as a computer readable storage medium or a tangible storage device.

[0085] Moreover, steps or blocks representing one or more information transmissions may correspond to information transmissions between software and / or hardware modules in the same physical device, however, other information transmissions may be between software and / or hardware modules in different physical devices.

[0086] Various aspects and embodiments have been disclosed herein, and other aspects and embodiments will be apparent to those of ordinary skill in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, the true scope of which is indicated by the following claims. [Explanation of symbols]

[0087] 100 Systems 105 Object 110 Imaging device, camera 120 light source 130 Sample Container 135 Sample medium 140a Region, first volume 140b Region, second volume A set (stack) of 200 images 210a Top image 210b Second image 250 projected images 300 Depth Map 301 Images, Projected Images 305 Specific Pixels 310a Narrow depth of field image 310b Narrow depth of field image 310c narrow depth of field image 310d narrow depth of field image 315a pixels 315b pixels 315c pixels 315d pixels Set of 320 texture values 325 Peak 400 projected images 405 Specific Pixels 410a Narrow depth of field image, first image 410b Narrow depth of field image, second image 410c Narrow depth of field image, third image 410d Narrow depth of field image 415a pixels 415b pixels 415c pixels 500 images, input images 510 First Segmentation Map 515a First Area 515b Second Area 515c Third Region 520 Depth Map 525a Area, 1st patch 525b Area 530 Second Segmentation Map 535a area 535b area 535c area 535d area 535e area 535f Single Area 600 Automatic Imaging System 610 Frame 620 Sample Container 625 wells 630 Sample Container Tray 640 Imaging Device 650 Actuating Gantry 700 Computing system, computing device 702 Communication Interface 704 User Interface 706 Processor 707 Sensors 708 Data Storage Device 710 System Bus 712 Data 718 Program Instructions 720 Application Program 722 Operating Systems

Claims

1. 1. A method for generating a projection image of a three-dimensional sample, comprising: acquiring a set of images of the sample, each image of the set of images corresponding to a respective focal plane within the sample; applying a filter to each image of the set of images to determine a respective depth value for each pixel of an output image of the sample, a given depth value representing a depth within the sample at which contents of the sample may be imaged in focus; and determining an image value for each pixel of the output image based on the depth value of the pixel of the output image, wherein determining an image value for a particular pixel of the output image comprises: (i) identifying an image of the set of images that corresponds to the depth value of the particular pixel; and (ii) determining the image value for the particular pixel based on a pixel of the identified image having a location within the identified image that corresponds to the particular pixel, the method comprising: generating a first segmentation map of the samples based on the output image; and generating a second segmentation map of the samples by further dividing at least one segment of the first segmentation map based on the determined depth value. The step of further dividing at least one segment of the first segmentation map further comprises: selecting a set of depth values ​​from the determined depth values ​​that correspond to a particular region of the first segmentation map; identifying at least two clusters within the selected set of depth values; and further dividing the particular region based on the identified at least two clusters.

2. 2. The method of claim 1 , wherein the filter is a texture filter, and applying the filter to a particular image of the set of images comprises determining, for a particular pixel of the particular image, at least one of a standard deviation, an entropy, or a numerical range of pixels of the particular image that are in a neighborhood of the particular pixel.

3. The method of claim 1 or 2, further comprising generating a depth map for the sample based on the depth values.

4. 4. The method of claim 1, further comprising the step of spatially pre-processing the determined depth values, wherein determining the image value for each pixel of the output image based on the depth values ​​of the pixels of the output image comprises determining the image value for each pixel of the output image based on the spatially pre-processed depth values ​​of the pixels of the output image.

5. 5. The method of claim 1, wherein the set of images is a set of bright field images of the sample.

6. The method of claim 1 , wherein the set of images is a set of fluorescent images of the sample.

7. 7. The method of claim 1, wherein the sample contains at least one three-dimensional cultured multicellular structure.

8. 8. The method of claim 7, wherein the at least one three-dimensional cultured multicellular structure comprises at least one of an organoid or a tumor spheroid embedded in an extracellular matrix.

9. 9. The method of claim 8, wherein the at least one three-dimensional cultured multicellular structure comprises an organoid embedded in an extracellular matrix, the extracellular matrix having a shape of an extracellular matrix dome.

10. 9. The method of claim 8, wherein the at least one three-dimensional cultured multicellular structure comprises at least one of a hepatocyte organoid, a pancreatic cell organoid, or an intestinal cell organoid.

11. 11. The method of claim 1, wherein the step of determining the image value for the particular pixel of the output image additionally includes the step of (iii) identifying two or more additional images of the set of images that correspond to depth values ​​within a neighborhood of the depth value of the particular pixel, and the step of determining the image value for the particular pixel based on the pixel of the identified image having a location within the identified image that corresponds to the particular pixel includes the step of performing a pixel-by-pixel operation on pixels of the identified image and the two or more additional images having locations within their respective identified images that correspond to the particular pixel.

12. 1. A method for generating a projection image of a three-dimensional sample, comprising: acquiring a set of images of the sample, each image of the set of images corresponding to a respective focal plane within the sample; applying a filter to each image of the set of images to determine a respective depth value for each pixel of a depth map, the depth values ​​representing depths within the sample at which contents of the sample may be imaged in focus, and applying the filter to each image of the set of images includes generating the projection image of the three-dimensional sample by determining, for a particular pixel of the image, a weighted measure of texture of a set of pixels in a neighborhood of the particular pixel to determine which pixel of the set of images is in focus; determining an image value for each pixel of an output image based on the depth value of a corresponding pixel of the depth map.

13. A method for generating a projection image of a three-dimensional sample, comprising: acquiring a set of images of the sample, each image of the set of images corresponding to a respective focal plane within the sample; applying a filter to each image of the set of images to determine a respective depth value for each pixel of a depth map, the depth values ​​representing depths within the sample at which contents of the sample may be imaged in focus; determining an image value for each pixel of an output image based on the depth value of a corresponding pixel of the depth map; selecting a set of depth values ​​from the determined depth values ​​that correspond to a particular region of a first segmentation map; identifying at least two clusters within the selected set of depth values; and further dividing the particular region based on the identified at least two clusters.

14. 14. The method of claim 12 or 13, wherein the filter is a texture filter and applying the filter to a particular image of the set of images comprises determining, for a particular pixel of the particular image, at least one of the standard deviation, entropy, or numerical range of pixels of the particular image that are in a neighborhood of the particular pixel.

15. generating a first segmentation map of the samples based on the output image; and generating a second segmentation map of the samples by further dividing at least one region of the first segmentation map based on the determined depth value.

16. The step of further dividing at least one region of the first segmentation map further comprises: selecting a set of depth values ​​from the determined depth values ​​that correspond to a particular region of the first segmentation map; identifying at least two clusters within the selected set of depth values; and further dividing the particular region based on the identified at least two clusters.

17. 17. The method of claim 12, wherein the set of images is a set of bright field images of the sample.

18. 17. The method of claim 12, wherein the set of images is a set of fluorescent images of the sample.

19. 19. The method of any one of claims 12 to 18, wherein the sample contains at least one organoid.

20. 1. A method for segmenting an image of a sample, comprising: acquiring an image of the sample; obtaining a depth map of the content of the sample; generating a first segmentation map of the sample based on the image; generating a second segmentation map of samples based on the depth map by further dividing at least one region of the first segmentation map; The depth map includes a plurality of depth values, and further dividing at least one region of the first segmentation map includes: selecting a set of depth values ​​from said depth values ​​that correspond to a particular region of said first segmentation map; identifying at least two clusters within the selected set of depth values; and further dividing the particular region based on the identified at least two clusters.

21. 21. The method of claim 20, wherein the sample contains at least one organoid.

22. A non-transitory computer-readable medium having stored thereon instructions executable by at least one processor to perform operations, comprising: acquiring a set of images of a sample, each image of the set of images corresponding to a respective focal plane within the sample; applying a filter to each image of the set of images to determine a respective depth value for each pixel of an output image of the sample, a given depth value representing a depth within the sample at which contents of the sample may be imaged in focus; determining an image value for each pixel of the output image based on the depth value of the pixel of the output image, wherein determining an image value for a particular pixel of the output image includes: (i) identifying an image of the set of images that corresponds to the depth value of the particular pixel; and (ii) determining the image value for the particular pixel based on a pixel of the identified image having a location within the identified image that corresponds to the particular pixel; The operation, generating a first segmentation map of the samples based on the output image; and generating a second segmentation map of the samples by further dividing at least one segment of the first segmentation map based on the determined depth value. Further dividing at least one segment of the first segmentation map includes: selecting a set of depth values ​​from the determined depth values ​​that correspond to a particular region of the first segmentation map; identifying at least two clusters within the selected set of depth values; and further dividing the particular region based on the identified at least two clusters.

23. 23. The non-transitory computer-readable medium of claim 22, wherein the filter is a texture filter and applying the filter to a particular image of the set of images includes determining, for a particular pixel of the particular image, at least one of a standard deviation, an entropy, or a numerical range of pixels of the particular image that are in a neighborhood of the particular pixel.

24. 24. The non-transitory computer-readable medium of claim 22 or 23, wherein the operations further include generating a depth map for the sample based on the depth values.

25. The operation, 25. The non-transitory computer-readable medium of claim 22, further comprising: spatially pre-processing the determined depth values, and determining the image value for each pixel of the output image based on the depth values ​​of the pixels of the output image further comprises determining the image value for each pixel of the output image based on the spatially pre-processed depth values ​​of the pixels of the output image.

26. 26. The non-transitory computer-readable medium of claim 22, wherein determining the image value for the particular pixel of the output image additionally includes (iii) identifying two or more additional images of the set of images that correspond to depth values ​​within a neighborhood of the depth value of the particular pixel, and determining the image value for the particular pixel based on the pixel of the identified image having a location within the identified image that corresponds to the particular pixel includes performing pixel-by-pixel operations on pixels of the identified image and the two or more additional images having locations within their respective identified images that correspond to the particular pixel.

27. A non-transitory computer-readable medium having stored thereon instructions executable by at least one processor to perform operations, comprising: acquiring a set of images of a sample, each image of the set of images corresponding to a respective focal plane within the sample; applying a filter to each image of the set of images to determine a respective depth value for each pixel of a depth map, a given depth value representing a depth within the sample at which contents of the sample may be imaged in focus; determining an image value for each pixel of an output image based on the depth value of a corresponding pixel of the depth map; The operation, generating a first segmentation map of the sample based on the output image; generating a second segmentation map of the samples by further dividing at least one region of the first segmentation map based on the determined depth value; Further dividing at least one region of the first segmentation map includes: selecting a set of depth values ​​from the determined depth values ​​that correspond to a particular region of the first segmentation map; identifying at least two clusters within the selected set of depth values; and further dividing the particular region based on the identified at least two clusters.

28. 28. The non-transitory computer-readable medium of claim 27, wherein the filter is a texture filter, and wherein applying the filter to a particular image of the set of images includes determining, for a particular pixel of the particular image, at least one of a standard deviation, an entropy, or a numerical range of pixels of the particular image that are in a neighborhood of the particular pixel.

29. A non-transitory computer-readable medium having stored thereon instructions executable by at least one processor to perform operations, comprising: Obtaining an image of the sample; and obtaining a depth map of the contents of the sample; generating a first segmentation map of the sample based on the image; generating a second segmentation map of samples by further dividing at least one region of the first segmentation map based on the depth map; The depth map includes a plurality of depth values, and further dividing at least one region of the first segmentation map includes: selecting a set of depth values ​​from the depth values ​​that correspond to a particular region of the first segmentation map; identifying at least two clusters within the selected set of depth values; and further dividing the particular region based on the identified at least two clusters.

30. one or more processors; and a non-transitory computer-readable medium having stored thereon instructions executable by at least one of the one or more processors to perform operations, the non-transitory computer-readable medium comprising: acquiring a set of images of a sample, each image of the set of images corresponding to a respective focal plane within the sample; applying a filter to each image of the set of images to determine a respective depth value for each pixel of an output image of the sample, a given depth value representing a depth within the sample at which contents of the sample may be imaged in focus; determining an image value for each pixel of the output image based on the depth value of the pixel of the output image, wherein determining an image value for a particular pixel of the output image includes: (i) identifying an image of the set of images that corresponds to the depth value of the particular pixel; and (ii) determining the image value for the particular pixel based on a pixel of the identified image having a location within the identified image that corresponds to the particular pixel; The operation includes: generating a first segmentation map of the sample based on the output image; generating a second segmentation map of the samples by further dividing at least one segment of the first segmentation map based on the determined depth value; Further dividing at least one segment of the first segmentation map includes: selecting a set of depth values ​​from the determined depth values ​​that correspond to a particular region of the first segmentation map; identifying at least two clusters within the selected set of depth values; and further dividing the particular region based on the identified at least two clusters.

31. 31. The system of claim 30, wherein the filter is a texture filter, and applying the filter to a particular image of the set of images includes determining, for a particular pixel of the particular image, at least one of a standard deviation, an entropy, or a numerical range of pixels of the particular image that are in a neighborhood of the particular pixel.

32. 32. The system of claim 30 or 31, wherein the operations further comprise generating a depth map for the sample based on the depth values.

33. The operation includes:

33. The system of claim 30, further comprising spatially pre-processing the determined depth values, wherein determining the image value for each pixel of the output image based on the depth values ​​of the pixels of the output image comprises determining the image value for each pixel of the output image based on the spatially pre-processed depth values ​​of the pixels of the output image.

34. 34. The system of claim 30, wherein determining the image value for the particular pixel of the output image additionally includes (iii) identifying two or more additional images of the set of images that correspond to depth values ​​within a neighborhood of the depth value of the particular pixel, and determining the image value for the particular pixel based on the pixel of the identified image having a location within the identified image that corresponds to the particular pixel includes performing pixel-by-pixel operations on pixels of the identified image and the two or more additional images having locations within their respective identified images that correspond to the particular pixel.

35. one or more processors; and a non-transitory computer-readable medium having stored thereon instructions executable by at least one of the one or more processors to perform operations, the non-transitory computer-readable medium comprising: acquiring a set of images of a sample, each image of the set of images corresponding to a respective focal plane within the sample; applying a filter to each image of the set of images to determine a respective depth value for each pixel of a depth map, a given depth value representing a depth within the sample at which contents of the sample may be imaged in focus; determining an image value for each pixel of an output image based on the depth value of a corresponding pixel of the depth map; selecting a set of depth values ​​from the determined depth values ​​that correspond to a particular region of a first segmentation map; identifying at least two clusters within the selected set of depth values; and further dividing the particular region based on the identified at least two clusters.

36. 36. The system of claim 35, wherein the filter is a texture filter, and applying the filter to a particular image of the set of images includes determining, for a particular pixel of the particular image, at least one of a standard deviation, an entropy, or a numerical range of pixels of the particular image that are in a neighborhood of the particular pixel.

37. The operation includes: generating a first segmentation map of the sample based on the output image; and generating a second segmentation map of the samples by further dividing at least one region of the first segmentation map based on the determined depth value.

38. one or more processors; and a non-transitory computer-readable medium having stored thereon instructions executable by at least one of the one or more processors to perform operations, the non-transitory computer-readable medium comprising: Obtaining an image of the sample; and obtaining a depth map of the contents of the sample; generating a first segmentation map of the sample based on the image; generating a second segmentation map of samples by further dividing at least one region of the first segmentation map based on the depth map; The depth map includes a plurality of depth values, and further dividing at least one region of the first segmentation map includes: selecting a set of depth values ​​from the depth values ​​that correspond to a particular region of the first segmentation map; identifying at least two clusters within the selected set of depth values; and further dividing the particular region based on the identified at least two clusters.

Images