How to classify cells

Abstract

The disclosure provides example embodiments for automatically or semi-automatically classifying cells in microscopic images of a biological specimen. These embodiments include methods for selecting a training set for development of a classifier model. The disclosed selection embodiments enable retraining of the classifier model using training examples that have been subjected to the same or similar culture conditions as the target specimen. These selection embodiments can reduce the amount of human effort required to specify the training examples. The disclosed embodiments further include classification of individual cells based on metrics determined for the cells using phase-contrast and defocused bright-field images. These metrics can include size, shape, texture, and intensity-based metrics. These metrics are determined based on underlying image segmentation. In some embodiments, the segmentation is based on phase-contrast and / or defocused bright-field images of the biological specimen.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is an international application claiming priority to U.S. Patent Application No. 17 / 099,983, filed November 17, 2020, which is incorporated herein by reference. Also incorporated by reference are U.S. Patent Application No. 16 / 265,910, filed February 1, 2019, and U.S. Patent Application No. 16 / 950,368, filed November 17, 2020.

[0002] background

[0002] Currently known methods for segmenting cells in biological specimens require fluorescently labeled proteins (e.g., thresholding nuclear-localized proteins such as histones for marker-controlled segmentation algorithms). Alternative label-free techniques exist, such as ptychography-based methods, lateral shearing interferometry, and digital holography, but these techniques require complex image acquisition setups and complex image formation algorithms with long processing times. Another label-free technique involves deep learning algorithms (e.g., convolutional neural networks), which require extensive training with large datasets of images and slow processing times. Other methods use bright-field images in out-of-focus conditions, which require dedicated hardware such as pinhole apertures, and do not allow cell-by-cell segmentation.

[0003]

[0003] Classification of cells in microscopic images (e.g., cells whose location and extent within the image have been determined by segmentation) can facilitate a variety of applications, including evaluating the effects of various experimental conditions by quantifying the effect in terms of an increase or decrease in the number of cells present in a specimen and / or the proportion of cells corresponding to various conditions (e.g., differentiated vs. undifferentiated). Cell classification can be performed manually, but such manual segmentation can be expensive in terms of time and effort and can result in inaccurate classification of cells. Furthermore, while automated methods are available, these methods may require fluorescently labeled proteins that can interfere with the natural physiology of the cells or may require the provision of a large set of training examples to train the automated algorithm. Summary of the Invention [Means for solving the problem]

[0004] overview In one aspect, an example method for cell classification is disclosed. The method includes: (i) acquiring a set of images of a plurality of biological specimens, the set of images including at least one image of each specimen of the plurality of biological specimens; (ii) acquiring a representation of a first set of cells within the plurality of biological specimens and acquiring a representation of a second set of cells within the plurality of biological specimens, wherein the first set of cells is associated with a first state and the second set of cells is associated with a second state; and (iii) determining a first plurality of sets of metrics based on the set of images, the representation of the first set of cells, and the representation of the second set of cells, wherein the first plurality of sets of metrics includes a set of metrics for each cell of the first set of cells. (iv) generating a model using a supervised learning algorithm based on the first plurality of sets of metrics to distinguish between cells in the first set of cells and cells in the second set of cells, thereby generating a training model; (v) determining a second plurality of sets of metrics based on the set of images, the second plurality of sets of metrics including a set of metrics for each cell present in the target specimen; and (vi) classifying cells in the target specimen, wherein classifying the cells includes applying the training model to the set of metrics for the cells.

[0005] In another aspect, an example method for cell classification is provided. The method includes: (i) acquiring three or more images of a target specimen, the target specimen including one or more cells centered on a focal plane for the target specimen, the three or more images including a phase-contrast image, a first bright-field image, and a second bright-field image, where the first bright-field image represents an image of the target specimen focused at a first defocus distance above the focal plane and the second bright-field image represents an image of the target specimen focused at a second defocus distance below the focal plane; (ii) determining cell images of the target specimen based on the first and second bright-field images; (iii) determining a target segmentation map for the target specimen based on the cell images and the phase-contrast image; (iv) determining a set of metrics for each cell present in the target specimen based on the two or more images of the target specimen and the target segmentation map; and (v) classifying cells in the target specimen, where classifying the cells includes applying the set of cell metrics to a training classifier.

[0006] In yet another aspect, an example method for cell classification is provided, the method including: (i) acquiring two or more images of a target specimen, the target specimen including one or more cells centered at a focal plane for the target specimen, the two or more images including a phase contrast image and one or more bright field images, the one or more bright field images including at least one bright field image representing an image of the target specimen not focused at the focal plane; (ii) determining a set of metrics for each cell present in the target specimen based on the two or more images; and (iii) classifying cells in the target specimen by applying a training model to the set of metrics for the cells.

[0007]

[0007] In another aspect, a non-transitory computer-readable medium is provided that is configured to store at least computer-readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform controller operations that implement any of the above-mentioned methods.

[0008]

[0008] In yet another aspect, a system for examining a biological specimen is provided, comprising: (i) an optical microscope; (ii) a controller including one or more processors; and (iii) a non-transitory computer-readable medium configured to store at least computer-readable instructions that, when executed by the controller, cause the controller to perform controller operations that implement any of the above-described methods.

[0009]

[0009] The described features, functions, and advantages can be achieved independently in various examples, or may be combined in yet other examples, with further details to be found in the following description and with reference to the drawings.

[0010] BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a functional block diagram of a system according to one example implementation. [Figure 2]

[0012] 1 shows a block diagram of a computing device and a computer network, according to an example implementation. [Figure 3]

[0013] 1 shows a flowchart of a method according to an example implementation. [Figure 4]

[0014] 1 shows an image of a biological specimen, according to an example implementation. [Figure 5]

[0015] 10 shows an image of another biological specimen, according to an example implementation. [Figure 6A]

[0016] Experimental results of cell-by-cell segmentation masks generated by the example implementation for cell image responses after a 24-hour course of HT1080 fibrosarcoma apoptosis following camptothecin (CPT, cytotoxic) treatment are shown. [Figure 6B]

[0017] 6A shows cell subsets sorted based on red (NucLight Red, a cell health indicator, "NucRed") and green fluorescence (caspase 3 / 7, an apoptosis indicator). [Figure 6C]

[0018] The implementation of Figure 6A shows that there was a decrease in the red population after CPT treatment indicating loss of viable cells, an increase in red and green fluorescence indicating early apoptosis, and an increase in green fluorescence after 24 hours indicating late apoptosis. [Figure 6D]

[0019] FIG. 6B shows the concentration-response time course of the early apoptotic population (percentage of total cells showing red and green fluorescence) according to the implementation of FIG. 6A. [Figure 6E]

[0020] Experimental results of cell-by-cell segmentation masks generated by the example implementation for cell image responses after 24 hours of HT1080 fibrosarcoma apoptosis following cyclohexamide (CHX, cytostatic) treatment are shown. [Figure 6F]

[0021] FIG. 6E shows cell subsets sorted based on red (NucLight Red, a cell health indicator, "NucRed") and green fluorescence (caspase 3 / 7, an apoptosis indicator) according to the implementation. [Figure 6G]

[0022] The implementation in Figure 6E shows a lack of apoptosis after CHX treatment, but a reduction in cell number. [Figure 6H]

[0023] FIG. 6E shows the concentration-response time course of the early apoptotic population (percentage of total cells showing red and green fluorescence) according to the implementation of FIG. 6E. [Figure 7A]

[0024] 1 shows a cell-by-cell segmentation mask imposed onto a phase contrast image for label-free cell counting of adherent cells using cell-by-cell segmentation analysis generated by an example implementation. Varying densities of A549 cells labeled with NucLight Red reagent were analyzed with both label-free cell-by-cell analysis and red nucleus counting analysis confirming label-free counting over time. [Figure 7B]

[0025] 7B shows the cell-by-cell segmentation mask from FIG. 7A without a phase contrast image in the background. [Figure 7C]

[0026] 7B shows the time course of phase number and NucRed number data across density according to the implementation of FIG. 7A. [Figure 7D]

[0027] 7B shows the correlation of count data over a 48-hour period according to the implementation of FIG. 7A, showing an R value of 1 with a slope of 1. [Figure 8]

[0028] 1 shows a flowchart of a method according to an example implementation. [Figure 9]

[0029] 1 shows a flowchart of a method according to an example implementation. [Figure 10]

[0030] 1 shows a flowchart of a method according to an example implementation. [Figure 11]

[0031] 1 shows an example of a microscope image and an associated example of a segmentation map. [Figure 12A]

[0032] 1 shows an example of an annotated microscope image. [Figure 12B]

[0033] 1 shows an example of an annotated microscope image. [Figure 13]

[0034] 1 shows an example of a schematic diagram of wells in a multi-well specimen plate. [Figure 14A]

[0035] The experimental predictive accuracy of the methods described herein is illustrated. [Figure 14B]

[0035] The experimental predictive accuracy of the methods described herein is illustrated. [Figure 15A]

[0036] The experimental predictive accuracy of the methods described herein is illustrated. [Figure 15B]

[0036] The experimental predictive accuracy of the methods described herein is illustrated. [Figure 16A]

[0037] 1 illustrates the experimental predictive accuracy of the methods described herein compared to label-based classification. [Figure 16B]

[0037] The experimental predictive accuracy of the methods described herein compared to label-based classification is illustrated. [Figure 16C]

[0037] The experimental predictive accuracy of the methods described herein compared to label-based classification is illustrated. DETAILED DESCRIPTION OF THE INVENTION

[0012]

[0038] The drawings are for illustrative purposes only and the invention is not limited to the arrangements and instrumentality shown in the drawings.

[0013] Detailed Description I. Overview

[0039] Microscopic imaging of biological specimens can facilitate numerous analyses of the specimen's contents and its response to various applied experimental conditions. Such analyses can include counting cells after sorting them to determine the effect of the applied conditions. For example, a specimen can include a set of differentiated cells and a set of undifferentiated cells, and analysis of the specimen can include determining the proportion of differentiated cells, e.g., to determine the effectiveness of applied conditions on differentiation of the undifferentiated cells. To perform such analyses, it is necessary to localize each cell in the specimen and then classify each cell. Such classification processes can be performed manually. However, manual classification can be very expensive, time-consuming, and can result in incorrect classifications.

[0014]

[0040] The embodiments described herein demonstrate various methods for automatically classifying cells based on phase-contrast images, bright-field images, composites of phase-contrast and / or bright-field images, or other microscopic images of the cells. Some of these embodiments involve using a designated set of cells in one or more biological specimens to train a model for classifying the cells. Such trained models can then be applied to additional cells to classify those additional cells. To classify a particular cell, a set of metrics is determined for the cell based on one or more images representing the cell. Such metrics can include metrics related to the size and / or shape of the cell. Additionally or alternatively, such metrics may relate to the texture or intensity of the cell as shown in one or more phase-contrast, bright-field, fluorescent, or composite images. For example, one or more of the metrics can relate to the texture of the cell (e.g., brightness or intensity variations and / or structure of variations across an area of ​​the cell) in a fluorescent image or some other variety of images (e.g., phase-contrast, bright-field). The determined set of metrics for the cell can then be applied to a training model to classify the cell.

[0015]

[0041] The set of cells used to train the model can be identified in a variety of ways. In some examples, cells can be manually designated by a user. This can include a user manually designating all wells of a multi-well specimen plate. Additionally or alternatively, a user can manually designate individual cells within one or more biological specimens. In yet another example, a user can designate a set of cells by specifying a time point, e.g., a first time point before all cells in the specimen belong to a first set (e.g., an undifferentiated set) and a second time point after all cells in the specimen belong to a second set (e.g., a differentiated set). In some examples, cells can be designated automatically or semi-automatically. This can include identifying sets of cells based on fluorescent images of the cells (e.g., fluorescent signals above a threshold can be assigned to a first group, while fluorescent signals below a threshold can be assigned to a second group). In another example, an unsupervised or semi-supervised learning algorithm can collect or otherwise assemble cells into sets that can later be used to train a classifier.

[0016] II. Architecture Examples

[0042] 1 is a block diagram illustrating an operating environment 100 that may include or involve, for example, an optical microscope 105 and a biological specimen 110 having one or more cells. Methods 300, 800, 900, and 1000 in FIGS. 3-5, 8, 9, and 10, described below, illustrate embodiments of methods that may be implemented within this operating environment 100.

[0017]

[0043] FIG. 2 is a block diagram illustrating an example computing device 200, according to an example implementation, configured to interface directly or indirectly with operating environment 100. Computing device 200 may be used to perform the functions of the methods illustrated in FIGS. 3-5, 8, 9, and 10 described below. In particular, computing device 200 may be configured to perform one or more functions, including, for example, an image generation function based in part on images obtained by optical microscope 105. Computing device 200 includes a processor 202, a communications interface 204, a data storage device 206, an output interface 208, and a display 210, each connected to a communications bus 212. Additionally, computing device 200 may include hardware to enable communications within computing device 200 and between computing device 200 and other devices (e.g., not shown). The hardware may include, for example, a transmitter, a receiver, and an antenna.

[0018]

[0044] Communications interface 204 may be a wireless interface and / or one or more wired interfaces that enable both short-range and long-range communications with one or more networks 214 or one or more remote computing devices 216 (e.g., tablet 216a, personal computer 216b, laptop computer 216c, and mobile computing device 216d). Such wireless interfaces may provide communications under one or more wireless communications protocols, such as Bluetooth, WiFi (e.g., Institute of Electrical and Electronics Engineers (IEEE) 802.11 protocol), Long Term Evolution (LTE), cellular communications, Near Field Communications (NFC), and / or other wireless communications protocols. Such wired interfaces may include an Ethernet interface, a Universal Serial Bus (USB) interface, or a similar interface that communicates via a wire, twisted pair, coaxial cable, optical link, fiber optic link, or other physical connection to a wired network. Thus, communications interface 204 may be configured to receive input data from one or more devices and may also be configured to transmit output data to other devices.

[0019]

[0045] Additionally, communication interface 204 may include user input devices such as a keyboard, keypad, touch screen, touchpad, computer mouse, trackball, and / or other similar devices.

[0020]

[0046] Data storage 206 may include or take the form of one or more computer-readable storage media readable or accessible by processor 202. Computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other memory or disk storage, that may be integrated in whole or in part with processor 202. Data storage 206 is considered a non-transitory computer-readable medium. In some examples, data storage 206 may be implemented using a single physical device, such as one optical, magnetic, organic, or other memory or disk storage unit, while in other examples, data storage 206 may be implemented using two or more physical devices.

[0021]

[0047] Thus, data storage device 206 is a non-transitory computer-readable storage medium that stores executable instructions 218 thereon. Instructions 218 include computer-executable code. When instructions 218 are executed by processor 202, processor 202 performs functions. Such functions include, but are not limited to, receiving a brightfield image from optical microscope 105 and generating a phase contrast image, a confluence mask, a cell image, a seed mask, a cell-by-cell segmentation mask, and a fluorescence image.

[0022]

[0048] Processor 202 may be a general-purpose processor or a special-purpose processor (e.g., a digital signal processor, an application-specific integrated circuit, etc.). Processor 202 may receive input from communication interface 204, process the input, and generate output that is stored in data storage device 206 and output to display 210. Processor 202 may be configured to execute executable instructions 218 (e.g., computer-readable program instructions) that are stored in data storage device 206 and that are executable to provide the functionality of computing device 200 as described herein.

[0023]

[0049] The output interface 208 outputs information to a display 210 or other component. Thus, the output interface 208 may be similar to the communication interface 204 and may be a wireless interface (e.g., a transmitter) or even a wired interface. The output interface 208 may, for example, send commands to one or more controllable devices.

[0024]

[0050] 2 may represent, for example, a local computing device 200a in the operating environment 100 in communication with the optical microscope 105. This local computing device 200a may perform one or more of the steps of methods 300, 800, 900, and 1000 described below, may receive input from a user, and / or may transmit image data and user input to the computing device 200 to perform all or a portion of the steps of methods 300, 800, 900, and / or 1000. Furthermore, in one optional example embodiment, the Incucyte® platform may be utilized to perform one or more of methods 300, 800, 900, and 1000, and includes the combined functionality of the computing device 200 and the optical microscope 105.

[0025]

[0051] FIG. 3 illustrates a flowchart of an example method 300 for achieving cell-by-cell segmentation for one or more cells of a biological specimen 110, according to an example implementation. FIGS. 8, 9, and 10 illustrate flowcharts of example methods 800, 900, and 1000, respectively, for achieving cell-by-cell classification of one or more cells of a biological specimen 110, according to example implementations. The methods 300, 800, 900, and 1000 illustrated in FIGS. 3, 8, 9, and 10 illustrate example methods usable, for example, in the computing device 200 of FIG. 2. Furthermore, a device or system may be used or configured to perform the logical functions illustrated in FIGS. 3, 8, 9, and / or 10. In some cases, components of a device and / or system may be configured to perform a function, for example, to configure and structure the components with hardware and / or software to enable such performance. Components of a device and / or system may be adapted to perform a function, capable of performing a function, or arranged to be suitable for performing a function, for example, when operated in a particular manner. Methods 300, 800, 900, 1000 may include one or more operations, functions, or acts as illustrated by one or more of the numbered blocks (e.g., blocks 305-330). Although the blocks are illustrated in sequential order in the figures, some of these blocks may be performed in parallel and / or in a different order than described herein. Additionally, various blocks may be combined into fewer blocks, divided into additional blocks, and / or removed based on the desired implementation.

[0026]

[0052] For this and other processes and methods disclosed herein, the flowchart should be understood to illustrate the functionality and operation of one possible implementation of the example. In this regard, each block may represent a module, segment, or portion of program code, including one or more instructions executable by a processor to implement a specific logical function or step in the process. The program code may be stored, for example, in any type of computer-readable medium or data storage device, such as a storage device including a disk or hard drive. Furthermore, the program code may be encoded in a machine-readable format on a computer-readable storage medium or other non-transitory medium or article of manufacture. The computer-readable medium may include, for example, a non-transitory computer-readable medium or memory, such as a computer-readable medium that stores data for a short-term period, e.g., a register memory, a processor cache, and a random access memory (RAM). The computer-readable medium may also include a non-transitory medium, such as a secondary or permanent long-term storage device, such as a read-only memory (ROM), an optical or magnetic disk, or a compact disc read-only memory (CD-ROM). The computer readable medium may also be any other volatile or non-volatile storage system. The computer readable medium may also be considered, for example, a tangible computer readable storage medium.

[0027]

[0053] Additionally, within other processes and methods disclosed herein, each block in Figures 3, 8, 9, and 10 may represent circuitry that is hardwired to perform a particular logical function in the process. As will be appreciated by a reasonably skilled artisan, alternative implementations that may perform functions out of the order shown or described, including substantially simultaneously or in reverse order, depending on the functionality involved, are included within the scope of examples of the present disclosure.

[0028] III. Method Example

[0054] As used herein, a "bright-field image" refers to an image obtained through a microscope based on a biological specimen that is illuminated from below so that light waves pass through a transparent portion of the biological specimen. Various brightness levels are then captured in the bright-field image.

[0029]

[0055] As used herein, a "phase-contrast image" refers to an image obtained directly or indirectly via a microscope based on a biological specimen illuminated from below, capturing the phase shift of light passing through the biological specimen due to differences in the refractive index of different portions of the biological specimen. For example, as light waves travel through a biological specimen, the light wave amplitude (i.e., brightness) and phase change in a manner that depends on the properties of the biological specimen. As a result, a phase-contrast image has brightness intensity values ​​associated with pixels that vary such that denser regions with a high refractive index appear darker in the resulting image and thinner regions with a low refractive index appear brighter in the resulting image. Phase-contrast images can be generated through a number of techniques, including from a Z-stack of bright-field images.

[0030]

[0056] As used herein, a "Z-stack" or "Z-sweep" of bright-field images refers to a digital image processing method that combines multiple images taken at different focal lengths to provide a composite image that has a greater depth of field (i.e., focal plane thickness) than any of the individual source bright-field images.

[0031]

[0057] As used herein, "focal plane" means a plane positioned perpendicular to the axis of an optical microscope lens in which a biological specimen can be observed at optimum focus.

[0032]

[0058] As used herein, "defocus distance" means the distance above or below the focal plane such that the biological specimen is observable out of focus.

[0033]

[0059] As used herein, "confluence mask" means a binary image that identifies pixels as belonging to one or more cells in a biological specimen, such that pixels corresponding to one or more cells are assigned a value of 1 and the remaining pixels corresponding to the background are assigned a value of 0, or vice versa.

[0034]

[0060] As used herein, "cell image" means an image generated based on at least two bright field images acquired at different planes to enhance cell contrast against the background.

[0035]

[0061] As used herein, "seed mask" means an image with binary pixelation generated based on a set pixel intensity threshold.

[0036]

[0062] As used herein, a "cell-by-cell segmentation mask" refers to an image having binary pixelation (i.e., each pixel is assigned a value of 0 or 1 by a processor) to display each cell of the biological specimen 110 as a distinct region of interest. Advantageously, a cell-by-cell segmentation mask allows for label-free counting of displayed cells, allows for determination of the total area of ​​individual adherent cells, allows for analysis based on cell texture metrics and cell shape descriptors, and / or allows for detection of individual cell boundaries, including adherent cells that tend to form in sheets, where each cell may contact many other neighboring cells in the biological specimen 110.

[0037]

[0063] As used herein, a "region growing iteration" refers to a single step in an iterative image segmentation method that defines a region of interest ("ROI") by taking one or more initially identified individual pixels or sets of pixels (i.e., seeds) and iteratively expanding the seeds by adding neighboring pixels to the set. The processor utilizes a similarity metric to determine which pixels to add to the expanded region, and a stopping criterion is defined for the processor to determine when the region growing is complete.

[0038]

[0064] As used herein, "trained model" means a prediction and / or classification model (e.g., artificial neural network, Bayesian predictor, decision tree) whose parameters (e.g., weights, filter bank coefficients), structure (e.g., number of hidden layers and / or units, pattern of interconnections of such units), or other characteristics of its configuration have been trained (e.g., by reinforcement learning, by gradient descent, by analytically determining maximum likelihood values ​​of the model parameters) based on a set of training data to generate an output that predicts class membership (e.g., alive / dead, differentiated / undifferentiated) of cells.

[0039]

[0065] Referring now to Figures 3-5, a method 300 is illustrated using the computing device of Figures 1 and 2. The method 300 includes, at block 305, a processor 202 generating at least one phase-contrast image 400 of a biological specimen 110 including one or more cells centered on a focal plane for the biological specimen 110. Next, at block 310, the processor 202 generates a confluence mask 410 in the form of a binary image based on the at least one phase-contrast image 400. Next, at block 315, the processor 202 receives a first bright-field image 415 of one or more cells in the biological specimen 110 at a defocus distance above the focal plane and a second bright-field image 420 of one or more cells in the biological specimen 110 at a defocus distance below the focal plane. Next, at block 320, the processor 202 generates a cell image 425 of one or more cells in the biological specimen based on the first bright-field image 415 and the second bright-field image 420. At block 325, the processor 202 generates a seed mask 430 based on the cell image 425 and the at least one phase contrast image 400. Further, at block 330, the processor 202 generates an image of one or more cells in the biological specimen showing a cell-by-cell segmentation mask 435 based on the seed mask 430 and the confluence mask 410.

[0040]

[0066] 3 , in block 305, a processor 202 for generating at least one phase-contrast image 400 of a biological specimen 110 containing one or more cells centered on a focal plane for the biological specimen 110 includes a processor 202 that receives a Z-sweep of a bright-field image and then generates at least one phase-contrast image 400 based on the Z-sweep of the bright-field image. In various embodiments, the biological specimen 110 may be distributed within multiple wells in a well plate representing an experimental set.

[0041]

[0067] In one optional embodiment, the method 300 includes a processor 202 that receives at least one fluorescent image within the cell-by-cell segmentation mask 435 and then calculates the fluorescent intensity of one or more cells in the biological specimen 110. In this embodiment, the fluorescent intensity corresponds to the level of a protein of interest, e.g., an antibody that labels a cell surface marker such as CD20, or an Annexin V reagent that induces fluorescence corresponding to cell death. Furthermore, determining the fluorescent intensity within individual cell boundaries increases subpopulation discrimination and allows for the calculation of subpopulation-specific metrics (e.g., the average area and eccentricity of all dying cells as defined by the presence of Annexin V).

[0042]

[0068] In another embodiment, the processor 202 generating the confluence mask 410 in the form of a binary image based on at least one phase contrast image 400 in block 310 includes the processor 202 applying one or more filters of a local texture filter or intensity filter to enable identification of pixels belonging to one or more cells in the biological specimen 110. Examples of filters may include, but are not limited to, a local range filter, a local entropy filter, a local standard deviation filter, a local intensity filter, and a Gabor wavelet filter. Examples of the confluence mask 410 are shown in Figures 4 and 5.

[0043]

[0069] In another optional embodiment, the optical microscope 105 determines the focal plane of the biological specimen 110. Furthermore, in various embodiments, the defocus distance may be in the range of 20 μm to 60 μm. The optimal defocus distance is determined based on the optical properties of the objective lens used, including the magnification and working distance of the objective lens.

[0044]

[0070] In a further embodiment shown in FIG. 5 , the processor 202 generating the cell image 425 based on the first bright-field image 415 and the second bright-field image 420 in block 320 includes the processor 202 enhancing the first bright-field image 415 and the second bright-field image 420 based on the third bright-field image 405 centered on the focal plane using at least one of pixel-wise mathematical operations or feature detection. One example of a pixel-wise mathematical operation includes addition, subtraction, multiplication, division, or any combination of these operations. The processor 202 then calculates transformation parameters to align the first bright-field image 415 and the second bright-field image 420 to the at least one phase-contrast image 400. The processor 202 then combines the brightness level for each pixel of the aligned second bright-field image 420 with the brightness level of the corresponding pixel in the aligned first bright-field image 415, thereby forming the cell image 425. The combination of the brightness levels for each pixel can be achieved via any of the mathematical operations described above. The technical effect of generating cell image 425 is to remove bright field artifacts (eg, shadows) and enhance image contrast to increase cell detection relative to seed mask 430 .

[0045]

[0071] In another optional embodiment, in block 320, the processor 202 that generates the cell image 425 of one or more cells in the biological specimen 110 based on the first bright-field image 415 and the second bright-field image 420 includes the processor 202 receiving one or more user-defined parameters that determine one or more threshold levels and one or more filter sizes. The processor 202 then applies one or more smoothing filters to the cell image 425 based on the one or more user-defined parameters. The technical effect of the smoothing filters is to further increase the accuracy of cell detection in the species mask 430 and increase the likelihood of assigning one species to each cell. The smoothing filter parameters are selected to suit different adherent cell morphologies (e.g., flat vs. round shape, protruding cells, clustered cells, etc.).

[0046]

[0072] In a further optional embodiment, the processor 202 generating the seed mask 430 based on the cell image 425 and the at least one phase-contrast image 400 in block 325 includes the processor 202 modifying the cell image 425 to identify each pixel at or above a threshold pixel intensity as a cell-type pixel, thereby resulting in a seed mask 430 having a binary pixelation. The technical effect of the binary pixelation of the seed mask is to enable comparison with a corresponding binary pixelation of the confluence mask. Furthermore, the binary pixelation of the seed mask is utilized as a starting point for the region-growing iterations described below. For example, in yet another optional embodiment, the seed mask 430 may have multiple seeds, each corresponding to a single cell in the biological specimen 110. In this embodiment, method 300 further includes processor 202 comparing seed mask 430 and confluence mask 410, removing one or more regions from seed mask 430 that are not located in the area of ​​confluence mask 410, and removing one or more regions from confluence mask 410 that do not contain one of the multiple species in seed mask 430, prior to processor 202 generating an image of one or more cells in the biological specimen representing cell-by-cell segmentation mask 435. The technical effect of these removed regions is to eliminate small bright objects (e.g., cellular debris) that generate seeds, enhancing species discrimination utilized in the region-growing iterations described below.

[0047]

[0073] In a further optional embodiment, the processor 202 that generates an image of one or more cells in the biological specimen 110 showing a cell-by-cell segmentation mask 435 based on the seed mask 430 and the confluence mask 410 at block 330 includes a processor 202 that performs region-growing iterations for each of the active set of species. The processor 202 then repeats the region-growing iterations for each species in the active set of species until the expanded region for a given species reaches one or more boundaries of the confluence mask 410 or overlaps with the expanded region of another species. The active set of species is selected by the processor 202 for each iteration based on characteristics of the values ​​of corresponding pixels in the cell image. Furthermore, a technical effect of using not only bright-field images 415, 420, 405 but also at least one phase-contrast image 400 is that species correspond to both bright spots in the cell image 425 and areas of high texture in the phase-contrast image 400 (i.e., overlap of the confluence mask 410 with the seed mask 430, described in more detail below). Another technical effect that results from using not only brightfield images 415, 420, 405, but also the confluence mask 410 and at least one phase contrast image is, as one example, increased accuracy in identifying individual cell locations and cell boundaries in the cell-by-cell segmentation mask 435, which can advantageously quantify features such as cell surface protein expression.

[0048]

[0074] In yet another optional embodiment, the method 300 may include the processor 202 applying one or more filters in response to user input to remove objects based on one or more cell texture metrics and cell shape descriptors. The processor 202 then modifies the image of the biological specimen showing the cell-by-cell segmentation mask in response to application of the one or more filters. Examples of cell texture metrics and cell shape descriptors include, but are not limited to, cell size, perimeter, eccentricity, fluorescence intensity, aspect ratio, solidity, Feret diameter, phase difference entropy, and phase difference standard deviation.

[0049]

[0075] In a further optional embodiment, the method 300 may include the processor 202 determining a cell count for the biological specimen 110 based on an image of one or more cells in the biological specimen 110 exhibiting a cell-by-cell segmentation mask 435. Advantageously, the cell count is recognized as a result of distinct cell boundaries, as shown, for example, in the cell-by-cell segmentation mask 435 shown in FIG. 4 . In one optional embodiment, the one or more cells in the biological specimen 110 are one or more of adherent and non-adherent cells. In a further embodiment, the adherent cells may include one or more of various cancer cell lines, including human lung cancer cells, fibrocarcinoma cells, breast cancer cells, and ovarian cancer cells, or human microvascular cell lines, including human umbilical vein cells. In an optional embodiment, the processor 202 applies a smoothing filter to non-adherent cells, including human immune cells such as PMBCs and Jurkat cells, that is different from the filter applied to the adherent cells, and performs region-growing iterations in a manner that improves approximation of cell boundaries.

[0050]

[0076] As one example, a non-transitory computer-readable medium storing program instructions that, when executed by the processor 202, result in the performance of a set of operations, including the processor 202 generating at least one phase-contrast image 400 of a biological specimen 110 including one or more cells based on at least one bright-field image 405 centered on a focal plane for the biological specimen 110. The processor 202 then generates a confluence mask 410 in the form of a binary image based on the at least one phase-contrast image 400. The processor 202 then receives a first bright-field image 415 of one or more cells in the biological specimen 110 at a defocus distance above the focal plane and a second bright-field image 420 of one or more cells in the biological specimen 110 at a defocus distance below the focal plane. The processor 202 then generates a cell image 425 of the one or more cells based on the first bright-field image 415 and the second bright-field image 420. The processor 202 further generates a seed mask 430 based on the cell image 425 and the at least one phase-contrast image 400. Additionally, the processor 202 generates an image of one or more cells in the biological specimen 110 showing a cell-by-cell segmentation mask 435 based on the seed mask 430 and the confluence mask 410 .

[0051]

[0077] In one optional embodiment, the non-transitory computer-readable medium further includes a processor 202 that receives at least one fluorescent image and calculates the fluorescent intensity of one or more cells in the biological specimen within the cell-by-cell segmentation mask.

[0052]

[0078] In another optional embodiment, the non-transitory computer readable medium further includes a processor 202 that generates a seed mask 430 based on the cell image 425 and the at least one phase contrast image 400. Additionally, the non-transitory computer readable medium further includes a processor 202 that modifies the cell image 425 to identify each pixel at or above a threshold pixel intensity as a cell-type pixel, thereby resulting in a seed mask 430 having a binary pixelation.

[0053]

[0079] In a further optional embodiment, seed mask 430 has multiple species each corresponding to a single cell. Additionally, the non-transitory computer-readable medium further includes processor 202 for comparing seed mask 430 and confluence mask 410, removing one or more regions from seed mask 430 that are not located in an area of ​​confluence mask 410, and removing one or more regions from confluence mask 410 that do not include one of the multiple species in seed mask 430, prior to processor 202 generating an image of one or more cells in biological specimen 110 showing cell-by-cell segmentation mask 435.

[0054]

[0080] In yet another optional embodiment, the program instructions causing the processor 202 to generate an image of one or more cells in the biological specimen 110 showing a cell-by-cell segmentation mask 435 based on the species mask 430 and the confluence mask 410 include the processor 202 performing region-growing iterations for each of the active set of species. The non-transitory computer-readable medium then further includes the processor 202 repeating the region-growing iterations for each species in the active set of species until the grown region for a given species reaches one or more boundaries of the confluence mask 410 or overlaps with the grown region of another species.

[0055]

[0081] The non-transitory computer-readable medium further includes a processor 202 that applies one or more filters in response to user input to remove objects based on one or more cell texture metrics and cell shape descriptors. Additionally, the processor 202 modifies the image of the biological specimen 110 showing the cell-by-cell segmentation mask 435 in response to application of the one or more filters.

[0056]

[0082] Referring now to FIG. 8, an exemplary method 800 for cell classification is illustrated using the computing devices of FIGS. 1 and 2. Method 800 includes, at block 805, a processor (e.g., processor 202) acquiring a set of images of a plurality of biological specimens, the set of images including at least one image of each specimen of the plurality of biological specimens. Next, at block 810, the processor acquires representations of a first set of cells within the plurality of biological specimens, acquires representations of a second set of cells within the plurality of biological specimens, and associates the first set of cells with a first state and the second set of cells with a second state. Next, at block 815, the processor determines a first plurality of sets of metrics based on the set of images, the representations of the first set of cells, and the representations of the second set of cells, the first plurality of sets of metrics including a set of metrics for each cell of the first set of cells and a set of metrics for each cell of the second set of cells. At block 820, the processor uses a supervised learning algorithm to generate a model based on the first plurality of sets of metrics to distinguish between cells in the first set of cells and cells in the second set of cells, thereby generating a training model. At block 825, the processor determines a second plurality of sets of metrics based on the set of images, the second plurality of sets of metrics including a set of metrics for each cell present in the target specimen. Then, at block 830, the processor classifies the cells in the target specimen, where classifying the cells includes applying the training model to the set of metrics for the cells. Method 800 may include additional steps or features.

[0057]

[0083] Referring now to FIG. 9, another exemplary method 900 for cell classification is illustrated using the computing device of FIGS. 1 and 2. The method 900 includes, at block 905, a processor (e.g., processor 202) acquiring three or more images of a target specimen, the target specimen including one or more cells centered on a focal plane for the target specimen, the three or more images including a phase-contrast image, a first bright-field image, and a second bright-field image, where the first bright-field image represents an image of the target specimen focused at a first defocus distance above the focal plane, and the second bright-field image represents an image of the target specimen focused at a second defocus distance below the focal plane. Next, at block 910, the processor determines cell images of the target specimen based on the first and second bright-field images. Next, at block 915, the processor determines a target segmentation map for the target specimen based on the cell images and the phase-contrast image. At block 920, the processor determines a set of metrics for each cell present in the target specimen based on the two or more images of the target specimen and the target segmentation map. Next, the processor classifies the cells in the target specimen, where classifying the cells includes applying the set of cell metrics to the training classifier, at block 925. Method 900 can include additional steps or features.

[0058]

[0084] Referring now to FIG. 10 , another exemplary method 1000 for cell classification is illustrated using the computing device of FIGS. 1 and 2 . Method 1000 includes, at block 1005, a processor (e.g., processor 202) acquiring two or more images of a target specimen, the target specimen including one or more cells centered at a focal plane for the target specimen, the two or more images including a phase-contrast image and one or more bright-field images, the one or more bright-field images including at least one bright-field image representing an image of the target specimen not focused at the focal plane. Next, at block 1010, the processor determines a set of metrics for each cell present in the target specimen based on the two or more images. Next, at block 1015, the processor classifies cells in the target specimen by applying a training model to the set of metrics for the cells. Method 1000 may include additional steps or features.

[0059]

[0085] As described above, a non-transitory computer-readable medium storing program instructions available upon execution by processor 202 to cause the performance of any of the functions of the methods described above.

[0060] IV. Experimental Results

[0086] An example implementation can track cell health in subpopulations over time. For example, Figure 6A shows experimental results of cell-by-cell segmentation masks generated by an example implementation for phase-contrast image responses after 24 hours of HT1080 fibrosarcoma apoptosis following camptothecin (CPT, a cytotoxic agent) treatment. Cell health was determined using multiplexed readouts of Incucyte® NucLight Red (a nuclear viability marker) and unperturbed Incucyte® Caspase 3 / 7 Green Reagent (an apoptosis indicator). Figure 6B shows cell subsets classified based on red and green fluorescence by the implementation of Figure 6A using the Incucyte® cell-by-cell analysis software tool. Figure 6C shows that the implementation of Figure 6A showed a decrease in the red population after CPT treatment, indicating loss of viable cells, an increase in red and green fluorescence, indicating early apoptosis, and an increase in green fluorescence after 24 hours, indicating late apoptosis. Figure 6D shows the concentration-response time course of the early apoptotic population (percentage of total cells showing red and green fluorescence) according to the implementation of Figure 6A. Values ​​shown are the mean ± SEM of three wells.

[0061]

[0087] In another example, Figure 6E shows experimental results of cell-by-cell segmentation masks generated by the example implementation for cell image responses after 24 hours of HT1080 fibrosarcoma apoptosis following cyclohexamide (CHX, cytostatic) treatment. Cell health was determined with multiplexed readouts of Incucyte® NucLight Red (a nuclear viability marker) and unperturbed Incucyte® Caspase 3 / 7 Green Reagent (an apoptosis indicator). Figure 6F shows cell subsets classified based on red and green fluorescence by the implementation of Figure 6E using the Incucyte® cell-by-cell analysis software tool. Figure 6G shows the lack of apoptosis after CHX treatment, although there was a reduction in cell number (data not shown). Figure 6H shows the concentration-response time course of the early apoptotic population (percentage of total cells showing red and green fluorescence) by the implementation of Figure 6E. Values ​​shown are means ± SEM of three wells.

[0062]

[0088] Figure 7A shows a cell-by-cell segmentation mask imposed onto a phase-contrast image for label-free cell counting of adherent cells using cell-by-cell segmentation analysis generated by an example implementation via Incucyte® software. Various densities of A549 cells labeled with NucLight Red reagent were analyzed with both label-free cell-by-cell analysis and red nucleus counting analysis, which confirmed label-free counting over time. Figure 7B shows the cell-by-cell segmentation mask from Figure 7A without the background phase-contrast image. Figure 7C shows the time course of phase and red count data across densities using the implementation of Figure 7A. Figure 7D shows the correlation of count data over 48 hours using the implementation of Figure 7A, showing an R value of 1 with a slope of 1. This has been replicated across various cell types. Values ​​shown are the mean ± SEM of four wells.

[0063] V. Examples of Cell Classification

[0089] Algorithmic classification of cells based on images of specimens containing the cells can facilitate a variety of applications. This can include quantifying properties of cells and / or cell specimens, quantifying the response of a cell specimen to applied experimental conditions (e.g., toxicity or efficacy of a putative drug or treatment), or assessing some other information about the specimen. Cell classification facilitates such applications by being able to determine the number of cells of each class within a specimen. Such classification may include two-class classification, or classification into more than two classes. In some examples of classification, cells may be classified as live or dead, as stem cells or mature cells, as undifferentiated cells or differentiated cells, as wild-type cells or mutant cells, as epithelial or mesenchymal, as normal or as cells morphologically altered by an applied compound (e.g., altered by application of a cytoskeleton-targeted therapeutic compound), or between two or more additional or alternative classifications. Cells may also be assigned multiple classes, each selected from multiple different enumerated sets of classes. For example, cells can be classified as live (from the possible classes of "live" and "dead") and as differentiated (from the possible classes of "differentiated" and "undifferentiated").

[0064]

[0090] The embodiments described herein achieve a classification of a particular cell by determining a set of metrics for the cell. The set of metrics is determined from one or more microscopic images of the cell. One or more defocused bright-field images of the cell, or a composite image determined from the microscopic image of the cell and / or in combination with a phase-contrast image of the cell, are particularly useful in determining such metrics. For example, one or more metrics for the cell can be determined from each of the phase-contrast image of the cell and the cell image (determined as described above) of the cell. Determining the set of metrics generally involves segmenting the image to determine which portions of the image correspond to the cell. The segmentation itself is determined based on one or more of the images as described elsewhere herein. Furthermore, the segmentation may be used to determine one or more of the metrics (e.g., cell size, one or more metrics related to cell shape, etc.). The set of metrics is then applied to a model to classify the cell.

[0065]

[0091] 11 shows an example of a per-cell segmentation mask (bright line) imposed on a phase-contrast image 1100 of a biological specimen containing many cells, including an example cell 1110. The per-cell segmentation mask outlines a portion of the phase-contrast image 1100 that corresponds to the cell 1110, which is shown by a dark line 1150 that indicates the portion of the per-cell segmentation mask that corresponds to the example cell 1110. As shown in the portion 1150 of the per-cell segmentation mask that outlines the example cell 1110 (e.g., a size-related metric, a shape-related metric), the portion of the phase-contrast image 1100 within the dark line 1150 can be used to determine one or more metrics (e.g., a texture-related metric, an intensity-related metric) for the example cell 1110.

[0066]

[0092] Segmenting one or more microscopic images of a biological specimen to localize cells within the specimen may be accomplished using one or more of the methods described above. Additionally or alternatively, one or more microscopic images of the specimen may be applied to a convolutional neural network that has been trained to generate such a segmentation map. This may include applying a phase-contrast image of the specimen and an image of the cells.

[0067]

[0093] The segmentation map may be used to determine a size metric for the cell, which may include using the segmentation map to determine the area of ​​the cell, the number of pixels of the image occupied by the cell, the percentage of the pixels and / or area of ​​the image occupied by the cell, the perimeter of the cell, the maximum Feret diameter of the cell, or some other metric related to the size of the cell.

[0068]

[0094] The segmentation map may be used to determine one or more shape descriptor metrics for the cell, such as the circularity of the cell, the circularity of the cell's convex hull, or the percentage of the cell's convex hull occupied by the cell, the aspect ratio of the cell (i.e., the ratio of the cell's maximum length to its orthogonal axis), the geographic centroid of the cell, the intensity-weighted centroid of the cell, or the difference between those two centroids, or some other metric related to cell shape.

[0069]

[0095] The additional metrics may include metrics related to the texture and / or intensity of the cell, as shown in one or more microscopic images of the cell. Such microscopic images of the cell may include phase-contrast, bright-field, fluorescent, or other images of the cell. The images may include composite images. Such composite images may include a cell image generated as described above from two or more bright-field images focused at different planes relative to the cellular contents of the biological specimen. Another example of a composite image is a composite of a phase-contrast image and one or more bright-field images (e.g., a composite of a phase-contrast image and a cell image). Determining such texture- or intensity-based metrics may include determining the metrics based on pixels of the images corresponding to particular cells according to a segmentation map.

[0070]

[0096] A texture metric may be determined from the variation and / or texture across a set of pixels representing a cell. This may include calculating one or more metrics for each neighborhood; for example, for a given pixel, a texture value may be determined based on the set of pixels surrounding the given pixel within a specified distance. Such neighboring texture values ​​may then be averaged across the pixel for the cell to obtain an overall texture value for the cell. Such texture values ​​may include a range value, which is the difference between the maximum and minimum intensity values ​​within the set of pixels; a variance or standard deviation; entropy; a contrast value, which is a measure of the local variation present in the set of pixels; a uniformity value, which is a measure of the uniformity of the set of pixels; and / or several other texture-based measurements.

[0071]

[0097] The intensity-based metric can include the mean brightness of cells in the image, the standard deviation of the brightness of cells in the image, the minimum brightness of cells in the image, the maximum brightness of cells in the image, the brightness of a specified percentile of pixels of cells in the image, a kurtosis or skewness measure of the distribution of brightness values ​​across cells in the image, or some other metric based on intensity, or the intensity variation of cells in one or more images.

[0072]

[0098] Once a set of metrics has been determined for a particular cell, the set of metrics can be used to classify the cell. This can include applying the set of metrics to a training model. Such a model can include at least one of principal component analysis, independent component analysis, support vector machines, artificial neural networks, lookup tables, regression trees, ensembles of regression trees, decision trees, ensembles of decision trees, k-nearest neighbors, Bayesian inference, or logistic regression.

[0073]

[0099] The output of the model can be a representative representation of the determined class of cells after applying the set of metrics to the model. Alternatively, the model can output one or more values ​​indicative of the class of the cell. Such values ​​can then be compared to a threshold to classify the cell. For example, if the model output value is greater than the threshold, the cell can be classified as "viable," while if the model output value is less than the threshold, the cell can be classified as "dead." The value of such a threshold can be determined by an algorithm, for example, as part of the process of training the model based on training data. Additionally or alternatively, the threshold can be set by a user. For example, a user can adjust the threshold based on visual feedback in one or more microscopic images indicating the classification of cells in the image. A user can adjust the threshold after generating an initial threshold through algorithmic processing.

[0074] 12A and 12B illustrate an example of a substantially real-time or otherwise iterative process by a user in adjusting a threshold and receiving visual feedback regarding the effect of the adjustment on the classification of cells in a biological specimen. FIG. 12A illustrates elements of an example user interface during a first time period. The example user interface includes a first annotated image 1200a (e.g., an annotated phase-contrast image) of a biological specimen. The first annotated image 1200a illustrates cells in the specimen and is annotated to indicate the classification of the cells according to a first value of the threshold. As shown in FIG. 12A, a first class of cells is indicated by red coloring and a second class of cells is indicated by blue coloring.

[0075] The threshold value can then be updated by user input to a second value. Such input can include a user pressing a real or virtual button to increase or decrease the threshold value, manipulating a keypad or other means to enter a value for the threshold, moving a slider or dial to adjust the value for the threshold, or performing some other user input action to adjust the threshold to a second value. The second value of the threshold is then applied to reclassify the cells in the specimen. This reclassification is then visually presented to the user in the form of an updated second annotated image 1200b of the biological specimen, shown in FIG. 12B. The second annotated image 1200b is annotated to show the cells in the specimen and indicate the classification of the cells according to the updated second value of the threshold. The classification of some cells changes due to the adjustment of the threshold, and the second annotated image 1200b reflects this change. This updating process can be performed multiple times. For example, the update process can be performed at a rate of once every 20 milliseconds or some other rate to approximate real-time updates of cell classification as a result of the user adjusting the threshold.

[0076] [000102] The model used to classify cells can be trained using a supervised training method and an appropriate training dataset. The training dataset includes a set of metrics determined for each cell in two or more groups of training cells. Each group of training cells corresponds to a respective class or set of classes that the model can be trained to distinguish. The set of metrics in the training dataset can be determined as described above by determining the set of metrics for a particular training cell in a particular group based on one or more microscopic images of the particular training cell.

[0077] [000103] In some examples, training cells can be placed in wells of the same multiwell specimen plate that contain target cells to be classified based on the training cells. This has the advantage of training a model on training cells that have been exposed to the same or similar environmental or other conditions as the target cells without requiring manual annotation of a large number of individual cells. Alternatively, training cells can be placed in wells of a first multiwell specimen plate, and target cells can be placed in wells of a second, different multiwell specimen plate. Such first and second multiwell specimen plates can be cultured in the same incubator or otherwise exposed to the same or similar environmental conditions.

[0078] [000104] The various images and / or metrics used to train the model can be the same as or different from the various images and / or metrics applied to the training model to classify unknown cells. For example, a fluorescent marker can be present in the biological specimen containing the training cells but absent from the specimen containing the unknown target cells to be classified. This can improve model training while avoiding the complexity or hassle of adding fluorescent markers to the target specimen. Additionally or alternatively, fluorescent markers can be used to assign training cells to groups prior to training the model.

[0079] [000105] The training cells in the two (or more) groups of training cells can be identified in various ways. In some examples, the groups of training cells can be manually identified by a user. This can include a user manually indicating individual cells for each of the two or more groups. Such indication can be performed using a user interface that displays images of cells in a biological specimen, with or without the images already segmented. Additionally or alternatively, a user can manually indicate all wells in a multi-well specimen plate as corresponding to each class for training. Any cells detected in wells indicated in this manner are assigned to the corresponding class to train the model. A user can indicate such wells based on knowledge of the state of the wells. For example, a particular well may contain a substance that causes cell death, and then a user can indicate wells containing cells belonging to the "dead" class to train the model. Indicating groups of training cells in this well-by-well manner has the advantage of requiring a relatively small amount of user time and effort (e.g., compared to a user indicating individual cells for training).

[0080] [000106] FIG. 13 illustrates elements of an example user interface 1300 that can be used by a user to indicate one or more wells of a multiwell specimen plate as corresponding to one of two or more classes that a model can later be trained to distinguish. The user interface 1300 illustrates the relative positions of the wells of the multiwell specimen plate, with each well represented by a square. Additional information about each well may be provided. Such additional information may include information about the well contents, conditions applied to the wells, images of the well contents, or some other information. The user can then indicate a set of wells as corresponding to each class. As shown, the user indicates a first set 1310a of wells as corresponding to a first class (e.g., a "survival" class) and a second set 1310b of wells as corresponding to a second class (e.g., a "dead" class).

[0081] [000107] Note that showing a set of cells (e.g., by showing individual cells, by showing all wells of a multi-well specimen plate, by showing cells in conjunction with an automated or semi-automated method) can include showing cells at one or more specified time points. For example, showing a first set of cells can include showing a well at a first time point (e.g., to show the set of viable cells if all or most of the cells in the well are viable), and showing a second set of cells can include showing the same well at a second time point (e.g., to show the set of dead cells if all or most of the cells in the well are dead).

[0082] [000108] Prior to training a model using the resulting training data, the represented set of cells, or the set of metrics determined from the set of cells, can be filtered or otherwise modified. This can be done to reduce the time or number of iterations required to fit the data, resulting in a more accurate model without overfitting the training data, or to otherwise improve the training model and / or the process of training the model. Such filtering or other preprocessing steps can include synthetically balancing the training set of cells, subsampling the training set of cells, and / or normalizing the values ​​of the decision metrics (e.g., normalizing each decision metric so that the population of values ​​of the metric across all cells in the training data occupies a standard range and / or fits a specified distribution).

[0083] Additionally or alternatively, the groups of training cells can be identified by an algorithm or otherwise automatically or semi-automatically. This can include identifying the groups of training cells using the presence or absence of a fluorescent marker. This can include obtaining a fluorescent image of a biological specimen containing the fluorescent marker, and based on the fluorescent image, identifying first and second groups of cells in the specimen according to whether the cells have a mean fluorescent intensity greater than or less than a threshold level, respectively.

[0084] [000110] In another example, an unsupervised training process can be used to classify cells in training images. This can include identifying two or more populations of cells in the training images. A user can then manually classify a limited number of cells as belonging to each class selected from a set of two or more classes. These manually classified cells can be cells that have already been clustered by the unsupervised training process or can be novel cells. Manual classification can then be used to assign clusters of cells to appropriate classes within the set of two or more classes. Manual classification can be cell-by-cell, whole-well-by-well, or some other method of manual classification of cells.

[0085] [000111] The description of different advantageous configurations has been presented for purposes of illustration and description and is not intended to be exhaustive or limited to examples in the form disclosed. Many modifications and variations will be apparent to those skilled in the art. Moreover, different advantageous examples may offer different advantages over other advantageous examples. A selected example or examples have been chosen and described to best explain the principles and practical applications of the examples and to enable those skilled in the art to understand the disclosure for various examples with various modifications as appropriate for the particular use contemplated.

[0086] VI. Experimental Classification Results [000112] Cell classification improves when using one or more metrics determined from cell images of cells (i.e., a composite image determined from two or more defocused brightfield images). Figures 14A and 14B show the accuracy of cell classification as live or dead across many specimens treated with camptothecin (a cytotoxic compound capable of causing cell death, "CMP") or an experimental control compound ("VEH"). Figure 14A shows classification based on a set of metrics determined from a cell-by-cell segmentation mask (e.g., area, perimeter) of the specimen and a phase-contrast image of the specimen (e.g., phase-contrast mean intensity). Figure 14B shows classification based on the above metrics and additional metrics determined from cell images of the specimen (e.g., cell image mean intensity). The overall accuracy across all cells depicted in Figures 14A and 14B increased from 0.82 to 0.94, and the F1 statistic (using live cells as the "positive" class) increased from 0.89 to 0.96.

[0087] [000113] Figures 15A and 15B show the effect of this improved accuracy of classifying cells as live or dead on determined cell mortality rates in many specimens over time. Figure 15A shows a sample of determined cell mortality rates over time as determined from cell-by-cell segmentation masks (e.g., zones, perimeters) of the specimen and phase contrast images of the specimen (e.g., phase contrast average intensity). The red line is the rate as determined by the training model, while the blue line is the true rate. Figure 15B shows a sample of determined cell mortality rates over time as determined from the training model using the metrics described above and additional metrics determined from cell images of the specimen (e.g., cell image average intensity).

[0088] [000114] The classification method described herein facilitates cell classification with accuracy approaching that of fluorophore-based methods. This allows for accurate classification without the high cost, complexity, or experimental tediousness that can accompany the use of fluorescent labels. In experiments, A549 cells were treated with increasing concentrations of the cytotoxic compound camptothecin (0.1-10 μM) for 72 hours in the presence of Annexin V reagent. Cells were classified as dead or alive based on their fluorescent annexin response (live cells = low fluorescence, dead cells = high fluorescence). The results of the Annexin V-based classification are shown in Figure 16A. We used the metric-based method described herein to train a model using label-free features of dead cells (10 μM, 72 hours) and live cells (vehicle, 0-72 hours). This model was then applied to classify cells as live or dead to obtain a percentage of dead cells that could be compared to the Annexin V response. The results of this label-free classification are shown in Figure 16B. Figure 16C shows an overlay of concentration-response curves of % mortality at 72 hours as calculated using Annexin V or the label-free method, demonstrating that the predicted responses across the concentration range were comparable and that EC50 values ​​were similar (Annexin V EC 50 =6.6 E-07;Unlabeled EC50=5.3 E-07 M -1 ).

Claims

1. 1. A method for sorting cells, comprising: acquiring a set of images of a plurality of biological specimens, each of the biological specimens including one or more first cells centered on a first focal plane, the set of images including, for each of the biological specimens, a first phase contrast image and a first bright field image not focused at the first focal plane; obtaining a representation of a first set of cells within the plurality of biological specimens and obtaining a representation of a second set of cells within the plurality of biological specimens, wherein the first set of cells is associated with a first state and the second set of cells is associated with a second state; determining a first plurality of sets of metrics based on the set of images, the representation of the first set of cells, and the representation of the second set of cells, the first plurality of sets of metrics including a set of metrics for each cell of the first set of cells and a set of metrics for each cell of the second set of cells; generating a model using a supervised learning algorithm based on the first plurality of sets of metrics to distinguish between cells in the first set of cells and cells in the second set of cells, thereby generating a training model; determining a second plurality of sets of metrics based on the set of images, the second plurality of sets of metrics including a set of metrics for each cell present in the target specimen; classifying cells in the target specimen, wherein classifying the cells comprises applying the training model to the set of metrics for the cells; Including, The method, wherein the target specimen includes one or more second cells centered on a second focal plane, and the images in the set of images showing the target specimen include a second phase contrast image and a second bright field image not focused at the second focal plane.

2. 2. The method of claim 1, wherein applying the training model to the set of metrics for the cell comprises generating a model output value based on the set of metrics for the cell, and wherein classifying the cell further comprises comparing the model output value to a threshold.

3. displaying an annotated image of the target specimen, the annotated image of the target specimen including an indication of the cells and the classification of the cells; receiving a user input indicating an update threshold; reclassifying the cell by comparing the model output value to an update threshold; displaying an updated annotated image of the target specimen, the updated annotated image of the target specimen including an indication of the cells and the reclassification of the cells; The method of claim 2 further comprising:

4. 4. The method of claim 1, wherein determining the set of metrics for the cells comprises determining at least one of a size metric, a shape descriptor metric, a texture metric, or an intensity-based metric.

5. 5. The method of claim 1, wherein the training model comprises at least one of principal component analysis, independent component analysis, support vector machines, artificial neural networks, lookup tables, regression trees, ensembles of regression trees, decision trees, ensembles of decision trees, k-nearest neighbors, Bayesian inference, or logistic regression.

6. the one or more bright field images include a first bright field image and a second bright field image, the first bright field image representing an image of the target specimen focused at a first defocus distance beyond the focal plane, and the second bright field image representing an image of the target specimen focused at a second defocus distance below the focal plane; 2. The method of claim 1, wherein the method further comprises determining cellular images of the target specimen based on the first and second brightfield images, and wherein determining the set of metrics for the cells comprises determining at least one metric based on the cellular images.

7. 7. The method of claim 1, wherein a fluorescent marker is present in cells of the first set of cells and in cells of the second set of cells, and wherein the fluorescent marker is absent from the target specimen.

8. placing the first set of cells, the second set of cells, and the target specimen all within wells of a multi-well specimen plate; 8. The method of claim 1, further comprising displaying a representation of relative positions of wells of the multiwell specimen plate, wherein the first set of cells are present in a first set of wells of the multiwell specimen plate and the second set of cells are present in a second set of wells of the multiwell specimen plate, and wherein obtaining the representation of the first set of cells and the representation of the second set of cells comprises receiving user input indicating the relative positions of the first set of wells and the relative positions of the second set of wells within the multiwell specimen plate after displaying the representation of the relative positions of wells of the multiwell specimen plate.

9. 9. The method of claim 1, further comprising preprocessing the first plurality of sets of metrics by performing at least one of: normalizing at least one metric in the first plurality of sets of metrics; synthetically balancing the first plurality of sets of metrics between the set of metrics for each cell of the first set of cells and the set of metrics for each cell of the second set of cells; and subsampling the first plurality of sets of metrics before generating the training model.

10. the first set of cells and the second set of cells comprise a fluorescent marker, the set of images of the plurality of biological specimens comprises at least one fluorescent image representing the first set of cells and at least one fluorescent image representing the second set of cells, and obtaining the representation of the first set of cells in the plurality of biological specimens comprises using the at least one fluorescent image representing the first set of cells to identify the first set of cells; 10. The method of claim 1, wherein obtaining the representation of the second set of cells in the plurality of biological specimens comprises using the at least one fluorescent image indicative of the second set of cells to identify the second set of cells.

11. 11. The method of any one of claims 1 to 10, wherein classifying the cells in the target comprises at least one of classifying the cells as live or dead, classifying the cells as stem cells or mature cells, classifying the cells as epithelial or mesenchymal, or classifying the cells as undifferentiated or differentiated cells.

12. 1. A method for sorting cells, comprising: acquiring a set of images of a plurality of biological specimens, each of the biological specimens including one or more first cells centered on a first focal plane, the set of images including, for each of the biological specimens, a first phase contrast image and a first bright field image not focused at the first focal plane; generating a model that distinguishes cells using a supervised learning algorithm based on the set of images, thereby generating a training model; acquiring three or more images of a target specimen, the target specimen including one or more second cells centered on a focal plane for the target specimen, the three or more images including a phase contrast image, a first bright field image, and a second bright field image, the first bright field image representing an unfocused image of the target specimen at a first defocus distance beyond the focal plane, and the second bright field image representing an unfocused image of the target specimen at a second defocus distance below the focal plane; determining a cellular image of the target specimen based on the first brightfield image and the second brightfield image; determining a target segmentation map for the target specimen based on the cell image and the phase contrast image; determining a set of metrics for each cell present in the target specimen based on the three or more images of the target specimen and the target segmentation map; classifying cells in the target specimen, wherein classifying the cells comprises applying the set of metrics of the cells to the training model; A method comprising:

13. The method of claim 12 , wherein determining the set of metrics for the cells comprises determining at least one of a size metric, a shape descriptor metric, a texture metric, or an intensity-based metric.

14. 14. The method of claim 12 or 13, wherein determining the set of metrics of the cell comprises determining at least one metric of the set of metrics of the cell based on the phase contrast image.

15. 15. The method of claim 12, wherein determining the target segmentation map for the target specimen based on the first and second brightfield images comprises applying at least the first and second brightfield images and the phase contrast image to a convolutional neural network.

16. 1. A method for sorting cells, comprising: acquiring a set of images of a plurality of biological specimens, each of the biological specimens including one or more first cells centered on a first focal plane, the set of images including, for each of the biological specimens, a first phase contrast image and a first bright field image not focused at the first focal plane; generating a model that distinguishes cells using a supervised learning algorithm based on the set of images, thereby generating a training model; acquiring two or more images of a target specimen, the target specimen comprising one or more cells centered at a focal plane for the target specimen, the two or more images comprising a phase contrast image and one or more bright field images, the one or more bright field images comprising at least one bright field image representing an image of the target specimen not focused at the focal plane; determining a set of metrics for each cell present in the target specimen based on the two or more images; classifying cells in the target specimen by applying a training model to the set of metrics for the cells; A method comprising:

17. the two or more images of the target specimen include a first brightfield image and a second brightfield image, the first brightfield image representing an image of the target specimen focused at a first defocus distance beyond the focal plane, and the second brightfield image representing an image of the target specimen focused at a second defocus distance below the focal plane; 17. The method of claim 16, wherein the method further comprises determining cellular images of the target specimen based on the first brightfield image and the second brightfield image, and wherein determining the set of metrics for the cells comprises determining at least one metric based on the cellular images.

18. A non-transitory computer readable medium configured to store at least computer readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform controller operations that implement the method of any one of claims 1 to 17.

19. 1. A system for assaying a biological specimen, comprising: An optical microscope; a controller including one or more processors; a non-transitory computer readable medium configured to store at least computer readable instructions that, when executed by the controller, cause the controller to perform controller operations that implement the method of any one of claims 1 to 17; A system including:

Images