Systems and methods for autofocus and automated cell counting using artificial intelligence

Abstract

A system and method for autofocus using artificial intelligence includes (i) capturing multiple monochrome images across a nominal focus range, (ii) identifying one or more binding components in each monochrome image, (iii) sorting the identified binding components based on the number of pixels associated with each binding component, (iv) generating focus quality estimates for at least some of the sorted binding components using a machine learning module, and (iv) calculating a target focus position based on the focus quality estimates of the evaluated binding components. The calculated target focus position can be used to perform cell counting using artificial intelligence, such as by (i) generating a seed likelihood image and a whole-cell likelihood image based on the output of a convolutional neural network, and (ii) generating a mask indicative of object quantity and / or pixel location based on the seed likelihood image.

Description

[Technical Field]

[0001] The present disclosure relates generally to microscopy image analysis and, more particularly, to image analysis solutions for autofocus and cell counts driven by artificial intelligence (AI). [Background technology]

[0002] Cytometry is the counting of cells and / or the measurement of cell characteristics. Various devices and methods are used in the field of cytometry to measure characteristics such as cell number, cell size, cell morphology, and cell life cycle phase. Cytometry can also involve the measurement of various cellular components, such as nucleic acids, the presence of specific proteins, cell typing and / or differentiation (e.g., viability counting), and various medical diagnostic applications.

[0003] There is a continuing need for improvement in the field of cytometry and related fields of image analysis. Summary of the Invention

[0004] Embodiments of the present disclosure provide systems and methods for autofocus and automated cell counting that offer one or more advantages over conventional approaches. For example, one or more embodiments may include a method for autofocusing an imaging system that includes capturing multiple images (e.g., monochrome images) across a nominal focus range, identifying one or more binding components in each image, sorting the identified binding components based on the number of pixels associated with each binding component, using a machine learning module to evaluate focus quality estimates for at least some of the sorted binding components, and calculating a target focus position based on the evaluated focus quality estimates for the binding components. While the term "monochrome image" is used in most of the examples presented herein, it will be understood that the same principles and features can be utilized in applications involving other image types (e.g., color images). Thus, the examples described herein need not be limited to applications involving only monochrome images.

[0005] In one aspect, sorting the identified connected components may additionally or alternatively be based on the circularity and / or brightness associated with each connected component.

[0006] In one aspect, capturing the plurality of monochrome images includes capturing a monochrome 8-bit image (e.g., a 1280 x 960 monochrome 8-bit image) at each z-axis step across the nominal focus range, where each z-axis step is a predetermined size that is automatically determined based on a corresponding sample holder detected in the imaging system.

[0007] In one aspect, the act of identifying one or more connected components in each monochrome image includes thresholding each monochrome image to obtain a resulting binary image for each monochrome image, applying one or more morphological operators to the resulting binary image, and defining one or more connected components in the resulting binary image. In one aspect, the thresholding is based on a difference between a minimum image and a maximum image of a nominal focus range. In a further or alternative aspect, the act of applying the one or more morphological operators includes applying one or more of morphological closing, morphological opening, and / or foreground hole-fill. In one aspect, the morphological closing includes 2x2 morphological closing, and / or the morphological opening includes 2x2 morphological opening. Additionally or alternatively, the foreground hole-fill includes 8-connected foreground hole-fill.

[0008] In one embodiment, the act of identifying one or more binding components in each monochrome image includes measuring first and second binary moments of each binding component, fitting a corresponding ellipse having equivalent moments to each binding component, and creating a second binary image including the corresponding ellipse. In one embodiment, the method further includes measuring each binding component in the second binary image and removing any binding components having an ellipse minor axis less than 15 μm, or less than 10 μm, preferably less than 7 μm, and more preferably less than 5 μm.

[0009] In one aspect, the act of sorting the identified connected components includes counting the number of pixels in each connected component, calculating a median pixel count across one or more connected components, and sorting the connected components based on the corresponding absolute difference in the number of pixels from the median pixel count. In one aspect, the connected components are sorted in ascending (or descending) order by the corresponding absolute difference in the number of pixels from the median pixel count.

[0010] In one aspect, the act of sorting the identified one or more binding components includes ordering the calculated joint probabilities that the binding components (i) represent a cell and, optionally, (ii) represent that the cell is alive. The binding components with the greatest joint probabilities can be sorted for further evaluation of focus quality estimates using a machine learning module.

[0011] In one aspect, the act of sorting the identified one or more connected components includes determining one or more of the circularity or brightness of each connected component and sorting the connected components based on the comparative circularity and / or brightness determined for each of the connected components.

[0012] In one aspect, the method for autofocusing an imaging system can include forming a z-stack of specific pixel dimensions (e.g., 32x32 or other suitable size) for each connected component. In one aspect, the machine learning module includes an artificial neural network that receives as input the pixel z-stack for each connected component. In one aspect, the artificial neural network includes multiple feature identification layers having a design pattern of convolutional, linear, and max-pooling layers. In one aspect, the convolutional layer includes a 3x3 convolutional layer, the linear layer includes a ReLu nonlinear function, and / or the max-pooling layer includes a 2x2 max-pooling layer or an average pooling layer.

[0013] In one embodiment, the artificial neural network includes a plurality of feature identification layers followed by a long-short-term memory (LSTM) processing layer, where the LSTM layer processes the z-stack of each connected component in an iterative manner. In one embodiment, the artificial neural network includes a final linear layer that is combined with the output of the LSTM layer to define a focus quality estimate.

[0014] Embodiments of the present disclosure further include a computer system configured to autofocus an imaging system. In one aspect, the computer system configured to autofocus an imaging system includes one or more processors and one or more hardware storage devices storing computer-executable instructions that, when executed by the one or more processors, configure the computer system to perform any one or more of the methods disclosed herein.

[0015] Embodiments of the present disclosure may further include a method for performing automated cell counting, including the acts of acquiring an image, defining a set of one or more tiles based on the image, processing the one or more tiles using a convolutional neural network, and constructing multiple pseudo-probability maps based on the output of the convolutional neural network. The multiple pseudo-probability maps may include at least one or more seed likelihood images. The acts may further include generating one or more masks based on the one or more seed likelihood images. The one or more masks may define pixel locations of one or more objects represented in the one or more seed likelihood images. The one or more masks may indicate / provide cell counts.

[0016] In one aspect, a method for performing automated cell counting includes performing connected component analysis on one or more seed likelihood images to determine cell counts.

[0017] In one aspect, the plurality of pseudo-probability maps further comprises one or more whole cell likelihood images, and the method of performing automated cell counting comprises generating one or more segmented images based on at least the one or more whole cell likelihood images and the one or more masks, wherein the one or more segmented images indicate / provide a cell count.

[0018] In one aspect, the image is captured at a target focal position.

[0019] In one embodiment, the convolutional neural network is trained on a training data set including images from multiple focal planes, preferably for focus positions ranging from -2 to +2 z positions, to increase the robustness of the method and reduce the sensitivity of the method to autofocus output accuracy. The training data set may include ground truth outputs including one or more whole-cell binary masks and / or one or more seed masks. The images in the training data set may include preprocessed images.

[0020] In one embodiment, the image comprises a 2592x1944 monochrome 8-bit image with a pixel size of 0.871 μm. Other types of images (e.g., having different dimensions, bit values, and / or pixel sizes) may also be utilized.

[0021] In one aspect, the method for performing automated cell counting further includes performing one or more preprocessing operations. The one or more preprocessing operations can be performed based on the image to generate a preprocessed image. A set of one or more tiles can be extracted from the preprocessed image, or the one or more preprocessing operations can be performed on one or more tiles.

[0022] The one or more pre-processing operations may include a down-sampling operation, which may utilize an averaging filter, which may reduce the image size by at least a factor of two.

[0023] The one or more preprocessing operations may include a background removal operation. In some cases, the background removal operation includes estimating the background by calculating a local average for each pixel within a radius of each pixel and subtracting the background from the image. In some cases, the background removal operation includes calculating respective minima in each image region of the image or downsampled image and fitting a surface through the respective minima.

[0024] The one or more pre-processing operations may include a pixel or voxel intensity normalization operation. The pixel or voxel intensity normalization operation may include a global normalization operation. The pixel or voxel intensity normalization operation may include a kernel-based normalization operation.

[0025] In one aspect, at least some of the one or more preprocessing operations are performed as a batch process on one or more tiles provided as input to the convolutional neural network.

[0026] In one aspect, defining one or more sets of tiles includes forming a reflected image, decomposing the reflected image into one or more sets of tiles with pixel overlap, and storing the one or more sets of tiles in a tensor array. Reflecting the downsampled image can extend its edges by a number of pixels (e.g., 4-12 pixels, or 6-10 pixels, or 8 pixels), and the reflected image can be decomposed into sets of tiles with pixel overlap. In a non-limiting example, the reflected image is decomposed into 130 tiles of 128 x 128 pixels with 8-pixel overlap. The tensor array can include a 130 x 1 x 128 x 128 tensor array or other suitable tensor array based on the number of tiles, image size, and other image characteristics.

[0027] In one embodiment, the convolutional neural network comprises a U-Net convolutional neural network. The output of the convolutional neural network may be stored in a suitable tensor array (e.g., a 130x4x128x128 tensor array).

[0028] In one aspect, constructing the multiple pseudo-probability maps includes converting the output of the convolutional neural network into a tensor (e.g., 8-bit format) and image stitching the tensor. Converting the output into a tensor may include multiplying a tensor array (e.g., a 130x4x128x128 tensor array) by a multiplier (e.g., 255). Image stitching the tensor may include constructing several (e.g., four) full-size pseudo-probability maps from the tiles. The multiple pseudo-probability maps may each include: (i) one or more live cell seed likelihood images, preferably indicating one or more center locations of live cells; (ii) one or more live cell global likelihood images; (iii) one or more dead cell seed likelihood images, preferably indicating one or more center locations of dead cells; and (iv) one or more dead cell global likelihood images.

[0029] In one embodiment, generating the one or more masks includes applying a threshold greater than 75% likelihood. The threshold may correspond to a threshold of 192 (0.75 x 255) in pixel intensities in the plurality of pseudo-probability maps. Other thresholds may be utilized, such as, by way of non-limiting example, thresholds in the range of about 50% to about 90% likelihood. Generating the one or more masks may further include applying bound component labeling to detect bound regions, and the one or more masks may be generated using a deep learning algorithm.

[0030] In some cases, generating the one or more segmented images includes applying a watershed transformation to a distance map calculated from pixel locations of one or more objects represented in one or more seed likelihood images and segmented by one or more whole-cell likelihood images.

[0031] In one aspect, the method of performing an automated cell count further comprises determining a cell number based on the output of the convolutional neural network. The method of performing an automated cell count may further comprise displaying the cell number based on the output of the convolutional neural network.

[0032] In one aspect, the method for performing automated cell counting further includes performing one or more feature calculation operations using the one or more segmented images. The one or more feature calculation operations may include ellipse fitting cells within the one or more segmented images. The ellipse fitting may include measuring object size (e.g., in μm), constructing a histogram of object size, pixel intensity, and / or calculating object circularity.

[0033] In one aspect, the image comprises a monochrome image.

[0034] Embodiments of the present disclosure further include computer systems configured to perform automated cell counting. For example, a computer system for performing automated cell counting can include one or more processors and one or more hardware storage devices having stored thereon computer-executable instructions that, when executed by the one or more processors, configure the computer system to perform any one or more methods of performing automated cell counting disclosed herein.

[0035] Accordingly, systems and methods are disclosed for autofocusing an imaging system and / or performing automated cell counting. In some embodiments, the systems and methods disclosed herein enable automated cell viability counting.

[0036] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an index of the scope of the claimed subject matter.

[0037] Additional objects and advantages of the present disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the present disclosure. The features and advantages of the present disclosure may be realized and obtained by means of the instruments and combinations disclosed herein. These and other features of the present disclosure will become more fully apparent from the following description and appended claims, or may be learned by the practice of the disclosure as set forth hereinafter. [Brief explanation of the drawings]

[0038] To illustrate how the above-mentioned and other advantages and features of the present disclosure can be obtained, a more particular description of the present disclosure, briefly described above, will be rendered by reference to specific embodiments thereof, which are illustrated in the accompanying drawings. It will be understood that these drawings illustrate only typical embodiments of the disclosure and therefore should not be considered as limiting its scope. The present disclosure will be described and explained with additional specificity and detail using the following accompanying drawings: [Figure 1] FIG. 1 shows a perspective view of an imaging system configured to perform one or more of the methods disclosed herein, including methods for artificial intelligence (AI)-assisted autofocus and automated cell viability counting, in accordance with one or more embodiments of the present disclosure. [Figure 2] FIG. 2 is an exemplary flow diagram illustrating various actions that may be performed by the imaging system of FIG. 1 to facilitate AI-assisted autofocus and automated cell viability counting, in accordance with one or more embodiments of the present disclosure. [Figure 3] 2 shows a schematic diagram of various exemplary components within the imaging system of FIG. 1 in accordance with one or more embodiments of the present disclosure. [Figure 4] FIG. 10 is an exemplary flow diagram illustrating various acts for determining a target focal position in accordance with one or more embodiments of the present disclosure. [Figure 5A] 1 shows a simplified diagram of a canonical neural network as known in the art. [Figure 5B] 5B shows an isolated portion of the neural network shown in FIG. 5A. [Figure 6]FIG. 1 is a block diagram representing an example design of an artificial neural network for facilitating AI-assisted autofocus, in accordance with one or more embodiments of the present disclosure. [Figure 7] FIG. 1 is an exemplary flow diagram showing various acts for performing automated, AI-assisted cell viability counting using a target focal position, in accordance with one or more embodiments of the present disclosure. [Figure 8] 1 provides a conceptual diagram of background removal. [Figure 9] FIG. 1 shows a block diagram representing an exemplary design of a U-Net convolutional neural network for facilitating AI-assisted determination of cell viability, in accordance with one or more embodiments of the present disclosure. [Figure 10] 1 provides additional conceptual diagrams of the inputs and outputs associated with a U-Net convolutional neural network. [Figure 11] An example is shown of using the watershed transform to generate a segmented image based on a seed likelihood image and a whole-cell likelihood image. [Figure 12] 10 shows an example representation utilizing one-object operations to generate a segmented image based on a seed likelihood image and a whole-cell likelihood image. [Figure 13] 10 shows an example representation utilizing a deep learning module to generate a segmented image based on a seed likelihood image and a whole-cell likelihood image. [Figure 14] 1 illustrates an exemplary image displayed after implementing a method for AI-assisted autofocus and automated cell viability counting, according to one or more embodiments of the present disclosure. [Figure 15A] Shown are the same base images assessed and annotated with the number and location of live and dead cells within each image, as determined by a biologist (Figure 15A). [Figure 15B] The same base images are shown evaluated and annotated with the number and location of live and dead cells within each image, as determined by prior art automated cell identification and viability methods (FIG. 15B). [Figure 15C]The same base images are shown evaluated and annotated with the number and location of live and dead cells within each image, as determined by the AI-assisted autofocus and automated cell counting methods disclosed herein (FIG. 15C). DETAILED DESCRIPTION OF THE INVENTION

[0039] Before describing embodiments of the present disclosure in detail, it should be understood that the present disclosure is not limited to particularly illustrated system, apparatus, method, and / or process parameters, which may, of course, vary. Thus, while particular embodiments of the present disclosure will be described in detail with reference to particular configurations, parameters, components, elements, etc., the description is illustrative and should not be construed as limiting the scope of the disclosure. Additionally, the terminology used herein is for the purpose of describing embodiments and is not necessarily intended to limit the scope of the disclosure.

[0040] It will be understood that systems, devices, methods, and / or processes according to certain embodiments of the present disclosure may include, incorporate, or otherwise include properties or features (e.g., components, members, elements, parts, and / or portions) described in other embodiments disclosed and / or described herein. Thus, various features of certain embodiments may be compatible with, combined with, included in, and / or incorporated into other embodiments of the present disclosure. Thus, the disclosure of a particular feature in connection with a particular embodiment of the present disclosure should not be construed as limiting the application or inclusion of that feature to the particular embodiment. Rather, it will be understood that other embodiments may include such features, members, elements, parts, and / or portions without necessarily departing from the scope of the present disclosure.

[0041] Furthermore, unless otherwise understood or stated, implicitly or explicitly, it is understood that for any given component or embodiment described herein, any of the possible candidates or alternatives listed for that component may generally be used individually or in combination with each other. Additionally, unless otherwise understood or stated, implicitly or explicitly, it will be understood that any listing of such candidates or alternatives is merely illustrative and not limiting.

[0042] Additionally, unless otherwise indicated, numbers expressing quantities, components, distances, or other measurements used in the specification and claims should be understood as being modified by the term "about." As used herein, the terms "about," "approximately," "substantially," or their equivalents refer to an amount or condition that is close to the specific stated amount or condition that still performs the desired function or achieves the desired result. For example, the terms "approximately," "about," and "substantially" may refer to an amount or condition that deviates from the specifically stated amount or condition by less than 10%, or less than 5%, or less than 1%, or less than 0.1%, or less than 0.01%.

[0043] Accordingly, unless indicated to the contrary, the numerical parameters set forth in the specification and appended claims are approximations that can vary depending upon the desired properties sought to be obtained by the subject matter presented herein. At the very least, and not as an attempt to limit the application of the doctrine of equivalents to the scope of the claims, each numerical parameter should at least be construed in light of the number of reported significant digits and by applying ordinary rounding techniques. Notwithstanding that the numerical ranges and parameters setting forth the broad scope of the subject matter presented herein are approximations, the numerical values ​​set forth in the specific examples are reported as precisely as possible. However, any numerical value inherently contains certain errors necessarily resulting from the standard deviation found in their respective testing measurements.

[0044] The headings and sub-headings used herein are for organizational purposes only and are not meant to be used to limit the scope of the description or the claims. Overview of systems and methods for AI-assisted autofocus and cell viability counting

[0045] Cell viability counting has traditionally been a manual, time-intensive process. Advances in the field of image processing have enabled imaging systems to automate some of these manual tasks or otherwise reduce the amount of time and manual effort associated with determining cell concentration in a sample, particularly with regard to ascertaining the proportional number of live / dead cells within a given sample (or viability counting). While these innovations are promising, they ultimately fall short of the accuracy and precision of manual cell viability counting.

[0046] Prior art systems and methods for analyzing cell viability in a sample suffer from many drawbacks. For example, many cell viability systems require the use of dyes, labels, or other compounds to determine the viability of cells in a sample. The use of many of these compounds often requires specialized, expensive, and / or bulky equipment to retrieve the resulting readout. Thus, the equipment is unlikely to be readily available and / or likely to be conveniently located within the laboratory space.

[0047] Some prior art systems rely on computationally intensive algorithms to identify and differentiate between live and dead cells within a sample. Unfortunately, these systems generally require access to dedicated, robust computing resources, such as graphics processing units (GPUs), or distributed computing resources that can more quickly compute and process the large amounts of data associated with known image recognition and processing algorithms. Unfortunately, access to distributed computing resources and network or cloud computing environments that may enable more rapid analysis of image-acquired cell viability data is not always available or feasible. Indeed, the confidential nature of some laboratory samples and / or ongoing experiments has led to an increase in security measures that prevent access to networked computing resources, such as the cloud, including situations where such resources and analytical capabilities are provided by third parties and privacy concerns exist regarding laboratory samples.

[0048] There is a lack of systems that are compact, relatively inexpensive, and conveniently located and accessible from a common workplace without occupying a large benchtop footprint or requiring an isolated, dedicated environment. Adding a GPU or otherwise increasing the computing power of an existing system is impractical for some devices. Additional computing power requires additional space within the system, generates additional heat, and increases the overall cost of the system, both upfront and operational (although the principles disclosed herein may be implemented on devices utilizing one or more GPUs). However, without additional computing resources to process image data, conventional systems cannot provide the desired results in a timely manner, if at all. Known cell viability counting systems are unable to focus on live cells while ignoring dust, debris, manufacturing defects in the bottom of the counting chamber, and other non-cellular particles on microscopic images. These artifacts confound the results and prevent current systems from accurately and precisely distinguishing cellular particles from non-cellular particles. As a result, current cell viability counting systems are ineffective at generating consistent and accurate measurements of cell viability within a sample.

[0049] Therefore, there is a need for a cost-effective, compact, and safe benchtop imaging system that performs accurate and rapid autofocus and cell viability counting of relevant samples.

[0050] As alluded to above, prior art systems and methods for autofocusing and analyzing cell viability within a sample suffer from many drawbacks. In particular, there has traditionally been a high computational cost associated with image processing, which has heretofore prevented non-specialized laboratory equipment from quickly and accurately focusing on and identifying live / dead cells within a sample. Problematically, previously known methods rely on the evaluation of higher resolution images in an attempt to use the additional data provided thereby to resolve the differences between live and dead cells. This implementation necessarily requires a significant investment in additional computational resources, such as a dedicated GPU battery or access to a large-scale distributed computing environment.

[0051] There is a lack of systems that are compact, relatively inexpensive, and conveniently located and accessible from a common workplace without occupying a large benchtop footprint or requiring an isolated, dedicated environment. Adding a GPU or otherwise increasing the computing power of an existing system is impractical for some devices. Additional computing power requires additional space within the system, generates additional heat, and increases the overall cost of the system, both upfront and operational. The embodiments described herein may be implemented on devices that utilize one or more GPUs. However, in certain applications, without additional computing resources to process image data, conventional systems cannot provide the desired results in a timely manner, if at all. Known cell viability counting systems are unable to focus on live cells while ignoring dust, debris, manufacturing defects in the bottom of the counting chamber, and other non-cellular particles on microscopic images. These artifacts confound the results and prevent current systems from accurately and precisely distinguishing cellular particles from non-cellular particles. As a result, current cell viability counting systems are ineffective at generating consistent and accurate measurements of cell viability within a sample.

[0052] The systems and methods disclosed herein solve one or more of the problems noted in the art and advantageously enable rapid autofocus (e.g., for mixtures of live and dead cells) using minimal processing cycles and without the need for additional processing hardware. The improved autofocus methods disclosed herein are powered by artificial intelligence and allow for rapid and reliable identification of a target focus position in any given sample. This rapid, low-cost method of identifying a target focus position within a sample is incorporated into many of the disclosed cell viability counting methods as a first step to enable rapid (e.g., less than 20 seconds, preferably less than 10 seconds) identification and / or display of a representation of the cell viability count.

[0053] FIG. 1 shows a perspective view of an imaging system 100 configured to perform one or more of the methods disclosed herein. For example, the imaging system 100 of FIG. 1 is operable to facilitate a method associated with flow diagram 200 for AI-assisted autofocus and / or automated cell viability counting, as disclosed by the exemplary flow diagram of FIG. 2. As shown, the imaging system 100 includes a housing 102 that encloses and protects a microscope and a computing system used to perform autofocus and cell viability counting. The housing 102 includes a slide port / stage assembly 106 operable to accept a cell counting slide into the imaging system (act 202). Once received therein, the imaging system 100 then determines a target focal position for imaging cells on the cell counting slide (act 204) and performs automated cell viability counting using the target focal position (act 206). A representation of the cell viability counting is displayed on the imaging system 100 (act 208), for example, using the display 104. This representation and / or other data associated with the automated cell viability counting may be removed from the imaging system 100 through user interaction with the communications module 108, which in some embodiments may include a USB port or other data exchange port as known in the art, and / or may be stored on a separate device.

[0054] In light of the present disclosure, it will be understood that the principles described herein may be implemented utilizing any suitable imaging system and / or any suitable imaging modality. The specific examples of imaging systems and imaging modalities discussed herein are provided by way of example and as a means of illustrating features of the disclosed embodiments. Accordingly, the embodiments disclosed herein are not limited to any particular microscope system or microscope application, but may be implemented in a variety of contexts, such as bright-field imaging, fluorescence microscopy, flow cytometry, confocal imaging (e.g., 3D confocal imaging, or any type of 3D imaging), etc. For example, the principles discussed herein may be implemented using a flow cytometry system to provide or improve cell counting capabilities. As another example, cell count and / or viability data obtained according to the techniques of the present disclosure may be used to complement fluorescence data to improve accuracy in distinguishing between different cells.

[0055] Furthermore, in light of the present disclosure, it will be understood that any number of the principles described herein may be implemented in a variety of fields. For example, a system may implement the cell counting techniques discussed herein without necessarily also implementing the autofocus, cell viability, and / or feature detection techniques described herein.

[0056] 3 shows a schematic diagram of various exemplary components within the imaging system 100 of FIG. 1 in accordance with one or more embodiments of the present disclosure. For example, FIG. 3 illustrates that the imaging system 100 may include a computer system 110 and a microscope system 120 included therein. FIG. 3 conceptually depicts the computer system 110 and the microscope system 120 disposed within the housing 102 of the imaging system 100. However, in light of the present disclosure, it should be understood that any portion of the computer system 110 or the microscope system 120 may be disposed at least partially outside the housing 102 within the scope of the disclosed embodiments.

[0057] FIG. 3 illustrates that the computer system 110 of the imaging system 100 can include various components, such as a processor 112, a hardware storage device 114, a controller 116, a communication module 108, and / or a machine learning module 118.

[0058] Processor 112 may comprise one or more sets of electronic circuitry, including any number of logic units, registers, and / or control units, to facilitate the execution of computer-readable instructions (e.g., instructions forming a computer program). Such computer-readable instructions may be stored in hardware storage devices 114, which may include physical system memory, and may be volatile, non-volatile, or some combination thereof. Further details related to processors (e.g., processor 112) and computer storage media (e.g., hardware storage devices 114) are provided below.

[0059] Controller 116 may comprise any suitable software components (e.g., a set of computer-executable instructions) and / or hardware components (e.g., application-specific integrated circuits or other dedicated hardware components) operable to control one or more physical devices of imaging system 100, such as portions of microscope system 120 (e.g., positioning mechanism 128).

[0060] Communications module 108 may comprise any combination of software or hardware components operable to facilitate communication between on-system components / devices and / or with off-system components / devices. For example, communications module 108 may comprise a port, bus, or other physical connection for communicating with other devices (e.g., a USB port, an SD card reader, and / or other devices). Additionally or alternatively, communications module 108 may comprise a system operable to wirelessly communicate with external systems and / or devices through any suitable communications channel, such as, but not limited to, Bluetooth, ultra-wideband, WLAN, infrared communications, etc.

[0061] The machine learning module 118 may also comprise any combination of software or hardware components operable to facilitate processing using a machine learning model or other artificial intelligence-based structure / architecture. For example, the machine learning module 118 may include hardware components or computer-executable instructions operable to execute functional blocks and / or processing layers configured in the form of, for example, a single-layer neural network, a feedforward neural network, a radial basis function network, a deep feedforward network, a recurrent neural network, a long short-term memory (LSTM) network, a gated recurrent unit, an autoencoder neural network, a variational autoencoder, a denoising autoencoder, a sparse autoencoder, a Markov chain, a Hopfield neural network, a Boltzmann machine network, a restricted Boltzmann machine network, a deep belief network, a deep convolutional network (or convolutional neural network), a deconvolutional neural network, a deep convolutional inverse neural network, a generative adversarial network, a liquid state machine, an extreme learning machine, an echo state network, a deep residual network, a Kohonen network, a support vector machine, a neural Turing machine, or the like, as non-limiting examples.

[0062] As shown in FIG. 3, the imaging system 100 includes a microscope system 120 having an image sensor 122, an illumination source 124, an optical train 126, a slide port / stage assembly 106 for receiving a sample slide, and a positioning mechanism 128.

[0063] The image sensor 122 is positioned in the optical path of the microscope system and configured to capture images of the sample, which will be used in the disclosed methods to identify the target focus position and subsequently perform automated cell viability counting. As used herein, the terms "image sensor" or "camera" refer to any applicable image sensor compatible with the apparatus, systems, and methods described herein, including, but not limited to, charge-coupled devices, complementary metal-oxide-semiconductor devices, N-type metal-oxide-semiconductor devices, Quanta image sensors, scientific complementary metal-oxide-semiconductor devices, and combinations of the foregoing.

[0064] The optical train 126 may include one or more optical elements configured to direct light from the illumination source 124 onto the received cell counting slide to facilitate viewing of the cell counting slide. The optical train 126 may also be configured to direct light scattered, reflected, and / or emitted by the specimen in the cell counting slide toward the image sensor 122. The illumination source 124 may be configured to emit various types of light, such as white light or light in one or more specific wavelength bands. For example, the illumination source 124 may include a light cube (e.g., a Thermo Fisher EVOS™ light cube), which can be installed and / or replaced within a housing for any desired set of illumination wavelengths.

[0065] The positioning mechanism 128 may include any of an x-axis motor, a y-axis motor, and a z-axis motor, which are operable to adjust components of the optical train 126 and / or the image sensor 122 accordingly.

[0066] 3 further illustrates that, in some instances, imaging system 100 includes a display 104. Figure 3 illustrates that display 104 may be in direct or indirect communication with various other components of imaging system 100, such as computer system 110 or its microscope system 120 (e.g., as indicated by the three-headed arrow in Figure 3). For example, imaging system 100 may capture images using components of microscope system 120, the captured images may be processed and / or stored using components of computer system 110 (e.g., processor 112, hardware storage device 114, machine learning module 118, etc.), and the processed and / or stored images may be displayed on display 104 for viewing by one or more users.

[0067] As described herein, components of imaging system 100 may facilitate AI-assisted autofocus for samples contained within a cell counting slide imaged by imaging system 100, as well as AI-assisted cell viability counting for samples contained within the cell counting slide. In some cases, a representation of the results of the AI-assisted cell viability counting (which may be performed according to a target focus position determined via AI-assisted autofocus) may be displayed on display 104 of imaging system 100 within a short period of time (e.g., within about 20 seconds or less, or within about 10 seconds or less) after initiating the autofocus and cell viability counting process for a cell counting slide inserted into imaging system 100.

[0068] In view of the present disclosure, it should be understood that the imaging system may include additional or alternative components to those shown and described with reference to FIG. 3, and that such components may be organized and / or distributed in various manners. Systems and methods for facilitating AI-assisted autofocus

[0069] As described above, facilitating artificial intelligence-powered autofocus and cell viability counting includes determining a target focal position for imaging cells on a cell counting slide (act 204, as described above with reference to FIG. 2). Determining a target focal position for imaging cells on a cell counting slide may be associated with various acts. FIG. 4 shows an exemplary flow diagram illustrating various acts associated with act 204 from flow diagram 200 for determining a target focal position for imaging cells on a cell counting slide. While the acts shown in flow diagram 200 may be illustrated and / or described in a particular order, a specific ordering is not required unless specifically stated or required, as some acts depend on the completion of other acts before they can be performed. Furthermore, it should be noted that not all acts depicted in the flow diagrams are necessary to facilitate the disclosed methods, including the methods for AI-assisted autofocus and automated cell viability counting disclosed herein.

[0070] Act 204a related to act 204 includes capturing multiple monochrome images over a nominal focus range. In some implementations, act 204 is performed by imaging system 100 using processor 112, hardware storage device 114, and / or controller 116 of computer system 110, and image sensor 122, illumination source 124, optical train 126, slide port / stage assembly 106, and / or positioning mechanism of microscope system 120.

[0071] As an example, imaging system 100 may employ processor 112 in conjunction with one or more sensors to identify a sample holder (e.g., a cell counting slide) placed within slide port / stage assembly 106. In some cases, imaging system 100 automatically identifies the type of sample holder (e.g., a disposable cell counting slide or a reusable cell counting slide) and automatically determines image capture settings based on the detected sample holder type. For example, imaging system 100 may identify an image z-axis step height / size and / or an initial nominal focus range based on whether the cell counting slide is disposable or reusable and / or based on other attributes of the cell slide (e.g., a determined sample holder, coverslip, and / or other substrate thickness).

[0072] Additionally, imaging system 100 can utilize processor 112 and / or controller 116 to cause positioning mechanism 128 to position optical train 126 relative to slide port / stage assembly 106 to facilitate capture of images of the cell counting slide (e.g., a sample within the cell counting slide). Imaging system 100 can capture images of the cell counting slide using image sensor 122 and illumination source 124 in combination with any of the above. Furthermore, imaging system 100 may capture additional images of the cell counting slide under different relative positioning of slide port / stage assembly 106 and optical train 126 to obtain multiple monochrome images across a nominal focus range. In some implementations, the monochrome images comprise 8-bit images with a resolution of 1280 x 960, each of which is captured at each z-axis step across the nominal focus range.

[0073] Any instructions for performing the actions described herein and / or data used or generated / stored in connection with performing the actions described herein (e.g., cell counting slide type, nominal focus range, z-axis step, monochrome image, etc.) may be stored in hardware storage device 114 in a volatile or non-volatile manner.

[0074] Act 204b, related to act 204, includes identifying one or more connected components within each monochrome image. In some implementations, imaging system 100 utilizes one or more of processor 112, hardware storage device 114, and / or machine learning module 118 to identify connected components within each monochrome image.

[0075] As used herein, "connectivity" as used in "connected component" refers to which pixels are considered neighbors of a target pixel. After a suitable digitized image is available (e.g., from multiple monochrome images acquired pursuant to act 204a), all connected components in the image are first identified. A connected component is a set of pixels of a single value, e.g., a value representing black, and a path can be formed from any pixel in the set to any other pixel in the set without leaving the set, e.g., by traversing only black pixels. Generally speaking, a connected component can be either "4-connected" or "8-connected." In the case of 4-connected, the path can only travel horizontally or vertically, so there are four possible directions. Thus, two diagonally adjacent black pixels are not 4-connected unless another horizontally or vertically adjacent black pixel acts as a bridge between the two. In the case of 8-connected, the path between pixels can also proceed diagonally. One embodiment uses 8-connected components, although 4-connected components can also be identified and used.

[0076] For example, in some implementations, identifying connected components in each monochrome image is performed by thresholding the monochrome image to obtain a resulting binary image for each monochrome image. The thresholding may be performed based on the difference between the minimum and maximum images in the nominal focus range described above with reference to act 204a. Various methods for thresholding are within the scope of this disclosure, such as the "upper triangle" thresholding method.

[0077] In addition to thresholding, identifying connected components of each monochrome image may include applying one or more morphological operators to each of the resulting binary images. Morphological operations / operators include, by way of non-limiting example, morphological closing, morphological opening, The morphological closing operation may include fjords (C-shaped artifacts or irregularities) present in the monochrome image to approximate the cell shape in the binary image, based on the assumption that cells should have a round shape. In one embodiment, the morphological closing operation utilizes a 2x2 mask size.

[0078] Furthermore, in some cases, cells may appear in the binary image as including tendrils (e.g., side branches) that extend beyond the cell wall due to improper or suboptimal illumination, focusing, image sensing, and / or post-processing (e.g., thresholding). Thus, in some implementations, the system may perform a morphological opening operation to trim or remove tendrils from the binary image to improve the approximation of the cell shape in the binary image. In one embodiment, the morphological opening operation is performed using a 2x2 mask size.

[0079] Additionally, foreground holes may appear in the set of binary images based on the appearance of cells from different focus positions (e.g., different z-height focus positions at which multiple monochrome images were captured according to act 204a). Accordingly, in some instances, a foreground hole-fill operation may be performed to expand connected pixels in a manner that fills such foreground holes. In one embodiment, the foreground hole-fill operation is an 8-connection foreground hole-fill operation, while in some implementations, the foreground hole-fill operation is a 4-connection foreground hole-fill operation.

[0080] In some cases, after performing the desired morphological operations (e.g., morphological opening, morphological closing, foreground hole-fill) on each binary image, imaging system 100 may define connected components within each binary image in preparation for further processing. However, in some cases, additional operations may be performed to identify connected components in preparation for further processing.

[0081] For example, in some implementations, the imaging system 100 measures first and second binary moments for each binding component defined within each of the binary images. In some cases, the first and second binary moments of a particular binding component may be correlated with the axes of an ellipse, which may serve as an approximation of the cell shape. Thus, for a particular binding component, the imaging system 100 can fit an ellipse to the particular binding component, the ellipse having moments based on the first and second binary moments measured for the particular binding component.

[0082] In this manner, the imaging system 100 can fit an ellipse to each connected component defined in each of the above-mentioned binary images. In some implementations, the imaging system 100 generates a second binary image from each of the above-mentioned binary images (e.g., each binary image generated from thresholding each monochrome image). The second binary image can include connected components generated / defined based on ellipse fitting from first and second binary moments measured for each connected component from the above-mentioned binary images. In this regard, in some cases, the ellipse-based connected components of the second binary image can assist the imaging system 100 in approximating cell shapes.

[0083] In some implementations, according to the present disclosure, the second binary image is used for further processing to determine the target focus position. However, in some implementations, the imaging system 100 generates a third set of binary images based on the first binary image and the second binary image and uses the third set of binary images for further processing to determine the target focus position. The third set of binary images can be generated by merging or taking the union of each binary image (or an initial binary image generated by thresholding each corresponding monochrome image) with its corresponding second binary image.

[0084] Whether the imaging system 100 utilizes the first, second, or third binary image set for further processing to determine the target focal position, the imaging system 100, in some implementations, may measure each binding component and remove any binding components that do not meet a predetermined size condition. For example, the imaging system 100 may remove binding components from any of the first, second, or third binary images with a minor axis length or diameter (e.g., an elliptical minor axis length) of less than 40 μm. In some implementations, the predetermined size condition is selected based on the type of cells being counted. For example, for T cells, B cells, NK cells, and / or monocytes, the imaging system 100 may remove binding components with a minor axis diameter outside the range of approximately 2 to 40 μm, while for other cells, the imaging system 100 may remove binding components with a minor axis diameter outside the range of approximately 2 to 8 μm. Such minor axis length diameter may be set, for example, at 2 μm, 4 μm, 6 μm, 8 μm, 10 μm, 15 μm, 20 μm, 25 μm, 30 μm, 35 μm, 40 μm, 45 μm, or 50 μm, or a range having endpoints defined by any two of the foregoing values, depending on the needs of a particular application.

[0085] Act 204c, associated with act 204, includes sorting the identified connected components based on the number of pixels associated with each connected component. As described above, the identified connected components sorted according to act 204c may be from the third set of binary images, the second binary image, or the first binary image described above with reference to act 204b. In some implementations, imaging system 100 utilizes processor 112 to sort the identified connected components based on the number of pixels associated with each connected component.

[0086] Sorting the identified connected components according to act 204c may include various acts / steps. For example, in some instances, sorting the identified connected components may include counting the number of pixels included in each connected component, calculating a median pixel count for the connected components, and sorting the connected components based on the median pixel count (e.g., based on the absolute difference between the number of pixels in each respective connected component and the median pixel count).

[0087] In some cases, imaging system 100 utilizes one or more other inputs in addition to, or as an alternative to, pixel counts to sort the identified bound components. For example, imaging system 100 may utilize pixel brightness of the imaged components and / or a measure of the relative circularity of the imaged components. Based on the brightness and / or circularity of the imaged components, imaging system 100 may determine a probability that the imaged component is a cell and another probability that the cell is alive (e.g., a live cell or a dead cell). These two probabilities may be combined (e.g., by multiplying the two probabilities together) to form a joint probability that the imaged component is a live cell (e.g., a probability that the component is a cell based on brightness alone and a probability that the component is a cell based on circularity alone).

[0088] The binding components may be sorted in ascending or descending order. In some cases, sorting the binding components in ascending order has the effect of moving binding components that approximate the typical size of cells to the top of the list and binding components that are more likely to approximate other objects (e.g., clumps and debris particles) to the bottom of the list.

[0089] Additionally, in some instances, imaging system 100 modifies the list by removing list elements that are unlikely to approximate normal or expected cell sizes. For example, imaging system 100 can utilize a Gaussian or other distribution about the list median to remove certain list elements, such as list elements that are outside a predetermined distance or difference from the list median. In other examples, imaging system 100 uses the maximum list value as a starting point for determining which list elements to remove.

[0090] Act 204d, related to act 204, includes using a machine learning module to evaluate focus quality estimates for at least some of the sorted connected components. In some implementations, imaging system 100 utilizes processor 112 and / or machine learning module 118 to evaluate focus quality estimates for at least some of the sorted connected components.

[0091] By way of background, artificial intelligence, at its core, attempts to model human thought or intelligence to solve complex or difficult problems. Machine learning is a form of artificial intelligence that utilizes computer models and some form of feedback to "learn" from data without a complex set of prescribed rules. Most machine learning algorithms can be classified based on the type of feedback used in the learning process. For example, in unsupervised learning models, unlabeled data is input into the machine learning algorithm, from which a general structure is extracted. Unsupervised learning algorithms can be powerful tools for clustering data sets. On the other hand, supervised learning models utilize labeled input data to train the machine learning algorithm and "learn" the model to reliably predict desired outcomes. Thus, supervised machine learning models can be powerful tools for classifying data sets or performing regression analysis.

[0092] Neural networks encompass a broad category of machine learning algorithms that attempt to simulate complex thinking and decision-making processes with the aid of computers by selecting suitable network topologies and processing functions that mimic the biological functions of the brain. Neural networks are dynamic enough to be used in a wide range of supervised and unsupervised learning models. Today, there are hundreds of different neural network models and numerous connection characteristics and functions. Nevertheless, the main way neural networks function remains the same.

[0093] For example, FIG. 5A shows a simplified schematic diagram of a canonical neural network. FIG. 5B further provides an isolated portion of the neural network shown in FIG. 5A. As shown, so-called input neurons are located on the input side of the neural network and are connected to hidden neurons. Each neuron has one or several weighted inputs, either in the form of external signals or as outputs from other neurons. Positive and negative weights are possible. The sum of the weighted inputs is transferred via a transfer function to one or more output values, which control or serve as output values ​​for other neurons. The hidden neuron shown in FIG. 5A (and as a single node in FIG. 5B) is connected to an output neuron (or output node in the case of FIG. 5B). Of course, the hidden neuron region can have a much more complex structure and can consist of several interlinked levels.

[0094] The set of all neurons of a function is called a layer (e.g., the input layer). Some of the more influential parameters of a neural network, apart from its topology, include neuron base potentials and the strength of the connections between neurons. To set the parameters, a representative training set is iteratively evaluated by the network. After each evaluation cycle, weights and base potentials are changed and set anew. The iterations are performed until the average failure rate falls below a predetermined minimum value or a predefined problem-related termination criterion is reached.

[0095] As described above, imaging system 100 can utilize machine learning to evaluate focus quality estimates for at least some of the sorted connected components (e.g., sorted according to act 204D of FIG. 4 ). Imaging system 100 can use the identified connected components to generate inputs for a machine learning model. For example, in some implementations, the system localizes a pixel window for at least some of the connected components of the binary images (e.g., the third set of binary images). The pixel window can have various sizes, such as 32×32 pixels or another size. Those skilled in the art will understand, in light of this disclosure, that the size of the pixel window can depend on the imaging application for which imaging system 100 is being employed (e.g., the type of cell being imaged).

[0096] The imaging system 100 may also determine a z-stack for each bound component based on the pixel window of each respective bound component. For example, for a pixel window for a particular bound component identified in a particular binary image associated with a particular z-height (or focal position), the imaging system 100 may identify corresponding pixel windows (e.g., pixel windows having the same pixel coordinates) in other binary images associated with z-heights that are offset from the particular z-height of the particular binary image for the particular bound component (e.g., offset by +1 z-step, −1 z-step, +2 z-step, −2 z-step, etc.), and this collection of pixel windows may form a z-stack for the particular bound component.

[0097] In view of the present disclosure, it will be appreciated that the size of a particular z-stack may vary in different implementations. For example, a z-stack may include several pixel windows (from binary images associated with adjacent z-heights) in the range of about 3 to 11 or more, and the number of pixel windows (as well as the size of the pixel windows) for a z-stack may also depend on the imaging application and / or cell type being imaged.

[0098] Imaging system 100 can form z-stacks for any number of bound components. For example, imaging system 100 may form z-stacks for a predetermined number (e.g., 32 or another number) of bound components included in a sorted list of bound components (e.g., sorted as described above with reference to act 204d of FIG. 4). Imaging system 100 can provide z-stacks for various identified bound components as input to a machine learning module, which can evaluate a focus quality estimate based on the z-stacks.

[0099] Those skilled in the art will appreciate, in light of the present disclosure, that various machine learning models can be employed to evaluate focus quality estimates based on one or more sorted connected components (e.g., based on a z-stack as described above). FIG. 6 illustrates one exemplary neural network that imaging system 100 can utilize to facilitate focus quality assessment in accordance with the present disclosure. In particular, FIG. 6 illustrates an exemplary block diagram of an artificial neural network 600 for facilitating AI-assisted autofocus. Artificial neural network 600 can be trained in a supervised or semi-supervised manner using training data that includes z-stacks for connected components as input training data and clearly identified target focus positions as ground truth output. In some instances, the training data can include a pixel window of the z-stack or a z-stack with a smaller z-step between images to improve the robustness of artificial neural network 600 for assessing focus quality.

[0100] 6 illustrates input data 602, which, as described above, can include any number of z-stacks (e.g., 1, 2, ... 32, or more). The artificial neural network 600 may receive the input data 602 and process it using one or more feature identification layers 604. As shown in FIG. 6, the feature identification layer 604 may comprise various components, such as, for example, a convolutional layer 606, a linear layer 608, and a max-pooling layer 610 (or, in some cases, an average pooling layer). As a non-limiting example, in some implementations, the convolutional layer 606 is a 3x3 convolutional layer, the linear layer 608 is a ReLu nonlinear function, and the max-pooling layer is a 2x2 max-pooling layer.

[0101] 6 shows an implementation in which artificial neural network 600 includes three substantially identical feature identification layers, specifically feature identification layer 604, feature identification layer 612, and feature identification layer 614. However, in other implementations, artificial neural network 600 may include any number of feature identification layers having the same or at least partially different components.

[0102] 6 also illustrates an implementation in which the artificial neural network 600 includes a long short-term memory (LSTM) processing layer 616 following the feature identification layer. The LSTM may enable processing of z-stacks provided as input in a bidirectional manner. Implementing the LSTM processing layer 616 into the artificial neural network 600 may, in some cases, improve the accuracy of the artificial neural network 600 for assessing focus quality; however, implementing the LSTM processing layer 616 may be process-intensive and / or time-consuming. Therefore, in some implementations, the artificial neural network 600 omits the LSTM processing layer 616 to conserve computation time and / or resources (e.g., to enable a total computation time of less than 20 seconds or about 10 seconds or less).

[0103] 6 shows that the artificial neural network 600 includes a final linear layer 618 following the LSTM process layer 616 (or following the feature identification layer 614 in implementations that omit the LSTM process layer 616). The final linear layer 618 may be configured to provide an output 620, which may comprise a focus quality estimate for each z-height represented in a particular z-stack provided as input to the artificial neural network 600. For each particular z-stack provided as input to the artificial neural network 600, the imaging system 100 can identify a particular z-height (or focus position) from the output 620 that represents a target focus position for the particular z-stack.

[0104] By providing multiple z-stacks to the artificial neural network 600, the imaging system 100 can obtain a corresponding focus quality estimate as output 620 for each z-stack, as well as a respective target focus position for each z-stack. Act 204e, related to act 204, includes calculating a target focus position based on the focus quality estimates of the evaluated binding components. As described above, the focus quality estimates (from output 620 from the artificial neural network 600) for the multiple z-stacks can provide a respective target focus position for each z-stack. The target focus position can be generated or defined based on the focus quality estimates (or the respective target focus positions for each z-stack) in various ways. For example, in some cases, the imaging system 100 defines a median, mode, or mean of the various respective target focus positions for each z-stack to select an overall target focus position for the cell counting slide imaged by the imaging system 100 according to act 204a described above. In this manner, the imaging system 100 can utilize artificial intelligence to facilitate autofocus in an improved, computationally inexpensive, and / or rapid manner. Systems and methods for facilitating AI-assisted cell viability counting

[0105] As described above, facilitating artificial intelligence-driven cell viability counting may include performing an automated cell viability count using a target focal position (act 206, as described above with reference to FIG. 2). Performing an automated cell viability count using a target focal position can be associated with various acts. FIG. 7 shows an example flow diagram depicting various acts associated with act 206 from flow diagram 200 for performing an automated cell viability count using a target focal position.

[0106] While the present disclosure focuses, at least in some respects, on performing cell viability counting using a target focus position, it should be understood in light of the present disclosure that the principles discussed herein may be implemented independently of one another. For example, the principles discussed herein related to cell counting may be implemented without necessarily also implementing the autofocus and / or cell viability analysis techniques discussed herein. Similarly, the autofocus techniques discussed herein may be implemented without necessarily also implementing the cell counting and / or viability analysis techniques discussed herein. For example, a flow cytometer may, in some cases, implement at least some of the cell counting techniques discussed herein without first performing the autofocus techniques discussed herein. Embodiments of the present disclosure may also utilize the autofocus techniques discussed herein periodically (e.g., once a day or once a week in a laboratory) while utilizing cell counting aspects on each sample processed; such embodiments have the advantage of saving time and / or are applicable to applications where autofocusing each sample may not be possible.

[0107] Act 206a associated with act 206 includes acquiring an image. In some implementations, act 206 is performed by imaging system 100 using processor 112, hardware storage device 114, machine learning module 118, and / or controller 116 of computer system 110, and image sensor 122, illumination source 124, optical train 126, slide port / stage assembly 106, and / or positioning mechanism of microscope system 120.

[0108] In some implementations, the image associated with act 206a includes a monochrome image. By way of example, imaging system 100 can use various components of computer system 110 and microscope system 120 to obtain a target focal position according to act 204 (and related acts 204a-204e), as described above with reference to FIG. 2 (and FIG. 4). Using processor 112 and / or controller 116, imaging system 100 can cause positioning mechanism 128 to position optical train 126 relative to slide port / stage assembly 106 according to the target focal position. Using image sensor 122 and illumination source 124 in any combination described above, imaging system 100 can capture an image of the cell counting slide using the identified target focal position. In some implementations, the captured image of the cell counting slide at the target focal position is a 2592 × 1944 monochrome 8-bit image with a pixel size of 0.871 μm. Aside from the foregoing examples, images used for cell counting as described herein may have any suitable image size or aspect ratio and may be acquired utilizing any suitable imaging device with any suitable pixel size and / or other image sensor characteristics. In some implementations, the images have an image size of 96x96, 250x250, or another image size.

[0109] Any instructions for performing the acts described herein and / or data used or generated / stored in connection with performing the acts described herein (e.g., monochrome images, cell viability counting data) may be stored in hardware storage device 114 in a volatile or non-volatile manner.

[0110] As mentioned above, the present example focuses on monochrome images associated with a target focus position, but in other examples, monochrome may be associated with any focus position according to act 206a.

[0111] Act 206b, related to act 206, includes generating a preprocessed image by performing one or more preprocessing operations. The preprocessed image, in some cases, may include a preprocessed monochrome image. Various preprocessing operations are within the scope of this disclosure. For example, the one or more preprocessing operations of act 206b may include one or more of downsampling (or downaveraging), background removal, and / or intensity normalization operations. In some cases, a user may select which preprocessing functions, if any, to perform pursuant to act 206b.

[0112] In some implementations, imaging system 100 utilizes processor 112 to perform one or more downsampling operations in accordance with act 206b. For example, in some cases, imaging system 100 performs downsampling via pixel decimation or by applying an averaging filter (e.g., generating each output pixel based on the average intensity of each neighboring pixel). Imaging system 100 may downsample the captured monochrome image by a predetermined downsampling factor, such as a factor of about 1.5 to about 4, or about 2, in each image dimension (e.g., reducing a 2592 x 1944 captured image to a 1296 x 972 downsampled image). While any downsampling factor is within the scope of this disclosure, in some cases, a high downsampling factor (e.g., a factor of 4 or greater) may cause some features to be removed from and / or distorted in the downsampled preprocessed image.

[0113] As described above, one or more preprocessing operations of act 206b may include a background removal operation. In some instances, the acquired image may have a high or uneven background. Removing the background may facilitate easier distinction of objects of interest within the acquired image. In some instances, background removal is tailored to low- or no-fluorescence samples. Thus, as described above, the user may select whether to perform background removal as part of act 206b (e.g., by checking or unchecking a box associated with performing background removal).

[0114] 8 is a conceptual diagram of background removal, where a raw image 802 contains both a background 804 and an object 806. Background removal can be conceptualized as generating an estimated background 808 and subtracting the estimated background 808 from the raw image 802 to generate a background-removed image 810.

[0115] Background removal can be facilitated in various ways. For example, background removal may be performed according to a low-pass filtering method, such as by estimating the background by calculating a local mean within a radius r at each pixel and subtracting the estimated background from the image. The radius r may be defined as the input maximum object size (e.g., constrained between 1 and 255). The subtraction may result in a fixed mean value. Low-pass filtering background removal may be performed according to a selected object detection mode, such as light / dark, dark / light, and / or other. The fixed mean value may be 0 for light / dark mode, a maximum value for dark / light mode, and / or an intermediate value for other modes.

[0116] In some cases, low-pass filtering methods are adapted for use in images with channels that serve to identify objects. Low-pass filtering methods can be performed efficiently in images with high-contrast edges (e.g., brightly stained nuclei). Low-pass filtering methods can also be useful for removing background caused by "dirty" samples (e.g., containing out-of-plane fluorescence, such as "floating cells"). In some cases, the parameter value for low-pass filter background removal corresponds to the area sampled to determine the amount of background to remove. Lower values ​​may be considered more aggressive and may be constrained to within the diameter of the largest object. Higher values ​​may be considered more conservative and may be within a multiple of the diameter of the largest object.

[0117] In some cases, background removal may be performed using a surface fitting method, which may include dividing the input image into a grid, calculating the respective minimum values ​​in each image region, and fitting a surface through the respective minimum values. In some cases, using the minimum values ​​(e.g., rather than the average values) preserves the true intensity values ​​(but may take longer to calculate).

[0118] In some cases, surface fitting methods are adapted to images requiring intensity-dependent measurements, such as fluorophore-specific antibody labeling. Surface fitting methods may additionally or alternatively be adapted to preserve the dim edges of whole-cell staining. Similar to low-pass filtering methods, parameter values ​​for surface fitting background removal may correspond to the area sampled to determine the amount of background to remove, with lower values ​​being considered aggressive and higher values ​​being considered conservative.

[0119] In some instances, a user interface is provided that allows a user to select whether and / or what type of background removal to perform pursuant to act 206b.

[0120] As mentioned above, the one or more preprocessing operations of act 206b may include a normalization operation. A normalization operation may be performed to normalize the intensities present within image pixels (or voxels in the case of 3D images) to reduce the impact of intensity variations from object to object on subsequent processes. The normalization operation may include global normalization (e.g., implementing the same image statistics across the entire image) or a kernel-based normalization operation.

[0121] Any combination of the aforementioned preprocessing operations (e.g., downsampling / downaveraging, background removal, normalization) may be performed in accordance with act 206b. If multiple preprocessing operations are performed, the preprocessing operations may be performed in any suitable order. As described above, the output of the preprocessing operations is one or more preprocessed (monochrome) images (e.g., a downsampled / downaveraged image, a background removed image, a normalized image, a downsampled / downaveraged and background removed image, a downsampled / downaveraged and normalized image, a background removed and normalized image, a downsampled background removed and normalized image, etc.). The one or more preprocessed images may be utilized for subsequent operations (e.g., act 206c, act 206d, etc.).

[0122] Preprocessing operations may be performed on the monochrome image described above with reference to act 206a. In some cases, preprocessing operations are performed on one or more tiles (e.g., via batch processing) as described below with reference to act 206c. In light of the present disclosure, it will be understood that at least some preprocessing steps may be implemented in the functionality of a convolutional neural network (CNN) as described below with reference to act 206d, or may utilize a functional architecture that is at least partially independent from a CNN.

[0123] In some cases, no pre-processing operations are performed, and the raw data (eg, the monochrome image of act 206a) is used directly in subsequent operations (eg, act 206c, act 206d, etc.).

[0124] Act 206c, associated with act 206, includes defining one or more sets of tiles based on the preprocessed image (as described above, the preprocessed image may include a preprocessed monochrome image). Act 206c, in some cases, is performed by imaging system 100 utilizing processor 112. Defining one or more sets of tiles may include various acts, such as reflecting the preprocessed monochrome image and dilating its edges by multiple pixels (or voxels). Performing a reflect operation on the downsampled image may generate a reflected image having edges dilated by 6 pixels, 8 pixels, 16 pixels, or another pixel value. The number of pixels / voxels for dilating the downsampled image to obtain the reflected image may be selected based on various factors, such as desired processing time, image size, etc.

[0125] Imaging system 100 may then decompose the reflected image into a set of one or more tiles with a predetermined pixel overlap (e.g., 4, 6, 8, 10, 12, 14, or 16 pixel overlap, or another pixel value overlap as desired. In one example, imaging system 100 decomposes the reflected image into 130 tiles having a pixel size of 128x128 (or another number of tiles of another or the same pixel size).

[0126] In some cases, imaging system 100 may be configured to store one or more sets of tiles in a tensor array to generate input for a machine learning module (e.g., the CNN of act 206d) to facilitate automated cell counting. For example, the one or more sets of tiles described above with reference to act 206c may be stored in a 130x1x128x128 tensor array (or a tensor array having other dimensions, depending on the image tile size and number). As described below, the tensor array may serve as input for a machine learning module to perform automated cell counting and / or viability determination.

[0127] Act 206d, related to act 206, includes processing one or more tiles using a convolutional neural network (CNN). If the one or more tiles are stored in a tensor array, act 206d may include processing the tensor array using a CNN. In some implementations, imaging system 100 utilizes processor 112 and / or machine learning module 118 to process the tensor array described above with reference to act 206d. A brief context for artificial intelligence and machine learning is provided above with reference to act 204d of FIG. 4 and FIGS. 5A and 5B.

[0128] Those skilled in the art will appreciate, in light of the present disclosure, that various machine learning models can be employed to process the tensor array in accordance with act 206d. FIG. 9 illustrates one exemplary neural network that imaging system 100 can utilize to facilitate processing of one or more tiles as part of performing automated cell counting and / or viability determination. In particular, FIG. 9 illustrates a U-Net convolutional neural network 900. FIG. 9 illustrates a U-Net convolutional neural network 900 receiving as input one or more tiles or tiles 906 of a tensor array (e.g., from act 206d), and the U-Net convolutional neural network 900 of FIG. 9 illustrates neural network processing configured to be performed on each tile of the one or more tiles or tensor array. The U-Net convolutional neural network 900 can be configured to receive image inputs, such as rectangular images (e.g., 16-bit images) of various sizes. For example, the image size can fall within a range of approximately 150 x 150 to approximately 250 x 250 in one or both dimensions. It will be understood that the range may vary for different imaging modalities. For example, for flow cytometry applications, the input image may be in the range of about 96x96 to about 248x248.

[0129] Additionally, U-Net convolutional neural network 900 may be configured to receive various numbers and / or types of image inputs. For example, U-Net convolutional neural network 900 may be configured to receive as simultaneous input a batch of images (e.g., a z-stack of images or a subset of a z-stack) and / or a batch of multiple images of different imaging modalities (e.g., one or more conventional images / volumes and one or more phase-contrast images / volumes associated with the same z-position).

[0130] The U-Net convolutional neural network 900 of FIG. 9 includes a down layer 902 and an up layer 904. FIG. 9 also illustrates that the down layer 902 and the up layer 904 may include various components. As a non-limiting example, FIG. 9 illustrates an implementation in which the down layer 902 includes a feature identification layer (e.g., two feature identification layers) including a convolution layer (e.g., a 2D convolution layer), a batch normalization layer (e.g., a 2D batch normalization layer), and a ReLu nonlinear function layer. FIG. 9 also illustrates the down layer 902 as including a max pooling layer (e.g., a 2D max pooling layer). The down layer 902 can facilitate incremental downsampling of the input image (e.g., tile 906) (e.g., reducing the image size by a factor of four at each step). In some instances, as shown in FIG. 9, the imaging system 100 can apply 2D batch normalization before applying the components of the down layer 902.

[0131] 9 also illustrates up-layers 904 as including two sets of an upsampling layer (e.g., a 2D upsampling layer), a convolution layer (e.g., a 2D convolution layer), a batch normalization layer (e.g., a 2D batch normalization layer), and a ReLu nonlinear function layer. Up-layers 904 can facilitate incremental upsampling (e.g., increasing the image size by a factor of four at each step) of downsampled tiles (e.g., tiles downsampled according to down-layers 902), and imaging system 100 can utilize combination layers 908 to perform the upsampling (e.g., based on corresponding images used or generated during down-layer 902 processing). In some instances, as shown in FIG. 9, imaging system 100 may apply a sigmoid function after applying the components of up-layers 904.

[0132] In some cases, the U-Net convolutional neural network 900 is trained with a set of training data including images from multiple focal planes for the identified target focal position (or other focal positions). For example, in some implementations, the convolutional neural network 900 is trained for z-heights ranging from -2 to +2 z-steps for the identified target focal position to increase the robustness of the convolutional neural network 900 and reduce the sensitivity of the convolutional neural network 900 to autofocus output accuracy (e.g., according to act 204 described above). The U-Net convolutional neural network 900 receives training data including monochrome images (and / or image tiles) as training inputs and / or manually located and / or identified cell counts, viability determinations, cell segmentations, cell areas (e.g., μm for individual cells and / or cell combinations) as ground truth outputs. 2The training data can be supervised or semi-supervised using information such as cell size (units), cell pseudo-diameter (e.g., cell diameter if the singlet were perfectly round, in μm), whole-cell binary mask, seed mask, and / or other information (e.g., tags indicating whether the cell image was processable and / or whether the cell borders the region of interest). The ground truth can be obtained through human annotation / labeling / tagging / segmentation and / or generated at least in part using images captured using an appropriate imaging modality (e.g., brightfield, fluorescence). The training data can include images capturing various types of cells, such as IMMUNO-TROL, macrophages, Jurkat, CAR-T, PBMC, and / or others. The images can be associated with one or more different Z-positions. In some cases, the images in the training data set are preprocessed according to preprocessing operations (e.g., downsampling / downaveraging, background removal, normalization, etc.) that may be performed during end use. The training input can additionally or alternatively include images with debris, allowing the model to distinguish between live / dead cells (including clumps of cells) and debris. In some cases, the training data includes control objects such as glass beads (e.g., 1 μm, 2.5 μm, 3.6 μm, 5.5 μm, 9.9 μm, 14.6 μm, 30.03 μm) that are similar in size to live or dead cells to enable the U-Net convolutional neural network 900 to robustly distinguish between cells and objects that are not cells but share similar physical properties (e.g., size and shape) as cells.

[0133] As previously mentioned, U-Net convolutional neural network 900 may be configured to receive various numbers and / or types of image inputs. Correspondingly, U-Net convolutional neural network 900 may be trained using a batch of images (e.g., a z-stack of images or a subset of a z-stack) as simultaneous inputs and / or a batch of multiple images of different imaging modalities (e.g., one or more conventional images / volumes and one or more phase-contrast images / volumes associated with the same z-location) as simultaneous inputs.

[0134] In light of the present disclosure, it will be appreciated that the U-Net convolutional neural network 900 (or any other artificial intelligence module described herein) may be further trained and / or refined after its initial training on a set of training data as described above. For example, the U-Net convolutional neural network 900 may be further trained on training data obtained for a particular field of cellular analysis.

[0135] As shown in FIG. 9 , the output of the U-Net convolutional neural network 900 may include or be used to generate various pseudo-probability maps 910. For example, FIG. 9 illustrates the output pseudo-probability maps 910 of the U-Net convolutional neural network 900 as including: (i) a live cell location probability map (e.g., a live cell seed likelihood image) indicating the center locations of live cells, (ii) a live cell mask probability map (a live cell global likelihood image), (iii) a dead cell location probability map (a dead cell seed likelihood image) indicating the center locations of dead cells, and (iv) a dead cell mask probability map (a dead cell global likelihood image). For example, the pseudo-probability maps 910 in FIG. 9 represent the predicted / likely locations and shapes / sizes of live and dead cells represented in tiles 906 of the tensor array obtained according to act 206d.

[0136] As described above, U-Net convolutional neural network 900 may operate on each tile (e.g., tile 906) of the tensor array obtained according to act 206d. Thus, in the example of a 130×1×128×128 tensor array, U-Net convolutional neural network 900 may generate four pseudo-probability maps for each single tile represented in the tensor array, and these pseudo-probability maps (output from U-Net convolutional neural network 900) may become stored in a 130×4×128×128 tensor array.

[0137] While the example presented in FIG. 9 and described with reference to act 206d includes outputting (or generating based on) a particular set of pseudo-probability maps via U-Net convolutional neural network 900, other ranges / types of U-Net CNN outputs are within the scope of this disclosure. For example, in some cases, the output of the U-Net CNN does not distinguish between live and dead cells, but generates a seed likelihood image indicating cell location and / or a whole-cell likelihood image indicating cell shape / size that includes information on both live and dead cells (if both are present). For example, in some implementations, the cell counting function is performed independently of cell shape / size / feature detection, such that the U-Net CNN outputs at least a seed likelihood image, and connected component analysis can be performed to determine cell number (e.g., without determining cell viability and / or cell shape / size / feature) based on the seed likelihood image. Such functionality may be desirable in a flow cytometry system, as a non-limiting example.

[0138] A further conceptual representation of the inputs and outputs associated with a U-Net CNN is provided in Figure 10. Figure 10 shows an exemplary input image 1002, which may include a preprocessed monochrome image or a preprocessed monochrome image tile. The input image 1002 is used as an input to a U-Net CNN 1004, which corresponds in principle to the U-Net convolutional neural network 900 described above. Figure 10 also shows an exemplary output 1006, which includes a seed likelihood image 1008 and a whole-cell likelihood image 1010 overlaid on top of each other.

[0139] While this example focuses, at least in some respects, on utilizing U-Net CNN to obtain pseudo-probability maps, in some embodiments, other modules such as machine learning driven texture detection and / or kernel detection methods may be utilized.

[0140] Act 206e, related to act 206, includes constructing a plurality of pseudo-probability maps based on the output of the convolutional neural network, where the plurality of pseudo-probability maps includes at least a seed likelihood image and / or a whole-cell likelihood image. As described above, in some instances, the output from U-Net convolutional neural network 900 may comprise a 130×4×128×128 tensor array (or a tensor array of other size appropriate for the particular input used). In some implementations, imaging system 100 may construct the plurality of pseudo-probability maps of act 206e by converting the output of the convolutional neural network (e.g., a 130×4×128×128 tensor array) into 8-bit format tensors and image stitching the 8-bit format tensors. For example, converting a 130x4x128x128 tensor array to 8-bit format may include multiplying the tensor array by a suitable multiplier such as 255, and stitching the 8-bit tensor format image may include constructing four full-size pseudo-probability maps from the tiles in the 8-bit tensor format. In some implementations, the four resulting pseudo-probability maps may comprise a representation similar to pseudo-probability map 910 of Figure 9 (e.g., live cell locations, live cell mask, dead cell locations, and dead cell mask) or the output of Figure 10 (e.g., seed likelihood image and whole-cell likelihood image).

[0141] Act 206f associated with act 206 includes generating one or more masks based on the one or more seed likelihood images. For example, in some instances, imaging system 100 may generate a binarized location map by thresholding four full-size pseudo-probability maps (e.g., corresponding to pseudo-probability map 910 of FIG. 9 ). The one or more masks may define or indicate pixel locations of one or more objects (e.g., cells) represented in the one or more seed likelihood images. Thus, in some instances, the one or more masks may indicate cell counts.

[0142] Various techniques for generating the one or more masks are within the scope of this disclosure, such as applying a likelihood threshold of 75% or greater. In some cases, utilizing a threshold of 192 or greater on pixel intensities represents a pseudo-probability of 0.75×255 or greater. Various thresholds are within the scope of this disclosure, such as, by way of non-limiting example, a threshold in the range of about 50% to about 90%. Connected component labeling may further be applied to detect connected regions in the output image. One or more masks may be generated based on the connected component labels. Additionally or alternatively, one or more masks may be generated using a deep learning algorithm.

[0143] In some implementations, the imaging system 100 performs additional image processing operations on the binarized location map, such as dilation processing, to attempt to capture potential cells that have been ignored through other processing steps.

[0144] Act 206g, related to act 206, includes generating one or more segmented images based on at least one or more whole-cell likelihood images and one or more masks. The one or more segmented images can indicate / provide cell counts and / or cell viability. In some implementations, imaging system 100 generates the one or more segmented images via a watershed transform applied to a distance map calculated from pixel / voxel locations of objects (e.g., cells) represented in one or more seed likelihood images. The one or more segmented images may be separated by one or more whole-cell likelihood images (e.g., separated by the objects' respective masks) such that each pixel is assigned to an object or background. Binary or intensity-based watershed methods are within the scope of this disclosure. Figure 11 shows an example representation utilizing a watershed transform 1106 to generate a segmented image 1108 based on a seed likelihood image 1102 and a whole-cell likelihood image 1104. Different objects represented by seed likelihood image 1102 are numerically labeled (e.g., "1," "2," "3"). Segmented image 1108 shows distinct objects (correspondingly labeled “1,” “2,” and “3” in segmented image 1108) extended from different objects (objects 1, 2, and 3) in seed likelihood image 1102.

[0145] In addition to or as an alternative to the watershed method, other techniques can be used to facilitate the generation of one or more segmented images. For example, in some embodiments, one or more segmented images may be generated that treat a grouping of cells as a single object, such as by assigning all pixels in the seed likelihood image with a likelihood greater than 75% to a mask structure (other thresholds may be utilized, as discussed above). Such an approach may be considered a "one-object" approach, allowing downstream calculations to be performed on a single object (rather than on individual objects). Such an approach may be beneficial when computational resources are limited. Cell morphology measurements of the individual objects may be used to verify the number of seeds identified by the seed likelihood image described above. Figure 12 shows an exemplary representation utilizing a one-object operation 1206 to generate a segmented image 1208 based on a seed likelihood image 1202 and a whole-cell likelihood image 1204. The different objects represented by the seed likelihood image 1202 are numerically labeled (e.g., "1," "2," "3"). Segmented image 1208 shows a unit object (labeled "1" in segmented image 1108) constructed from different objects (objects 1, 2, and 3) in seed likelihood image 1202.

[0146] As another example, one or more segmented images may be generated using a deep learning module. The deep learning module may be trained using training data including a seed / whole-cell likelihood image input and a segmented image ground truth output. In some cases, using a deep learning module may enable accurate segmentation of overlapping cells. FIG. 13 shows an exemplary representation of using a deep learning module 1306 to generate a segmented image 1308 based on a seed likelihood image 1302 and a whole-cell likelihood image 1304. The different objects represented by the seed likelihood image 1302 are numerically labeled (e.g., “1,” “2,” “3”). The segmented image 1308 shows distinct objects (correspondingly labeled “1,” “2,” and “3” in the segmented image 1308) that extend from the different objects (objects 1, 2, and 3) in the seed likelihood image 1302.

[0147] The one or more segmented images may, in some cases, indicate or provide the basis for cell counting and / or cell viability and may be displayed on a user interface to inform one or more users of the cell count and / or cell viability represented on the cell counting slide. In some cases, in addition to cell counting, one or more feature calculation operations may be performed using the one or more segmented images to determine one or more characteristics of the detected cells. The features may be extracted for each cell. For example, the imaging system 100 may perform ellipse fitting on the cells represented in the one or more segmented images. The ellipse fitting on the cells of the one or more segmented images may utilize the techniques described above for fitting ellipses to connected components (e.g., see act 204b of FIG. 4). Ellipses fitted to the cells of the one or more segmented images may facilitate obtaining advantageous data related to the cells imaged on the cell counting slide, such as providing a measure of object size (e.g., in μm), providing a histogram of object size, providing pixel intensity, providing a basis for calculating object circularity, and / or other. Further examples of features that may be obtained for detected cells include object number, object center (e.g., object center per dimension, such as x center, y center, z center), object width (e.g., width of the smallest bounding box containing the object), object height (e.g., height of the smallest bounding box containing the object), pixel size (e.g., pixel size per dimension, such as x pixel size, y pixel size, z pixel size), area, perimeter, perimeter to area, fiber length (e.g., the length of the object measured along its spine), fiber width (e.g., the width of the object estimated from the area and length), centroid (e.g., x centroid, y centroid), orientation (e.g., the orientation of a bounding box aligned to the object), coherency (e.g., a measure of the arrangement of substructures within the object), major radius, minor radius, radius of gyration (e.g., along the z axis), box length (e.g., the length of the bounding box in which the object is located), box width (e.g., the width of the bounding box in which the object is located), length to width ratio, box fill ratio,The number of pixels that make up each object, object intensity (e.g., maximum intensity, minimum intensity, total intensity, mean intensity, standard deviation of intensity, skewness of intensity, kurtosis of intensity, entropy of intensity, etc.), radial intensity moments (e.g., mean radial intensity, standard deviation of radial intensity, skewness of radial intensity, kurtosis of radial intensity, radial distance, etc.), object co-occurrence (e.g., maximum probability of intensity distribution of all pixels in the mask, contrast, entropy, angular second moment of two-dimensional co-occurrence, etc.), object size (e.g., diameter of a circle with an area equal to the area of ​​the object), equivalent spherical metric (e.g., diameter, surface area, or volume of an equivalent circle or sphere), moments of an equivalent ellipse, etc. The area-equivalent features may include, for example, area (e.g., the ratio of the length to the width of an equivalent circle or sphere, the volume of an ellipse generated by rotating an area-equivalent ellipse around its major or minor axis, etc.), object distance (e.g., the distance from an object to its nearest neighbor, the average distance from an object to all other objects, the standard deviation of the distances from an object to all other objects, etc.), object gradient ratio (e.g., the intensity gradient of an inner or outer region within an object mask, the ratio of the intensity gradient between the inner and outer regions within the object mask), surface area density (e.g., the total difference in intensity of all pixels within an object normalized by its area), and / or others. Any of the foregoing features may be weighted in an appropriate manner (e.g., according to pixel intensity). Such data may be obtained for live and / or dead cells represented in a cell counting slide and, in some cases, stored in the consumer report and prepared / provided for export (e.g., using the communication module 108).

[0148] Additionally, as described above with reference to FIG. 2, fitting an ellipse to one or more segmented images may enable the system to display a representation of the cell viability count (e.g., according to act 208 of flow diagram 200 of FIG. 2).

[0149] For example, referring now to Figure 14, images are shown that represent exemplary results after processing by the disclosed AI-assisted autofocus and automated cell viability counting system. The live / dead cell locations and masks obtained as output from processing individual tiles through an artificial neural network (e.g., the U-Net convolutional neural network shown in Figure 9) are used to generate a pseudo-probability map, and after a series of image processing and ellipse fitting steps as discussed above, the resulting ellipse map, containing solid ellipse markers for live cells 1402 and dashed ellipse markers for dead cells 1404, is overlaid on the corresponding bright-field image 1400 to clearly identify live and dead cells. In some embodiments, live and dead cells are visually distinguished in one or more images viewable on a display associated with the imaging system (e.g., display 104 of imaging system 100 of Figure 1). For example, live cells may be outlined, shaded, and / or otherwise highlighted in a particular color (e.g., green), while dead cells may be outlined, shaded, and / or otherwise highlighted in a different color (e.g., red).

[0150] Because the disclosed systems and methods can utilize information within a z-stack that is not readily apparent from a single z-stack slice (e.g., polarity reversal phenomena), the disclosed systems can advantageously autofocus on a mixture of live and dead cells using minimal processing cycles and without the need for additional processing hardware. This improved method of autofocus allows for rapid and reliable identification of the target focus position in any given sample, and is a useful first step toward enabling the disclosed methods of automated cell viability counting. As with the autofocus methods described above, the disclosed methods of automated cell counting and / or cell viability counting disclosed herein are significant improvements over competing prior art systems and methods.

[0151] For example, as shown in Figures 15A-15C, the same base image was evaluated and annotated with the number and location of live and dead cells within each image. The image shown in Figure 15A was evaluated by a biologist with experience annotating the number and location of live and dead cells. The image in Figure 15A was determined to be accurately and precisely annotated and served as a positive control for a comparison of accuracy and precision between prior art methods of automated cell identification and viability and the exemplary AI-assisted autofocus and automated cell viability counting method as disclosed herein.

[0152] As shown in Figure 15B, Figure 15B shows an image evaluated by a prior art automated cell identification and viability method and annotated with the number and location of live and dead cells, and the prior art method is unable to accurately segment and identify the number of cells within cell clusters, is unable to consistently identify monodispersed cells as single cells, is unable to accurately distinguish between live and dead cells, and is unable to ignore debris, instead identifying the portion of debris within the observation area as a collection of clustered and monodispersed live and dead cells.

[0153] In contrast to prior art methods (e.g., as illustrated in FIG. 15B), the systems and methods disclosed herein for AI-assisted autofocus and automated cell viability counting adequately segment and count live / dead cells (compared to biologist-annotated controls) and also avoid debris, as shown in FIG. 15C. As illustrated in FIGS. 15A-15C (and replicated across 12 different cell types—data not shown), the disclosed solutions demonstrated significant improvements in cell segmentation and debris discrimination, as well as viability counting accuracy, relative to conventional image processing methods. Such improvements have been repeatedly shown to occur across monodispersed and clumped cell specimens.

[0154] Example Data Structure Details The following description provides example data structures that may be associated with the various components / elements disclosed herein. An image may be defined by an image class, which may include an array of pixel values ​​and accompanying metadata. The pixel values ​​may take various forms, such as Byte, SByte, UInt16, Int16, Int32, UInt32, Int64, UInt64, Single, Double, etc. The metadata may include, by non-limiting example, pixel data type, bits per pixel, bytes per pixel, pixel offset, stride (e.g., allowing for cropping of areas of the image without pixel copying), XYZ position within the well (e.g., in micrometers), XYZ pixel size, image dimensions (width, height), acquisition time, intensity control settings (e.g., exposure time, gain, binning), and / or others. The image class may support grayscale and / or color (RGB). If color is used, each pixel may have a corresponding color value (e.g., an RGB color value or a value according to another color system). Image classes may be serializable and / or convertible to different types (e.g., between color and RGB). Image readers can be used to read / write image classes from standard file formats via any stream (file, memory, pipe).

[0155] A mask may define a list of pixel indices and / or a bounding box to which the pixels belong. The mask class can be used as input (e.g., to define truth data) and output (e.g., to define the location of an object). The bounding box may be the smallest bounding box that contains the object, thereby allowing the mask definition to be maintained independent of the field from which it is acquired. Thus, in some implementations, a mask can be extracted from one image set (e.g., one time point, one pass) and applied to a second image set (e.g., another time point, another pass), even if the second image set is not associated with the exact same location as the first image set (e.g., the second image set may be associated with a different magnification and / or XYZ location). The mask class can provide methods for converting pixel lists between bounding box definitions. In some cases, images can be alternately adjusted to match a mask (e.g., instead of adjusting the mask to match the image). For example, the image class can provide a way to "crop" a bounding box to match a mask (e.g., cropping can only change the pixel offset and stride without actually copying the image pixel values). Dilation methods can be used to convert from a mask class to a shape (e.g., ellipse, rectangle, polygon) and / or vice versa. In some cases, converting a mask to a shape may be lossy (e.g., "best fit"), but may still provide a convenient way to display the mask.

[0156] The output data can include a variety of data types such as double, integer, Boolean, datetime, double enumeration, integer enumeration, string enumeration, string, binary, 1D, 2D, or ND arrays of any of the above.

[0157] Computer system of the present disclosure It will be appreciated that computer systems are increasingly taking on a wide variety of forms. In this description and in the claims, the term "computer system" or "computing system" is broadly defined to include any device or system, or combination thereof, having at least one physical and tangible processor and physical and tangible memory capable of having computer-executable instructions that can be executed by the processor. By way of example and not limitation, the term "computer system" or "computing system" as used herein is intended to include a portion of a disclosed imaging system that is in electrical communication with its associated optical and mechanical components and operable to perform the various autofocus and / or cell viability counting methods disclosed herein. Thus, the term "computer system" or "computing system" as used herein can execute operational commands related to stage / sample movement in addition to controlling automated image capture and processing (e.g., determining target focus position and / or performing automated cell viability counting). It should be understood that the computing systems disclosed herein are components of the disclosed imaging systems unless otherwise specified.

[0158] The memory components of the disclosed computer systems may take any form and may depend on the nature and form of the computing system. Memory may be physical system memory, including volatile memory, non-volatile memory, or a combination of the two. The term "memory" may be used herein to refer to non-volatile mass storage devices, such as physical storage media.

[0159] The computing systems disclosed herein are understood to have stored thereon a number of structures, often referred to as "executable components." For example, the memory of a computing system may include executable components. The term "executable component" is a name for a structure that is well understood by those skilled in the art of computing, as being a structure that may be software, hardware, or a combination thereof.

[0160] For example, when implemented in software, those skilled in the art will understand that the structure of an executable component may include software objects, routines, methods, etc. that can be executed by one or more processors on a computing system, whether such executable components reside on the computing system's heap or whether the executable components reside on a computer-readable storage medium. The structure of the executable component resides on a computer-readable medium in a manner that, when executed by one or more processors of the computing system, is operable to cause the computing system to perform one or more functions, such as the functions and methods described herein. Such structure may be directly computer-readable by a processor, such as when the executable component is binary. Alternatively, the structure may be structured to be sequentially interpretable (whether in a single step or multiple steps) and / or compiled to generate a binary that is directly sequentially interpretable by a processor.

[0161] The term "executable component" will also be well understood by those skilled in the art to include structure that is implemented exclusively or nearly exclusively in hardware logic components, such as within a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a program-specific standard product (ASSP), a system-on-a-chip system (SOC), a complex programmable logic device (CPLD), or other specialized circuitry. Thus, the term "executable component" is a term for structure that is well understood by those skilled in the art of computing, whether implemented in software, hardware, or a combination thereof.

[0162] Terms such as "component," "service," "engine," "module," "control," "generator," and the like may also be used in this description. As used in this description, these terms, with or without modifiers, are intended to be synonymous with the term "executable component," and as such, have a structure that is well understood by those skilled in the computing arts.

[0163] While not all computing systems require a user interface, in some embodiments, a computing system includes a user interface for use in communicating information to and from a user. A user interface may include output mechanisms as well as input mechanisms. The principles described herein are not limited to the exact output or input mechanisms, which as such depend on the nature of the device. However, output mechanisms may include, for example, speakers, displays, tactile output, etc. Examples of input mechanisms include, for example, a microphone, a touchscreen, a camera, a keyboard, a stylus, a mouse or other pointer input, any type of sensor, etc.

[0164] Accordingly, embodiments described herein may include or utilize special purpose or general purpose computing systems. Embodiments also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media that can be accessed by a general purpose or special purpose computer system. Computer-readable media that store computer-executable instructions are physical storage media. Computer-readable media that carry computer-executable instructions are transmission media. Thus, by way of example, and not limitation, embodiments disclosed or contemplated herein may include at least two distinctly different kinds of computer-readable media: storage media and transmission media.

[0165] The computer-readable storage medium may include RAM, ROM, EEPROM, solid state drive ("SSD"), flash memory, phase-change memory, and the like. These include computer-readable storage media such as memory, "PCM," CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or other physical and tangible storage media that can be used to store desired program code in the form of computer-executable instructions or data structures, and that can be accessed and executed by a general-purpose or special-purpose computing system to implement the disclosed functions of the present invention. For example, computer-executable instructions can be embodied on one or more computer-readable storage media to form a computer program product.

[0166] Transmission media can include a network and / or data links that can carry desired program code in the form of computer-executable instructions or data structures and that can be accessed and executed by a general-purpose or special-purpose computer. Combinations of the above are also intended to be included within the scope of computer-readable media.

[0167] Furthermore, program code in the form of computer-executable instructions or data structures may be automatically transferred from transmission media to computer storage media (or vice versa) upon reaching various computer system components. For example, computer-executable instructions or data structures received over a network or data link may be buffered in RAM within a network interface module (e.g., a "NIC") and eventually transferred to computer system RAM and / or a less volatile storage medium of the computer system. Thus, it should be understood that storage media may be included in computing system components that also (or even primarily) utilize transmission media.

[0168] Those skilled in the art will further appreciate that a computing system may also include communication channels that enable it to communicate with other computing systems, for example, over a network. However, as provided above, the computing system of the present disclosure is preferably a component of the disclosed imaging system. Thus, while a computing system may include communication channels that enable network communication (e.g., for file and / or data transfer, configuration or updating of firmware and / or software associated with the computing system), it should be understood that the computing systems disclosed herein are intended to perform the disclosed methods locally, rather than in a distributed systems environment linked over a network (either by hardwired data links, wireless data links, or a combination of hardwired and wireless data links) where processing and / or memory may be distributed among various networked computing systems.

[0169] Although the subject matter described herein may be presented in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the features or acts described above. Rather, the described features and acts are disclosed as exemplary forms of implementing the claims. Additional Terms and Definitions

[0170] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. The terms and expressions used herein are used as terms of description and not of limitation, and the use of such terms and expressions is not intended to exclude equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the present disclosure as claimed. Thus, while the present invention has been specifically disclosed in part by preferred embodiments, exemplary embodiments, and optional features, it should be understood that modifications and variations of the concepts disclosed herein may be resorted to by those skilled in the art, and such modifications and variations are considered to be within the scope of the present disclosure. The specific embodiments provided herein are examples of useful embodiments of the present invention, as well as various modifications and / or variations of the features of the present invention shown herein; additional applications of the principles set forth herein, which will occur to those skilled in the relevant art and in possession of this disclosure, may be made to the illustrated embodiments and should be considered to be within the scope of the present disclosure.

[0171] Furthermore, unless a feature is described as requiring another feature in combination, any feature herein can be combined with any other feature of the same or different embodiments disclosed herein. Moreover, various well-known aspects of example systems, methods, devices, etc. have not been described in particular detail herein to avoid obscuring aspects of the example embodiments. However, such aspects are also contemplated herein.

[0172] As used herein, unless otherwise understood or stated, implicitly or explicitly, words appearing in the singular include their plural equivalents, and words appearing in the plural include their singular equivalents. Accordingly, it should be noted that, as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the content clearly dictates otherwise. For example, a reference to a single referent (e.g., "widget") includes one, two, or more referents unless otherwise understood or stated, implicitly or explicitly. Similarly, a reference to a plural referent should be construed as including a single referent and / or multiple referents unless the content and / or context clearly dictates otherwise. For example, a reference to a plural referent (e.g., "widget") does not necessarily require a plurality of such referents. Instead, it will be understood that one or more referents are contemplated herein, regardless of the presumed number of referents, unless otherwise specified.

[0173] All references cited in this application are incorporated herein by reference in their entirety to the extent they do not contradict the disclosure of this application. It will be apparent to those skilled in the art that methods, devices, device elements, materials, procedures, and techniques other than those specifically described herein can be applied to the practice of the invention broadly disclosed herein without resort to undue experimentation. All art-known functional equivalents of the methods, devices, device elements, materials, procedures, and techniques specifically described herein are intended to be encompassed within this disclosure.

[0174] When a group, such as an ingredient, is disclosed herein, it is understood that all individual members of the disclosed group and all subgroups thereof are separately disclosed. When a Markush group or other group is used herein, all individual members of the group and all possible combinations and subcombinations of the group are intended to be individually included in the disclosure. All formulations or combinations of ingredients described or exemplified herein can be used to implement preferred and / or alternative embodiments of the disclosure, unless otherwise specified. Whenever a range is given in the specification, all intermediate ranges and subranges, as well as all individual values ​​included in the given range, are intended to be included in the disclosure.

[0175] All changes that come within the meaning and range of equivalency of the claims are to be embraced within their scope.

Claims

1. 1. A method for autofocusing an image, comprising: receiving a sample holder in an imaging system; automatically determining a z-axis step size for a nominal focus range based at least in part on the sample holder; receiving a plurality of images spanning the nominal focus range; generating input data based on the plurality of images, the input data being generated via one or more image processing operations; utilizing the input data as input to a machine learning module configured to provide a target focus position output in response to input generated from images associated with one or more nominal focus ranges; obtaining a target focal position based on the output of the machine learning module; configuring the imaging system to capture an image using the target focal position; The method comprises: identifying one or more connected components in each image as the input data; sorting the identified connected components based on the number of pixels associated with each connected component; generating focus quality estimates for at least some of the sorted connected components using the machine learning module; identifying the target focus position based on the focus quality estimates of the sorted connected components; The method further comprises:

2. The method of claim 1 , wherein the one or more image processing operations include connected component analysis.

3. The method of claim 1 , wherein capturing a plurality of images comprises capturing a 1280×960 8-bit image at each z-axis step across the nominal focus range.

4. identifying one or more connected components in each image; thresholding each image to obtain a resulting binary image for each image; applying one or more morphological operators to the resulting binary image; and defining one or more connected components in the resulting binary image.

5. The method of claim 4 , wherein thresholding is based on the difference between the minimum and maximum images of the nominal focus range.

6. The method of claim 4 or 5, wherein applying the one or more morphological operators comprises applying one or more of a morphological closing, a morphological opening, and / or a foreground hole fill.

7. The method of claim 6 , wherein the morphological closing comprises a 2×2 morphological closing.

8. The method of claim 6 or 7, wherein the morphological opening comprises a 2x2 morphological opening.

9. The method of any one of claims 6 to 8, wherein the foreground hole-fill comprises an 8-combined foreground hole-fill.

10. identifying one or more connected components in each image; measuring first and second binary moments of each coupled component; fitting a corresponding ellipse with equivalent moments to each connected component; and creating a second binary image containing the corresponding ellipse.

11. 11. The method of claim 10, further comprising measuring each bound component in the second binary image and removing any bound components having an ellipse minor axis less than 10 μm, less than 7 μm, or less than 5 μm.

12. Sorting the identified one or more linked components Counting the number of pixels in each connected component; calculating a median number of pixels across the one or more connected components; and sorting the one or more connected components based on a corresponding absolute difference of the number of pixels from the median pixel number.

13. The method of claim 12 , wherein the one or more connected components are sorted in ascending order by the corresponding absolute difference of the number of pixels from the median pixel number.

14. Sorting the identified one or more linked components determining one or more of the circularity or brightness of each combined component; and sorting the one or more connected components based on a comparative circularity and / or brightness determined for each of the connected components.

15. The method of any one of claims 1 to 14, further comprising forming a 32x32 pixel z-stack for each combined component.

16. The method of claim 15 , wherein the machine learning module comprises an artificial neural network that receives as input the 32×32 pixel z-stack of each connected component.

17. 17. The method of claim 16, wherein the artificial neural network comprises a plurality of feature identification layers having a design pattern of convolutional layers, linear layers, and max pooling layers.

18. 18. The method of claim 17, wherein the convolutional layer comprises a 3x3 convolutional layer.

19. The method of claim 17 , wherein the linear layer comprises a ReLu nonlinear function.

20. 20. The method of any one of claims 17 to 19, wherein the max pooling layer comprises a 2x2 max pooling layer.

21. 21. The method of any one of claims 17 to 20, wherein the artificial neural network comprises a long short-term memory (LSTM) processing layer following the plurality of feature identification layers.

22. 22. The method of claim 21, wherein the LSTM process layer processes the z-stack of each coupled component in an iterative manner.

23. 23. The method of claim 21 or 22, wherein the artificial neural network comprises a final linear layer, the final linear layer being combined with the output of the LSTM process layer to define the focus quality estimate.

24. 24. A computer system configured to autofocus an imaging system, the computer system comprising: one or more processors; and one or more hardware storage devices storing computer-executable instructions that, when executed by the one or more processors, configure the computer system to perform the method of any one of claims 1 to 23.

25. One or more hardware storage devices storing instructions executable by one or more processing devices of an imaging system to configure said imaging system to perform the method of any one of claims 1 to 23.

Images