Image-based cell sorting systems and methods

Abstract

To provide image-based cell sorting systems and methods.SOLUTION: Disclosed are systems, devices and methods for imaging and image-based sorting of particles in a flow system, including cells in a flow cytometer. In some aspects, a system includes a particle flow device to flow particles through a channel, an imaging system to obtain image data of a particle during flow through the channel, a processing unit to determine a property associated with the particle and to produce a control command for sorting the particle based on sorting criteria associated with particle properties, and an actuator to direct the particle into one of a plurality of output paths of the particle flow device in real-time.SELECTED DRAWING: Figure 1A

Description

[Technical field]

[0001] STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT This invention was made with Government support under Grant No. 1R43DA042636-01 awarded by the National Institutes of Health (NIH). The Government has certain rights in the invention.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This patent document claims priority to and benefit of U.S. Provisional Patent Application No. 62 / 348,511, entitled "FLOW CYTOMETER WITH IMAGE-BASED CELL SORTING," filed June 10, 2016. The entire contents of the aforementioned patent application are incorporated by reference as part of the disclosure of this patent document.

[0003] This patent document relates to flow cytometry devices and techniques, as well as systems, devices and techniques for particle sorting in fluids, including applications in chemical or biological testing and diagnostic assays. [Background technology]

[0004] Flow cytometry is a technique for detecting and analyzing particles, such as live cells, as they flow through a fluid. For example, a flow cytometer device can be used to characterize the physical and biochemical properties of cells and / or biochemical molecules or molecular clusters based on their optical, electrical, acoustic and / or magnetic responses as they are successively interrogated. Typically, flow cytometry uses an external light source to interrogate particles, from which optical signals caused by one or more interactions between the input light and the particles, such as forward scatter, side scatter, and fluorescence, are detected. Properties measured by flow cytometry include the relative size, granularity, and / or fluorescence intensity of the particles.

[0005] Cell sorting, including cell sorting at the single cell level, has become an important feature in the field of flow cytometry as researchers and clinicians become more interested in studying and purifying specific cells, such as stem cells, circulating tumor cells, and rare bacterial species. Cell sorting can be achieved by a variety of techniques.

[0006] Flow cytometry devices and systems can be implemented based on microfluidic technology for research assays and diagnostics as well as clinical applications. Microfluidic devices are instruments that can control the behavior of very small amounts of fluid (e.g., nL, pL, and fL, etc.) through channels with relatively small dimensions, e.g., dimensions in the sub-millimeter range. Microfluidic devices can be implemented to obtain various analytical measurements, including molecular diffusion values, chemical binding coefficients, pH values, fluid viscosity, molecular reaction kinetics, etc. Microfluidic devices can be built on microchips, also called lab-on-a-chip, to detect, separate, and analyze biological samples. For example, microfluidic devices may use biological fluids or solutions containing cells or cell parts to diagnose diseases. For example, inside the microfluidic channels of a microfluidic flow cytometer, particles including cells, beads, and macromolecules can be interrogated according to their optical, electrical, acoustic, and / or magnetic responses using flow cytometry techniques. Summary of the Invention [Means for solving the problem]

[0007] The technology disclosed in this patent document can be implemented to provide methods, devices, and systems for generating images of particles in a flow system. In certain configurations, the disclosed technology can be used to image particles in real time and then sort the particles, including cells, based on spatial information from the images. The disclosed techniques can be applied to the generation of cell images and cell sorting in flow cytometers. In application, the disclosed technology can be used to detect and sort cells based on fluorescence intensity and / or scattering intensity by taking into account spatial information such as the spatial distribution of fluorescence.

[0008] In implementations, for example, the disclosed systems have the high throughput of a flow cytometer and the high spatial resolution of an imaging cytometer, and cell images are generated at rates fast enough to accommodate real-time cell sorting in flow systems, e.g., based on physical and / or physiological properties of cells rather than simply detected events.

[0009] In some embodiments, the image-based particle sorting system comprises a particle flow device constructed to include a substrate, a flow channel formed on the substrate, the flow channel operable to flow cells along a flow direction into a first region of the flow channel, and two or more output paths branching from the flow channel at a second region within the flow channel proximate to the first region; an imaging system interfaced with the particle flow device and operable to acquire image data associated with the cells when the cells are within the first region during flow through the flow channel; and a data processing and control unit in communication with the imaging system, the data processing and control unit processing image data acquired by the imaging system to determine one or more characteristics associated with the cells from the processed image data, and determining one or more characteristics associated with the cells. and a data processing and control unit configured to generate control commands based on a comparison of the image signal data corresponding to the cells to a sorting criterion, the control commands being generated while the cells are flowing through the flow path and indicating a sorting decision determined based on one or more cell attributes ascertained from the image signal data corresponding to the cells; and an actuator operatively coupled to the particle flow device and in communication with the actuator, the actuator operable to direct the cells to one of two or more output paths based on the control command, wherein the system is operable to sort each of the cells while they are flowing through the flow path within a time frame of 15 ms or less from a first time of image capture by the imaging system to a second time of particle direction by the actuator.

[0010] In some embodiments, an image-based particle sorting method includes acquiring image signal data of cells flowing through a flow path of a particle flow device, processing the image signal data to generate an image dataset representing images of the cells, analyzing the generated image dataset to identify one or more characteristics of the cells from the processed image data, evaluating the one or more identified characteristics of the cells with a sorting criterion to generate control commands to sort the cells based on one or more cellular attributes ascertained from the image signal data corresponding to the cells in the cell stream in the particle flow device, and directing the cells to one of a plurality of output paths of the particle flow device based on the control commands.

[0011] In some embodiments, an image-based particle sorting system includes a particle flow device constructed to include a substrate, a flow channel formed on the substrate, the flow channel operable to flow particles along a flow direction into a first region of the flow channel, and two or more output paths branching from the flow channel at a second region within the flow channel proximate to the first region; an imaging system interfaced with the particle flow device and operable to acquire image data associated with the particle when the particle is within the first region during flow through the flow channel; and a data processing and control unit in communication with the imaging system, the data processing and control unit processing the image data acquired by the imaging system to generate a processed image. The system includes a data processing and control unit including a processor configured to determine one or more properties associated with the particles from the data and generate a control command based on a comparison of the determined one or more properties with a sorting criterion; and an actuator operatively coupled to the particle flow device and in communication with the actuator, the actuator operable to direct the particles to one of two or more output paths based on the control command, wherein the system is operable to sort each of the particles as they flow through the flow path within a time frame of 15 ms or less from a first time of image capture by the imaging system to a second time of particle direction by the actuator.

[0012] In some embodiments, a method for image-based sorting of particles includes acquiring image signal data of particles flowing through a flow path of a particle flow device, processing the image signal data to generate an image dataset representing an image of the particles, analyzing the generated image dataset to identify one or more characteristics of the particles from the processed image data, generating a control command by evaluating the one or more identified characteristics with a sorting criteria, and directing the particles to one of a plurality of output paths of the particle flow device based on the control command.

[0013] These and other aspects of the disclosed technology and its implementations and applications are described in more detail in the drawings, specification, and claims. [Brief description of the drawings]

[0014] [Figure 1A] 1 shows a diagram of an exemplary embodiment of an image-based particle sorting system in accordance with the disclosed technology; [Figure 1B] FIG. 1 shows a block diagram of an exemplary embodiment of a data processing and control unit of an image-based particle sorting system in accordance with the disclosed technology. [Figure 2A] FIG. 1 shows a diagram of an exemplary embodiment of an image-based cell sorting microfluidic system in accordance with the disclosed technology. [Figure 2B] FIG. 1 shows a diagram of an exemplary embodiment of an image-based cell sorting microfluidic system in accordance with the disclosed technology. [Figure 2C] FIG. 1 shows a diagram of an exemplary embodiment of an image-based cell sorting microfluidic system in accordance with the disclosed technology. [Figure 3A] 1 shows exemplary image data captured and processed from PMT signals to generate cell images for image-based particle sorting techniques according to the disclosed technology. [Figure 3B] 1 shows exemplary image data captured and processed from PMT signals to generate cell images for image-based particle sorting techniques according to the disclosed technology. [Figure 4A] 1 shows a diagram of an exemplary embodiment of a process for an image-based particle sorting method according to the present technique. [Figure 4B] 1 shows a diagram of an exemplary embodiment of a process for an image-based particle sorting method according to the present technique. [Figure 4C] 1 shows a diagram of an exemplary embodiment of a process for an image-based particle sorting method according to the present technique. [Figure 4D] 1 shows intensity-time plots illustrating an implementation of an exemplary cell detection process based on processing of time domain PMT signals associated with a single cell passing through the illumination area of ​​an exemplary system in accordance with the present technology. [Figure 5A] FIG. 1 shows a diagram of an exemplary implementation of a process for determining identified feature(s) of detected cells for image-based cell sorting. [Figure 5B] 1 shows data plots depicting results of an implementation of an exemplary image reconstruction technique in accordance with the present technique. [Figure 6] 1 shows a data plot depicting the fluorescence intensity signal of cells based on an exemplary image reconstruction. [Figure 7] 1 shows an example of a reconstructed image of a cell. [Figure 8A] Exemplary fluorescent microscopy images of transfected but non-drug treated and transfected and drug treated cells are shown. [Figure 8B] Exemplary fluorescent microscopy images of transfected but non-drug treated and transfected and drug treated cells are shown. [Figure 9A] 1 shows exemplary fluorescent cell images taken by an image-based cell sorter system of transfected but non-drug treated and transfected and drug treated cells. [Figure 9B] 1 shows exemplary fluorescent cell images taken by an image-based cell sorter system of transfected but non-drug treated and transfected and drug treated cells. [Figure 10] 13 shows exemplary calculated fluorescence area histograms of all events and sorted cells from an exemplary implementation. [Figure 11] FIG. 1 shows a diagram of an exemplary embodiment of an image-based cell sorting microfluidic system in accordance with the disclosed technology. [Figure 12] 12 shows a flow diagram of an exemplary image processing implementation using the system shown in FIG. [Figure 13A] 13 shows a flow diagram depicting exemplary data through the exemplary image processing steps shown in FIG. 12. [Figure 13B] 13 shows an exemplary distribution plot of the subsets using an exemplary Receiver Operating Characteristic (ROC) technique for evaluating extracted parameters such as cell morphology parameters. [Figure 14A]13A-13C show cell images of non-translocated and translocated cells, respectively, captured by an exemplary system and reconstructed by an exemplary image reconstruction technique according to the disclosed technology. [Figure 14B] 13A-13C show cell images of non-translocated and translocated cells, respectively, captured by an exemplary system and reconstructed by an exemplary image reconstruction technique according to the disclosed technology. [Figure 15] For the implementation of the cell sorting criterion, we show an example of a hyperplane that exhibits the separation of two sets of cells. [Figure 16A] 1A-1D show cell images of non-translocated and translocated cells, respectively, captured by an exemplary system via a microscope. [Figure 16B] 1A-1D show cell images of non-translocated and translocated cells, respectively, captured by an exemplary system via a microscope. [Figure 17A] 13A-13C show cell images of G1 and G2 / M phase cells, respectively, captured by an exemplary system and reconstructed by an exemplary image reconstruction technique according to the disclosed technology. [Figure 17B] 13A-13C show cell images of G1 and G2 / M phase cells, respectively, captured by an exemplary system and reconstructed by an exemplary image reconstruction technique according to the disclosed technology. [Figure 18] For the implementation of the cell sorting criterion, we show an example of a hyperplane that exhibits the separation of two sets of cells. [Figure 19A] 13A-13C show cell images of G1 phase cells and G2 / M cells, respectively, captured by an exemplary system and reconstructed by an exemplary image reconstruction technique according to the disclosed technology. [Figure 19B] 13A-13C show cell images of G1 phase cells and G2 / M cells, respectively, captured by an exemplary system and reconstructed by an exemplary image reconstruction technique according to the disclosed technology. [Figure 20] 12 shows a flow diagram of an exemplary image processing implementation for sorting based on the number of beads bound to the cell membrane of cells implemented by the system shown in FIG. 11 . [Figure 21]13 shows examples of grayscale cell images processed by an exemplary image processing module with different numbers of beads. [Figure 22] Histograms of bead image area for different numbers of beads are shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Some existing flow cytometer devices and systems detect and sort cells based on fluorescence intensity and / or scattering intensity without considering spatial information such as the spatial distribution of fluorescence. Some progress has been made in developing techniques for generating cell images for flow cytometers with high throughput and high spatial resolution. However, cell images are not generated at a rate fast enough to be useful for applications, particularly cell sorting in flow systems, due to, for example, the amount of computation required to generate the cell images. Thus, existing state-of-the-art cell sorting capabilities are "detection-only" systems that do not "screen" the detected cells based on meaningful and sensitive criteria.

[0016] In application, the disclosed technology can be implemented in a specific manner in the form of a method, system, and device for image-based cell sorting in flow cytometry using (a) real-time image acquisition of fast moving cells by efficient data processing techniques utilizing mathematical algorithms implemented in FPGA and / or GPU, and simultaneously (b) "gating" techniques based on spatial characteristics of particles as sorting criteria from real-time acquired images. Unlike conventional flow cytometers that use the fluorescence intensity of selected biomarkers as criteria for cell sorting, the method, system, and device according to the disclosed technology allow for various user-defined gating criteria including spatial characteristics.

[0017] Examples of image-based gating criteria include cell contours, cell size, cell shape, size and shape of cell internal structures such as cell nuclei, fluorescence patterns, fluorescence color distribution, etc. For example, a user can draw the cells they wish to separate, and the system will act accordingly. With such a unique capability, users such as researchers can track many important biological processes by the localization of specific proteins within cytosol, nucleus, or cell membrane domains and subdomains. Since every cell population has a degree of heterogeneity at the genomic level (e.g., mutations, epigenetics) or environmental level (e.g., asymmetric division, morphogen gradients), the identification and extraction of single cells by their unique spatial features will contribute greatly to the fields of immunology, tumor heterogeneity, stem cell differentiation, and neuronal analysis.

[0018] In some embodiments, the image-based particle sorting system includes a particle flow device, such as a flow cell or a microfluidic device, integrated with a particle sorting actuator, a high-speed high-sensitivity optical imaging system, and a real-time cell image processing and sorting control electronic system. For example, the objective of the disclosed methods, systems, and devices is to accomplish the entire process of (i) particle (e.g., cell) image capture, (ii) image feature reconstruction from time-domain signals, and (iii) particle sorting decision and actuator sorting action within a latency of less than 15 ms to meet the needs of real-time particle sorting. In some implementations described herein, the total latency is less than 8 ms (e.g., 7.8 ms), in some implementations, the total latency is less than 6 ms (e.g., 5.8 ms), and in some implementations, the total latency is less than 3.5 ms (e.g., 3.3 ms). For cell sorting implementations, for example, the disclosed methods, systems, and devices can image, analyze, and sort cells by image features specific to cell cycle, protein localization, gene localization, DNA damage, and other cellular characteristics that may be associated with various diseases or pathogens.

[0019] FIG. 1A shows a diagram of an exemplary embodiment of an image-based particle sorting system 100 according to the present technology. The system 100 includes a particle flow device 110, an imaging system 120 interfaced with the particle flow device 110, a data processing and control unit 130 in communication with the imaging system 120, and an actuator 140 in communication with the data processing and control unit 130, operatively coupled to the particle flow device 110. The particle flow device 110 is constructed to include a flow path 111 through which particles flow along a flow direction to an interrogation area 115, and image data is acquired by the imaging system 120 for each particle in the interrogation area 115. The data processing and control unit 130 is configured to process the image data and determine one or more characteristics associated with the particle to generate a control command for sorting the particle. The control command is provided to the actuator 140, which is interfaced with the particle flow device 110 at a sorting area of ​​the device 110 such that the actuator operates to sort a particular particle into an output flow path corresponding to the control command. System 100 implements real-time image-based particle sorting, where particles are imaged by imaging system 120 in an interrogation area and sorted by actuator 140 in a sorting area in real-time based on determined characteristics analyzed by data processing and control unit 130.

[0020] The system 100 is user programmable to sort each particle based on user-defined criteria that may be related to one or more of a number of characteristics exhibited by each individual particle that are analyzed in real time by the data processing and control unit 130. Some exemplary user-defined criteria include, but are not limited to, the amount and / or size of a sub-feature of or on an individual particle (e.g., sub-particles attached to a live cell, including particles that have been phagocytosed by the cell or attached to the cell), the morphology of the individual particle, and / or the size of the individual particle. In this manner, the system 100 can evaluate and sort particles by characteristics such as characteristics of live cells, including sorting by physiological function of the cell (e.g., uptake of a particle or substance by the cell, or phagocytosis of a particle by the cell), by cell damage, by protein localization, or by other cell characteristics.

[0021] FIG. 1B illustrates a block diagram of an exemplary embodiment of the data processing and control unit 130. In various implementations, the data processing and control unit 130 is embodied on one or more personal computing devices, including, for example, a desktop or laptop computer, one or more computing devices in a computer system or communication network accessible via the Internet, including servers and / or databases in the cloud (referred to as the "cloud"), and / or one or more mobile computing devices, such as a smartphone, tablet, or wearable computing device, including a smartwatch or smart glasses. The data processing and control unit 130 includes a processor 131 for processing data and a memory 132 in communication with the processor 131 for storing and / or buffering data. For example, the processor 131 can include a central processing unit (CPU) or a microcontroller unit (MCU). In some implementations, the process 131 can include a field programmable gate array (FPGA) or a graphics processing unit (GPU). For example, memory 132 may include and store processor-executable code that, when executed by processor 131, configures data processing and control unit 130 to perform various operations, such as receiving information, commands, and / or data, processing information and data, such as from imaging system 120, and transmitting or providing processed information / data to another device, such as actuator 140. To support various functions of data processing and control unit 130, memory 132 may store information and data, such as instructions, software, values, images, and other data that is processed or referenced by processor 131. For example, the storage functions of memory 132 may be implemented using various types of random access memory (RAM) devices, read only memory (ROM) devices, flash memory devices, and other suitable storage media.In some implementations, the data processing and control unit 130 includes an input / output (I / O) unit 133 for interfacing the processor 131 and / or memory 132 to other modules, units, or devices. In some implementations, such as for mobile computing devices, the data processing and control unit 130 includes a wireless communication unit, such as a transmitter (Tx) or a transmitter / receiver (Tx / Rx) unit. For example, in such an embodiment, the I / O unit 133 can interface the processor 131 and memory 132 with the wireless communication unit, for example, to utilize various types of wireless interfaces compatible with typical data communication standards that can be used in the data processing and control unit 130's communication with other devices, such as between one or more computers in the cloud and a user device. Data communication standards include, but are not limited to, Bluetooth, Bluetooth Low Energy (BLE), ZigBee, IEEE 802.11, Wireless Local Area Network (WLAN), Wireless Personal Area Network (WPAN), Wireless Wide Area Network (WWAN), WiMAX, IEEE 802.16 (Worldwide Interoperability for Microwave Access (WiMAX)), 3G / 4G / LTE cellular communication methods, and parallel interfaces. In some implementations, the data processing and control unit 130 can interface with other devices using a wired connection via the I / O unit 133. The data processing and control unit 130 can also interface with other external interfaces, data storage sources, and / or visual or audio display devices, etc. to retrieve and transfer data and information, which can be processed by the processor 131, stored in the memory 132, or presented on a display device or output unit of an external device.

[0022] 2A-2C show diagrams of an image-based cell sorting microfluidic system 200 according to some embodiments of the image-based particle sorting system 100. System 200 includes a microfluidic device 210 for flowing particles through an optical interrogation channel for sorting, an imaging system 220 for acquiring image data of the particles within an illuminated area of ​​the interrogation channel, a data processing and control system 230 for processing the acquired image data in real time to determine sorting commands, and an actuator 240 for gating the particles in the microfluidic device 210 based on the determined sorting commands.

[0023] As shown in Figures 2A and 2C, a microfluidic device 210 is constructed to include a substrate 213 having passages forming a microfluidic sample channel 211 and a microfluidic sheath channel 212 converging onto the sample channel 211. In an implementation, for example, the sample channel is configured to carry particles (e.g., cells) suspended in a fluid flowing in a flow direction, and the sheath channel 212 is configured to provide a sheath flow of the fluid to hydrodynamically focus the suspended particles in the fluid before flowing through an illumination area 215 of the microfluidic device 210. In some embodiments, for example, the substrate 213 can be formed of a bulk material, such as, for example, polydimethylsiloxane (PDMS), bonded to a base substrate, such as a glass base substrate or a base substrate of another material.

[0024] The imaging system 220 of the system 200 includes a light source 221, e.g., a laser, for providing input or probe light to the illumination area 215 of the microfluidic device 210, and an optical imager 222 for acquiring images of illuminated particles in the illumination area 215. An exemplary optical imager 222 includes a spatial filter (SF) 224, an absorption filter (EF) 225, and an objective lens 223 (e.g., of a microscope or other optical imaging device) optically coupled to a photomultiplier tube (PMT) 226, as shown in FIG. 2A. In some implementations, for example, the imaging system 220 includes one or more light directing elements 229 for directing the input light to the illumination area 215 of the microfluidic device 210. In the example shown in FIG. 2A, the light directing element 229 includes a dichroic mirror disposed with the light source 221 and the optical imager 222 to direct the input light to the illumination area 215.

[0025] In some implementations of imaging system 220, light source 221 (e.g., a laser) is configured to generate a fluorescence excitation signal that is incident on illumination area 215 to cause fluorescence emission by the particles. Optical imager 222 captures the optical output fluorescence emission signal so that an image of the particles can be generated.

[0026] The data processing and control system 230 of the system 200 is configured to communicate with the optical imager 222, for example, via a PMT, to rapidly process the imaged particles and perform sorting control in real time based on the processed image of each imaged particle. In some implementations of the data processing and control unit 230, the FPGA processing unit is configured to rapidly process the image signal data received by the optical imager 222. An example of such an implementation includes a Virtex-II (xc2v3000) FPGA platform with a compiler Xilinx 10.1, which may be provided via a National Instrument chassis Crio-9104 to execute algorithms according to the data processing method of the present technology.

[0027] The actuator 240 of the system 200 is configured to communicate with the real-time data processing and control system 230 to gate particles flowing through the gate area 217 of the sample channel 211 into two or more output channels 218 of the microfluidic device. In some embodiments, for example, the distance between the illumination area 215 and the gate area 217 can be in the range of 50 μm to 1 mm. In an implementation, the actuator 240 receives sorting commands from the data processing and control system 230 in real time such that the imaging system 220 and the data processing and control system 230 operate to capture and process images of each particle as it flows through the illumination area 215. As a result, the actuator 240 receives and executes the sorting commands to gate each particle accordingly. For example, in some implementations, the actuator 240 includes a piezoelectric actuator coupled to the substrate 213 to cause a deflection that moves the particle in the gate area 217 in a particle direction to direct the particle along a trajectory into one of two or more of the output channels 218.

[0028] In an implementation of system 200, for example, suspended single cells are hydrodynamically focused within a microfluidic channel by a sheath flow, ensuring that the cells move at a uniform velocity through the center of the fluidic channel. Fluorescent emission is detected by a PMT 226 in a wide-field fluorescence microscope configuration, such as the example shown in FIG. 2A. In this example, to accommodate the geometry of the microfluidic device, the laser beam is introduced for optical interrogation by 52 degree reflection off a small dichroic mirror (DM) 229 placed in front of a 50x objective lens 223 (e.g., NA=0.55, working distance=13 mm).

[0029] FIG. 2B illustrates an exemplary embodiment of a spatial filter (SF) 225 inserted in the detection path just at the image plane of the optical imager 222 of the imaging system 220. The spatial filter design includes a pattern with a number of spaced slits. In some embodiments, for example, the spatial filter 225 includes a pattern of apertures with uniform dimensions, which pattern of apertures encodes the waveform of the light received by the optical imager. In the example shown in FIG. 2B, the pattern includes ten 100 μm×50 μm slits spaced in a closely spaced manner. In some embodiments, for example, the spatial filter 225 includes a pattern of apertures with different longitudinal and lateral dimensions relative to the flow direction across the microfluidic channel, such that a waveform is encoded by the optical imager to enable optical detection of particle position in at least two dimensions within the illuminated area 215 of the microfluidic channel 211. An example of a two-dimensional spatially varying spatial filter is provided in U.S. Pat. No. 9,074,978 B2, entitled "Optical Space-Time Encoding Techniques in Microfluidic Devices," the entire contents of which are incorporated herein by reference for all purposes.

[0030] Referring back to FIG. 2A, imaging system 220 shows only one PMT for detection of fluorescent signals, however, more PMTs can be added to optical imager 222 and more excitation laser beams can be added to light source 221 to generate multi-color fluorescent cell images as needed.

[0031] The real-time data processing and control system 230 includes a control loop system implemented using a field programmable gate array (FPGA) to process the captured images and generate corresponding sorting commands for each particle. The data processing and control system 230 includes image processing and image-based particle sorting algorithms executable by the FPGA to provide automated cell image generation and accurate sorting by the system 200. For example, once a sorting decision is made by the FPGA algorithm, an exemplary on-chip integrated piezoelectric lead zirconate titanate (PZT) actuator 240 is actuated to apply fluid pressure to a nozzle structure of the sorting junction. In some implementations, the exemplary PZT actuator 240 is configured to perform fluid displacement operations in response to the sorting command within a time frame of 0.1 ms or less per operation. For example, fluid displacement by the PZT actuator in less than 0.1 ms exhibits single-cell hydrodynamic manipulation capabilities with high throughput.

[0032] Other examples of particle flow device and / or actuator features that may be used in exemplary embodiments of the devices, systems, and methods according to the disclosed technology are provided in U.S. Patent No. 9,134,221 B2, entitled "Fluid Flow Cytometry Apparatus and Particle Sensing Based on Signal Encoding," the entire contents of which are incorporated by reference as part of this disclosure for all purposes. Other examples of optical imaging system features that may be used in exemplary embodiments of the devices, systems, and methods according to the present invention are provided in U.S. Patent Application No. 15 / 514,930, a national stage application under 35 U.S.C. 371 based on PCT Application No. PCT / US2015 / 053368, published as WO 2016 / 054293 A1, entitled "Imaging Flow Cytometry Using Space-Time Conversion," the entire contents of which are incorporated by reference as part of this disclosure for all purposes.

[0033] 3A and 3B show exemplary results of PMT signals processed by the exemplary data processing and control system 230 using the exemplary algorithm described in FIG. 4 and implemented by the exemplary FPGA, and fluorescent cell images constructed from the PMT signals. FIG. 3A shows an exemplary time-domain PMT output signal of fluorescence from an A549 cell stained with CellTrace CFSE. FIG. 3B shows the corresponding processed (e.g., resized) fluorescent image depicting the actual size of the cells. Numbered regions demarcated by dashed lines in the figure demonstrate the correspondence between the time-domain signal and the resulting image. Sizes are labeled in FIG. 3B.

[0034] The spatial resolution of the restored image in the x (horizontal) direction depends on the number of slits on the spatial filter, and in the y (cell progression) direction it depends on the sampling rate and cell flow velocity. In the original image restored by the imaging flow cytometer (shown in Figure 3B), the effective pixel size is 2 μm in the x direction and about 0.4 μm in the y direction. The restored image represents an area of ​​20 μm × 20 μm in the object plane in the microfluidic channel.

[0035] 4A shows a diagram of an exemplary embodiment of a method 400 for image-based particle sorting. Implementation of method 400 can be performed by various embodiments of image-based particle sorting system 100 according to the present technology, such as system 200 and system 1100.

[0036] The method 400 includes a process 405 for capturing, by the imaging system 120, image data of particles flowing through a flow path in the particle flow device 110, for example at the interrogation area 115. For example, the process 405 can include continuously capturing images at a predetermined rate or a varying rate, which may be based on the particle flow rate in the flow path. In some implementations, the process 405 includes receiving an image capture command from the data processing and control unit 230 at a controller of the imaging system 120 to affect an image processing protocol to acquire image data. For example, the data processing and control unit 130 can modify one or more parameters of an image capture protocol executed by the imaging system 120 in a real-time implementation of the system.

[0037] The image capture rate can be related to the amount of data for a single particle. For example, the system can determine the image capture rate and / or other parameters of the image capture protocol based at least in part on (a) the particle flow rate of the particle flow device 110 and (b) the electronic sampling rate, e.g., based on what resolution is desired, which can be used to determine the amount of data for a single particle. The image capture rate can be related to the data recording / computing capabilities of the data processing and control unit 130 and / or the controller (e.g., processor) of the imaging system 120. For example, a faster analog-to-digital conversion (ADC) and a larger memory capacity can increase the image capture rate. For example, the image capture protocol, including image capture and processing parameters, can be selected at least in part based on the design of the spatial filter, since in some implementations the optical output from the spatial filter can affect the processing complexity of the algorithm. In particular, optical factors such as magnification, optical filters, imaging modes, etc. typically have little, if any, effect on the image capture rate, but are important for obtaining input data that is processed to generate an overall result, e.g., a determination of the particle characteristics that are evaluated to determine the sorting decision or other analysis.

[0038] Method 400 includes a process 410 for receiving image data in data processing and control unit 130 and from imaging system 120, the received data relating to particles imaged by imaging system 120 flowing through a flow path of particle flow device 110. For example, in some implementations of process 410, data processing and control unit 130 receives time domain signal data, e.g., light intensity, from one or more PMTs of imaging system 120 for each particle imaged within an illuminated area on particle flow device 110. Method 400 includes a process 420 for processing the data (e.g., image signal data) to generate an image dataset representing an image of the particle imaged by imaging system 120. For example, in some implementations of process 420, the received image signal data is pre-processed by filtering the data, reconstructing an image based on the filtered data using an image reconstruction algorithm according to the present technique, and / or resizing the reconstructed image, e.g., the reconstructed image is converted to binary image data. Method 400 includes a process 430 for analyzing the generated image data set to identify one or more characteristics of the imaged particles based on predetermined criteria and determining a sorting command based on the one or more identified characteristics. In some implementations of the method, method 400 includes a process 415 for first processing the received image signal data to detect the presence of particles in the illuminated area and then processing the data according to process 420 to generate an image data set, which is then analyzed according to process 430. Method 400 includes a process 440 for providing a sorting command to actuator 140. Method 400 includes a process 445 for executing the sorting command by actuator 140 to direct particles flowing through a sorting area of ​​a flow channel of particle flow device 110 to a corresponding output flow channel of device 110.For example, in implementations of method 400, imaging system 120 and data processing and control system 230 perform processes 405 and 410, 420, 430, and 440, respectively, in real time such that actuator 140 receives and executes sorting commands to appropriately direct particles, e.g., gate each particle into the appropriate output channel of device 110, within the short period of time between image capture (process 405) and activation of particle gating (process 445). For example, in some implementations of process 445, actuator 140 includes a piezoelectric actuator coupled to flow device 110 that creates a hydrodynamic deflection in the fluid to move particles along desired trajectories into desired output channels.

[0039] In some embodiments, the data processing and control unit 130 includes software modules corresponding to one or any combination of the processes 410, 420, 430, and 440 stored in the memory 132 and executable by the processor 131 to implement the processes 410, 420, 430, and / or 440. FIG. 4B shows an exemplary diagram of the data processing modules of a system according to the present technology. For example, the data processing and control unit 130 can include a particle detection module 461, an image processing module 463, and a sorting decision module 465. As shown by the exemplary diagram of FIG. 4B, when a particle (e.g., a cell) is detected by implementing an algorithm executable by the particle detection module 461, the data processing and control unit 130 proceeds to implement the image processing module 463 if a particle (e.g., a cell) is detected. Otherwise, the data processing and control unit 130 goes back and continues to record an image via, for example, a PMT readout. By extracting parameters from the cell images and comparing those values ​​to predefined sorting criteria, e.g., performed by image processing module 463, a sorting decision module 465 is implemented to decide whether to trigger actuator 140. In some embodiments, data processing and control unit 130 includes an image recording module 467 that at least partially controls imaging system 120 to implement process 405, which may include, for example, initiating and / or adapting an image capture protocol or settings of the imaging system. In some implementations, such as with imager 222, image recording module 467 is operable to control the recording of PMT readouts at a certain length.

[0040] In some implementations, parameters related to particle characteristics (e.g., cell characteristics) are extracted based on the type of parameter(s) to be extracted. For example, for different morphological parameters, a process for extracting parameters related to cell morphological characteristics may include at least some of the following techniques: The process may include analyzing the image area, including, for example, determining the number of pixels having a value of "1" in the binary image. The number of pixels is the image area. The process may include analyzing the perimeter, including, for example, determining the number of pixels on the image contour when the image contour is detected. The process may include analyzing the x-direction diameter, including, for example, determining the number of "1" pixels in each row representing the x-direction (e.g., in the binary image). For example, the maximum count may be determined as the x-direction diameter. The process may include analyzing the y-direction diameter, including, for example, determining the number of "1" pixels in each column representing the y-direction (e.g., in the binary image). For example, the maximum count may be determined as the y-direction diameter.

[0041] FIG. 4C shows a diagram of an exemplary implementation of the process 430 for detecting cells for image-based cell sorting, e.g., an implementation of the particle detection module 461. In some implementations, for example, the particle detection module 461 searches for a portion of the time-domain PMT signal that has an integrated fluorescence intensity (herein referred to as "brightness") greater than a preset threshold. For example, in some embodiments, the cell movement speed in the flow cell or microfluidic device is 0.08 m / s and a sampling rate of 200 kHz is used, such that the particle sorting and image processing algorithms integrate every 500 consecutive data points to obtain a brightness value. Whenever the brightness is greater than a first preset threshold, e.g., threshold 1, this means that the cell is in the field of view of the optical system in the image plane. The algorithm determines the time derivative of the brightness, which is then compared to a second preset threshold, e.g., threshold 2. If the time derivative is less than threshold 2, the algorithm considers the cell to be sufficiently within the field of view, and the exemplary process 430 continues by determining the identified feature(s) of the detected cell, which may result in, for example, in the implementation of image processing module 463.

[0042] 4D shows a brightness-time plot depicting an exemplary cell detection process implementation based on processing a time domain PMT signal associated with a single cell passing through the illumination area of ​​the system. As shown in the plot, a cell is detected as entering the imaging area (e.g., interrogation area 115) based on a threshold (e.g., brightness > 1.0) just after the 1.176 s time marker, and the system captures image signal data until the cell is detected as leaving the imaging area based on a threshold (e.g., brightness < 1.0) after the 1.179 s time marker.

[0043] FIG. 5A shows a diagram of an exemplary implementation of process 430, e.g., an implementation of image processing module 463, for determining identified feature(s) of detected cells for image-based cell sorting. In the example shown in FIG. 5A, process 420 is implemented after some aspects of process 430, e.g., in particular after analyzing image signal data to determine whether cells are detected. In some implementations, e.g., as shown in FIG. 5A, after partially analyzing the received image signal data to determine the presence of cells, image processing module 463 pre-processes the image signal data (e.g., time-domain PMT signals) including filtering or other signal processing techniques. For example, in some implementations, the received image signal data is low-pass filtered to remove high-frequency noise. In some examples, a 10th order Hamming window for low-pass filtering is used, which is one example of a specific low-pass filter that can be applied. Process image module 463 can implement other features of process 420, including image reconstruction techniques (e.g., based on exemplary algorithms described herein, including Equation (1)) to convert the time-domain signal into an image representing a two-dimensional spatial distribution of the fluorescence of the cells. The reconstructed cell image is then resized and converted to a binary image for fidelity purposes. For binary images, an optional open filter technique is applied, for example to remove spurious noise, and the image processing module 463 detects one or more cell features after the open filter is applied (e.g., cell wall or membrane detection). In some implementations, the open filter can be applied after the cell feature(s) are detected. The image processing module 463 extracts parameters based on the detected cell features that describe aspects of the cells, such as cell morphology, which can be compared to values ​​from a user's pre-set sorting criteria for determining subsequent sorting commands (e.g., gating commands).

[0044] An exemplary implementation of the method 400 using the system 200 is described below with respect to an exemplary study demonstrating image-based cell sorting. In the exemplary study, the imaging system 220 included a 100 mW 488 nm laser (e.g., iBeam-SMART, Toptica) with an elliptical beam shape and Gaussian energy distribution, which was collimated, focused, and then expanded to illuminate an area of ​​100 μm (x-direction) × 250 μm (y-direction) from the illumination area 215 on the microfluidic device 210. Fluorescence and scattered light passing through a small dichroic mirror (e.g., ThorLabs) with a cutoff wavelength of 500 nm was collected via a 50x, 0.55 NA objective lens (e.g., Mituyoyo). The light intensity signal of each channel was acquired by a PMT (e.g., H9307-02, Hamamatsu).

[0045] For example, an image reconstruction technique implemented by the image processing module 463 was used for space-time transformation of image data that can be mathematically formulated as follows:

number

number

number

[0046] An exemplary image reconstruction technique that may be implemented by the image processing module 463 includes, for example, determining two variables for a flowing cell: the number of sampling points in each peak, and the starting point of the time-domain image signal data (e.g., PMT signal). For example, since cells do not move at a perfectly uniform speed, the number of sampling points in each peak varies slightly. Also, since the cell movement speed and the cell position in the image area vary slightly (e.g., 20 μm×20 μm image area set based on the exemplary spatial filter), the starting point of the time-domain PMT readout varies as well. In the exemplary image reconstruction technique, m is referred to as the starting point of the PMT readout, and n is referred to as the number of points in each peak. For example, n ranges from 46 to 51 based on the variability of the cell speed. Thus, m ranges from 0 to 519-10n. The exemplary image reconstruction technique sweeps m and n to ensure the best combination for reconstructing the cell image. For example, the sum of the intensity at the starting point of each peak is calculated for every combination in the sweep. The combination with the smallest sum is the correct answer for image reconstruction. After calculating m and n, the signals recorded by both PMTs are synchronized and can be used to reconstruct both brightfield and fluorescent images. An exemplary image reconstruction technique calculates the values ​​based on equation (4).

number

[0047] 5B shows an exemplary data plot depicting the results of an implementation of the image reconstruction technique, e.g., showing exemplary results of a search for the best combination of starting point and number of sampling points for each peak. The black "*" symbols shown in the data plot represent the starting points of each peak found by the reconstruction algorithm.

[0048] In general, when a spatial filter as described above (e.g., spatial filter 224 in FIG. 2B) is inserted in the image plane, the fluorescence from different parts of the cell will pass through different slits at different times. As a result, the waveform of the fluorescence signal from the PMT will contain a series of patterns separated in the time domain, with each part of the time domain signal corresponding to the fluorescence signal generated by each particular state of the cell. After receiving the light intensity profile across each slit, a cell image of the whole cell can be constructed by splicing all the profiles together. In the exemplary embodiment of the system 200 shown in FIGS. 2A-2C, the spatial filter includes ten 100 μm×50 μm rectangular slits arranged in sequence. For example, with a 50× objective (M=50), the filter design allows the construction of a fluorescence or scattering image of a moving cell, for example, 20 μm×20 μm or smaller, using an algorithm including Equation (3). This requires minimal computational effort and is suitable for high throughput and real-time image-based cell classification and sorting. For example, using a 50x / 0.55NA objective lens, a sampling rate of 500 kHz for acquiring the PMT signal, and a cell migration speed of 0.2 m / s given by a sample flow rate of 12 μL / min and a sheath flow rate of 120 μL / min, the effective size of a pixel in the y-direction is smaller than the Rayleigh criterion.

number

[0049] In this exemplary study, the spatial filter design was drawn in AutoCAD and printed onto a transparency mask at 20,000 dots per inch (dpi). A layer of negative photoresist (e.g., NR9-1500PY, Futurrex, Inc.) was spun onto a 6-inch glass wafer at 3,000 revolutions per minute (rpm). The wafer was heated on a hotplate at 150°C for 3 minutes and then illuminated with UV light (e.g., EVG620NT, EV The glass wafer was exposed to UV light from the 300 nm UV-exposed 100 nm thick 1000 nm thick 1000 nm thick 15 ...

[0050] For this exemplary study, HEK293T human embryonic kidney cell samples were transfected with pEGFP-GR plasmid (e.g., Addgene). After 3 days of continuous culture, 1 μM dexamethasone (e.g., Sigma-Aldrich) was added to the medium. After 60 min of incubation, HEK293T cells were harvested, fixed with 4% paraformaldehyde, washed, and resuspended in 1x phosphate-buffered saline (PBS). Before each imaging experiment, the suspension was diluted in PBS to a concentration of 200 cells / μL.

[0051] The exemplary embodiment of the image-based cell sorting system 200 used in the exemplary study included a spatial mask with 10 slits and utilized a cell flow velocity of approximately 0.08 m / s. The image area was set to 20 μm × 20 μm. With this design, the PMT signal from the cell's fluorescence appears to be 10 separate peaks, and the number of sampling points for each peak is approximately 50. However, since the velocities of different cells are not completely uniform, the number of sampling points for each cell varies slightly, and the starting point of each peak also needs to be determined. Therefore, so that the cell image can be successfully reconstructed, the data processing algorithm was configured to take such varying cell velocities into account, i.e., to search for the starting point and the number of sampling points for each peak within a certain range. Based on the variability of the cell flow velocity, the number of sampling points for each peak typically ranged from 46 to 51. Therefore, the total number of points in the cell image is 460 to 510. For each cell, once the particle detection module 461 determined that a cell was coming, a PMT signal of 520 sampling points in length was recorded. For example, to ensure the best combination of starting point and number of sampling points for each peak, the algorithm swept through the starting points of each peak with numbers of sampling points ranging from 46 to 51. For each combination in the sweep, the sum of the intensities at all starting points was calculated, and the combination with the smallest sum was selected as the correct answer for image reconstruction.

[0052] Figure 6 shows a data plot depicting the fluorescence intensity signal of a cell based on an exemplary image reconstruction. The exemplary results show searching for the best combination of starting point and number of sampling points for each peak.

[0053] FIG. 7 shows an example of a reconstructed image of a cell, where the criteria used to determine the cell boundary using, for example, the fluorescent area to generate an image of the cell. For the open filter, which is an erosion followed by a dilation, a 3×3 neighborhood was used. In the cell wall or membrane detection step, all pixels in the binary cell image are scanned. For a pixel with a non-zero intensity, the exemplary algorithm checked all 9 pixels in its 3×3 neighborhood. For example, if the counted number was greater than 0 and less than 8, this pixel was determined as a pixel on the cell wall or membrane by counting its neighboring pixels with a non-zero intensity.

[0054] This exemplary technique was implemented at least in part using an FPGA. Table 1 shows the rounded up approximate latency results of each step in the exemplary implementation of the data processing algorithm, with the total latency being well within 3.3 ms. The exemplary data processing algorithm is flexible to be implemented on multiple platforms, including in parallel. For example, taking advantage of the parallel processing power of a graphics processing unit (GPU), the algorithm can also be implemented on the GPU, for example, by either the CUDA architecture by Nvidia or OpenCL by AMD. Since the image soring algorithm has a very short execution time combined with the parallel processing power of the GPU, the sorting throughput is further improved. In this patent document, the formulation for generating cell images is described in the form of an exemplary method, but it is worth noting that the algorithm may be slightly different when using a spatial filter design different from the 10-slit spatial mask mentioned above. However, the overall working procedure of the system is the same.

[0055] Table 1 shows exemplary data depicting the performance of modules in an exemplary FPGA design, for example, latency expressed in time (ms). [Table 1]

[0056] Exemplary sorting results from an exemplary study are described. For example, to demonstrate the feasibility of the exemplary real-time image-based cell sorter system, a sorting test was performed on a mixture of cells with fluorescent cytoplasm and cells with fluorescent nuclei. Normal human embryonic kidney cells, HEK293T cells, after transfection with pEGFP-GR plasmid, express GFP, which can be excited by a 488 nm laser and has an emission peak at 509 nm in its cytoplasm. After 1 μM dexamethasone treatment for 1 hour, the fluorescence translocates from the cytoplasmic region to the cell nucleus.

[0057] Figures 8A and 8B show exemplary fluorescence microscopy images of transfected but drug-untreated cells (Figure 8A) and transfected and drug-treated cells (Figure 8B). The images in Figures 8A and 8B represent an area of ​​20 μm x 20 μm. As shown by the exemplary and representative microscopy images of transfected HEK293T cells without drug treatment (Figure 8A) and treated cells (Figure 8B), the magnitude of the fluorescence intensity of both treated and untreated cells has a fairly wide range, but the drug-treated cell images have a smaller fluorescence area.

[0058] As shown in the images of Figures 8A and 8B, the left column of both rows shows the fluorescent images, the center column shows the bright field images, and the right column shows the overlay images. Row 1 shows sample cells in which the fluorescence was distributed in the cytoplasm (e.g., untreated with dexamethasone). Row 2 shows sample cells in which the fluorescence was distributed in the nucleus (treated with dexamethasone). The cells are HEK293T human embroynic kidney cells transfected with pEGFP-GR plasmid.

[0059] 9A and 9B show exemplary fluorescent cell images taken by the image-based cell sorter system of transfected but non-drug treated cells (FIG. 9A) and transfected and drug treated cells (FIG. 9B). The exemplary images represent an area of ​​20 μm x 20 μm.

[0060] FIG. 10 shows exemplary calculated fluorescence area histograms of all events and sorted cells from an exemplary study.

[0061] In the exemplary study, the sorting criteria was set to 84 μm so that drug-treated cells were sorted by the exemplary system. 2 In this exemplary study, it is possible that not all drug-treated cells were sorted due to variability in cell size, nuclear size, and cellular uptake of the drug, but purity was ensured as all sorted cells had fluorescence only in the nucleus.

[0062] As demonstrated in an exemplary implementation of an exemplary embodiment of system 200, the disclosed systems and techniques provide a high-throughput flow cytometer with cell sorting capabilities based on single-cell fluorescent and light scattering images. By implementing the cell image capture, processing, and sorting operations in the FPGA and / or GPU, exemplary results show that the aforementioned design provides millisecond-level overall processing latency.

[0063] 11 shows a diagram of an image-based cell sorting microfluidic system 1100 according to some embodiments of the image-based particle sorting system 100. The system 1100 includes a microfluidic device 1110 for flowing particles through an optical interrogation channel for sorting, an imaging system 1120 for acquiring image data of the particles within an illuminated area of ​​the interrogation channel, a data processing and control system 1130 for processing the acquired image data in real time to determine a sorting command, and an actuator 1140 for gating the particles in the microfluidic device 210 based on the determined sorting command. In some implementations of the exemplary system 1110, the microfluidic device 1110 and the actuator 1140 can include the microfluidic device 210 and the actuator 240, respectively.

[0064] The imaging system 1120 of the system 1100 includes a light source 1121, e.g., a laser, for providing input or probe light to an illumination area of ​​the microfluidic device 1110, and an optical imager 1122 for acquiring images of illuminated particles in the illumination area 215. The exemplary optical imager 1122 includes a spatial filter (SF) 1124, absorption filters (EF) 1125A and 1125B, and an objective lens 1123 (e.g., of a microscope or other optical imaging device) optically coupled to a photomultiplier tube (PMT) 1126, as shown in FIG. In the exemplary implementation shown in FIG. 11, the imaging system 1120 includes dichroic mirrors (DM) 1129A and 1129B, where DM 1129A is disposed with the light source to direct input light to an illuminated area on the microfluidic device 1110, and DM 1129B is disposed in the optical path with optical imager 1122 to direct a portion of the optical output signal to PMT 1126B via EF 1125B, and an undirected portion of the optical output signal proceeds to PMT 1126A via EF 1125A. In some implementations, the light source 1121 (e.g., a laser) is configured to generate a fluorescence excitation signal that is incident on the illuminated area to cause fluorescence emission by the particles. The optical imager 1122 captures the optical output fluorescence emission signal at PMT 1126A and 1126B so that an image of the particles can be generated.

[0065] In an exemplary implementation of the system 1100, the suspended single cells are hydrodynamically focused in the sorting channel of the microfluidic device 1110 by the sheath flow, ensuring that the cells move at equal speed through the center of the fluidic channel. For example, both the fluorescence emission and the bright field signal can be detected by multiple photomultiplier tubes, e.g., PMT 1126A and 1126B. To accommodate the geometry of the microfluidic device, the 488 nm laser beam from the laser 1121 is introduced into the optical interrogation by 52° reflection by a small dichroic mirror 1129A placed in front of a 50x objective lens (e.g., NA=0.55, working distance=13 mm). In some exemplary embodiments, the system 1100 includes an optical light source 1128 (e.g., an LED, such as the exemplary 405 nm LED shown in FIG. 11) that can be placed on the opposite side of the channel to generate a bright field image, and the light can be focused at the laser illumination location. A spatially encoded filter, e.g., SF 1124, is inserted at the image plane in the detection path. Dichroic mirror 1129B splits the fluorescent and brightfield light collected by the objective based on spectrum to route the desired emission bands to their respective PMTs. The spatial resolution of the reconstructed image is determined by the filters. The resolution in the x (lateral) direction depends on the number of slits in the filter, and the resolution in the y (cell progression) direction depends on the sampling rate and the flow velocity of the cells. In the exemplary embodiment of system 1100 shown in FIG. 11, the effective pixel size is 2 μm in the x direction and about 0.4 μm in the y direction.

[0066] FIG. 12 illustrates a flow diagram of an exemplary image processing implementation of an exemplary embodiment of the system 1100. The data processing and control unit 1130 includes an image processing module for processing the fluorescent and bright field images captured by the imaging system 1120. For example, an exemplary image processing module for implementing aspects of the processes 420 and 430 of the method 400 includes pre-processing the received image signal data, for example, from the multiple PMTs 1126A and 1126B. In such an implementation, the pre-processing technique can be performed after the detection of a cell is determined based on an analysis of the received image signal data. In the example shown in FIG. 12, the PMT signals of the fluorescent and bright field images are low-pass filtered to remove high frequency noise. In some implementations, for example, a 10th order Hamming window is used for low-pass filtering. The exemplary image processing module is configured to execute an image reconstruction algorithm for reconstructing both the bright field and fluorescent images from the time domain PMT signals, for example, into a two-dimensional image. Since the bright field and fluorescent signals are generated by the same slit, they are synchronized by the data processing and control unit 1130. In some implementations, an image reconstruction algorithm is invoked once for both the bright field image and the fluorescence image. In some implementations, for example, the reconstructed image is resized to 50x50 pixels. In some implementations, for example, the grayscale image is converted to a binary image based on an intensity threshold. In some implementations, for example, the binary image is filtered by an open filter to remove spurious noise. In some implementations, for example, a cell contour algorithm is invoked to detect contours in both binary images. In some implementations, for example, morphological parameters are extracted based on the processed images, and a sorting decision is made based on the extracted parameters.

[0067] Figure 13A shows a flow diagram depicting exemplary data from the exemplary image processing steps shown in Figure 12. In the figure, the exemplary image processing module generates reconstructed and resized images of the detected individual cells including image contour features that allow for extraction of cell feature parameters such as cell morphology from the generated images.

[0068] The algorithm is implemented on an FPGA (National Instrument cRIO-9039). Table 2 shows an exemplary round-up estimated latency result of each step implemented by an exemplary image processing module of the data processing and control unit 1130. Image processing of brightfield and fluorescent images is performed in parallel. As shown in the table, in this example, the total latency of image processing is about 5.8 ms.

[0069] Table 2 shows example data depicting the performance of modules in an example FPGA design for an example implementation of system 1100, eg, latency expressed in time (ms). [Table 2]

[0070] For example, the time latency of image processing can be reduced, thereby improving the sorting throughput. For example, with a more powerful FPGA with more computational resources, the image processing module can be further parallelized to improve the time latency. The exemplary image processing algorithms can alternatively or additionally be implemented on the GPU (e.g., in parallel with the FPGA), such as by either the CUDA architecture by Nvidia or OpenCL by AMD. Because the exemplary image processing algorithms are data parallel, by utilizing the parallel processing capabilities of the GPU, the processing speed can be much accelerated, and as a result, much higher throughput can be achieved.

[0071] In some implementations, for example, the data processing and control unit 1130 can be configured to evaluate the extracted morphological parameters, for example using a receiver operating characteristic (ROC), and the top parameters are selected for real-time fractionation.

[0072] In some implementations, the method for evaluating the extracted morphological parameters using the ROC technique includes the following. For example, the ROC technique can be applied to the processed data from the image data obtained by the flow cell sample passing through the system. For example, the cell images can be separated into two subsets, for example, possibly by manual identification. As an example, the two subsets can be translocated cells (e.g., sorted group) and non-translocated cells (e.g., unsorted group). For each cell, a morphological parameter is extracted. The ROC technique can include generating a distribution of each parameter of both subsets. This technique includes generating a curve that is the ROC curve of the distribution. This technique includes calculating the area under the curve (AUC), i.e., the integral of the curve. This technique includes evaluating how the parameters apply to the classification, and for example, the parameter with the larger AUC can be selected as the most suitable parameter for classification.

[0073] 13B shows an exemplary distribution plot of the subsets using an exemplary receiver operating characteristic (ROC) technique for evaluating extracted parameters such as cell morphology parameters. In the example shown, the ROC includes TP, FP, FN, TN (TP is true positive, FP is false positive, FN is false negative, TN is true negative). The exemplary curve represents the ROC curve. The parameters with larger AUC are better for classification.

[0074] After the parameters for sorting have been selected, a support vector machine (SVM) can be used to generate a nonlinear hyperplane using the selected parameters as the gating criteria.

[0075] As an example, after the images are separated into subsets, such as two exemplary subsets, selected parameters are calculated for each image. Each image corresponds to an n-dimensional vector (n is the number of selected parameters). An exemplary SVM technique is implemented to generate a boundary in the n-dimensional space that separates the subsets. For example, when making a sorting decision, the selected parameters can be calculated for each cell, for example, to know which side of the boundary the cell is on (e.g., which subset the cell belongs to). Then, a sorting decision can be made.

[0076] Exemplary extracted morphological parameters are shown in Table 3. [Table 3]

[0077] An exemplary implementation of method 400 using system 1100 is described below with respect to an exemplary study demonstrating image-based cell sorting, including sorting based on protein translocation, cell cycle based sorting, and sorting based on the number of beads bound to the cell membrane.

[0078] One exemplary study involved the sorting of HEK-297T human embryonic kidney cells transposed with pEGFP-GR plasmid, for example, the implementation of protein transposition-based sorting. The exemplary study demonstrated the ability to distinguish and sort HEK-297T human embryonic kidney cells transposed with pEGFP-GR plasmid from transposed and non-transposed mixtures. In this study, HEK-293T human embryonic kidney cells were transfected with GR-GFP and separated into two plates. One plate of cells was untreated, so that the fluorescence remained in the cytoplasm. The other plate of cells was treated with a drug, so that the fluorescence moved from the cytoplasm to the nucleus. Both types of cells were mixed and flowed through the system, and based on the implementation of the exemplified method, only the transposed cells were sorted and collected.

[0079] In the exemplary study, the recorded PMT signals were processed by a data processing and control unit running algorithms implemented, for example, by Matlab code. Based on the PMT signals, both fluorescent and bright field images of each cell were reconstructed, and morphological parameters of each cell were extracted and recorded. The extracted morphological parameters were used for supervised machine learning to generate real-time image-based cell sorting criteria.

[0080] In an exemplary study, reconstructed cell images were separated into two subsets by manual identification: one subset of cells were non-translocated cells with fluorescence in the cytoplasm, and the other subset of cells were translocated cells with fluorescence in the nucleus.

[0081] Figures 14A and 14B show cell images of non-translocated and translocated cells captured by an exemplary system and reconstructed by Matlab code. Figure 14A shows an image of a non-translocated cell, and Figure 14B shows an image of a translocated cell. In the images of Figures 14A and 14B, the image on the left is a fluorescent image of the cell, the image in the middle is a bright-field image of the cell, and the image on the right is an overlay image containing the detected bright-field contours.

[0082] In the exemplary study, morphological parameters were evaluated based on the two annotated subsets using receiver operating characteristics (ROC). The top three parameters were selected for real-time sorting. For example, the top three parameters in this case were fluorescence area / bright field area (e.g., area ratio), fluorescence perimeter / bright field perimeter (e.g., perimeter ratio), and fluorescence area.

[0083] In an exemplary study, a sorting criterion using three selected parameters was used for real-time sorting of cells in the system. A three-dimensional nonlinear hyperplane was formed by a support vector machine (SVM) that separated two cell subsets based on the top three selected parameters.

[0084] FIG. 15 shows an example of a hyperplane exhibiting separation of two sets of cells (eg, translocated and non-translocated cells) for the implementation of the sorting criterion.

[0085] The two types of cells were mixed 50:50 and diluted to 200 / μL using phosphate buffered saline (PBS). The mixed sample was flowed through an exemplary image-based sorting system and sorted based on the real-time module. The cells moved at a speed of 8 cm / s given by a sample flow rate of 6 μL / min and a sheath flow rate of 60 μL / min. The exemplary imager included a microscope with a CCD camera that was used to capture both fluorescent and bright field images of the collected cells.

[0086] Figure 16A and Figure 16B show cell images of non-translocated cells and translocated cells captured by an exemplary system through a microscope. Figure 16A shows a microscope image of a non-translocated cell, and Figure 16B shows a microscope image of a translocated cell. In the images of Figure 16A and Figure 16B, the image on the left is a fluorescent image of the cell captured by a microscope, the image in the middle is a bright field image of the cell by a microscope, and the image on the right is an overlay image.

[0087] One exemplary study involved the implementation of cell cycle-based sorting, such as sorting MDCK Madin-Darby canine kidney epithelial cells in the G2 / M phase. In this study, MDCK Madin-Darby canine kidney epithelial cells were fixed and cell nuclei were stained with propidium iodide (PI). The fixed and stained MDCK cells were run through an exemplary image-based sorting system, and only G2 / M phase cells were sorted and collected.

[0088] Similar to the exemplary study for protein translocation-based sorting, the recorded PMT signals were processed to reconstruct cell images and extract morphological parameters. The reconstructed cell images were separated into two subsets by manual identification. One subset of cells was in the G1 phase, in which nuclear components are confined within the nuclear membrane. The other subset of cells was in the G2 / M phase, in which the nuclear membrane breaks down and nuclear components are distributed intracellularly.

[0089] Figures 17A and 17B show cell images of G1 and G2 / M phase cells captured by an exemplary system and reconstructed by Matlab code. Figure 17A shows an image of a G1 phase cell, and Figure 17B shows an image of a G2 / M phase cell. In the images of Figures 17A and 17B, the image on the left is a fluorescent image of the cell, the image in the middle is a bright field image of the cell, and the image on the right is an overlay image containing the detected bright field contours.

[0090] Similar to the exemplary study for protein translocation-based fractionation, morphological parameters were evaluated using ROC and a three-dimensional nonlinear hyperplane was formed by SVM. The top three morphological parameters were fluorescence area / bright field area (e.g., area ratio), fluorescence area, and fluorescence perimeter / bright field perimeter (e.g., perimeter ratio).

[0091] FIG. 18 shows an example of a hyperplane exhibiting separation of two sets of cells (eg, cells in G1 phase and cells in G2 / M) for the implementation of the sorting criterion.

[0092] MDCK cells were diluted to 200 / μL using PBS. Samples were flowed through the exemplary system and sorted based on the real-time module. Cells moved at a speed of 8 cm / s given by a sample flow rate of 6 μL / min and a sheath flow rate of 60 μL / min. The exemplary imager included a microscope with a CCD camera that was used to capture both fluorescent and bright field images of collected cells.

[0093] Figures 19A and 19B show cell images of G1 and G2 / M cells captured by an exemplary system via a microscope. Figure 19A shows a microscope image of a G1 cell, and Figure 19B shows a microscope image of a G2 / M cell. In the images of Figures 19A and 19B, the image on the left is a fluorescent image of the cell captured by the microscope, the image in the middle is a bright field image of the cell by the microscope, and the image on the right is an overlay image.

[0094] One exemplary study included the implementation of sorting HEK-297T human embryonic kidney cells based on the number of beads bound to the cells, e.g., on the cell membrane. In this study, HEK-297T human embryonic kidney cells were bound with fluorescent beads and stained with a CFSE kit. The fluorescence of the cells is at 520 nm and the fluorescence of the beads is at 645 nm. The two wavelengths of fluorescent signals were routed by a dichroic mirror and detected by two PMTs. The cells were sorted based on the number of bound beads. The image processing module for this application was modified accordingly to process the images.

[0095] FIG. 20 shows a flow diagram of an exemplary image processing implementation for sorting based on the number of beads bound on the cell membrane of cells implemented by an exemplary embodiment of the system 1100. The data processing and control unit 1130 includes an image processing module for processing the fluorescent and bright field images captured by the imaging system 1120. For example, an exemplary image processing module for implementing aspects of processes 420 and 430 of the method 400 includes pre-processing the received image signal data, for example, from the multiple PMTs 1126A and 1126B. As shown, for example, the fluorescent signals at both wavelengths were low-pass filtered. In this exemplary study, a 520 nm fluorescent signal was used for image reconstruction. The reconstructed bead images were resized to 50×50 pixels. A top-hat transformation was implemented to remove the image background. In this exemplary implementation, a 7×7 pixel neighborhood was used for the top-hat transformation. In this exemplary implementation, the grayscale images were converted to binary images. Morphological parameters were extracted. In this example, the beads have a relatively uniform size, so the image area was selected as the sorting criterion. In some implementations, for example, morphological parameters are extracted based on the processed images and fractionation decisions are made based on the extracted parameters.

[0096] Table 4 shows that the total latency of real-time image processing is 7.8 ms. An exemplary module was implemented using FPGA for bead counting. [Table 4]

[0097] Figure 21 shows examples of grayscale cell images processed by image processing modules with different numbers of beads. As shown in Figure 21, the images in the left column are fluorescent images of beads, the images in the middle column are fluorescent images of cells, and the images in the right column are overlay images.

[0098] FIG. 22 shows the histograms of bead image area for different numbers of beads. EXAMPLES

[0099] The following examples are illustrative of some embodiments according to the present technology. Other exemplary embodiments of the present technology may be presented before or after the examples listed below.

[0100] In some embodiments (Example A1) according to the present technology, an image-based particle sorting system includes a particle flow device constructed to include a substrate, a flow channel formed on the substrate, the flow channel operable to flow particles along a flow direction into a first region of the flow channel, and two or more output paths branching from the flow channel at a second region within the flow channel proximate to the first region; an imaging system interfaced with the particle flow device and operable to acquire image data associated with the particle when the particle is within the first region during flow through the flow channel; and a data processing and control unit in communication with the imaging system, the data processing and control unit processing the image data acquired by the imaging system, a data processing and control unit including a processor configured to determine one or more properties associated with the particles from the processed image data and generate a control command based on a comparison of the determined one or more properties with a sorting criterion; and an actuator operatively coupled to the particle flow device and in communication with the actuator, the actuator operable to direct the particles to one of two or more output paths based on the control command, wherein the system is operable to sort each of the particles as they flow through the flow path within a time frame of 15 ms or less from a first time of image capture by the imaging system to a second time of particle direction by the actuator.

[0101] Example A2 includes the system of example A1, where the particle flow device includes a microfluidic device or flow cell integrated with an actuator on a substrate of the microfluidic device or flow cell.

[0102] Example A3 includes the system of Example A1, wherein the actuator includes a piezoelectric actuator coupled to the substrate and operable to cause a deflection that moves the particle in a direction within the second region that directs the particle along a trajectory to one of the two or more output paths.

[0103] Example A4 includes the system of Example A1, wherein the imaging system includes one or more light sources for providing input light to a first region of the particle flow device and an optical imager for capturing image data from particles illuminated by the input light in the first region.

[0104] Example A5 includes the system of example A4, in which the one or more light sources include at least one of a laser or a light emitting diode (LED).

[0105] Example A6 includes the system of example A4, where the optical imager includes a microscope objective optically coupled to a spatial filter, an absorptive filter, and a photomultiplier tube.

[0106] Example A7 includes the system of Example A6, wherein the optical imager further includes one or more light-guiding elements for directing input light to the first region, for directing light emitted or scattered by the particles to optical elements of the optical imager, or both.

[0107] Example A8 includes the system of example A7, in which the light-directing element includes a dichroic mirror.

[0108] Example A9 includes the system of example A6, wherein the optical imager includes two or more photomultiplier tubes for generating two or more corresponding signals based on two or more bands or types of light emitted or scattered by the particle.

[0109] Example A10 includes the system of example A1, in which the processor of the data processing and control unit includes a field programmable gate array (FPGA), a graphics processing unit (GPU), or a parallel FPGA and GPU.

[0110] Example A11 includes the system of Example A1, wherein the data processing and control unit is configured to receive image data (e.g., including time domain signal data) relating to particles imaged within a first region on the particle flow device, process the image data to generate an image dataset representing an image of the particles, analyze the generated image dataset to extract one or more parameters from the image dataset related to one or more characteristics of the particles, and determine a control command based on a comparison of the extracted one or more parameters with one or more thresholds of a sorting criterion.

[0111] Example A12 includes the system of Example A11, wherein the data processing and control unit is configured to process the image data to generate an image dataset by filtering the image data, reconstructing a first image based on the filtered data, and resizing the reconstructed first image to generate a second image, wherein the second image includes binary image data.

[0112] Example A13 includes the system of Example A1, wherein the particle includes a cell, and the one or more properties associated with the cell include an amount or size of a feature of the cell or a feature on the cell, one or more subparticles attached to the cell, or a particular morphology of the cell or a portion of a cell.

[0113] Example A14 includes the system of example A1, wherein the particles include cells and the sorting criteria include cell contour, cell size, cell shape, nucleus size, nucleus shape, fluorescence pattern, or fluorescence color distribution.

[0114] Example A15 includes the system of Example A1, wherein the particle includes a cell and the one or more properties associated with the cell include a physiological property of the cell including a cell cycle phase, expression or localization of a protein by the cell, damage to the cell, or phagocytosis of a substance or subparticle by the cell.

[0115] In some embodiments (Example A16) according to the present technology, an image-based method for sorting particles includes acquiring image signal data of particles flowing through a flow path of a particle flow device, processing the image signal data to generate an image dataset representing an image of the particles, analyzing the generated image dataset to identify one or more characteristics of the particles from the processed image data, generating a control command by evaluating the one or more identified characteristics on a sorting basis, and directing the particles to one of a plurality of output paths of the particle flow device based on the control command.

[0116] Example A17 includes the method of example A16, wherein the acquiring, processing, analyzing, generating, and directing operations are performed within a time frame of 15 ms or less while the cells are flowing through the channel.

[0117] Example A18 includes the method of example A16, wherein generating the control command includes extracting one or more parameters from the processed image data of the particles related to the identified one or more characteristics of the particles, and comparing the extracted one or more parameters from the image to one or more thresholds of the sorting criteria.

[0118] Example A19 includes the method of example A16, wherein processing the image signal data to generate an image dataset includes filtering the image signal data, reconstructing a first image based on the filtered data, and resizing the reconstructed first image to generate a second image, the second image including binary image data.

[0119] Example A20 includes the method of example A16, in which processing the image signal data includes detecting a presence of a particle before generating an image data set representing an image of the particle.

[0120] Example A21 includes the method of example A20, where detecting the presence of a particle includes calculating a brightness value related to a magnitude of signal intensity of the image signal data, evaluating the brightness value with a first threshold, determining a derivative value when the brightness value exceeds the first threshold, evaluating that the derivative value exceeds a second threshold, and determining that the derivative value exceeds the second threshold.

[0121] Example A22 includes the method of Example A16, wherein the particles flowing through the flow path are cells, and the one or more properties associated with the cells include an amount or size of a feature of the cell or a feature on the cell, one or more subparticles attached to the cell, or a particular morphology of the cell or a portion of a cell.

[0122] Example A23 includes the method of example A16, wherein the particles flowing through the flow path are cells, and the sorting criteria include cell contour, cell size, cell shape, nucleus size, nucleus shape, fluorescence pattern, or fluorescence color distribution.

[0123] Example A24 includes the method of Example A16, wherein the particles flowing through the flow path are cells, and the one or more properties associated with the cells include physiological properties of the cells including cell cycle phase, expression or localization of a protein by the cell, damage to the cell, or phagocytosis of a substance or subparticle by the cell.

[0124] In some embodiments (Example A25) according to the present technology, a method for obtaining image-based cell sorting in a flow cytometry device includes recording an image, analyzing the recorded image to determine whether a cell is detected, and when a determination that a cell is detected is determined, processing the image, and making a decision based on a result of the processing whether to trigger an actuator in the flow cytometry device.

[0125] Example A26 includes the method of example A25, further comprising resuming image recording when it is determined that no cells are detected.

[0126] Example A27 includes the method of example A25, where determining whether a cell is detected includes calculating a brightness value; determining whether the brightness value exceeds a first threshold; if a determination is made that the brightness value exceeds the first threshold, determining a derivative value; determining whether the derivative value exceeds a second threshold; and if a determination is made that the derivative value exceeds the second threshold, providing an indication that a cell is detected, and if no determination is made that the brightness value exceeds the first threshold or the derivative value exceeds the second threshold, providing an indication that a cell is not detected or providing no indication.

[0127] In some embodiments (Example A28) according to the present technology, a particle imaging and flow system includes a substrate; a flow path formed on the substrate, the flow path operable to flow particles along a flow direction into a first region of the flow path; an imaging system interfaced with the particle flow device and operable to acquire image data associated with the particles when the particles are in the first region while flowing through the flow path; and a data processing and control unit in communication with the imaging system, the data processing and control unit configured to process the image data acquired by the imaging system to determine one or more properties associated with the particles and generate an analyzed dataset including data indicative of the determined one or more properties of the particles.

[0128] Example A29 includes the system of example A28, wherein the particle flow device includes a microfluidic device or a flow cell.

[0129] Example A30 includes the system of example A28, wherein the imaging system includes one or more light sources for providing input light to a first region of the particle flow device and an optical imager for capturing image data from particles illuminated by the input light in the first region.

[0130] Example A31 includes the system of example A30, wherein the one or more light sources include at least one of a laser or a light emitting diode (LED).

[0131] Example A32 includes the system of example A30, where the optical imager includes a microscope objective optically coupled to a spatial filter, an absorptive filter, and a photomultiplier tube.

[0132] Example A33 includes the system of example A32, wherein the optical imager further includes one or more light-guiding elements (e.g., dichroic mirrors) for directing input light to the first region, for directing light emitted or scattered by the particles to optical elements of the optical imager, or both.

[0133] Example A34 includes the system of example A30, wherein the optical imager includes two or more photomultiplier tubes for generating two or more corresponding signals based on two or more bands or types of light emitted or scattered by the particle.

[0134] Example A35 includes the system of example A28, in which the processor of the data processing and control unit includes a field programmable gate array (FPGA), a graphics processing unit (GPU), or a parallel FPGA and GPU.

[0135] Example A36 includes the system of example A28, wherein the data processing and control unit is configured to receive image data (e.g., including time domain signal data) relating to particles imaged within a first region on the particle flow device, process the image data to generate an image dataset representing an image of the particles, and analyze the generated image dataset to extract one or more parameters from the image dataset relating to one or more characteristics of the particles.

[0136] Example A37 includes the system of example A36, wherein the data processing and control unit is configured to process the image data to generate an image dataset by filtering the image data, reconstructing a first image based on the filtered data, and resizing the reconstructed first image to generate a second image, wherein the second image includes binary image data.

[0137] Example A38 includes the system of Example A28, wherein the particle includes a cell, and the one or more properties associated with the cell include an amount or size of a feature of the cell or a feature on the cell, one or more subparticles attached to the cell, or a particular morphology of the cell or a portion of a cell.

[0138] Example A39 includes the system of example A28, wherein the particles include cells and the sorting criteria include cell contour, cell size, cell shape, nuclear size, nuclear shape, fluorescence pattern, or fluorescence color distribution.

[0139] Example A40 includes the system of Example A28, wherein the particle includes a cell and the one or more properties associated with the cell include a physiological property of the cell including a cell cycle phase, expression or localization of a protein by the cell, damage to the cell, or phagocytosis of a substance or subparticle by the cell.

[0140] In some embodiments (Example B1) according to the present technology, the system includes a microfluidic device integrated with an on-chip sorting actuator, high speed and high sensitivity imaging optics, and a real-time cell imaging and sorting control electronic system.

[0141] Example B2 includes the system of Example B1, where the real-time cell imaging and sorting control electronic system is configured to allow one or more user-defined gating criteria.

[0142] Example B3 includes the system of example B1, wherein the one or more user-defined gating criteria include cell contour, cell size, cell shape, nuclear size and shape, fluorescence pattern, or fluorescence color distribution.

[0143] In some embodiments (Example B4) according to the present technology, a method for obtaining image-based cell sorting in a flow cytometry device includes recording an image, determining whether a cell is detected, processing the image if it is determined that a cell is detected, and making a decision based on the results of the processing whether to trigger an actuator in the flow cytometry device.

[0144] Example B5 includes the method of example B4, further comprising resuming image recording when it is determined that no cells are detected.

[0145] Example B6 includes the method of example B4, wherein determining whether a cell is detected includes calculating a brightness value, and if it is determined that the brightness value exceeds a first threshold, calculating a derivative value, and if it is determined that the derivative value exceeds a second threshold, providing an indication that a cell is detected.

[0146] Example B7 includes the method of example B4, which is implemented in one of an FPGA or a DSP.

[0147] Example B8 includes the system of Example B1, in which the real-time cell image processing and sorting control electronic system is implemented in an FPGA.

[0148] In some embodiments (Example C1) according to the present technology, an image-based particle sorting system includes a particle flow device constructed to include a substrate, a flow channel formed on the substrate, the flow channel operable to flow cells along a flow direction into a first region of the flow channel, and two or more output paths branching from the flow channel at a second region in the flow channel proximate to the first region; an imaging system interfaced with the particle flow device and operable to acquire image data associated with the cells when the cells are in the first region during flow through the flow channel; and a data processing and control unit in communication with the imaging system, the data processing and control unit processing the image data acquired by the imaging system to determine one or more characteristics associated with the cells from the processed image data, and determining the determined characteristics. The system includes a data processing and control unit including a processor configured to generate control commands based on a comparison of the one or more characteristics to a sorting criterion, the control commands being generated while the cells are flowing through the flow path and indicating a sorting decision determined based on one or more cell attributes ascertained from image signal data corresponding to the cells; and an actuator operatively coupled to the particle flow device and in communication with the actuator, the actuator operable to direct the cells to one of two or more output paths based on the control command, wherein the system is operable to sort each of the cells while they are flowing through the flow path within a time frame of 15 ms or less from a first time of image capture by the imaging system to a second time of particle direction by the actuator.

[0149] Example C2 includes the system of example C1, where the particle flow device includes a microfluidic device or flow cell integrated with an actuator on a substrate of the microfluidic device or flow cell.

[0150] Example C3 includes the system of Example C1, wherein the actuator includes a piezoelectric actuator coupled to the substrate and operable to generate a deflection that moves the cell in a direction within a second region that directs the cell along a trajectory to one of two or more output paths.

[0151] Example C4 includes the system of example C1, wherein the imaging system includes one or more light sources for providing input light to a first region of the particle flow device and an optical imager for capturing image data from cells illuminated by the input light in the first region.

[0152] Example C5 includes the system of example C4, in which the one or more light sources include at least one of a laser or a light emitting diode (LED).

[0153] Example C6 includes the system of example C4, where the optical imager includes a microscope objective optically coupled to a spatial filter, an absorptive filter, and a photomultiplier tube.

[0154] Example C7 includes the system of example C6, wherein the optical imager further includes one or more light-guiding elements for directing input light to the first region, for directing light emitted or scattered by the cells to optical elements of the optical imager, or both.

[0155] Example C8 includes the system of example C7, in which the light directing element includes a dichroic mirror.

[0156] Example C9 includes the system of example C6, wherein the optical imager includes two or more photomultiplier tubes for generating two or more corresponding signals based on two or more bands or types of light emitted or scattered by the cells.

[0157] Example C10 includes the system of example C1, in which the processor of the data processing and control unit includes a field programmable gate array (FPGA), a graphics processing unit (GPU), or a parallel FPGA and GPU.

[0158] Example C11 includes the system of Example C1, wherein the data processing and control unit is configured to receive image data including time domain signal data relating to cells imaged within a first region on the particle flow device, process the image data to generate an image dataset representing an image of the cells, analyze the generated image dataset to extract one or more parameters from the image dataset related to one or more characteristics of the cells, and determine a control command based on a comparison of the extracted one or more parameters with one or more thresholds of a sorting criterion.

[0159] Example C12 includes the system of example C11, wherein the data processing and control unit is configured to process the image data to generate an image dataset by filtering the image data, reconstructing a first image based on the filtered data, and resizing the reconstructed first image to generate a second image, wherein the second image includes binary image data.

[0160] Example C13 includes the system of Example C1, wherein the one or more characteristics associated with the cell include one or more of a feature of the cell or an amount or size of a feature on the cell, one or more particles attached to the cell, or a particular morphology of the cell or a portion of the cell.

[0161] Example C14 includes the system of example C1, wherein the sorting criteria includes cell contour, cell size, cell shape, nuclear size, nuclear shape, fluorescence pattern, or fluorescence color distribution.

[0162] Example C15 includes the system of Example C1, wherein the one or more properties associated with the cell include a physiological property of the cell including a cell cycle phase, expression or localization of a protein by the cell, expression or localization of a gene by the cell, damage to the cell, or phagocytosis of a substance or particle by the cell.

[0163] Example C16 includes the system of Example C15, wherein the determined damage to the cells includes DNA damage.

[0164] In some embodiments (Example C17) according to the present technology, an image-based particle sorting method includes acquiring image signal data of cells flowing through a flow path of a particle flow device, processing the image signal data to generate an image dataset representing an image of the cells, analyzing the generated image dataset to identify one or more characteristics of the cells from the processed image data, evaluating the one or more identified characteristics of the cells with a sorting criterion to generate a control command to sort the cells based on one or more cell attributes identified from the image signal data corresponding to the cells during cell flow in the particle flow device, and directing the cells to one of a plurality of output paths of the particle flow device based on the control command.

[0165] Example C18 includes the method of example C17, wherein the acquiring, processing, analyzing, generating, and directing operations are performed within a time frame of 15 ms or less while the cells are flowing through the channel.

[0166] Example C19 includes the method of example C17, wherein generating the control command includes extracting one or more parameters from the processed image data of the cell related to one or more identified characteristics of the cell, and comparing the extracted one or more parameters from the image with one or more thresholds of the sorting criteria.

[0167] Example C20 includes the method of example C17, wherein processing the image signal data to generate an image dataset includes filtering the image signal data, reconstructing a first image based on the filtered data, and resizing the reconstructed first image to generate a second image, wherein the second image includes binary image data.

[0168] Example C21 includes the method of example C17, in which processing the image signal data includes detecting a presence of a cell before generating an image dataset representing an image of the cell.

[0169] Example C22 includes the method of example C21, where detecting the presence of a cell includes calculating a brightness value related to a magnitude of signal intensity of the image signal data, evaluating the brightness value with a first threshold, determining a derivative value when the brightness value exceeds the first threshold, evaluating that the derivative value exceeds a second threshold, and determining that the derivative value exceeds the second threshold.

[0170] Example C23 includes the method of Example C17, wherein the one or more properties associated with the cell include one or more of a feature of the cell or an amount or size of a feature on the cell, one or more particles attached to the cell, or a particular morphology of the cell or a portion of a cell.

[0171] Example C24 includes the method of example C17, wherein the sorting criteria includes cell contour, cell size, cell shape, nuclear size, nuclear shape, fluorescence pattern, or fluorescence color distribution.

[0172] Example C25 includes the method of Example C17, wherein the one or more properties associated with the cell include a physiological property of the cell including cell cycle phase, expression or localization of a protein by the cell, expression or localization of a gene by the cell, damage to the cell, or phagocytosis of a substance or particle by the cell.

[0173] Example C26 includes the method of Example C25, wherein the determined damage to the cells includes DNA damage.

[0174] Implementations of the subject matter and functional operations described in this patent document can be implemented in various systems, digital electronic circuits, or computer software, firmware, or hardware, or one or more combinations thereof, including the structures disclosed herein and their structural equivalents. Implementations of the subject matter described herein can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory computer-readable medium for execution by or for controlling the operation of a data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of material effecting a machine-readable propagated signal, or one or more combinations thereof. The term "data processing unit" or "data processing device" encompasses all apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus can include, in addition to hardware, code that creates an environment for the execution of the computer program, such as code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations of these.

[0175] A computer program (also called a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple linked files (e.g., a file that stores one or more modules, subprograms, or portions of code). A computer program can be deployed to be executed on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0176] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs that perform functions by manipulating input data to generate output. The processes and logic flows may also be performed by, and an apparatus may be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0177] Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operatively connected to receive data from or transmit data to, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0178] This specification, together with the drawings, are intended to be considered illustrative, and by illustrative I mean exemplary. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly dictates otherwise. Additionally, the use of "or" is intended to include "and / or" unless the context clearly dictates otherwise.

[0179] Although this patent document contains many details, these should not be construed as limitations on the scope of any invention or claimed subject matter, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features that are described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in a particular combination and may initially be claimed as such, in some cases one or more features can be deleted from the claimed combination, and the claimed combination may relate to a subcombination or a variation of the subcombination.

[0180] Similarly, although operations are depicted in a particular order in the figures, this should not be understood as requiring that such operations be performed in the particular order or sequence shown, or that all illustrated operations be performed, to achieve desirable results.Furthermore, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0181] Various embodiments described herein are described in the general context of a method or process, which in one embodiment may be implemented by a computer program product including computer-executable instructions, such as program code, embodied in a computer-readable medium and executed by a computer in a network environment. Computer-readable media include removable and non-removable storage devices, including but not limited to read-only memory (ROM), random access memory (RAM), compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs, and the like. Thus, the computer-readable media described in this application includes non-transitory storage media. Generally, program modules may include routines, programs, objects, components, data structures, and the like that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of program code for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps or processes.

[0182] For example, one aspect of the disclosed embodiments relates to a computer program product embodied on a non-transitory computer-readable medium, the computer program product including program code for performing any one and / or all of the operations of the disclosed embodiments.

[0183] Only some implementations and examples have been described; other implementations, extensions, and variations can be made based on what is described and illustrated in this patent document. The present invention provides the following: (Item 1) 1. An image-based particle sorting system, comprising: a particle flow device constructed to include a substrate; a flow channel formed on the substrate, the flow channel operable to flow cells along a flow direction into a first region of the flow channel; and two or more output paths branching from the flow channel at a second region within the flow channel proximate to the first region; an imaging system interfaced with the particle flow device and operable to acquire image data associated with the cell when the cell is within the first region while flowing through the flow path; a data processing and control unit in communication with the imaging system, the data processing and control unit including a processor configured to process the image data acquired by the imaging system to determine one or more properties associated with the cells from the processed image data and to generate control commands based on a comparison of the determined one or more properties to a sorting criteria, the control commands being generated while the cells are flowing through the flow channel and indicative of a sorting decision determined based on one or more cell attributes ascertained from image signal data corresponding to the cells; an actuator operatively coupled to the particle flow device and in communication with the actuator, the actuator operable to direct the cells to one of the two or more output paths based on the control command; The system is operable to sort each of the cells while they are flowing through the channel within a time frame of 15 ms or less from a first time of image capture by the imaging system to a second time of particle orientation by the actuator. (Item 2) 2. The system of claim 1, wherein the particle flow device includes a microfluidic device or flow cell, and the microfluidic device or flow cell is integrated with the actuator on a substrate of the microfluidic device or flow cell. (Item 3) 2. The system of claim 1, wherein the actuator comprises a piezoelectric actuator coupled to the substrate and operable to create a deflection that moves the cell in a direction within the second region that directs the cell along a trajectory to the one of the two or more output paths. (Item 4) 2. The system of claim 1, wherein the imaging system includes one or more light sources for providing input light to the first region of the particle flow device and an optical imager for capturing the image data from the cells illuminated by the input light in the first region. (Item 5) 5. The system of claim 4, wherein the one or more light sources include at least one of a laser or a light emitting diode (LED). (Item 6) 5. The system of claim 4, wherein the optical imager includes a microscope objective lens optically coupled to a spatial filter, an absorption filter, and a photomultiplier tube. (Item 7) 7. The system of claim 6, wherein the optical imager further comprises one or more light directing elements for directing the input light to the first region, for directing light emitted or scattered by the cells to optical elements of the optical imager, or both. (Item 8) Item 8. The system of item 7, wherein the light directing element comprises a dichroic mirror. (Item 9) 7. The system of claim 6, wherein the optical imager comprises two or more photomultiplier tubes for generating two or more corresponding signals based on two or more bands or types of light emitted or scattered by the cells. (Item 10) 2. The system of claim 1, wherein the processor of the data processing and control unit comprises a field programmable gate array (FPGA), a graphics processing unit (GPU), or a parallel FPGA and GPU. (Item 11) 2. The system of claim 1, wherein the data processing and control unit is configured to receive the image data including time domain signal data associated with the cells imaged within the first region on the particle flow device, process the image data to generate an image dataset representing an image of the cells, analyze the generated image dataset to extract one or more parameters from the image dataset related to the one or more characteristics associated with the cells, and determine the control command based on a comparison of the extracted one or more parameters to one or more thresholds of the sorting criteria. (Item 12) 12. The system of claim 11, wherein the data processing and control unit is configured to process the image data to generate the image dataset by filtering the image data, reconstructing a first image based on the filtered data, and resizing the reconstructed first image to generate a second image, the second image comprising binary image data. (Item 13) 2. The system of claim 1, wherein the one or more characteristics associated with the cell include one or more of a feature of the cell or an amount or size of a feature on the cell, one or more particles attached to the cell, or a particular morphology of the cell or a portion of the cell. (Item 14) 2. The system of claim 1, wherein the sorting criteria include cell contour, cell size, cell shape, nuclear size, nuclear shape, fluorescence pattern, or fluorescence color distribution. (Item 15) The system of claim 1, wherein the one or more properties associated with the cell include a physiological property of the cell including a cell cycle phase, expression or localization of a protein by the cell, expression or localization of a gene by the cell, damage to the cell, or phagocytosis of a substance or particle by the cell. (Item 16) 16. The system of claim 15, wherein the determined damage to the cell comprises DNA damage. (Item 17) 1. A method for image-based particle sorting, comprising: acquiring image signal data of cells flowing through a flow channel of the particle flow device; processing the image signal data to generate an image dataset representative of an image of the cell; analyzing the generated image dataset to identify one or more characteristics of the cells from the processed image data; evaluating the one or more identified characteristics of the cells on a sorting basis to generate control commands to sort the cells based on one or more cell attributes ascertained from the image signal data corresponding to the cells during cell flow in the particle flow device; and directing the cells to one of a plurality of output paths of the particle flow device based on the control command. (Item 18) 20. The method of claim 17, wherein the acquiring, processing, analyzing, generating, and directing operations are performed within a time frame of 15 ms or less while the cells are flowing through the channel. (Item 19) 20. The method of claim 17, wherein generating the control command comprises extracting one or more parameters from the processed image data of the cells related to the identified one or more characteristics of the cells, and comparing the one or more parameters extracted from the images to the one or more thresholds of the sorting criteria. (Item 20) Processing the image signal data to generate the image data set comprises: filtering the image signal data; reconstructing a first image based on the filtered data; and 20. The method of claim 17, further comprising: resizing the reconstructed first image to generate a second image, the second image comprising binary image data. (Item 21) 18. The method of claim 17, wherein processing the image signal data comprises detecting the presence of the cell before generating the image dataset representing the image of the cell. (Item 22) Detecting the presence of the cell includes: calculating a luminance value related to a magnitude of signal strength of the image signal data; evaluating the luminance value with a first threshold; determining a derivative value when the luminance value exceeds the first threshold; evaluating when the differential value exceeds a second threshold; and and determining that the differential value exceeds the second threshold. (Item 23) 20. The method of claim 17, wherein the one or more properties associated with the cell include one or more of a feature of the cell or an amount or size of a feature on the cell, one or more particles attached to the cell, or a particular morphology of the cell or a portion of the cell. (Item 24) 18. The method of claim 17, wherein the sorting criteria comprises cell contour, cell size, cell shape, nuclear size, nuclear shape, fluorescence pattern, or fluorescence color distribution. (Item 25) 18. The method of claim 17, wherein the one or more properties associated with the cell include a physiological property of the cell including cell cycle phase, expression or localization of a protein by the cell, expression or localization of a gene by the cell, damage to the cell, or phagocytosis of a substance or particle by the cell. (Item 26) 26. The method of claim 25, wherein the determined damage to the cell comprises DNA damage.

Claims

[Claim 1] The invention as depicted in the drawings.

Images