Image-based unsupervised multi-model cell clustering

Abstract

This specification describes a framework that includes one or more feature extractors (models) and a cluster component for clustering. The framework supports (1) offline, image-based, unsupervised clustering, which replaces time-consuming manual gating, and (2) online, image-based single-cell sorting. During training, one or more cell image datasets, with or without ground truth, are used to train the one or more feature extractors. The feature extractors are based on neural networks containing several convolutional layers. Once trained, the feature extractors are used to extract cell image features for unsupervised cell clustering and sorting. Additionally, after the feature extractors are trained, they can be further refined using additional datasets.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation-in-part of co-pending U.S. patent application Ser. No. 17 / 222,131, entitled "A FRAMEWORK FOR IMAGE BASED UNSUPERVISED CELL CLUSTERING AND SORTING," filed April 5, 2021, and claims priority under 35 U.S.C. § 119 of U.S. Provisional Patent Application No. 63 / 116,065, entitled "UNSUPERVISED LEARNING FRAMEWORK FOR IMAGE BASED SINGLE CELL SORTING," filed November 19, 2020, which applications are incorporated herein by reference in their entireties for all purposes.

[0002] The present invention relates to cell sorting, and more particularly to image-based cell sorting. [Background technology]

[0003] Conventional fluorescence-activated cell sorting relies on labeling cells with fluorescent markers, which provides very limited morphological information. However, some applications require morphological information to accurately sort cells, and some applications do not allow for the use of fluorescent markers. Furthermore, conventional fluorescence-activated cell sorting (FACS) requires manual gating to establish sorting criteria based on fluorescent markers. However, manual gating is time-consuming and can introduce bias.

[0004] Several studies have proposed image-based cell sorting using deep neural networks or supervised learning based on manually created features. These studies assumed ground-truth cell images for training, which may not be available. Some software to assist the gating process relies on specific manually created features of fluorescent markers, but morphological information may not be sufficient for some applications and may not be suitable for some others. Summary of the Invention [Problem to be solved by the invention]

[0005] Image-based unsupervised multi-model cell clustering further develops clustering strategies to support the unsupervised cell clustering framework. The expanded framework has a model repository containing multiple models, such as a common fluorescence model, a specific fluorescence model, a specific bright-field model, and a single-cell isolation bright-field model. Multi-model clustering uses multiple models to extract cellular features, which can be combined for clustering. Models are selected based on the target application and the image channel of interest. Image-based unsupervised multi-model cell clustering is unique in that applications can use one or more models trained for different purposes to extract features for each cell. [Means for solving the problem]

[0006] In one aspect, a method includes performing offline initial image-based unsupervised clustering, training a plurality of models, each model of the plurality of models designed to extract features of each cell, and performing online image-based single-cell sorting. Each model of the plurality of models is different. The plurality of models is stored in a model repository. Each model of the plurality of models is based on a multilayer neural network. For a set of cell images, performing offline initial clustering includes using the plurality of models to extract features of the cell images, training a cluster component in an unsupervised manner using a given small subset of the cell images, and using the cluster component to determine a cluster for each given cell. Performing online image-based single-cell sorting includes utilizing the plurality of models to extract features of the cell images and using the cluster component to determine a cluster for each given cell. Clustering separates and groups different types of cells based on the extracted features. An example of a clustering algorithm is hierarchical density-based spatial clustering.

[0007] In another aspect, an apparatus includes: a non-transitory memory for storing an application for performing offline initial image-based unsupervised clustering; training a plurality of models, each model of the plurality of models designed to extract features of each cell; and performing online image-based single-cell sorting; and a plurality of processing units configured to process the application, the plurality of processing units including at least one central processing unit and at least one graphics processing unit. Each model of the plurality of models is different. The plurality of models are stored in a model repository. Each model of the plurality of models is based on a multilayer neural network. Performing offline initial clustering on a set of cell images includes extracting features of the cell images using the plurality of models, training a cluster component in an unsupervised manner using a small subset of the given cell images, and determining a cluster for each given cell using the cluster component. Performing online image-based single cell sorting includes utilizing the plurality of models to extract features from cell images and using a cluster component to determine a cluster for each given cell. Clustering separates and groups different types of cells based on the extracted features. One example of a clustering algorithm is hierarchical density-based spatial clustering.

[0008] In another aspect, a system includes a first computing device configured to perform offline initial image-based unsupervised clustering, train a plurality of models, each model of the plurality of models designed to extract features of each cell, and perform online image-based single-cell sorting; and a second computing device configured to transmit one or more images to the first computing device. Each model of the plurality of models is different. The plurality of models is stored in a model repository. Each model of the plurality of models is based on a multilayer neural network. For a set of cell images, performing offline initial clustering includes using the plurality of models to extract features of the cell images, training a cluster component in an unsupervised manner using a given small subset of the cell images, and using the cluster component to determine a cluster for each given cell. Performing online image-based single-cell sorting includes utilizing the plurality of models to extract features of the cell images and using the cluster component to determine a cluster for each given cell. Clustering separates and groups different types of cells based on the extracted features. One example of a clustering algorithm is hierarchical density-based spatial clustering. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a flowchart of a method for training a feature extractor, according to some embodiments. [Figure 2] 1 is a flowchart of a method for training a cluster component on a set of cell images, according to some embodiments. [Figure 3] 1 is a flowchart of a method for offline initial clustering on a set of cell images, according to some embodiments. [Figure 4] 1 is a flowchart of a method for online single cell sorting, according to some embodiments. [Figure 5] FIG. 1 is a block diagram of an exemplary computing device configured to implement an unsupervised image-based cell clustering and sorting framework, according to some embodiments. [Figure 6] 1 is a diagram illustrating a schematic overview of a biological sample analyzer according to some embodiments. [Figure 7] FIG. 1 illustrates an example model repository in accordance with some embodiments. [Figure 8] 1 is a flowchart of a method for training a cluster component on a set of cell images using multi-model feature extraction, according to some embodiments. [Figure 9] 1 is a flowchart of a method for offline initial clustering of a set of cell images with multi-model feature extraction, according to some embodiments. [Figure 10] 1 is a flowchart of a method for online single cell sorting with multi-model feature extraction, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0010] The methods and systems described herein include a learning framework that supports (1) offline, unsupervised image-based clustering, which replaces time-consuming manual gating, and (2) online, image-based single-cell sorting. This framework includes feature extraction and clustering. During training, one or more cell image datasets, with or without ground truth, are used to train a feature extractor. The feature extractor is based on a multilayer neural network. Once trained, the feature extractor is used to extract cell image features for unsupervised cell clustering and sorting. Multiple feature extractors using different sets of image channels can be trained and used for multi-model clustering. Additionally, after a feature extractor is trained, it can be further refined using additional datasets. The methods and systems described herein are the first to combine neural network-based feature extraction and clustering in an unsupervised learning framework for image-based offline cell clustering and online single-cell sorting. This replaces time-consuming manual gating in traditional FACS workflows as a tool for offline initial clustering. This improves or enables applications such as tools for online single cell sorting, which cannot be performed accurately without morphological information of the cells.

[0011] Conventional fluorescence-activated cell sorting (FACS) relies on labeling cells with fluorescent markers, which provides very limited morphological information about the cells. However, some applications require morphological information to accurately sort cells, and some applications are not suitable for fluorescent markers. The methods and systems described herein implement a framework that enables applications to cluster and sort cells based on cell images with or without fluorescent markers. Some studies have proposed image-based cell sorting using supervised learning based on deep neural networks or manually created features. These studies assumed cell images with ground truth for training, which may not be available. The methods and systems described herein implement a framework that enables training with or without ground truth.

[0012] Manual gating in traditional FACS is time-consuming and can be biased. Although software exists that assists the process but relies on specific manually created features, these features may not provide enough information as the image itself. The method and system described herein utilizes images and deep learning for better performance.

[0013] Described herein is a framework that includes a feature extractor and a cluster component for clustering, supporting offline image-based unsupervised clustering to replace time-consuming manual gating, and online image-based single-cell sorting.

[0014] FIG. 1 shows a flowchart of a method for training a feature extractor, according to some embodiments. In step 100, a cell image dataset is received. In some embodiments, the dataset may include images and / or video. The dataset may include information acquired using a particular imaging system (e.g., one or more cameras) and processed using one or more image / video processing algorithms. In some embodiments, the imaging system is part of a flow cytometer or other viewer for displaying and capturing images of cells. The dataset may be transmitted to a server for storage and then received (e.g., downloaded) at a computing device that implements the training method.

[0015] In step 102, a feature extractor is implemented. The feature extractor is used to extract features from the image. The feature extractor is based on a multi-layer neural network. In some embodiments, the feature extractor uses several convolutional layers followed by a pooling layer. To train the feature extractor, an exemplary approach is to use contrastive loss, which involves contrasting each sample with a set of positive and negative samples and calculating the loss. After the feature extractor is trained, additional datasets can be used to further refine the feature extractor.

[0016] In step 104, feedback from clustering is performed. In some embodiments, feedback clustering is optional. In some embodiments, the clusters optionally provide feedback for training the feature extractor. The clustering can utilize hierarchical density-based clustering or other clustering algorithms. The clustering separates and groups different types of cells based on extracted features. Hierarchical density-based spatial clustering (HDBSCAN) is an exemplary clustering algorithm that can handle an unknown number of classes. HDBSCAN performs noisy density-based clustering across epsilon values ​​and integrates the results to find stable clustering. Given a set of points in some space, HDBSCAN groups points that are densely packed together (e.g., points with many nearby neighbors) together. Although HDBSCAN is described herein, any clustering algorithm can be utilized.

[0017] Figure 2 shows a flowchart of a method for training a cluster component on a set of cell images, according to some embodiments. In some embodiments, after Phase II trains the cluster component, it can be used in offline initial clustering (Phase III) or online single-cell sorting (Phase IV).

[0018] In some embodiments, Phase III can also be used to train a cluster component as part of offline initial clustering. In such cases, Phase II is not used and the cluster component trained in Phase III is used for online single-cell sorting (Phase IV). Extending this example further, offline initial clustering is performed and the cluster component is trained simultaneously. A user can select a subset of clusters for further analysis and then use the trained cluster component for online single-cell sorting in Phase IV.

[0019] In step 200, a small set of cell images that does not include ground truth information is acquired. In some embodiments, the set of cell images includes images and / or video. The set of cell images may include information acquired using an imaging system and processed using one or more image / video processing algorithms. In some embodiments, the imaging system is part of a flow cytometer or other viewer for displaying and capturing images of cells.

[0020] In step 202, the feature extractor trained in step 102 is used to extract cellular features from the small set of cellular images.

[0021] A given set of cell images is used to train the cluster component in step 204. An exemplary algorithm for clustering is HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise), which has the advantage of handling an unknown number of clusters and allows for noise in the input.

[0022] Figure 3 shows a flowchart of a method for offline initial clustering of a set of cell images, according to some embodiments, which can be used to replace time-consuming manual gating in conventional FACS. Steps 300, 302, and 304 correspond to steps 200, 202, and 204, respectively. In step 306, the cluster trained in step 204 identifies clusters for all cells. In step 308, the user can select a subset of clusters of interest for further analysis.

[0023] Figure 4 shows a flowchart of a method for online single cell sorting, according to some embodiments. Step 400 shows a flow cytometer, a technology used to detect and measure the physical and chemical properties of a population of cells or particles. Next, in the process, a sample containing the cells or particles is suspended in a fluid and injected into a flow cytometer instrument.

[0024] In step 402, a trained feature extractor (e.g., one trained in step 102) extracts cellular features from each cellular image, which may include one or more channels of the image (e.g., bright field, dark field, etc.).

[0025] In step 404, the cluster component trained in step 204 is used to perform online sorting by classifying the cell features of each cell into a cell class in step 406. For online cell sorting, the implementation operates on an unknown number of clusters, allows for noise, and operates even when ground truth is not available.

[0026] Cell sorting involves removing cells from an organism and separating them according to their type. Image-based cell sorting allows cells to be separated based on extracted features of cell images. Real-time sorting can utilize cluster definitions. For example, the system compares cell features / components to determine which cluster a cell best matches.

[0027] Although Phases I through IV are described herein, in some embodiments, Phase II can be combined with Phase III. In some embodiments, the order of steps is important, e.g., performing Phase I before any other phases, or performing Phase IV only after Phases I through III.

[0028] FIG. 5 illustrates a block diagram of an exemplary computing device configured to implement an unsupervised image-based cell clustering and sorting framework, according to some embodiments. The computing device 500 can be used to acquire, store, compute, process, communicate, and / or display information, such as images and videos. The computing device 500 can implement any aspect of the unsupervised image-based cell clustering and sorting framework. In general, a hardware architecture suitable for implementing the computing device 500 includes a network interface 502, memory 504, a processor 506, I / O device(s) 508, a bus 510, and storage 512. The selection of the processor(s) is not critical as long as suitable processor(s) with sufficient speed are selected. The processor 506 can include multiple central processing units (CPUs). The processor 506 and / or hardware 520 can include one or more graphics processing units (GPUs) for efficient neural network-based feature extraction. Each GPU should be equipped with sufficient GPU memory to perform the feature extraction. The memory 504 can be any conventional computer memory known in the art. The storage device 512 can include a hard drive, CD-ROM, CDRW, DVD, DVDRW, high-definition disk / drive, ultra-high-definition drive, flash memory card, or any other storage device. The computing device 500 can include one or more network interfaces 502. Examples of network interfaces include a network card connected to an Ethernet or other type of LAN. The I / O device(s) 508 can include one or more of a keyboard, mouse, monitor, screen, printer, modem, touch screen, button interface, and other devices. The unsupervised image-based cell clustering and sorting framework application(s) 530 used to implement the framework are likely to be stored in the storage device 512 and memory 504 and processed as applications are normally processed.The computing device 500 may include more or fewer components than those shown in FIG. 5 . In some embodiments, unsupervised image-based cell clustering and sorting framework hardware 520 is included. While the computing device 500 of FIG. 5 includes hardware 520 and application 530 for the unsupervised image-based cell clustering and sorting framework, the unsupervised image-based cell clustering and sorting framework may be implemented in the computing device as hardware, firmware, software, or any combination thereof. For example, in some embodiments, the unsupervised image-based cell clustering and sorting framework application 530 is programmed into memory and executed using a processor. As another example, in some embodiments, the unsupervised image-based cell clustering and sorting framework hardware 520 is programmed hardware logic including gates specifically designed to implement the unsupervised image-based cell clustering and sorting framework.

[0029] In some embodiments, the unsupervised image-based cell clustering and sorting framework application(s) 530 include several applications and / or modules. In some embodiments, a module also includes one or more sub-modules. In some embodiments, fewer or additional modules may be included.

[0030] Examples of suitable computing devices include a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular / mobile phone, a smart appliance, a game console, a digital camera, a digital camcorder, a camera phone, a smartphone, a portable music player, a tablet computer, a mobile device, a video player, a video disc writer / player (e.g., a DVD writer / player, a high-definition disc writer / player, an ultra-high-definition disc writer / player), a television, a home entertainment system, an augmented reality device, a virtual reality device, smart jewelry (e.g., a smart watch), a vehicle (e.g., an autonomous vehicle), or any other suitable computing device.

[0031] FIG. 6 shows a schematic diagram of the overall configuration of a biological sample analyzer, according to some embodiments.

[0032] FIG. 6 shows an exemplary configuration of a biological sample analyzer of the present disclosure. The biological sample analyzer 6100 shown in FIG. 6 includes a light irradiation unit 6101 that irradiates light onto a biological sample S flowing through a flow path C, a detection unit 6102 that detects light generated by irradiating the biological sample S, and an information processing unit 6103 that processes information about the light detected by the detection unit. The biological sample analyzer 6100 is, for example, a flow cytometer or an imaging cytometer. The biological sample analyzer 6100 may include a sorting unit 6104 that sorts specific biological particles P in the biological sample. The biological sample analyzer 6100 including the sorting unit is, for example, a cell sorter.

[0033] (biological samples) The biological sample S may be a liquid sample containing biological particles. The biological particles may be, for example, cells or non-cellular biological particles. The cells may be living cells, more specific examples of which include blood cells such as red blood cells and white blood cells, and reproductive cells such as sperm and fertilized eggs. The cells may be directly collected from a sample such as whole blood, or may be cultured cells obtained after culturing. The non-cellular biological particles may be extracellular vesicles, particularly exosomes and microvesicles. The biological particles may be labeled with one or more labeling substances (e.g., dyes (particularly fluorescent dyes) and antibodies labeled with fluorescent dyes). It should be noted that the biological sample analyzer of the present disclosure may analyze particles other than biological particles, such as beads for calibration purposes.

[0034] (flow path) The flow channel C is designed to form a flow of the biological sample S. In particular, the flow channel C can be designed to form a flow in which biological particles contained in the biological sample are arranged substantially in a single row. The flow channel structure including the flow channel C can be designed to form a laminar flow. In particular, the flow channel structure is designed to form a laminar flow in which the flow of the biological sample (sample flow) is surrounded by the flow of sheath liquid. The design of the flow channel structure can be appropriately selected by those skilled in the art, or a known structure can be adopted. The flow channel C can be formed in a flow channel structure such as a microchip (a chip having flow channels on the order of micrometers) or a flow cell. The width of the flow channel C is 1 mm or less, and in particular, can be 10 μm or more and 1 mm or less. The flow channel C and the flow channel structure including the flow channel C can be formed from materials such as plastic or glass.

[0035] The biological sample analyzer of the present disclosure is designed so that light from a light irradiation unit 6101 is irradiated onto the biological sample flowing within flow channel C, particularly onto biological particles in the biological sample. The biological sample analyzer of the present disclosure can be designed so that the irradiation point of light on the biological sample is located within the flow channel structure in which flow channel C is formed, or so that the irradiation point is located outside the flow channel structure. An example of the former case is a structure in which light is emitted onto flow channel C within a microchip or flow cell. In the latter case, light can be irradiated onto biological particles after they have left the flow channel structure (particularly, the nozzle portion thereof), and for example, a jet-in-air type flow cytometer can be employed.

[0036] (Light irradiation unit) The light illumination unit 6101 includes a light source unit that emits light and a light guide optical system that guides the light to an illumination point. The light source unit includes one or more light sources. The type of light source(s) is, for example, a laser light source or an LED. The wavelength of the light to be emitted from each light source can be any wavelength of ultraviolet light, visible light, and infrared light. The light guide optical system includes optical components such as, for example, a beam splitter, a mirror, or an optical fiber. The light guide optical system can also include a group of lenses for focusing light, including, for example, an objective lens. There can be one or more illumination points where the biological sample and the light intersect. The light illumination unit 6101 can be designed to collect light emitted from one light source or different light sources onto one illumination point.

[0037] (detection unit) The detection unit 6102 includes at least one photodetector that detects light generated by emitting light on the bioparticles. The light to be detected can be, for example, fluorescence or scattered light (e.g., one or more of forward scattered light, back scattered light, and side scattered light). Each photodetector includes one or more light-receiving elements, for example, a photodetector array. Each photodetector can include, as the light-receiving elements, one or more photomultiplier tubes (PMTs) and / or photodiodes such as APDs and MPPCs. The photodetector includes, for example, a PMT array in which multiple PMTs are arranged in one dimension. The detection unit 6102 can also include an image sensor such as a CCD or CMOS. Using the image sensor, the detection unit 6102 can acquire images of the bioparticles (e.g., bright-field images, dark-field images, or fluorescent images).

[0038] The detection unit 6102 includes a detection optical system that allows light of a predetermined detection wavelength to reach a corresponding photodetector. The detection optical system includes a spectroscopic unit such as a prism or a diffraction grating, or a wavelength separation unit such as a dichroic mirror or an optical filter. The detection optical system is designed, for example, to disperse light generated by irradiating light onto bioparticles and detect the dispersed light using a number of photodetectors greater than the number of fluorescent dyes used to label the bioparticles. A flow cytometer including such a detection optical system is called a spectral flow cytometer. Furthermore, the detection optical system is designed, for example, to separate light corresponding to the fluorescence wavelength band of a specific fluorescent dye from the light generated by irradiating light onto the bioparticles and detect the separated light using a corresponding photodetector.

[0039] The detection unit 6102 may also include a signal processing unit that converts the electrical signal acquired by the photodetector into a digital signal. The signal processing unit may include an A / D converter as a device that performs the conversion. The digital signal acquired by the conversion performed by the signal processing unit may be transmitted to the information processing unit 6103. The digital signal may be treated by the information processing unit 6103 as data related to light (hereinafter also referred to as "light data"). The light data may be, for example, light data including fluorescence data. More specifically, the light data may be data of light intensity, and the light intensity may be light intensity data of light including fluorescence (the light intensity data may include feature quantities such as area, height, and width).

[0040] (Information Processing Unit) The information processing unit 6103 includes a processing unit that processes various types of data (e.g., optical data) and a storage unit that stores various types of data. When the processing unit acquires optical data corresponding to fluorescent dyes from the detection unit 6102, the processing unit can perform a fluorescence spillover correction (compensation process) on the light intensity data. In the case of a spectral flow cytometer, the processing unit also performs a fluorescence separation process on the optical data to acquire light intensity data corresponding to the fluorescent dyes. The fluorescence separation process can be performed, for example, by the unmixing method disclosed in Japanese Patent Application Laid-Open No. 2011-232259. When the detection unit 6102 includes an image sensor, the processing unit can acquire morphological information about bioparticles based on images acquired by the image sensor. The storage unit can be designed to store the acquired optical data. The storage unit can also be designed to store spectral reference data to be used in the unmixing process.

[0041] If the biological sample analyzer 6100 includes a sorting unit 6104 described below, the information processing unit 6103 can determine whether the biological particles should be sorted based on the optical data and / or morphological information. Then, the information processing unit 6103 can control the sorting unit 6104 based on the result of the determination, so that the biological particles can be sorted by the sorting unit 6104.

[0042] The information processing unit 6103 can be designed to output various types of data (e.g., optical data and images, etc.). For example, the information processing unit 6103 can output various types of data (e.g., two-dimensional plots or spectral plots, etc.) generated based on optical data. The information processing unit 6103 can also be designed to accept input of various types of data, for example, accepting a gating process on a plot by a user. The information processing unit 6103 can include an output unit (e.g., a display, etc.) or an input unit (e.g., a keyboard, etc.) and can perform output or input.

[0043] The information processing unit 6103 can be designed as a general-purpose computer, for example, as an information processing device including a CPU, a RAM, and a ROM. The information processing unit 6103 can be included in a housing including the light irradiation unit 6101 and the detection unit 6102, or can be located outside the housing. Furthermore, various processes or functions to be performed by the information processing unit 6103 can be realized by a server computer or a cloud connected via a network.

[0044] (Sorting unit) The sorting unit 6104 sorts bioparticles according to the results of the judgment made by the information processing unit 6103. The sorting method can be a method in which droplets containing bioparticles are generated by vibration, an electric charge is applied to the droplets to be sorted, and the movement direction of the droplets is controlled by electrodes. The sorting method can be a method in which bioparticles are sorted by controlling the movement direction of the bioparticles in a channel structure. The channel structure has, for example, a control mechanism based on pressure (injection or suction) or electric charge. An example of a channel structure can be a chip (for example, the chip disclosed in JP 2020-076736 A) in which channel C branches into a recovery channel and a waste channel downstream, and specific bioparticles are collected in the recovery channel.

[0045] In some embodiments, instead of having a single feature extractor (also called a model), multiple feature extractors (or models) can be implemented. Each model can function as a feature extractor. Multiple models can be trained, each for a specific purpose. In some embodiments, multiple models can be combined in the clustering stage.

[0046] Multi-model clustering uses multiple models to extract cellular features, which can then be combined for clustering. Models are selected based on the target application and image channels of interest. Multi-model clustering can use one or more models trained for different purposes to extract each cellular feature.

[0047] Multi-model clustering overcomes this issue when more channel images are needed than the original model was trained on. In single-model implementations, the model is retrained. Another issue is that training on many channel images can increase the training difficulty. Multi-model clustering allows clustering to use multiple models trained for different purposes. For example, multiple common FL models can be used to extract features for each given common FL channel, followed by a single-cell separation model trained to distinguish focused cells from noise (debris, aggregates, etc.).

[0048] FIG. 7 shows a diagram of an exemplary model repository according to some embodiments. The model repository 700 can include multiple models (feature extractors). A common fluorescence (FL) model 702 can be used to extract features for any FL channel. Specific FL models 704 (e.g., p65+DAPI for one application, CD19+CD45 for another) are each designed for a specific set of fluors as input for the specific application. A bright field (BF) model 706 for single cell isolation can be used to distinguish between noise and in-focus cells. A specific BF model 708 can be used for specific applications. A specific dark field (DF) model 710 can be used for specific applications. While several exemplary models are shown in FIG. 7, fewer or additional models can be implemented.

[0049] Phase I of the multi-model cell clustering implementation is similar to Phase I shown in FIG. 1. However, each model of the multiple models is trained. In some embodiments, each model is trained individually. In some embodiments, all models are trained together. In some embodiments, models are trained sequentially (e.g., one at a time). In some embodiments, multiple models are trained simultaneously (e.g., in parallel).

[0050] 8 shows a flowchart of a method for training a cluster component on a set of cell images using multi-model feature extraction, according to some embodiments. In some embodiments, after Phase II trains the cluster component, the cluster component can be used in offline initial clustering (Phase III) or online single-cell sorting (Phase IV).

[0051] In some embodiments, Phase III can also be used to train a cluster component as part of offline initial clustering. In such cases, Phase II is not used and the cluster component trained in Phase III is used for online single-cell sorting (Phase IV). Extending this example further, offline initial clustering is performed and the cluster component is trained simultaneously. A user can select a subset of clusters for further analysis and then use the trained cluster component for online single-cell sorting in Phase IV.

[0052] In step 800, a small set of cell images that does not include ground truth information is acquired. In some embodiments, the set of cell images includes images and / or video. The set of cell images may include information acquired using an imaging system and processed using one or more image / video processing algorithms. In some embodiments, the imaging system is part of a flow cytometer or other viewer for displaying and capturing images of cells.

[0053] In step 802, cellular features are extracted from the small set of cellular images using the multi-model feature extraction trained in Phase I. Multi-model feature extraction allows multiple different models to extract different cellular features. For example, multi-model feature extraction can be used to obtain fluorescent and bright-field features without the need to retrain the models. Feature extraction can be performed sequentially or in parallel. For example, fluorescent and bright-field features are extracted in parallel by different models.

[0054] In step 804, a cluster component is trained using a given set of cell images. An exemplary algorithm for clustering is HDBSCAN (Hierarchical Density-Based Spatial Clustering of Noisy Applications), which has the advantage of handling an unknown number of clusters and allows for noise in the input. In some embodiments, a small subset of the given cell images is used to train the cluster component. The cluster component can use multiple models to extract features of the cell images and use the combined features for clustering.

[0055] Figure 9 shows a flowchart of a method for offline initial clustering of a set of cell images using multi-model feature extraction, according to some embodiments. This can be used to replace time-consuming manual gating in traditional FACS. Steps 900, 902, and 904 correspond to steps 800, 802, and 804, respectively. In step 906, the clusters trained in step 804 identify clusters for all cells. In step 908, the user can select a subset of clusters of interest for further analysis. The clusters trained on a small subset of given cells in Phase II are used to identify clusters for all cells. Once the clusters are identified, the user can select the clusters of interest for further analysis.

[0056] Figure 10 shows a flowchart of a method for online single cell sorting with multi-model feature extraction, according to some embodiments. Step 1000 shows a flow cytometer, a technology used to detect and measure the physical and chemical properties of a population of cells or particles. Next, in the process, a sample containing the cells or particles is suspended in a fluid and injected into the flow cytometer instrument.

[0057] In step 1002, multiple trained feature extractors / models (e.g., those trained in Phase I) extract cellular features from each cellular image, which may include one or more channels of the image (e.g., bright field, dark field, etc.).

[0058] In step 1004, the cluster component trained in step 804 is used to perform online sorting by classifying the cell features of each cell into a cell class in step 1006. For online cell sorting, the implementation operates on an unknown number of clusters, allows for noise, and operates even when ground truth is not available.

[0059] Cell sorting involves removing cells from an organism and separating them according to their type. Image-based cell sorting allows cells to be separated based on extracted features of cell images. Real-time sorting can utilize cluster definitions. For example, the system compares cell features / components to determine which cluster a cell best matches.

[0060] Although Phases I through IV are described herein, in some embodiments, Phase II can be combined with Phase III. In some embodiments, the order of steps is important, e.g., performing Phase I before any other phases, or performing Phase IV only after Phases I through III.

[0061] To utilize the unsupervised image-based cell clustering and sorting framework described herein, an apparatus such as a flow cytometer including an imaging system (e.g., one or more cameras) can be used to acquire content, and the apparatus can process the acquired content. The unsupervised image-based cell clustering and sorting framework can be implemented with user assistance or automatically without user involvement.

[0062] In operation, the unsupervised image-based cell clustering and sorting framework can be used even when ground truth is not available, and feature extraction based on very few neural network layers improves the speed of extraction for real-time applications. Furthermore, the unsupervised image-based cell clustering and sorting framework enables applications to sort cells based on cell images with or without fluorescent markers.

[0063] Some embodiments of image-based unsupervised multi-model cell clustering 1. A method comprising: performing an offline initial image-based unsupervised clustering; training a plurality of models, each model of the plurality of models designed to extract features of each cell; performing online image-based single cell sorting; A method comprising:

[0064] 2. The method of claim 1, wherein each model in the plurality of models is different.

[0065] 3. The method of claim 1, wherein the plurality of models is stored in a model repository.

[0066] 4. The method of claim 1, wherein each model in the plurality of models is based on a multi-layer neural network.

[0067] 5. Performing offline initial clustering on a set of cell images is extracting features of the cell image using the plurality of models; training a cluster component in an unsupervised manner using a small subset of said given cell images; determining a cluster for each given cell using the cluster component; 2. The method of claim 1, comprising:

[0068] 6. Performing Online Image-Based Single-Cell Sorting extracting features of a cell image using the plurality of models; determining a cluster for each given cell using a cluster component; 2. The method of claim 1, comprising:

[0069] 7. The method of claim 1, wherein the clustering includes utilizing hierarchical density-based spatial clustering.

[0070] 8. The method of claim 7, wherein clustering separates and groups different types of cells based on the extracted features that comprise the definition of each cluster.

[0071] 9. An apparatus comprising: A non-transitory memory for storing an application, said application comprising: performing offline initial image-based unsupervised clustering; training a plurality of models, each model of the plurality of models designed to extract features of each cell; performing online image-based single cell sorting; a non-transitory memory for performing a plurality of processing units configured to process the applications, the plurality of processing units including at least one central processing unit and at least one graphics processing unit; An apparatus comprising:

[0072] 10. The apparatus of claim 9, wherein each model in the plurality of models is different.

[0073] 11. The apparatus of claim 9, wherein the plurality of models is stored in a model repository.

[0074] 12. The apparatus of claim 9, wherein each model in the plurality of models is based on a multi-layer neural network.

[0075] 13. Performing offline initial clustering on a set of cell images is extracting features of the cell image using the plurality of models; training a cluster component in an unsupervised manner using a small subset of said given cell images; determining a cluster for each given cell using the cluster component; 10. The apparatus of claim 9, comprising:

[0076] 14. Performing online image-based single-cell sorting extracting features of a cell image using the plurality of models; determining a cluster for each given cell using a cluster component; 10. The apparatus of claim 9, comprising:

[0077] 15. The apparatus of claim 9, wherein the clustering includes utilizing hierarchical density-based spatial clustering.

[0078] 16. The apparatus of claim 15, wherein clustering separates and groups different types of cells based on the extracted features that comprise the definition of each cluster.

[0079] 17. A system comprising: performing offline initial image-based unsupervised clustering; training a plurality of models, each model of the plurality of models designed to extract features of each cell; performing online image-based single cell sorting; a first computing device configured to: a second computing device configured to transmit one or more images to the first computing device; A system including:

[0080] 18. The system of claim 17, wherein each model in the plurality of models is different.

[0081] 19. The system of claim 17, wherein the plurality of models are stored in a model repository.

[0082] 20. The system of claim 17, wherein each model in the plurality of models is based on a multi-layer neural network.

[0083] 21. Performing offline initial clustering on a set of cell images is extracting features of the cell image using the plurality of models; training a cluster component in an unsupervised manner using a small subset of said given cell images; determining a cluster for each given cell using the cluster component; 18. The system of claim 17, comprising:

[0084] 22. Performing online image-based single-cell sorting extracting features of a cell image using the plurality of models; determining a cluster for each given cell using a cluster component; 18. The system of claim 17, comprising:

[0085] 23. The system of claim 17, wherein the clustering includes utilizing hierarchical density-based spatial clustering.

[0086] 24. The system of claim 23, wherein clustering separates and groups different types of cells based on the extracted features that comprise the definition of each cluster.

[0087] The present invention has been described with reference to specific embodiments incorporating details to facilitate an understanding of the principles of construction and operation of the invention. Reference herein to specific embodiments and their details is not intended to limit the scope of the claims appended hereto. Those skilled in the art will readily appreciate that various other modifications can be made to the embodiments chosen for illustration without departing from the spirit and scope of the invention as defined by the claims. [Explanation of symbols]

[0088] 100 Cell Image Dataset 102 Feature Extractor 104 Perform feedback from clustering A set of 200 ground truth-free cell images 202 Feature Extractor 204 Train Cluster Components A set of 300 ground truth-free cell images 302 Feature Extractor 304 Train Cluster Components 306 clusters identified 308 User picks cluster 400 Flow Cytometer 402 Feature Extractor 404 Use trained cluster component Classification into 406 cell classes 500 Computer equipment 502 network interface 504 memory 506 processor 508 I / O devices 510 Bus 512 Storage device 520 Unsupervised Image-Based Cell Clustering and Sorting Framework Hardware 530 Unsupervised Image-Based Cell Clustering and Sorting Framework Application 6100 Biological Sample Analyzer 6101 Light irradiation unit 6102 Detection Unit 6103 Information Processing Unit 6104 Sorting Unit 700 Model Repository 702 common FL model 704 Special FL Model 706 BF model for single cell isolation 708 Special BF Model 710 Special DF Model A set of 800 ground truth-free cell images 802 Multi-Model Feature Extraction 804 Train Cluster Components A set of 900 ground truth-free cell images 902 Multi-Model Feature Extraction 904 Train Cluster Components Identify 906 clusters 908 User selects cluster 1000 Flow Cytometer 1002 Multi-Model Feature Extraction 1004 trained cluster components used Classified into 1006 cell classes C flow path P bioparticles S Biological sample

Claims

1. 1. A method comprising: performing an offline initial image-based unsupervised clustering; training a plurality of models, each model of the plurality of models designed to extract features of each cell; performing online image-based single cell sorting; A method comprising:

2. The method of claim 1 , wherein each model in the plurality of models is different.

3. The method of claim 1 , wherein the plurality of models is stored in a model repository.

4. The method of claim 1 , wherein each model in the plurality of models is based on a multi-layer neural network.

5. Performing offline initial clustering on a set of cell images is extracting features of the cell image using the plurality of models; training a cluster component in an unsupervised manner using a small subset of the given cell images; determining a cluster for each given cell using the cluster component; 2. The method of claim 1, comprising:

6. The steps of performing online image-based single cell sorting include: extracting features of a cell image using the plurality of models; determining a cluster for each given cell using a cluster component; 2. The method of claim 1, comprising:

7. The method of claim 1 , wherein the clustering comprises utilizing hierarchical density-based spatial clustering.

8. The method of claim 7, wherein clustering separates and groups different types of cells based on the extracted features that comprise the definition of each cluster.

9. 1. An apparatus comprising: A non-transitory memory for storing an application, said application comprising: performing offline initial image-based unsupervised clustering; training a plurality of models, each model of the plurality of models designed to extract features of each cell; performing online image-based single cell sorting; a non-transitory memory for performing a plurality of processing units configured to process the applications, the plurality of processing units including at least one central processing unit and at least one graphics processing unit; 10. An apparatus comprising:

10. 10. The apparatus of claim 9, wherein each model in the plurality of models is different.

11. The apparatus of claim 9 , wherein the plurality of models is stored in a model repository.

12. 10. The apparatus of claim 9, wherein each model in the plurality of models is based on a multi-layer neural network.

13. Performing offline initial clustering on a set of cell images is extracting features of the cell image using the plurality of models; training a cluster component in an unsupervised manner using a small subset of said given cell images; determining a cluster for each given cell using the cluster component; 10. The apparatus of claim 9, comprising:

14. Performing online image-based single cell sorting is extracting features of a cell image using the plurality of models; determining a cluster for each given cell using a cluster component; 10. The apparatus of claim 9, comprising:

15. 10. The apparatus of claim 9, wherein the clustering comprises utilizing hierarchical density-based spatial clustering.

16. 16. The apparatus of claim 15, wherein clustering separates and groups different types of cells based on the extracted features that comprise the definition of each cluster.

17. 1. A system comprising: performing offline initial image-based unsupervised clustering; training a plurality of models, each model of the plurality of models designed to extract features of each cell; performing online image-based single cell sorting; a first computing device configured to: a second computing device configured to transmit one or more images to the first computing device; A system comprising:

18. 20. The system of claim 17, wherein each model in the plurality of models is different.

19. 20. The system of claim 17, wherein the plurality of models is stored in a model repository.

20. 20. The system of claim 17, wherein each model in the plurality of models is based on a multi-layer neural network.

21. Performing offline initial clustering on a set of cell images is extracting features of the cell image using the plurality of models; training a cluster component in an unsupervised manner using a small subset of said given cell images; determining a cluster for each given cell using the cluster component; 20. The system of claim 17, comprising:

22. Performing online image-based single cell sorting is extracting features of a cell image using the plurality of models; determining a cluster for each given cell using a cluster component; 20. The system of claim 17, comprising:

23. 20. The system of claim 17, wherein the clustering comprises utilizing hierarchical density-based spatial clustering.

24. 24. The system of claim 23, wherein clustering separates and groups different types of cells based on the extracted features that comprise the definition of each cluster.

Images