Image-based unsupervised multi-model cell clustering

Abstract

A framework is described herein that includes one or more feature extractors (models) and a clustering component for clustering. The framework supports (1) image-based off-line unsupervised clustering instead of time-consuming manual gating; and (2) image-based online single cell sorting. During training, one or more feature extractors are trained using one or more cell image data sets with or without reference truth values, the feature extractors based on a neural network comprising several convolutional layers. After training, a feature extractor is used for extracting the features of the cell images so as to perform unsupervised cell clustering and sorting. In addition, after the feature extractor has been trained, an additional data set may be used to further improve the feature extractor.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a continuation-in-part of co-pending U.S. patent application No. 17 / 222,131, filed on April 5, 2021, entitled “A FRAMEWORK FOR IMAGE BASED UNSUPERVISED CELL CLUSTERING AND SORTING,” which claims priority under 35 U.S.C. § 119(e) to U.S. provisional patent application serial number 63 / 116,065, filed on November 19, 2020, entitled “UNSUPERVISED LEARNING FRAMEWORK FOR IMAGE BASED SINGLE CELL SORTING,” which applications are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] The present invention relates to cell sorting. More particularly, the present invention relates to image-based cell sorting. Background Art

[0004] Traditional fluorescence activated cell sorting relies on labeling cells with fluorescent markers, which provides very limited information about cell morphology. However, some applications require cell morphology information to accurately sort cells, and some applications are not suitable for using fluorescent markers. In addition, traditional fluorescence activated cell sorting (FACS) uses manual gating to establish sorting criteria based on fluorescent markers. However, manual gating is time-consuming and may be biased.

[0005] Some studies have proposed image-based cell sorting using supervised learning based on deep neural networks or handcrafted features. They assume that cell images with ground truth are used for training, which may not be available. Some software that facilitates the gating process relies on specific handcrafted features of fluorescent markers, which may not have sufficient morphological information for some applications or may not be suitable for some other applications. Summary of the invention

[0006] Image-based unsupervised multi-model cell clustering further develops clustering strategies to support a framework for unsupervised cell clustering. The extended framework has a model repository containing more than one model, such as general fluorescence models, special fluorescence models, special bright field models, and single cell isolation bright field models. Multi-model clustering uses more than one model to extract cell features, which can be cascaded for clustering. Models are selected based on the target application and the image channels of interest. The uniqueness of image-based unsupervised multi-model cell clustering is that applications can use one or more models trained for different purposes to extract features for each cell.

[0007] In one aspect, a method includes performing offline initial unsupervised clustering based on images, training multiple models, wherein each of the multiple models is designed to extract features of each cell, and performing online single cell sorting based on images. Each of the multiple models is different. The multiple models are stored in a model repository. Each of the multiple models is based on a multi-layer neural network. Performing offline initial clustering of a set of cell images includes extracting features of the cell images using the multiple models, training a clustering component in an unsupervised manner using a small subset of the given cell images, and determining the cluster of each given cell using the clustering component. Performing online single cell sorting based on images includes extracting features of the cell images using the multiple models, and determining the cluster of each given cell using the clustering component. Clustering separates and groups different types of cells based on the extracted features. An example clustering algorithm is spatial clustering based on hierarchical density.

[0008] In another aspect, a device includes a non-transitory memory for storing an application and a plurality of processing units configured to process the application, the application being used to: perform image-based offline initial unsupervised clustering, train a plurality of models, wherein each of the plurality of models is designed to extract features of each cell, and perform image-based online single cell sorting, wherein the plurality of processing units include at least one central processing unit and at least one graphics processing unit. Each of the plurality of models is different. The plurality of models are stored in a model repository. Each of the plurality of models is based on a multi-layer neural network. Performing offline initial clustering of a set of cell images includes extracting features of the cell images using the plurality of models, training a clustering component in an unsupervised manner using a small subset of the given cell images, and determining a cluster of each given cell using the clustering component. Performing image-based online single cell sorting includes extracting features of the cell images using the plurality of models, and determining a cluster of each given cell using the clustering component. Clustering separates and groups different types of cells based on the extracted features. An example clustering algorithm is hierarchical density-based spatial clustering.

[0009] In another aspect, a system includes a first computing device and a second computing device, the first computing device being configured to: perform offline initial unsupervised clustering based on images, train multiple models, wherein each of the multiple models is designed to extract features of each cell, and perform online single cell sorting based on images, and the second computing device being configured to send one or more images to the first computing device. Each of the multiple models is different. The multiple models are stored in a model repository. Each of the multiple models is based on a multi-layer neural network. Performing offline initial clustering of a set of cell images includes extracting features of the cell images using the multiple models, training a clustering component in an unsupervised manner using a small subset of the given cell images, and determining the cluster of each given cell using the clustering component. Performing online single cell sorting based on images includes extracting features of the cell images using the multiple models, and determining the cluster of each given cell using the clustering component. Clustering separates and groups different types of cells based on the extracted features. An example clustering algorithm is spatial clustering based on hierarchical density. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A flow chart illustrating a method of training a feature extractor according to some embodiments.

[0011] Figure 2 A flow chart illustrating a method of training a clustering component for a set of cell images according to some embodiments.

[0012] Figure 3 A flow chart illustrating a method for offline initial clustering of a set of cell images according to some embodiments.

[0013] Figure 4 A flow chart illustrating a method for online single cell sorting according to some embodiments.

[0014] Figure 5 Illustrated is a block diagram of an exemplary computing device configured to implement an unsupervised, image-based cell clustering and sorting framework in accordance with some embodiments.

[0015] Figure 6 The illustrated diagram schematically illustrates a diagram of the overall configuration of a biological sample analyzer according to some embodiments.

[0016] Figure 7 A diagram illustrating an exemplary model repository in accordance with some embodiments.

[0017] Figure 8 A flow chart illustrating a method for training a clustering component for a set of cell images using multi-model feature extraction in accordance with some embodiments.

[0018] Fig. 9 A flow chart illustrating a method for offline initial clustering of a set of cell images using multi-model feature extraction in accordance with some embodiments.

[0019] Fig.10 A flow chart illustrating a method for online single cell sorting using multi-model feature extraction according to some embodiments. DETAILED DESCRIPTION

[0020] The methods and systems described herein include a learning framework that supports (1) offline image-based unsupervised clustering that replaces time-consuming manual gating, and (2) online image-based single cell sorting. The framework includes feature extraction and clustering. During training, a multi-layer neural network-based feature extractor is trained using one or more cell image datasets with or without ground truth. After training, the feature extractor is used to extract features of the cell images for unsupervised cell clustering and sorting. More than one feature extractor using different sets of image channels can be trained and used for multi-model clustering. In addition, after the feature extractor is trained, the feature extractor can be further improved using additional datasets. The methods and systems described herein combine neural network-based feature extraction and clustering for offline image-based cell clustering and online single cell sorting for the first time in an unsupervised learning framework. As a tool for offline initial clustering, it replaces time-consuming manual gating in conventional FACS workflows. As a tool for online single cell sorting, it improves or enables applications that cannot be accurately performed without cell morphology information.

[0021] Traditional fluorescence activated cell sorting (FACS) relies on labeling cells with fluorescent markers, which provides very limited information about cell morphology. However, some applications require cell morphology information to accurately sort cells, while some applications are not suitable for fluorescent markers. The methods and systems described herein implement a framework that allows applications to cluster and sort cells based on cell images with or without fluorescent markers. Some studies have proposed image-based cell sorting using supervised learning based on deep neural networks or handcrafted features. They assume that cell images with ground truth are used for training, which may not be available. The methods and systems described herein implement a framework that allows training with or without ground truth.

[0022] Manual gating in traditional FACS is time consuming and can be biased. Software exists to help with this process but relies on specific handcrafted features, which may not provide as much information as the image itself. The methods and systems described in this article leverage images and deep learning to achieve better performance.

[0023] Herein, a framework including a feature extractor and a clustering component for clustering is described. The framework supports offline image-based unsupervised clustering that replaces time-consuming manual gating as well as online image-based single-cell sorting.

[0024] Figure 1 A flow chart illustrating a method for training a feature extractor according to some embodiments. In step 100, a cell image data set is received. In some embodiments, a data set with or without a baseline truth value is received. In some embodiments, the data set includes images and / or videos. The data set can include information acquired using an imaging system (e.g., one or more cameras) and processed using one or more image / video processing algorithms. In some embodiments, the imaging system is part of a flow cytometer or other observer for observing and capturing images of cells. The data set can be sent to a server and stored in the server, and then received (e.g., downloaded to a computing device) at a computing device implementing the training method.

[0025] At step 102, a feature extractor is implemented. The feature extractor is used to extract features from the image. The feature extractor is based on a multi-layer neural network. In some embodiments, the feature extractor uses several convolutional layers followed by pooling layers. To train the feature extractor, an exemplary method is to use a contrastive loss, which includes comparing each sample to a set of positive and negative samples to calculate a loss. After the feature extractor is trained, additional data sets can be used to further improve the feature extractor.

[0026] In step 104, feedback from clustering is performed. In some embodiments, feedback clustering is optional. In some embodiments, clustering optionally provides feedback for training the feature extractor. Clustering can use hierarchical density-based clustering or other clustering algorithms. Clustering separates and groups cells of different types based on the extracted features. Hierarchical Density-Based Spatial Clustering (HDBSCAN) is an exemplary clustering algorithm that can handle an unknown number of classes. HDBSCAN performs density-based clustering of ε values ​​with noise and integrates the results to find stable clusters. Given a set of points in a certain space, HDBSCAN groups points that are densely packed together (e.g., points with many neighbors). Although HDBSCAN is described herein, any clustering algorithm may also be used.

[0027] Figure 2 Flowchart illustrating a method of training a clustering component for a set of cell images according to some embodiments.In some embodiments, Phase II trains the clustering component before it can be used for offline initial clustering (Phase III) or online single cell sorting (Phase IV).

[0028] In some embodiments, Phase III can also be used to train clustering components as part of offline initial clustering. In this case, Phase II is not used, and the clustering components trained in Phase III are used for online single cell sorting (Phase IV). Continuing with this example, offline initial clustering is performed while the clustering components are trained. The user can select a subset of clusters for further analysis, and then the trained clustering components can be used for online single cell sorting in Phase IV.

[0029] In step 200, a group of cell images without reference truth information is acquired. In some embodiments, the group of cell images includes images and / or videos. The group of cell images can include information acquired using an imaging system and processed using one or more image / video processing algorithms. In some embodiments, the imaging system is part of a flow cytometer or other observer for observing and capturing images of cells.

[0030] In step 202 , the feature extractor trained in step 102 is used to extract cell features from a small set of cell images.

[0031] In step 204, a clustering component is trained using a given set of cell images. An example algorithm for clustering is HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise), which has the advantages of handling an unknown number of clusters and allowing for noise in the input.

[0032] Figure 3 Flowchart illustrating a method for offline initial clustering of a set of cell images according to some embodiments. This can be used to replace time-consuming manual gating in traditional FACS. Steps 300, 302, and 304 correspond to steps 200, 202, and 204, respectively. In step 306, the clusterer trained in step 204 identifies clusters for all cells. In step 308, the user can select a subset of clusters of interest for further analysis.

[0033] Figure 4 A flow chart illustrating a method for online single cell sorting according to some embodiments. Step 400 shows a flow cytometer. Flow cytometry is a technique for detecting and measuring the physical and chemical properties of a population of cells or particles. In the process, a sample containing cells or particles is then suspended in a fluid and injected into a flow cytometer instrument.

[0034] In step 402, a feature extractor that has been trained (eg, trained in step 102) extracts cell features from each cell image, which may include images of one or more channels (eg, bright field, dark field, and others).

[0035] At step 404, the clustering component trained in step 204 is used for online sorting to classify the cellular features of each cell into cell classes at step 406. For online cell sorting, this implementation works for unknown numbers of clusters and tolerates noise, even when ground truth is unavailable.

[0036] Cell sorting involves removing cells from an organism and separating them by their type. In image-based cell sorting, cells can be separated based on features of an extracted cell image. Real-time sorting can utilize the definition of clusters. For example, the system compares features / components of cells and determines which cluster the cell most closely matches.

[0037] Although stages I to IV are described herein, in some embodiments, stage II can be combined with stage III. In some embodiments, the order of the steps is important, such as performing stage I before any other stage, or performing stage IV only after stages I-III.

[0038] Figure 5A block diagram of an exemplary computing device configured to implement unsupervised, image-based cell clustering and sorting frameworks according to some embodiments is represented. Computing device 500 can be used to obtain, store, calculate, process, transmit and / or display information such as images and videos. Computing device 500 can implement any aspect of unsupervised, image-based cell clustering and sorting frameworks. Typically, a hardware structure suitable for implementing computing device 500 includes a network interface 502, a memory 504, a processor 506, an I / O device 508, a bus 510 and a storage device 512. The choice of processor is not important, as long as a suitable processor with sufficient speed is selected. Processor 506 can include multiple central processing units (CPUs). Processor 506 and / or hardware 520 can include one or more graphics processing units (GPUs) for efficient feature extraction based on neural networks. Each GPU should be equipped with enough GPU memory for feature extraction. Memory 504 can be any conventional computer memory known in the art. Storage device 512 can include a hard drive, CDROM, CDRW, DVD, DVDRW, high-definition optical disk / drive, ultra-high-definition drive, flash memory card, or any other storage device. Computing device 500 can include one or more network interfaces 502. Examples of network interfaces include network cards connected to Ethernet or other types of LANs. I / O devices 508 can include one or more of the following: keyboard, mouse, monitor, screen, printer, modem, touch screen, button interface, and other devices. Unsupervised, image-based cell clustering and sorting framework application 530 for implementing the framework may be stored in storage device 512 and memory 504 and processed in the same manner as applications are normally processed. Computing device 500 can include Figure 5 More or fewer components as shown. In some embodiments, an unsupervised, image-based cell clustering and sorting framework hardware 520 is included. Although Figure 5 The computing device 500 in includes an application 530 and hardware 520 for an unsupervised, image-based cell clustering and sorting framework, but the unsupervised, image-based cell clustering and sorting framework can be implemented on the computing device using hardware, firmware, software, or any combination thereof. For example, in some embodiments, the unsupervised, image-based cell clustering and sorting framework application 530 is programmed in a memory and executed using a processor. In another example, in some embodiments, the unsupervised, image-based cell clustering and sorting framework hardware 520 is programmed hardware logic, including gates specifically designed to implement the unsupervised, image-based cell clustering and sorting framework.

[0039] In some embodiments, unsupervised, image-based cell clustering and sorting framework application 530 includes several applications and / or modules. In some embodiments, a module also includes one or more submodules. In some embodiments, fewer or more modules can be included.

[0040] Examples of suitable computing devices include personal computers, laptops, computer workstations, servers, mainframe computers, handheld computers, personal digital assistants, cellular / mobile phones, smart appliances, game consoles, digital cameras, digital video cameras, camera phones, smart phones, portable music players, tablet computers, mobile devices, video players, video disc writers / players (e.g., DVD writers / players, high-definition disc writers / players, ultra-high-definition disc writers / players), televisions, home entertainment systems, augmented reality devices, virtual reality devices, smart jewelry (e.g., smart watches), vehicles (e.g., self-driving vehicles), or any other suitable computing device.

[0041] Figure 6 The illustrated diagram schematically illustrates a diagram of the overall configuration of a biological sample analyzer according to some embodiments.

[0042] Figure 6 An example configuration of the biological sample analyzer of the present disclosure is shown. Figure 6 The biological sample analyzer 6100 shown in the figure includes: a light irradiation unit 6101, which irradiates the biological sample S flowing in the flow channel C with light; a detection unit 6102, which detects the light generated by irradiating the biological sample S; and an information processing unit 6103, which processes information about the light detected by the detection unit. The biological sample analyzer 6100 is, for example, a flow cytometer or an imaging cytometer. The biological sample analyzer 6100 may include a sorting unit 6104, which sorts out specific biological particles P in the biological sample. The biological sample analyzer 6100 including the sorting unit is, for example, a cell sorter.

[0043] (Biological Samples)

[0044] The biological sample S may be a liquid sample containing biological particles. The biological particles are, for example, cells or non-cellular biological particles. The cells may be living cells, and more specific examples thereof include blood cells such as red blood cells and white blood cells, and reproductive cells such as sperm and fertilized eggs. In addition, the cells may be cells collected directly from a sample such as whole blood, or may be cultured cells obtained after culture. Non-cellular biological particles are extracellular vesicles, or in particular, for example, exosomes and microvesicles. The biological particles may be labeled with one or more marker substances (such as dyes (particularly fluorescent dyes) and fluorochrome-labeled antibodies). Note that the biological sample analyzer of the present disclosure may analyze particles other than biological particles, and may analyze beads and the like for calibration and the like.

[0045] (Runner)

[0046] The flow channel C is designed so as to form a flow of the biological sample S. In particular, the flow channel C can be designed so as to form a flow in which the biological particles contained in the biological sample are basically arranged in a row. The flow channel structure including the flow channel C can be designed so as to form a laminar flow. In particular, the flow channel structure is designed so as to form a laminar flow in which the flow of the biological sample (sample flow) is surrounded by the flow of the cladding liquid. The design of the flow channel structure can be appropriately selected by those skilled in the art, or a known design can be adopted. The flow channel C can be formed in a flow channel structure such as a microchip (a chip with a flow channel of micrometer order) or a flow cell. The width of the flow channel C is 1 mm or less, or in particular, it can be not less than 10 μm and not more than 1 mm. The flow channel C and the flow channel structure including the flow channel C can be made of materials such as plastic or glass.

[0047] The biological sample analyzer of the present disclosure is designed so that the biological sample flowing in the flow channel C or in particular the biological microparticles in the biological sample are irradiated with light from the light irradiation unit 6101. The biological sample analyzer of the present disclosure may be designed so that the irradiation point of the light on the biological sample is located in the flow channel structure in which the flow channel C is formed, or may be designed so that the irradiation point is located outside the flow channel structure. An example of the former case may be a configuration in which light is emitted onto the flow channel C in a microchip or a flow cell. In the latter case, the biological microparticles after leaving the flow channel structure (in particular the nozzle portion of the flow channel structure) may be irradiated with light, and, for example, a jet flow cytometer may be used.

[0048] (Light irradiation unit)

[0049] The light irradiation unit 6101 includes a light source unit that emits light, and a light-guiding optical system that guides the light to the irradiation point. The light source unit includes one or more light sources. The type of the light source is, for example, a laser light source or an LED. The wavelength of the light emitted from each light source can be any wavelength of ultraviolet light, visible light, and infrared light. The light-guiding optical system includes, for example, optical components such as a spectroscope, a reflector, or an optical fiber. The light-guiding optical system may also include a lens group for focusing light, and, for example, an objective lens. There may be one or more irradiation points where the biological sample and the light intersect. The light irradiation unit 6101 may be designed to collect light emitted from one light source or different light sources to one irradiation point.

[0050] (Detection unit)

[0051] The detection unit 6102 includes at least one photodetector that detects light generated by emitting light onto the biological particles. For example, the light to be detected may be fluorescence or scattered light (such as one or more of the following: forward scattered light, backscattered light, and side scattered light). Each photodetector includes one or more light receiving elements, and, for example, has an array of light receiving elements. Each photodetector may include one or more photomultiplier tubes (PMTs) and / or photodiodes such as APDs and MPPCs as light receiving elements. The photodetector includes, for example, a PMT array in which a plurality of PMTs are arranged in a one-dimensional direction. The detection unit 6102 may also include an image sensor, such as a CCD or a CMOS. Through the image sensor, the detection unit 6102 may acquire an image of the biological particles (such as a bright field image, a dark field image, or a fluorescent image).

[0052] The detection unit 6102 includes a detection optical system that allows light of a predetermined detection wavelength to reach a corresponding photodetector. The detection optical system includes a spectroscopic unit such as a prism or a diffraction grating, or a wavelength separation unit such as a dichroic mirror or a filter. The detection optical system is designed, for example, to disperse light generated by light irradiation onto biological particles, and to detect the dispersed light with a larger number of photodetectors than the number of fluorescent dyes that label the biological particles. A flow cytometer including such a detection optical system is called a spectral flow cytometer. In addition, the detection optical system is designed, for example, to separate light corresponding to a fluorescent wavelength band of a specific fluorescent dye and light generated by light irradiation onto biological particles, and to cause a corresponding photodetector to detect the separated light.

[0053] The detection unit 6102 may further include a signal processing unit that converts the electrical signal obtained by the photodetector into a digital signal. The signal processing unit may include an A / D converter as a device for performing the conversion. The digital signal obtained by the conversion performed by the signal processing unit may be sent to the information processing unit 6103. The digital signal may be processed by the information processing unit 6103 as light-related data (hereinafter also referred to as "light data"). The light data may, for example, be light data including fluorescence data. More specifically, the light data may be light intensity data, and the light intensity may be light intensity data of light including fluorescence (the light intensity data may include feature quantities such as area, height, and width).

[0054] (Information Processing Unit)

[0055] The information processing unit 6103 includes, for example, a processing unit that processes various data (e.g., light data), and a storage unit that stores various data. In the case where the processing unit obtains light data corresponding to the fluorescent dye from the detection unit 6102, the processing unit can perform fluorescence leakage correction (compensation processing) on ​​the light intensity data. In the case of a spectral flow cytometer, the processing unit also performs fluorescence separation processing on the light data and obtains light intensity data corresponding to the fluorescent dye. The fluorescence separation processing can be performed, for example, by the unmixing method disclosed in JP 2011-232259A. In the case where the detection unit 6102 includes an image sensor, the processing unit can obtain morphological information about biological particles based on the image obtained by the image sensor. The storage unit can be designed to be able to store the acquired light data. The storage unit can be designed to be able to further store spectral reference data used in the unmixing processing.

[0056] In the case where the biological sample analyzer 6100 includes a sorting unit 6104 described later, the information processing unit 6103 can determine whether to sort the biological particles based on the optical data and / or the morphological information. Then, the information processing unit 6103 controls the sorting unit 6104 based on the result of the determination, and the biological particles can be sorted by the sorting unit 6104.

[0057] The information processing unit 6103 may be designed to output various data (e.g., light data and images). For example, the information processing unit 6103 may output various data generated based on the light data (e.g., a two-dimensional graph or a spectrogram). For example, the information processing unit 6103 may also be designed to accept input of various data and accept gated processing on the graph by the user. The information processing unit 6103 may include an output unit (e.g., a display) or an input unit (e.g., a keyboard) for output or input.

[0058] The information processing unit 6103 may be designed as a general-purpose computer, and may be designed as an information processing device including a CPU, a RAM, and a ROM, for example. The information processing unit 6103 may be included in a housing including the light irradiation unit 6101 and the detection unit 6102, or may be located outside the housing. In addition, various processes or functions performed by the information processing unit 6103 may be implemented by a server computer or a cloud connected via a network.

[0059] (Sorting unit)

[0060] The sorting unit 6104 sorts the biological particles according to the result of the determination made by the information processing unit 6103. The sorting method may be a method of generating droplets containing biological particles by vibration, applying electric charges to the droplets to be sorted, and controlling the travel direction of the droplets by electrodes. The sorting method may be a method of sorting by controlling the travel direction of the biological particles in a flow channel structure. The flow channel structure, for example, has a control mechanism based on pressure (injection or suction) or charge. An example of a flow channel structure may be a chip having a flow channel structure (for example, a chip disclosed in JP 2020-76736A), in which the flow channel C branches into a recovery flow channel and a waste liquid flow channel on the downstream side, and specific biological particles are collected in the recovery flow channel.

[0061] In some embodiments, instead of having a single feature extractor (also referred to as a model), multiple feature extractors (or models) can be implemented. Each model can function as a feature extractor. Multiple models can be trained for specific purposes. In some embodiments, multiple models can be combined in the clustering stage.

[0062] Multi-model clustering uses more than one model to extract cell features that can be cascaded for clustering. Models are selected based on the target application and the image channels of interest. Multi-model clustering is able to extract features for each cell using one or more models trained for different purposes.

[0063] Multi-model clustering overcomes the problem when more channel images are needed than to train the original model. In a single model implementation, the model would have to be retrained. Another problem is that training many channel images may increase the difficulty of training. Multi-model clustering allows clustering to use multiple models trained for different purposes. For example, multiple general FL models can be used to extract features for each given general FL channel, on top of which a single cell separation model trained to distinguish between focused cells and noise (such as debris, aggregates) can be used.

[0064] Figure 7The diagram illustrates a diagram of an exemplary model repository in accordance with some embodiments. Model repository 700 can contain multiple models (feature extractors). General fluorescence (FL) model 702 can be used to extract features for any FL channel. Special FL models 704 (e.g., p65+DAPI for one application, CD19+CD45 for another application) are designed for a specific set of fluorescence as input for a specific application. Bright field (BF) model 706 for single cell isolation can be used to distinguish between noise and focused cells. Special BF model 708 can be used for specific applications. Special dark field (DF) model 710 can be used for specific applications. Although Figure 7 Some example models are shown in , but fewer or additional models can be implemented.

[0065] Phase I of the multi-model cell clustering implementation is similar to Figure 1 In some embodiments, the plurality of models are trained in phase I. However, each of the plurality of models is trained. In some embodiments, each model is trained separately. In some embodiments, all models are trained together. In some embodiments, the models are trained sequentially (e.g., one after another). In some embodiments, the plurality of models are trained simultaneously (e.g., in parallel).

[0066] Figure 8 A flow chart illustrating a method for training a clustering component for a set of cell images using multi-model feature extraction according to some embodiments. In some embodiments, Phase II trains the clustering component before the clustering component can be used for offline initial clustering (Phase III) or online single cell sorting (Phase IV).

[0067] In some embodiments, Phase III can also be used to train clustering components as part of offline initial clustering. In such cases, Phase II is not used, and the clustering components trained in Phase III are used for online single cell sorting (Phase IV). Continuing with this example, offline initial clustering is performed while the clustering components are trained. The user can select a subset of clusters for further analysis, and then the trained clustering components can be used for online single cell sorting in Phase IV.

[0068] In step 800, a group of cell images without reference truth information is acquired. In some embodiments, the group of cell images includes images and / or videos. The group of cell images can include information acquired using an imaging system and processed using one or more image / video processing algorithms. In some embodiments, the imaging system is part of a flow cytometer or other observer for observing and capturing images of cells.

[0069] In step 802, cell features are extracted from a small set of cell images using the multi-model feature extraction trained in stage I. Multi-model feature extraction implements multiple different models for extracting different cell features. For example, using multi-model feature extraction, fluorescence features and bright field features can be obtained without having to retrain the model. Feature extraction can occur sequentially or in parallel. For example, fluorescence features and bright field features are extracted in parallel by different models.

[0070] In step 804, a clustering component is trained using a given set of cell images. An example algorithm for clustering is HDBSCAN (hierarchical density-based spatial clustering with noise), which has the advantages of handling an unknown number of clusters and allowing for noise in the input. In some embodiments, a small subset of a given cell image is used to train the clustering component. The clustering component can use multiple models to extract features of the cell images and use the cascaded features for clustering.

[0071] Fig. 9 The diagram illustrates a flow chart of a method for offline initial clustering of a group of cell images using multi-model feature extraction according to some embodiments. This can be used to replace the time-consuming manual gating in traditional FACS. Steps 900, 902 and 904 correspond to steps 800, 802 and 804, respectively. In step 906, the clustering trained in step 804 identifies clusters for all cells. In step 908, the user can select a subset of the cluster of interest for further analysis. The clustering uses the clustering trained with a small subset of given cells in Phase II to identify the clusters of all cells. Once a cluster is identified, the user can select a cluster of interest for further analysis.

[0072] Fig.10 Flowchart illustrating a method for online single cell sorting using multi-model feature extraction according to some embodiments. Step 1000 shows a flow cytometer. Flow cytometry is a technique for detecting and measuring the physical and chemical properties of a population of cells or particles. In the process, a sample containing cells or particles is then suspended in a fluid and injected into a flow cytometer instrument.

[0073] In step 1002, multiple trained (e.g., trained in stage I) feature extractors / models extract cell features from each cell image, which may include images of one or more channels (e.g., bright field, dark field, and others).

[0074] In step 1004, online sorting is performed using the clustering component trained in step 804, thereby classifying the cell features of each cell into cell classes in step 1006. For online cell sorting, this implementation works for unknown numbers of clusters and tolerates noise, even when ground truth is unavailable.

[0075] Cell sorting involves removing cells from an organism and separating them by their type. In image-based cell sorting, cells can be separated based on features of an extracted cell image. Real-time sorting can utilize the definition of clusters. For example, the system compares features / components of cells and determines which cluster the cell most closely matches.

[0076] Although stages I to IV are described herein, in some embodiments, stage II can be combined with stage III. In some embodiments, the order of the steps is important, such as performing stage I before any other stage, or performing stage IV only after stages I-III.

[0077] To utilize the unsupervised, image-based cell clustering and sorting framework described herein, a device such as a flow cytometer with an imaging system (e.g., one or several cameras) is used to acquire content, and the device is capable of processing the acquired content. The unsupervised, image-based cell clustering and sorting framework can be implemented with the help of a user, or automatically without user involvement.

[0078] In operation, the unsupervised, image-based cell clustering and sorting framework can be used even when ground truth is unavailable, and feature extraction based on very few neural network layers speeds up extraction to enable real-time applications. In addition, the unsupervised, image-based cell clustering and sorting framework allows applications to sort cells based on cell images with or without fluorescent markers.

[0079] Some Examples of Image-Based Unsupervised Multi-Model Cell Clustering

[0080] 1. A method comprising:

[0081] Perform offline initial unsupervised clustering based on images;

[0082] training a plurality of models, wherein each model of the plurality of models is designed to extract features of each cell; and

[0083] Perform image-based online single-cell sorting.

[0084] 2. A method according to clause 1, wherein each model in the plurality of models is different.

[0085] 3. The method of clause 1, wherein the plurality of models are stored in a model repository.

[0086] 4. A method according to clause 1, wherein each of the multiple models is based on a multi-layer neural network.

[0087] 5. A method according to clause 1, wherein performing offline initial clustering of a set of cell images comprises extracting features of the cell images using the multiple models, training a clustering component in an unsupervised manner using a small subset of the given cell images, and determining a cluster for each given cell using the clustering component.

[0088] 6. A method according to clause 1, wherein performing image-based online single-cell sorting comprises extracting features of the cell image using the multiple models, and determining the cluster of each given cell using a clustering component.

[0089] 7. A method according to clause 1, wherein clustering comprises utilizing hierarchical density-based spatial clustering.

[0090] 8. A method according to clause 7, wherein clustering is performed to separate and group different types of cells based on extracted features that comprise the definition of each cluster.

[0091] 9. A device comprising:

[0092] Non-transitory memory for storing applications for:

[0093] Perform offline initial unsupervised clustering based on images;

[0094] training a plurality of models, wherein each model of the plurality of models is designed to extract features of each cell; and

[0095] Perform image-based online single-cell sorting; and

[0096] A plurality of processing units configured to process the application, wherein the plurality of processing units include at least one central processing unit and at least one graphics processing unit.

[0097] 10. An apparatus according to clause 9, wherein each model of the plurality of models is different.

[0098] 11. An apparatus according to clause 9, wherein the plurality of models are stored in a model repository.

[0099] 12. An apparatus according to clause 9, wherein each of the plurality of models is based on a multi-layer neural network.

[0100] 13. An apparatus according to clause 9, wherein performing offline initial clustering of a set of cell images comprises extracting features of the cell images using the multiple models, training a clustering component in an unsupervised manner using a small subset of the given cell images, and determining a cluster for each given cell using the clustering component.

[0101] 14. An apparatus according to clause 9, wherein performing image-based online single cell sorting comprises extracting features of the cell image using the plurality of models, and determining a cluster for each given cell using a clustering component.

[0102] 15. An apparatus according to clause 9, wherein clustering comprises utilizing hierarchical density-based spatial clustering.

[0103] 16. Apparatus according to clause 15, wherein clustering separates and groups different types of cells based on extracted features comprising the definition of each cluster.

[0104] 17. A system comprising:

[0105] A first computing device, the first device being configured to:

[0106] Perform offline initial unsupervised clustering based on images;

[0107] training a plurality of models, wherein each model of the plurality of models is designed to extract features of each cell; and

[0108] Perform image-based online single-cell sorting; and

[0109] A second computing device is configured to send the one or more images to the first computing device.

[0110] 18. A system according to clause 17, wherein each model of the plurality of models is different.

[0111] 19. A system according to clause 17, wherein the plurality of models are stored in a model repository.

[0112] 20. A system according to clause 17, wherein each of the plurality of models is based on a multi-layer neural network.

[0113] 21. A system according to clause 17, wherein performing offline initial clustering of a set of cell images comprises extracting features of the cell images using the multiple models, training a clustering component in an unsupervised manner using a small subset of the given cell images, and determining a cluster for each given cell using the clustering component.

[0114] 22. A system according to clause 17, wherein performing online image-based single cell sorting comprises extracting features of the cell image using the plurality of models, and determining a cluster for each given cell using a clustering component.

[0115] 23. A system according to clause 17, wherein clustering comprises utilizing hierarchical density based spatial clustering.

[0116] 24. A system according to clause 23, wherein clustering separates and groups different types of cells based on extracted features that comprise the definition of each cluster.

[0117] The present invention has been described by way of specific embodiments including details in order to facilitate understanding of the principles of construction and operation of the present invention. Such reference to specific embodiments and details thereof herein is not intended to limit the scope of the claims appended hereto. It is clear to those skilled in the art that other various modifications may be made in the embodiments selected for illustration without departing from the spirit and scope of the present invention as defined in the claims.

Claims

1. A method comprising: Perform offline initial unsupervised clustering based on images; training a plurality of models, wherein each model of the plurality of models is designed to extract features of each cell; as well as Perform image-based online single-cell sorting.

2. The method of claim 1, wherein each of the plurality of models is different.

3. The method of claim 1, wherein the plurality of models are stored in a model repository.

4. The method of claim 1, wherein each of the plurality of models is based on a multi-layer neural network.

5. A method according to claim 1, wherein performing offline initial clustering of a set of cell images includes using the multiple models to extract features of the cell images, training a clustering component in an unsupervised manner using a small subset of given cell images, and using the clustering component to determine the cluster of each given cell.

6. The method of claim 1, wherein performing image-based online single-cell sorting comprises extracting features of cell images using the multiple models, and determining a cluster for each given cell using a clustering component.

7. The method of claim 1, wherein clustering comprises utilizing hierarchical density-based spatial clustering.

8. The method of claim 7, wherein clustering separates and groups different types of cells based on extracted features that comprise the definition of each cluster.

9. An apparatus comprising: Non-transitory memory for storing applications for: Perform offline initial unsupervised clustering based on images; training a plurality of models, wherein each model of the plurality of models is designed to extract features of each cell; and Perform image-based online single-cell sorting; and A plurality of processing units configured to process the application, wherein the plurality of processing units include at least one central processing unit and at least one graphics processing unit.

10. The apparatus of claim 9, wherein each of the plurality of models is different.

11. The apparatus of claim 9, wherein the plurality of models are stored in a model repository.

12. The apparatus of claim 9, wherein each of the plurality of models is based on a multi-layer neural network.

13. An apparatus according to claim 9, wherein performing offline initial clustering of a set of cell images includes using the multiple models to extract features of the cell images, training a clustering component in an unsupervised manner using a small subset of given cell images, and using the clustering component to determine the cluster of each given cell.

14. The apparatus of claim 9, wherein performing image-based online single-cell sorting comprises extracting features of cell images using the plurality of models, and determining a cluster for each given cell using a clustering component.

15. The apparatus of claim 9, wherein clustering comprises utilizing hierarchical density-based spatial clustering.

16. The apparatus of claim 15, wherein clustering separates and groups different types of cells based on extracted features that comprise the definition of each cluster.

17. A system comprising: A first computing device, the first device being configured to: Perform offline initial unsupervised clustering based on images; training a plurality of models, wherein each model of the plurality of models is designed to extract features of each cell; and Perform image-based online single-cell sorting; and A second computing device is configured to send the one or more images to the first computing device.

18. The system of claim 17, wherein each model of the plurality of models is different.

19. The system of claim 17, wherein the plurality of models are stored in a model repository.

20. The system of claim 17, wherein each of the plurality of models is based on a multi-layer neural network.

21. A system according to claim 17, wherein performing offline initial clustering of a set of cell images includes using the multiple models to extract features of the cell images, training a clustering component in an unsupervised manner using a small subset of given cell images, and using the clustering component to determine the cluster of each given cell.

22. The system of claim 17, wherein performing online image-based single cell sorting comprises extracting features of cell images using the plurality of models, and determining a cluster for each given cell using a clustering component.

23. The system of claim 17, wherein clustering comprises utilizing hierarchical density-based spatial clustering.

24. The system of claim 23, wherein clustering separates and groups different types of cells based on extracted features that comprise the definition of each cluster.

Images