Framework for Image-Based Unsupervised Cell Clustering and Sorting

Abstract

This specification describes a framework that includes a feature extractor and cluster components for clustering. The framework supports (1) offline, image-based, unsupervised clustering, which replaces time-consuming manual gating, and (2) online, image-based, single-cell sorting. During training, one or more cell image datasets, with or without ground truth, are used to train the feature extractor. The feature extractor is based on a neural network containing several convolutional layers. Once trained, the feature extractor is used to extract cell image features for unsupervised cell clustering and sorting. Additionally, after the feature extractor is trained, additional datasets can be used to further refine the feature extractor.

Description

Technical Field

[0001] [Cross - Reference to Related Applications] This application claims priority under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No. 63 / 116,065, filed on November 19, 2020, entitled "UNSUPERVISED LEARNING FRAMEWORK FOR IMAGE BASED SINGLE CELL SORTING", the entire disclosure of which is incorporated herein by reference for all purposes.

[0002] The present invention relates to cell sorting. More specifically, the present invention relates to image - based cell sorting.

Background Art

[0003] Conventional fluorescence - activated cell sorting relies on labeling cells with fluorescent markers, and the morphological information of cells is very limited. However, some applications require morphological information of cells to accurately sort cells, while some applications are not suitable for using fluorescent markers. Also, conventional fluorescence - activated cell sorting (FACS) uses manual gating to establish sorting criteria based on fluorescent markers. However, manual gating is time - consuming and may be biased.

[0004] Some studies have proposed image - based cell sorting that uses supervised learning based on deep neural networks or hand - crafted features. The studies assumed cell images that include ground truth for training, but ground truth may not be available. Any software that helps with the gating process relies on specific hand - crafted features of fluorescent markers, but such features may not have sufficient morphological information for some applications or may not be suitable for some other applications.

Summary of the Invention

Problems to be Solved by the Invention

[0005] This specification describes a framework that includes a feature extractor and a clustering component for clustering. The framework supports (1) offline image-based unsupervised clustering that replaces time-consuming manual gating, and (2) online image-based single-cell sorting. During training, the feature extractor is trained using one or more cell image datasets with or without ground truth. The feature extractor is based on a neural network that includes several convolutional layers. Once the feature extractor is trained, it is used to extract features of cell images for unsupervised cell clustering and sorting. Also, after the feature extractor is trained, an additional dataset can be used to further refine the feature extractor.

Means for Solving the Problems

[0006] In one aspect, the method includes performing offline initial image-based unsupervised clustering and performing online image-based single cell sorting. The method further includes training including receiving one or more cell image datasets with or without ground truth information and training a feature extractor. The feature extractor is based on a neural network including several convolutional layers. The step of performing offline initial clustering on a set of cell images includes using the feature extractor to extract features of the cell images, using a small subset of the given cell images to train cluster components, and using the cluster components to identify clusters of the given cell images in an unsupervised manner. The step of performing online image-based single cell sorting includes using the feature extractor to extract features of cell images and using the cluster components for unsupervised cell sorting. The method further includes, after the feature extractor is trained, refining the feature extractor using an additional dataset. Clustering can utilize hierarchical density-based spatial clustering or other clustering algorithms. Clustering separates and groups different types of cells based on the extracted features including the definition of each cluster.

[0007] In another aspect, the apparatus includes a non-transitory memory for storing an application for performing offline initial image-based unsupervised clustering and online image-based single-cell sorting, and a plurality of processing units configured to process the application, the plurality of processing units including at least one central processing unit and at least one graphics processing unit. The application is further for training, including receiving one or more cell image data sets, with or without ground truth information, to train a feature extractor. The feature extractor is based on a neural network including several convolutional layers. Performing offline initial clustering on a set of cell images includes using the feature extractor to extract features of the cell images, using a small subset of the given cell images to train cluster components, and using the cluster components to identify clusters of the given cell images in an unsupervised manner. Performing online image-based single-cell sorting includes using the feature extractor to extract features of cell images and using the cluster components for unsupervised cell sorting. The application is further for refining the feature extractor using an additional data set after the feature extractor has been trained. Clustering can utilize hierarchical density-based spatial clustering or other clustering algorithms. Clustering separates and groups different types of cells based on the extracted features including the definition of each cluster.

[0008] In another aspect, the system includes a first computer device configured to perform offline image-based unsupervised clustering and, based on the offline image-based unsupervised clustering, perform online image-based single-cell sorting, and a second computer device configured to transmit one or more images to the first computer device. The first computer device is configured for training, including receiving one or more cell image datasets with or without ground truth information and training a feature extractor. The feature extractor is based on a neural network including several convolutional layers. Performing offline initial clustering on a set of cell images includes using the feature extractor to extract features of the cell images, using a small subset of the given cell images to train cluster components, and using the cluster components to identify clusters of the given cell images in an unsupervised manner. Performing online image-based single-cell sorting includes using the feature extractor to extract features of cell images and using the cluster components for unsupervised cell sorting. After the feature extractor is trained, the first computer device is further configured to refine the feature extractor using an additional dataset. Clustering can utilize hierarchical density-based spatial clustering or other clustering algorithms. Clustering separates and groups different types of cells based on the extracted features including the definition of each cluster.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Embodiments for Carrying Out the Invention

[0010] The methods and systems described herein include a learning framework that supports (1) offline image-based unsupervised cell clustering that replaces time-consuming manual gating, and (2) online image-based single-cell sorting. This framework includes feature extraction and clustering. During training, a feature extractor is trained using one or more cell image datasets with or without ground truth. The feature extractor is based on a neural network that includes several convolutional layers. Once the feature extractor is trained, it is used to extract features of cell images for unsupervised cell clustering and sorting. Also, after the feature extractor is trained, an additional dataset can be used to further refine the feature extractor. The methods and systems described herein are the first to combine neural network-based feature extraction and clustering in an unsupervised learning framework for image-based offline cell clustering and online single-cell sorting. This replaces time-consuming manual gating in conventional FACS workflows as a tool for offline initial clustering. This improves or enables applications that cannot be accurately performed without morphological information of cells as a tool for online single-cell sorting.

[0011] Conventional fluorescence-activated cell sorting (FACS) relies on labeling cells with fluorescent markers, which provides very limited morphological information about the cells. However, some applications require morphological information about the cells to accurately sort them, while some applications are not suitable for fluorescent markers. The methods and systems described herein implement a framework that enables applications for clustering and sorting cells based on cell images with or without fluorescent markers. Some studies have proposed image-based cell sorting using supervised learning based on deep neural networks or handcrafted features. The studies assumed cell images that include ground truth for training, but it may not be available. The methods and systems described herein implement a framework that enables training with or without ground truth.

[0012] Manual gating in conventional FACS is time-consuming and may be biased. There is software that aids the process but relies on specific handcrafted features that may not provide sufficient information as an image itself. The methods and systems described herein utilize images and deep learning for better performance.

[0013] This document describes a framework that includes a feature extractor and a clustering component for clustering. The framework supports offline image-based unsupervised clustering that replaces time-consuming manual gating and online image-based single cell sorting.

[0014] Figure 1 shows a flowchart of a method for training a feature extractor according to some embodiments. At step 100, a cell image dataset is received. In some embodiments, a dataset with or without ground truth information is received. In some embodiments, the dataset includes images and / or videos. The dataset can be obtained using a specific imaging system (e.g., one or more cameras) and can include information processed using one or more image / video processing algorithms. In some embodiments, the imaging system is part of a flow cytometer or other viewer for displaying and capturing images of cells. The dataset can be sent to and stored on a server and then received (e.g., downloaded) by a computer device implementing the training method. The training can be performed without a teacher.

[0015] At step 102, a feature extractor is implemented. The feature extractor is used to extract features from the images. The feature extractor is based on a neural network. In some embodiments, the feature extractor uses several convolutional layers followed by a pooling layer. An exemplary approach for training the feature extractor is to use contrastive loss, which includes calculating a loss by comparing each sample with a set of positive and negative samples. After the feature extractor is trained, an additional dataset can be used to further refine the feature extractor.

[0016] In step 104, execute the feedback from the clustering. In some embodiments, the feedback clustering is optional. In some embodiments, the cluster optionally provides feedback for training the feature extractor. The clustering can utilize hierarchical density-based clustering or other clustering algorithms. The clustering separates and groups different types of cells based on the extracted features. Hierarchical density-based spatial clustering (HDBSCAN) is an exemplary clustering algorithm that can handle unknown classes. HDBSCAN performs density-based clustering including noise over epsilon values and integrates the results to find a stable clustering. Given a set of points in some space, HDBSCAN groups together points that are densely packed together (e.g., points having many neighboring adjacent points). Although HDBSCAN is described herein, any clustering algorithm can be utilized.

[0017] Figure 2 shows a flowchart of a method for training a cluster component for a set of cell images, according to some embodiments. In some embodiments, after Phase II trains the cluster component, the cluster component can be used in offline initial clustering (Phase III) or online single-cell sorting (Phase IV).

[0018] In some embodiments, Phase III can also be used to train the cluster component as part of the offline initial clustering. In such a case, Phase II is not used and the cluster component trained in Phase III is used for online single-cell sorting (Phase IV). Taking this example further, perform the offline initial clustering and at the same time train the cluster component. The user can select a subset of the clusters for further analysis and then use the trained cluster component for online single-cell sorting in Phase IV.

[0019] In step 200, a small set of cell images that do not contain ground truth information is obtained. In some embodiments, the set of cell images includes images and / or videos. The set of cell images is obtained using an imaging system and can include information processed using one or more image / video processing algorithms. In some embodiments, the imaging system is part of a flow cytometer or other viewer for displaying and capturing images of cells.

[0020] In step 202, cell features are extracted from the small set of cell images using the feature extractor trained in step 102.

[0021] In step 204, a given set of cell images is used to train cluster components. An exemplary algorithm for clustering is HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise), which has the advantage of handling an unknown number of clusters and allowing noise in the input.

[0022] Figure 3 shows a flowchart of a method of offline initial clustering for a set of cell images according to some embodiments. This can be used to replace the time-consuming manual gating in conventional FACS. Steps 300, 302, and 304 correspond to steps 200, 202, and 204, respectively. In step 306, the clusters trained in step 204 identify clusters for all cells. In step 308, the user can select a subset of the clusters of interest for further analysis.

[0023] Figure 4 shows a flowchart of a method of online single cell sorting according to some embodiments. Step 400 shows a flow cytometer. A flow cytometer is a technique used to detect and measure the physical and chemical properties of a population of cells or particles. Next, in the process, a sample containing cells or particles is suspended in a fluid and injected into a flow cytometer device.

[0024] In step 402, a trained feature extractor (e.g., the one trained in step 102) extracts cell features from each cell image, which can include one or more channels of the image (e.g., bright field, dark field, etc.).

[0025] In step 404, using the cluster components trained in step 204, online sorting is performed by classifying the cell features of each cell into cell classes in step 406. For online cell sorting, the implementation operates on unknown clusters, allows for noise, and operates even when ground truth is not available.

[0026] Cell sorting involves collecting cells from an organism and separating the cells according to their types. In image-based cell sorting, cells can be separated based on the extracted features of the cell images. Real-time sorting can utilize the definition of clusters. For example, the system compares the features / components of the cells to determine which cluster the cells most closely match.

[0027] Although phases I - IV are described herein, in some embodiments, phase II can be combined with phase III. In some embodiments, the order of steps, e.g., performing phase I before any other optional steps, or performing phase IV only after phases I - III, is important.

[0028] FIG. 5 shows a block diagram of an exemplary computer device configured to implement an unsupervised image-based cell clustering and sorting framework according to some embodiments. The computer device 500 can be used to acquire, store, compute, process, communicate, and / or display information such as images and videos. The computer device 500 can implement any of the aspects of the unsupervised image-based cell clustering and sorting framework. Generally, a hardware structure suitable for implementing the computer device 500 includes a network interface 502, a memory 504, a processor 506, an I / O device 508, a bus 510, and a storage device 512. The choice of the processor is not critical as long as a suitable processor with sufficient speed is selected. The processor 506 can include a plurality of central processing units (CPUs). The processor 506 and / or the hardware 520 can include one or more graphics processing units (GPUs) for efficient feature extraction based on neural networks. Each GPU should be equipped with sufficient GPU memory to execute feature extraction. The memory 504 can be any conventional computer memory known in the art. The storage device 512 can include a hard drive, a CDROM, a CDRW, a DVD, a DVDRW, a high-definition disk / drive, an ultra-high-definition drive, a flash memory card, or any other storage device. The computer device 500 can include one or more network interfaces 502. Examples of network interfaces include network cards connected to Ethernet or other types of LANs. The I / O device 508 can include one or more of the following, namely, a keyboard, a mouse, a monitor, a screen, a printer, a modem, a touch screen, a button interface, and other devices. The unsupervised image-based cell clustering and sorting framework application 530 used to implement the framework is stored in the storage device 512 and the memory 504 and is likely to be processed as an application is normally processed.The computer device 500 can include more or fewer components than shown in FIG. 5. In some embodiments, an unsupervised image-based cell clustering and sorting framework hardware 520 is included. The computer device 500 in FIG. 5 includes an application 530 and hardware 520 for an unsupervised image-based cell clustering and sorting framework, but the unsupervised image-based cell clustering and sorting framework can be implemented in the computer device as hardware, firmware, software, or any combination thereof. For example, in some embodiments, the unsupervised image-based cell clustering and sorting framework application 530 is programmed in memory and executed using a processor. In another example, in some embodiments, the unsupervised image-based cell clustering and sorting framework hardware 520 is programmed hardware logic that includes gates dedicated to implementing the unsupervised image-based cell clustering and sorting framework.

[0029] In some embodiments, the unsupervised image-based cell clustering and sorting framework application 530 includes several applications and / or modules. In some embodiments, a module also includes one or more sub-modules. In some embodiments, fewer or additional modules can be included.

[0030] Examples of suitable computer devices include personal computers, laptop computers, computer workstations, servers, mainframe computers, handheld computers, personal digital assistants, cellular / mobile phones, smart home appliances, gaming consoles, digital cameras, digital camcorders, camera phones, smartphones, portable music players, tablet computers, mobile devices, video players, video disc recorders / players (e.g., DVD recorders / players, high-definition disc recorders / players, ultra-high-definition disc recorders / players), televisions, home entertainment systems, augmented reality devices, virtual reality devices, smart jewelry (e.g., smart watches), vehicles (e.g., self-driving vehicles) or any other suitable computer device.

[0031] FIG. 6 shows a diagram schematically showing the overall configuration of a biological sample analyzer according to some embodiments.

[0032] FIG. 6 shows an exemplary configuration of the biological sample analyzer of the present disclosure. The biological sample analyzer 6100 shown in FIG. 6 includes a light irradiation unit 6101 that irradiates light onto a biological sample S flowing in a flow path C, a detection unit 6102 that detects light generated by irradiating the biological sample S, and an information processing unit 6103 that processes information about the light detected by the detection unit. The biological sample analyzer 6100 is, for example, a flow cytometer or an imaging cytometer. The biological sample analyzer 6100 can include a sorting unit 6104 that sorts specific biological particles P in the biological sample. The biological sample analyzer 6100 including the sorting unit is, for example, a cell sorter.

[0033] (Biological sample) The biological sample S can be a liquid sample containing biological particles. The biological particles can be, for example, cells or acellular biological particles. The cells can be living cells, and more specific examples thereof include blood cells such as red blood cells and white blood cells, and germ cells such as sperm and fertilized eggs. Also, the cells can be directly collected from a sample such as whole blood, or can be cultured cells obtained after culturing. The acellular biological particles are extracellular vesicles, particularly, for example, exosomes and microvesicles. The biological particles can be labeled using one or more labeling substances (for example, dyes (particularly fluorescent dyes), and antibodies labeled with fluorescent dyes). It should be noted that particles other than biological particles can be analyzed by the biological sample analyzer of the present disclosure, and beads or the like can be analyzed for calibration or the like.

[0034] (Flow path) The flow path C is designed such that a flow of the biological sample S is formed. In particular, the flow path C can be designed such that a flow is formed in which the biological particles contained in the biological sample are substantially arranged in a single row. The flow path structure including the flow path C can be designed such that a laminar flow is formed. In particular, the flow path structure is designed such that a laminar flow is formed in which the flow of the biological sample (sample flow) is surrounded by the flow of the sheath fluid. The design of the flow path structure can be appropriately selected by those skilled in the art, or a known structure can also be adopted. The flow path C can be formed in a flow path structure such as a microchip (a chip having a flow path on the order of micrometers) or a flow cell. The width of the flow path C is 1 mm or less, and particularly can be 10 μm or more and 1 mm or less. The flow path C and the flow path structure including the flow path C can be formed of a material such as plastic or glass.

[0035] The biological sample analyzer of the present disclosure is designed such that light from the light irradiation unit 6101 irradiates the biological sample flowing in the flow path C, particularly the biological particles in the biological sample. The biological sample analyzer of the present disclosure can be designed such that the light irradiation point on the biological sample is located within the flow path structure in which the flow path C is formed, or the irradiation point can be designed to be located outside the flow path structure. An example of the former case can be a structure in which light is emitted onto the flow path C in a microchip or a flow cell. In the latter case, light can be irradiated onto the biological particles after they exit the flow path structure (particularly, its nozzle portion), and for example, a jet-in-air type flow cytometer can be employed.

[0036] (Light irradiation unit) The light irradiation unit 6101 includes a light source unit that emits light and a light guiding optical system that guides the light to the irradiation point. The light source unit includes one or more light sources. The type of the light source can be, for example, a laser light source or an LED. The wavelength of the light to be emitted from each light source can be any wavelength of ultraviolet light, visible light, and infrared light. The light guiding optical system includes optical components such as, for example, a beam splitter, a mirror, or an optical fiber. The light guiding optical system can also include a lens group for condensing light, for example, an objective lens. There can be one or more irradiation points where the biological sample and the light intersect. The light irradiation unit 6101 can be designed to collect the light emitted from one light source or different light sources onto one irradiation point.

[0037] (Detection unit) The detection unit 6102 includes at least one photodetector that detects light generated by emitting light onto biological particles. The light to be detected can be, for example, fluorescence or scattered light (such as one or more of forward scattered light, backward scattered light, and side scattered light). Each photodetector includes one or more light receiving elements, for example, having a light receiving element array. Each photodetector can include, as the light receiving element, one or more photomultiplier tubes (PMTs) and / or photodiodes such as APDs and MPPCs. The photodetector includes, for example, a PMT array in which a plurality of PMTs are arranged in a one-dimensional direction. The detection unit 6102 can also include an image sensor such as a CCD or CMOS. Using the image sensor, the detection unit 6102 can acquire an image of the biological particles (such as a bright field image, a dark field image, or a fluorescence image).

[0038] The detection unit 6102 includes a detection optical system that causes light of a predetermined detection wavelength to reach the corresponding photodetector. The detection optical system includes a spectroscopic unit such as a prism or a diffraction grating, or a wavelength separation unit such as a dichroic mirror or an optical filter. The detection optical system is designed to, for example, disperse the light generated by irradiating the biological particles and detect the dispersed light using a number of photodetectors greater than the number of fluorescent dyes used for labeling the biological particles. A flow cytometer including such a detection optical system is called a spectral flow cytometer. Further, the detection optical system is designed to, for example, separate the light corresponding to the fluorescence wavelength band of a specific fluorescent dye from the light generated by irradiating the biological particles and cause the corresponding photodetector to detect the separated light.

[0039] The detection unit 6102 can also include a signal processing unit that converts an electrical signal acquired by a photodetector into a digital signal. The signal processing unit can include an A / D converter as a device that executes the conversion. The digital signal acquired by the conversion executed by the signal processing unit can be transmitted to the information processing unit 6103. The digital signal can be handled by the information processing unit 6103 as data related to light (hereinafter also referred to as "light data"). The light data can be, for example, light data including fluorescence data. More specifically, the light data can be data of light intensity, and the light intensity can be light intensity data of light including fluorescence (the light intensity data can include feature amounts such as area, height, and width).

[0040] (Information Processing Unit) The information processing unit 6103 includes a processing unit that executes processing of various types of data (e.g., light data) and a storage unit that stores various types of data, for example. When the processing unit acquires light data corresponding to a fluorescent dye from the detection unit 6102, the processing unit can execute fluorescence bleed-through correction (compensation process) on the light intensity data. In the case of a spectral flow cytometer, the processing unit can also execute a fluorescence separation process on the light data to acquire light intensity data corresponding to the fluorescent dye. The fluorescence separation process can be executed, for example, by an unmixing method disclosed in Japanese Patent Application Laid-Open No. 2011-232259. When the detection unit 6102 includes an image sensor, the processing unit can acquire morphological information about biological particles based on an image acquired by the image sensor. The storage unit can be designed to store the acquired light data. The storage unit can be designed to further store spectral reference data to be used in the unmixing process.

[0041] When the biological sample analyzer 6100 includes the sorting unit 6104 described below, the information processing unit 6103 can determine whether biological particles should be sorted based on optical data and / or morphological information. Next, the information processing unit 6103 controls the sorting unit 6104 based on the result of the determination, and the sorting unit 6104 can sort the biological particles.

[0042] The information processing unit 6103 can be designed to output various types of data (e.g., optical data and images, etc.). For example, the information processing unit 6103 can output various types of data (e.g., two-dimensional plots or spectral plots, etc.) generated based on optical data. The information processing unit 6103 can also be designed to accept inputs of various types of data, for example, accept the gating process on the plot by the user. The information processing unit 6103 includes an output unit (e.g., a display, etc.) or an input unit (e.g., a keyboard, etc.) and can perform output or input.

[0043] The information processing unit 6103 can be designed as a general-purpose computer, for example, designed as an information processing device including a CPU, a RAM, and a ROM. The information processing unit 6103 can be included within the housing containing the light irradiation unit 6101 and the detection unit 6102, or can be located outside the housing. Furthermore, various processes or functions to be executed by the information processing unit 6103 can be realized by a server computer or a cloud connected via a network.

[0044] (Sorting Unit) The sorting unit 6104 performs sorting of biological particles according to the result of determination executed by the information processing unit 6103. The sorting method can be a method in which droplets containing biological particles are generated by vibration, charges are applied to the droplets to be sorted, and the moving direction of the droplets is controlled by electrodes. The sorting method can be a method of sorting by controlling the moving direction of biological particles in a flow path structure. The flow path structure has, for example, a control mechanism based on pressure (injection or suction) or charge. An example of the flow path structure can be a chip having a flow path structure in which the flow path C branches into a recovery flow path and a waste liquid flow path on the downstream side, and specific biological particles are collected in the recovery flow path (for example, the chip disclosed in JP-A-2020-076736).

[0045] To utilize the teacherless image-based cell clustering and sorting framework described in this specification, a device such as a flow cytometer including a camera is used to acquire content, and the device can process the acquired content. The teacherless image-based cell clustering and sorting framework can be implemented automatically with user assistance or without user involvement.

[0046] During operation, even when ground truth is not available, the teacherless image-based cell clustering and sorting framework can be used, and feature extraction based on a very small number of neural network layers improves the extraction speed for real-time applications. Further, the teacherless image-based cell clustering and sorting framework enables an application to sort cells based on cell images with or without a fluorescent marker.

[0047] Some embodiments of a framework for image-based teacherless cell clustering and sorting 1. A method comprising: performing offline initial image-based teacherless clustering; and Performing online image-based single cell sorting, and A method comprising the steps of.

[0048] 2. The method according to item 1, further comprising training including receiving one or more cell image datasets with or without ground truth information and training a feature extractor.

[0049] 3. The method according to item 2, wherein the feature extractor is based on a neural network including several convolutional layers.

[0050] 4. The step of performing offline initial clustering on a set of cell images includes extracting features of the cell images using the feature extractor, training cluster components using a small subset of the given cell images, and identifying clusters of the given cell images in an unsupervised manner using the cluster components. The method according to item 2.

[0051] 5. The step of performing online image-based single cell sorting includes extracting features of cell images using the feature extractor and using the cluster components for unsupervised cell sorting. The method according to item 2.

[0052] 6. The method according to item 2, further comprising refining the feature extractor using an additional dataset after the feature extractor is trained.

[0053] 7. The method according to item 1, wherein clustering includes using hierarchical density-based spatial clustering.

[0054] 8. The method according to item 7, wherein clustering separates and groups different types of cells based on the extracted features including the definition of each cluster.

[0055] 9. An apparatus, Execute offline initial image-based unsupervised clustering, Execute online image-based single-cell sorting, A non-transitory memory for storing an application for, A plurality of processing units configured to process the application, the plurality of processing units including at least one central processing unit and at least one graphics processing unit, An apparatus including.

[0056] 10. The apparatus according to claim 9, wherein the application is further for training, including receiving one or more cell image data sets, with or without ground truth information, to train a feature extractor.

[0057] 11. The apparatus according to claim 10, wherein the feature extractor is based on a neural network including several convolutional layers.

[0058] 12. Executing offline initial clustering on a set of cell images includes extracting features of the cell images using the feature extractor, training cluster components using a small subset of the given cell images, and identifying clusters of the given cell images in an unsupervised manner using the cluster components. The apparatus according to claim 10, characterized by including.

[0059] 13. Executing online image-based single-cell sorting includes extracting features of cell images using the feature extractor and using the cluster components for unsupervised cell sorting. The apparatus according to claim 10, including.

[0060] 14. The apparatus according to claim 10, wherein the application is further for refining the feature extractor using an additional data set after the feature extractor is trained.

[0061] 15. The apparatus according to item 9, wherein clustering includes using hierarchical density-based spatial clustering.

[0062] 16. The apparatus according to item 15, wherein clustering separates and groups different types of cells based on the extracted features including the definition of each cluster.

[0063] 17. A system, configured to perform offline image-based unsupervised clustering, and perform online image-based single-cell sorting based on the offline image-based unsupervised clustering, a first computer device; and a second computer device configured to transmit one or more images to the first computer device. A system including the above.

[0064] 18. The system according to item 17, wherein the first computer device is configured for training including receiving one or more cell image datasets with or without ground truth information and training a feature extractor.

[0065] 19. The system according to item 18, wherein the feature extractor is based on a neural network including several convolutional layers.

[0066] 20. Performing offline initial clustering on a set of cell images includes extracting features of the cell images using the feature extractor, training cluster components using a small subset of the given cell images, and identifying clusters of the given cell images in an unsupervised manner using the cluster components, according to the system of item 18.

[0067] 21. Executing online image-based single-cell sorting includes extracting features of cell images using the feature extractor and using the cluster components for unsupervised cell sorting, for the system according to claim 18.

[0068] 22. The first computer device is further configured to refine the feature extractor using an additional dataset after the feature extractor is trained, for the system according to claim 18.

[0069] 23. Clustering includes using hierarchical density-based spatial clustering, for the system according to claim 17.

[0070] 24. Clustering separates and groups different types of cells based on the extracted features including the definition of each cluster, for the system according to claim 23.

[0071] For the purpose of facilitating the understanding of the structure and operating principle of the present invention, the present invention has been described with respect to specific embodiments incorporating detailed content. Such references to specific embodiments and their detailed content in this specification are not intended to limit the scope of the claims appended hereto. It will be readily understood by those skilled in the art that various other modifications can be made to the embodiments selected for illustration without departing from the spirit and scope of the present invention defined by the claims.

Explanation of Signs

[0072] 100 Receive cell image dataset 102 Implement feature extractor 104 Execute feedback from clustering 200 Obtain a set of cell images without ground truth information 202 Extract cell features 204 Train cluster components 300 Obtain a set of cell images without ground truth information 302 Extract cell features 304 Train cluster components 306 The trained clusters identify clusters 308 The user selects a subset of the clusters 400 Flow cytometer 402 The trained feature extractor extracts cell features 500 Computer device 502 Network interface 504 Memory 506 Processor 508 I / O device 510 Bus 512 Storage device 520 Unsupervised image-based cell clustering and sorting framework hardware 530 Unsupervised image-based cell clustering and sorting framework application 6101 Light irradiation unit 6102 Detection unit 6103 Information processing unit 6104 Sorting unit C Flow channel P Biological particle S Biological sample

Claims

1. A method comprising: training a feature extractor, the step of training the feature extractor comprising receiving one or more cell image datasets with or without ground truth information and training the feature extractor; performing offline initial image-based unsupervised clustering, the step of performing the offline initial image-based unsupervised clustering on a set of cell images comprising extracting features of the cell images using the feature extractor, training cluster components using a small subset of the given cell images, and identifying clusters of the given cell images in an unsupervised manner using the cluster components; performing online image-based single cell sorting, the step of performing online image-based single cell sorting comprising extracting features of cell images using the feature extractor and using the cluster components for unsupervised cell sorting; A method characterized by comprising the above steps.

2. The method according to claim 1, wherein the feature extractor is based on a neural network comprising several convolutional layers.

3. The method according to claim 1, further comprising the step of refining the feature extractor using an additional dataset after the feature extractor has been trained.

4. The method according to claim 1, wherein the clustering comprises utilizing hierarchical density-based spatial clustering.

5. The method according to claim 4, wherein the clustering separates and groups different types of cells based on the extracted features including the definition of each cluster.

6. An apparatus comprising: training a feature extractor, the step of training the feature extractor comprising receiving one or more cell image datasets with or without ground truth information and training the feature extractor; A step of performing offline initial image-based unsupervised clustering, wherein the step of performing the offline initial image-based unsupervised clustering on a set of cell images includes: using the feature extractor to extract features of the cell images; using a small subset of the given cell images to train cluster components; and using the cluster components to identify clusters of the given cell images in an unsupervised manner. A step of performing online image-based single cell sorting, which includes: using the feature extractor to extract features of cell images; and using the cluster components for unsupervised cell sorting. A non-transitory memory for storing an application for executing the above. A plurality of processing units configured to process the application, including at least one central processing unit and at least one graphics processing unit. An apparatus, characterized by including the above.

7. The apparatus according to claim 6, wherein the feature extractor is based on a neural network including several convolutional layers.

8. The apparatus according to claim 6, wherein the application is further for refining the feature extractor using an additional dataset after the feature extractor is trained.

9. The apparatus according to claim 6, wherein the clustering includes using hierarchical density-based spatial clustering.

10. The apparatus according to claim 9, wherein the clustering separates and groups different types of cells based on the extracted features including the definition of each cluster.

11. A system, A step of training a feature extractor, which includes receiving one or more cell image datasets with or without ground truth information and training the feature extractor. Steps for performing offline image-based unsupervised clustering, wherein the step of performing the offline initial image-based unsupervised clustering on a set of cell images includes: using the feature extractor to extract features of the cell images; using a small subset of the given cell images to train cluster components; and using the cluster components to identify clusters of the given cell images in an unsupervised manner. Steps for performing online image-based single-cell sorting, which include: using the feature extractor to extract features of cell images; and using the cluster components for unsupervised cell sorting. A first computer device configured to perform the above. A second computer device configured to transmit the one or more cell image datasets to the first computer device. A system characterized by including the above. Claim 12 The system according to claim 11, wherein the feature extractor is based on a neural network including several convolutional layers. Claim 13 The system according to claim 11, wherein the first computer device is further configured to refine the feature extractor using an additional dataset after the feature extractor is trained. Claim 14 The system according to claim 11, wherein clustering includes utilizing hierarchical density-based spatial clustering. Claim 15 The system according to claim 14, wherein clustering separates and groups different types of cells based on the extracted features including the definition of each cluster.

Images