Identification and classification module for sorting cells based on nuclear translocation of fluorescent signals

Abstract

The Image Activated Cell Sorting (IACS) classification workflow involves extracting features from cell images using a neural network-based feature encoder (or extractor), automatically clustering cells based on the extracted cell features, identifying clusters based on the cell images and selecting which cluster(s) to sort, fine-tuning a classification network based on the selected cluster(s), and, once refined, using the classification network to sort cells for real-time live sorting.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority under 35 U.S.C. § 119 to U.S. Provisional Patent Application No. 63 / 377,788, entitled "MODULE FOR IDENTIFICATION AND CLASSIFICATION TO SORT CELLS BASED ON THE NUCLEAR TRANSLOCATION OF FLUORESCENCE SIGNALS," filed September 30, 2022, which is incorporated herein by reference in its entirety for all purposes.

[0002] The present invention relates to cell sorting, and more particularly to image-based cell sorting. [Background technology]

[0003] Conventional fluorescence-activated cell sorting relies on labeling cells with fluorescent markers, which provides very limited information about cell morphology. However, some applications require cell morphology information for accurate cell sorting, and some applications do not allow for the use of fluorescent markers. Furthermore, conventional fluorescence-activated cell sorting (FACS) requires manual gating to establish sorting criteria based on fluorescent markers. However, manual gating is time-consuming and can introduce bias.

[0004] Several studies have proposed image-based cell sorting using deep neural networks or supervised learning based on manually created features. These studies assumed ground-truth cell images for training, which may not be available. Some software to assist the gating process relies on specific manually created features of fluorescent markers, but morphological information may not be sufficient for some applications and may not be suitable for some others. Summary of the Invention [Problem to be solved by the invention]

[0005] The Image Activated Cell Sorting (IACS) classification workflow involves extracting features from cell images using a neural network-based feature encoder (or extractor), automatically clustering cells based on the extracted cell features, identifying clusters based on the cell images and selecting which cluster(s) to sort, fine-tuning a classification network based on the selected cluster(s), and, once refined, using the classification network to sort cells for real-time live sorting. [Means for solving the problem]

[0006] In one aspect, the method includes extracting one or more features from a cell image using a neural network-based feature encoder; clustering one or more cells from the cell image based on the extracted one or more features to generate one or more clusters; identifying clusters of the one or more clusters to be sorted; fine-tuning a classification network based on the clusters; and performing real-time live sorting of a set of cells using the classification network. The one or more features include a target protein based on a fluorescent dye. The clustering of the one or more cells is based on the location of the target protein. When the target protein is in the cytoplasm, the one or more cells are clustered as dormant cells, and when the target protein is in the nucleus, the one or more cells are clustered as activated cells. Identifying the clusters to be sorted is based on a user manually identifying the clusters. Identifying the clusters to be sorted is based on machine learning to identify the clusters. Fine-tuning the classification network includes performing training with an additional dataset based on the clusters.

[0007] In another aspect, an apparatus includes: a non-transitory memory for storing an application for extracting one or more features from a cell image using a neural network-based feature encoder; clustering one or more cells from the cell image based on the extracted one or more features to generate one or more clusters; identifying clusters of the one or more clusters to be sorted; fine-tuning a classification network based on the clusters; and performing real-time live sorting of a set of cells using the classification network; and a processor configured to process the application. The one or more features include a target protein based on a fluorescent dye. Clustering the one or more cells is based on the location of the target protein. When the target protein is in the cytoplasm, the one or more cells are clustered as dormant cells, and when the target protein is in the nucleus, the one or more cells are clustered as activated cells. Identifying the clusters to be sorted is based on a user manually identifying the clusters. Identifying the clusters to be sorted is based on machine learning to identify the clusters. Fine-tuning the classification network includes performing training with additional datasets based on the clusters.

[0008] In another aspect, the system includes a first computing device configured to transmit one or more cell images to a second computing device; and a second computing device configured to extract one or more features from the one or more cell images using a neural network-based feature encoder, cluster one or more cells from the cell images based on the extracted one or more features to generate one or more clusters, identify clusters of the one or more clusters to be sorted, fine-tune a classification network based on the clusters, and perform real-time live sorting of a set of cells using the classification network. The one or more features include a target protein based on a fluorescent dye. Clustering the one or more cells is based on the location of the target protein. When the target protein is in the cytoplasm, the one or more cells are clustered as dormant cells, and when the target protein is in the nucleus, the one or more cells are clustered as activated cells. Identifying the clusters to be sorted is based on a user manually identifying the clusters. Identifying the clusters to be sorted is based on machine learning to identify the clusters. Fine-tuning the classification network includes performing training with additional datasets based on the clusters. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram of protein nuclear translocation during cell activation according to some embodiments. [Figure 2] 1 is a flowchart of an image-activated cell sorting (IACS) classification workflow according to some embodiments. [Figure 3] FIG. 1 is a diagram of an integrated nuclear transport module according to some embodiments. [Figure 4] FIG. 1 is a diagram of an integrated nuclear transport module according to some embodiments. [Figure 5] 1 is a diagram of the results of an exemplary implementation according to some embodiments. [Figure 6] FIG. 1 is a block diagram of an exemplary computing device configured to implement identification and classification implementations according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0010] The identification and classification implementation is used to sort cells based on nuclear translocation of fluorescent signals. The identification and classification implementation may utilize the clustering implementation described in U.S. Patent Application No. 18 / 070,352, entitled "IMAGE-BASED UNSUPERVISED MULTI-MODEL CELL CLUSTERING," filed November 28, 2022, which is incorporated herein by reference in its entirety for all purposes.

[0011] Figure 1 shows a diagram of protein nuclear translocation during cell activation, according to some embodiments. In this diagram, in a dormant cell 100, the nucleus 110 is labeled with a red fluorescent dye and the target protein 112 is labeled with a green fluorescent dye. In the diagram, an outline is shown to help distinguish the nucleus 110 from the protein 112, but in the real world, there may not be a clear dividing line. After activation, the activated cell 102 has the target protein 112 throughout the cell, including the nucleus, so the entire cell appears one color (e.g., green) or mostly one color.

[0012] Proteins are typically located in the cytoplasm or outside the nucleus of a cell and are fluorescently labeled. When a cell is activated (e.g., by disease, drugs, or other external stimuli), proteins translocate into the nucleus to promote gene and protein expression. For example, the nucleus is labeled with a red fluorescent dye, and a protein (or proteins) is labeled with a green fluorescent dye. The target protein typically belongs to the NF-κB family, but any other target protein can be used. The amount of fluorescent signal does not change. Rather, the location of the fluorescence changes, making imaging important rather than traditional flow cytometry, which simply measures the total fluorescent signal. In resting cells (also known as the cytoplasm), the green fluorescent signal (or other color) is located outside the nucleus and is typically very different from the nucleus (e.g., red / green contrast). In activated cells, the green fluorescent signal is seen to be spread throughout the cell, including the nucleus. While the total fluorescent signal does not change between resting and activated cells, the appearance of the cell clearly changes. Any type of image processing or analysis can be used to detect color changes and / or shifts (e.g., by detecting a particular shape or change in shape, detecting a shift in color, detecting a loss of color, detecting an increase in the amount of one color and a decrease in the amount of another color). Machine learning (ML) and / or artificial intelligence (AI) can be used to perform the image analysis / processing. The process can be permanent (e.g., the change in appearance remains) or temporary (e.g., a temporary change followed by a return to original appearance). For example, a user may see a response to a stimulus, but activated cells may revert to dormant cells after a period of time (e.g., 1 hour).

[0013] Nuclear transport assays are not possible with conventional cell sorters. However, cell imaging and imaging-based cell sorting make nuclear transport assays possible. Using nuclear transport assays, cell images can be sorted based on whether the cells remain dormant or are activated, and those cells can be further studied (e.g., single-cell genomics). For example, further analysis can be performed to determine why certain cells are activated or not. There are many pharmaceutical applications (e.g., new drug development).

[0014] FIG. 2 shows a flowchart of an image-activated cell sorting (IACS) classification workflow according to some embodiments. In step 200, a neural network-based feature encoder (or extractor) is used to extract features from a cell image (e.g., fluorescent dyes of a target protein, which may be in the cytoplasm or nucleus of the cell). For example, the neural network-based feature encoder can be trained to detect the location of a dye of a particular color, the amount of a dye of a particular color, the ratio of one dye to another, the location of one dye to another, the shape of a dye of a particular color, and / or any other training for detecting one or more features. Furthering this example, the neural network-based feature encoder is trained to detect a green fluorescent dye in the target protein and a red fluorescent dye in the nucleus. In some embodiments, the feature encoder is based on a multi-layer neural network. In some embodiments, the feature encoder uses multiple convolutional layers followed by a pooling layer. To train the feature encoder, an exemplary approach is to use a contrastive loss, which involves comparing each sample with a set of positive and negative samples and calculating a loss. After the feature encoder is trained, additional datasets can be used to further refine the feature encoder.

[0015] In step 202, cells are automatically clustered based on the extracted cellular features. For example, a neural network-based feature encoder can be trained to detect green fluorescent dye in the target protein and red fluorescent dye in the nucleus. When the green dye is located outside the cell with a red center, the cell can be classified / clustered as dormant, whereas when the green dye is dispersed throughout the cell with little or no red dye remaining, the cell can be classified / clustered as activated. Clustering separates and groups different types of cells based on the extracted features (e.g., target protein in the cytoplasm, target protein in the nucleus, or ambiguity as to where the target protein is located). In some embodiments, clustering is optional. In some embodiments, clustering optionally provides feedback to train the feature extractor. Clustering can utilize hierarchical density-based clustering or other clustering algorithms. Hierarchical density-based spatial clustering (HDBSCAN) is an exemplary clustering algorithm that can handle an unknown number of classes. HDBSCAN performs density-based clustering that includes noise in the epsilon value and integrates the results to find stable clustering. Given a set of points in a space, HDBSCAN groups points that are densely packed (e.g., points that have many nearby neighbors). While HDBSCAN is described here, any clustering algorithm can be used.

[0016] In step 204, a user can view the cell images, identify clusters, and select which cluster(s) or type(s) of cluster(s) to filter. For example, a user may want to focus on activated cells and perform further analysis on why those cells were activated (e.g., what do those cells have in common with each other to cause activation). In some embodiments, determining the type of cells within a cluster can be automated using ML / AI or another matching / identification implementation. Similarly, ML / AI can be used to select which clusters to filter. For example, if a drug is being tested and the AI ​​knows (from previous learning) that its goal is to understand why the drug is activated in certain cells, the AI ​​can automatically select the correct cluster for further filtering.

[0017] In step 206, the classification network (e.g., a neural network using AI) is fine-tuned based on the cluster(s) selected by the user or ML / AI. Fine-tuning can be implemented in any manner, such as by performing additional ML. For example, one or more additional datasets are used to train the classification network. The additional datasets can be associated with the selected clusters (e.g., if the clusters are activated cells, the classification network receives an additional dataset of activated cells for training or further fine-tuning).

[0018] Once refined, in step 208, a classification network (e.g., a neural network using AI) is used to sort cells for real-time live sorting. In some embodiments, cell sorting involves removing cells from an organism and separating them according to their type. Image-based cell sorting can separate cells based on extracted features of cell images (e.g., location of target proteins and / or amount of visible nuclei). Real-time sorting can utilize cluster definitions. For example, the system compares cellular features / components and determines which cluster a cell best matches.

[0019] In some embodiments, the order of the steps is changed. In some embodiments, fewer or additional steps are implemented. For example, if a user is performing a nuclear translocation assay, the clustering and supervised classifier has a pre-trained feature extractor to be used instead of the general use-case version of the workflow. The IACS classification workflow is further described in U.S. Patent Application No. 18 / 070,352, filed November 28, 2022, and entitled "Image-Based Unsupervised Multi-Model Cell Clustering."

[0020] Figure 3 shows a diagram of an integrated nuclear transport module according to some embodiments. A portion of the sample (e.g., 10,000-100,000 cells) is run to perform unsupervised clustering. Unsupervised clustering groups cells with similar image features and plots the events on a visualization (t-SNE, UMAP) where the clusters are color-coded. For example, although three major clusters (300, 302, and 304) exist, five clusters were found. Continuing this example, cluster 300 contains cells with nuclear signals, cluster 302 contains cells with cytoplasmic signals, and cluster 304 contains ambiguous cells (e.g., cells where the nucleus or cytoplasm is not clearly distinguishable).

[0021] Figure 4 shows a diagram of an integrated nuclear transport module according to some embodiments. A user can select a cluster to view sample cells. By viewing the sample cells, a user can determine what cells a cluster contains (e.g., cytoplasmic, nuclear, or other). In some embodiments, determining the type of cells within a cluster can be automated using ML and / or AI or another matching / identification implementation.

[0022] In some embodiments, after the user selects clusters, a supervised classifier is refined based on the cluster selection (taking 30 seconds to 1 minute). The supervised classifier is then used to make sorting decisions in real time. In an exemplary implementation, with results shown in Figure 5, nuclear and cytoplasmic cells were classified with greater than 98.4% precision and greater than 80% recall, with classification times per cell of less than 0.4 milliseconds.

[0023] The implementation of the identification and classification can be performed using a GPU-based neural network, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), an AI-based convolutional neural network, or any other implementation.

[0024] FIG. 6 illustrates a block diagram of an exemplary computing device configured to implement an identification and classification implementation according to some embodiments. The computing device 600 can be used to acquire, store, compute, process, communicate, and / or display information, such as images and videos. The computing device 600 can implement any of the aspects of the identification and classification. In general, a hardware configuration suitable for implementing the computing device 600 includes a network interface 602, memory 604, a processor 606, I / O device(s) 608, a bus 610, and storage 612. The selection of the processor(s) is not critical as long as suitable processor(s) with sufficient speed are selected. The processor 606 can include multiple central processing units (CPUs). The processor 606 and / or hardware 620 can include one or more graphics processing units (GPUs) for efficient feature extraction based on neural networks. Each GPU should be equipped with sufficient GPU memory to perform the feature extraction. The memory 604 can be any conventional computer memory known in the art. The storage device 612 may include a hard drive, CD-ROM, CDRW, DVD, DVDRW, high-definition disk / drive, ultra-high-definition drive, flash memory card, or any other storage device. The computing device 600 may include one or more network interfaces 602. An example of a network interface includes a network card connected to an Ethernet or other type of LAN. The I / O device(s) 608 may include one or more of a keyboard, mouse, monitor, screen, printer, modem, touch screen, button interface, and other devices. The identification and classification application(s) 630 used to implement the framework are likely stored in the storage device 612 and memory 604 and processed as applications are typically processed. The computing device 600 may include more or fewer components than those shown in FIG. 6 . In some embodiments, identification and classification hardware 620 is included.6 includes hardware 620 and application 630 for implementing the identification and classification, the identification and classification implementation may be implemented in the computer device as hardware, firmware, software, or any combination thereof. For example, in some embodiments, the identification and classification application 630 is programmed into memory and executed using a processor. As another example, in some embodiments, the identification and classification hardware 620 is programmed hardware logic that includes gates specifically designed to implement the identification and classification implementation.

[0025] In some embodiments, the identification and classification application(s) 630 include several applications and / or modules. In some embodiments, a module also includes one or more sub-modules. In some embodiments, fewer or additional modules may be included.

[0026] Examples of suitable computing devices include a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular / mobile phone, a smart appliance, a game console, a digital camera, a digital camcorder, a camera phone, a smartphone, a portable music player, a tablet computer, a mobile device, a video player, a video disc writer / player (e.g., a DVD writer / player, a high-definition disc writer / player, an ultra-high-definition disc writer / player), a television, a home entertainment system, an augmented reality device, a virtual reality device, smart jewelry (e.g., a smart watch), a vehicle (e.g., an autonomous vehicle), or any other suitable computing device.

[0027] To utilize the identification and classification implementations described herein, an apparatus such as a flow cytometer that includes an imaging system (e.g., one or more cameras or detectors) can be used to acquire content, and the apparatus can process the acquired content. Some imaging systems do not use cameras, but rather reconstruct images from pulse processing from photodiodes, photomultiplier tubes, or other implementations. The identification and classification implementations can be implemented with user assistance or automatically without user involvement.

[0028] In operation, compared to other implementations, the identification and classification implementation described herein is much more accurate and faster. For example, the identification and classification implementation described herein has an accuracy of over 98.4%, compared to approximately 90% accuracy for implementations based on Pearson's correlation coefficient.

[0029] Some embodiments of a module for discrimination and classification for sorting cells based on nuclear translocation of a fluorescent signal 1. A method comprising: extracting one or more features from the cell images using a neural network-based feature encoder; clustering one or more cells from the cell image based on the extracted one or more features to generate one or more clusters; identifying a cluster of the one or more clusters to be culled; fine-tuning a classification network based on the clusters; performing real-time live sorting of a set of cells using the classification network; A method comprising:

[0030] 2. The method of claim 1, wherein the one or more features include a target protein based on a fluorescent dye.

[0031] 3. The method of claim 2, wherein the step of clustering the one or more cells is based on the location of the target protein.

[0032] 4. The method of claim 3, wherein the one or more cells are clustered as dormant cells when the target protein is in the cytoplasm, and the one or more cells are clustered as activated cells when the target protein is in the nucleus.

[0033] 5. The method of claim 1, wherein the step of identifying the clusters to be culled is based on a user manually identifying the clusters.

[0034] 6. The method of claim 1, wherein the step of identifying the clusters to be sorted is based on machine learning to identify the clusters.

[0035] 7. The method of claim 1, wherein the step of fine-tuning the classification network includes performing training with an additional dataset based on the clusters.

[0036] 8. An apparatus comprising: A non-transitory memory for storing an application, said application comprising: extracting one or more features from the cell image using a neural network-based feature encoder; clustering one or more cells from the cell image based on the extracted one or more features to generate one or more clusters; identifying a cluster of the one or more clusters to be sorted; fine-tuning a classification network based on the clusters; performing real-time live sorting of a set of cells using the classification network; a non-transitory memory for performing a processor configured to process the application; An apparatus comprising:

[0037] 9. The device of paragraph 8, wherein the one or more features include targeting proteins based on fluorescent dyes.

[0038] 10. The apparatus of paragraph 9, wherein the clustering of the one or more cells is based on the location of the target protein.

[0039] 11. The device described in paragraph 10, wherein the one or more cells are clustered as dormant cells when the target protein is in the cytoplasm, and the one or more cells are clustered as activated cells when the target protein is in the nucleus.

[0040] 12. The apparatus of claim 8, wherein identifying the clusters to be culled is based on a user manually identifying the clusters.

[0041] 13. The apparatus of claim 8, wherein identifying the clusters to be sorted is based on machine learning to identify the clusters.

[0042] 14. The apparatus of claim 8, wherein fine-tuning the classification network includes performing training with additional datasets based on the clusters.

[0043] 15. A system comprising: a first computing device configured to transmit one or more cellular images to a second computing device; extracting one or more features from the one or more cell images using a neural network-based feature encoder; clustering one or more cells from the cell image based on the extracted one or more features to generate one or more clusters; identifying a cluster of the one or more clusters to be sorted; fine-tuning a classification network based on the clusters; performing real-time live sorting of a set of cells using the classification network; a second computer device configured to: A system including:

[0044] 16. The system of claim 15, wherein the one or more features include a target protein based on a fluorescent dye.

[0045] 17. The system described in paragraph 16, wherein the clustering of the one or more cells is based on the location of the target protein.

[0046] 18. The system described in paragraph 17, wherein the one or more cells are clustered as dormant cells when the target protein is in the cytoplasm, and the one or more cells are clustered as activated cells when the target protein is in the nucleus.

[0047] 19. The system of claim 15, wherein identifying the clusters to be culled is based on a user manually identifying the clusters.

[0048] 20. The system of claim 15, wherein identifying the clusters to be sorted is based on machine learning to identify the clusters.

[0049] 21. The system of claim 15, wherein fine-tuning the classification network includes performing training with additional datasets based on the clusters.

[0050] The present invention has been described with reference to specific embodiments incorporating details to facilitate an understanding of the principles of construction and operation of the invention. Reference herein to specific embodiments and their details is not intended to limit the scope of the claims appended hereto. Those skilled in the art will readily appreciate that various other modifications can be made to the embodiments chosen for illustration without departing from the spirit and scope of the invention as defined by the claims. [Explanation of symbols]

[0051] 100 dormant cells 102 Activated cells 110 Nuclear 112 Target Protein 200 feature encoder 202 Clustering 204 User selects cluster(s) 206 Supervised Classifier 208 Cell Sorting 300, 302, 304 clusters 600 Computer equipment 602 network interface 604 memory 606 processor 608 I / O Devices 610 Bus 612 Storage device 620 Identification and Classification Hardware 630 Identification and Classification Applications

Claims

1. 1. A method comprising: extracting one or more features from the cell images using a neural network-based feature encoder; clustering one or more cells from the cell image based on the extracted one or more features to generate one or more clusters; identifying a cluster of the one or more clusters to be culled; fine-tuning a classification network based on the clusters; using the classification network to perform real-time live sorting of a set of cells; A method comprising:

2. 10. The method of claim 1, wherein the one or more features include a target protein based on a fluorescent dye.

3. 3. The method of claim 2, wherein the step of clustering the one or more cells is based on the location of the target protein.

4. 4. The method of claim 3, wherein the one or more cells are clustered as dormant cells when the target protein is in the cytoplasm, and the one or more cells are clustered as activated cells when the target protein is in the nucleus.

5. 10. The method of claim 1, wherein identifying the clusters to be culled is based on a user manually identifying the clusters.

6. The method of claim 1 , wherein identifying the clusters to be culled is based on machine learning to identify the clusters.

7. The method of claim 1 , wherein the step of fine-tuning the classification network comprises performing training with additional datasets based on the clusters.

8. 1. An apparatus comprising: A non-transitory memory for storing an application, said application comprising: extracting one or more features from the cell images using a neural network-based feature encoder; clustering one or more cells from the cell image based on the extracted one or more features to generate one or more clusters; identifying a cluster of the one or more clusters to be sorted; fine-tuning a classification network based on the clusters; performing real-time live sorting of a set of cells using the classification network; a non-transitory memory for performing a processor configured to process the application; 10. An apparatus comprising:

9. 10. The device of claim 8, wherein the one or more features include a fluorescent dye-based target protein.

10. 10. The apparatus of claim 9, wherein the clustering of the one or more cells is based on the location of the target protein.

11. 11. The device of claim 10, wherein the one or more cells are clustered as dormant cells when the target protein is in the cytoplasm, and the one or more cells are clustered as activated cells when the target protein is in the nucleus.

12. 10. The apparatus of claim 8, wherein identifying the clusters to be culled is based on a user manually identifying the clusters.

13. The apparatus of claim 8 , wherein identifying the clusters to be culled is based on machine learning to identify the clusters.

14. 10. The apparatus of claim 8, wherein fine-tuning the classification network comprises performing training with additional datasets based on the clusters.

15. 1. A system comprising: a first computing device configured to transmit one or more cellular images to a second computing device; extracting one or more features from the one or more cell images using a neural network-based feature encoder; clustering one or more cells from the cell image based on the extracted one or more features to generate one or more clusters; identifying a cluster of the one or more clusters to be sorted; fine-tuning a classification network based on the clusters; performing real-time live sorting of a set of cells using the classification network; a second computer device configured to: A system comprising:

16. 16. The system of claim 15, wherein the one or more features include a fluorescent dye-based target protein.

17. 17. The system of claim 16, wherein the clustering of the one or more cells is based on the location of the target protein.

18. 18. The system of claim 17, wherein the one or more cells are clustered as dormant cells when the target protein is in the cytoplasm, and the one or more cells are clustered as activated cells when the target protein is in the nucleus.

19. 16. The system of claim 15, wherein identifying the clusters to be culled is based on a user manually identifying the clusters.

20. 16. The system of claim 15, wherein identifying the clusters to be culled is based on machine learning to identify the clusters.

21. 16. The system of claim 15, wherein fine-tuning the classification network comprises performing training with additional datasets based on the clusters.

Images