Automated detection and characterization of micro-objects in microfluidic devices
The convolutional neural network processes illuminated and non-illuminated images and generates pixel masks to identify micro objects, solving the problem of difficult micro object detection in microfluidic environments, achieving robust detection and characterization of micro objects, and supporting micro objects repositioning.
Patent Information
- Application Number
- CN201980036784.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-05-31
- Filing Date
- 2019-05-31
- Publication Date
- 2024-11-05
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In microfluidic environments, detection and characterization of micro objects exists difficulties, especially in non-uniform backgrounds, especially when the micro objects are not illuminated in fluorescent images, but are via variable or undetectable associated with the micro objects when visualizing the signal.
Machine learning algorithms such as convolutional neural networks are used to process illuminated and non-illuminated images, generate pixel masks to identify micro objects, and improve detection accuracy through image preprocessing and optical alignment.
It realizes robust detection and characterization of micro-objects, which can efficiently identify micro-objects in complex contexts and support repositioning of micro-objects.
Smart Images

Figure CN112204380B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure generally relates to an automatic method for detecting and characterizing micro-objects in an image. Specifically, the method may include a step for automatically detecting micro-objects (e.g., cells or beads) located within a microfluidic device in an illuminated image (e.g., a bright field image), and a step for using the position of the detected micro-object to measure the characteristics of the micro-object in a corresponding non-illuminated image (e.g., a fluorescent image, an infrared image, an ultraviolet image). Background Art
[0002] Embodiments Efficient and robust detection of micro-objects (such as biological cells or beads) on non-uniform or complex backgrounds is crucial for the automatic manipulation of micro-objects in microfluidic environments. Due to the translucent appearance of some micro-objects, non-uniform backgrounds with features of similar size to such micro-objects seriously affect detection. The challenge of detecting micro-objects in images such as fluorescent images is even more complex, in which the micro-objects are not illuminated, but instead visualized based on the amount of signals associated with each micro-object, which can be variable or undetectable. Some embodiments of the present disclosure relate to robust detection and characterization of micro-objects in microfluidic environments. Characterization can be an n-dimensional representation containing n different types of measurements, which can include one or more signals associated with the micro-object (e.g., fluorescent signals), optionally combined with one or more physical properties detectable in the illuminated image. Other embodiments of the present disclosure relate to repositioning micro-objects in a microfluidic environment based on the micro-object having a desired set of characteristics. Summary of the invention
[0003] In one aspect, a method for automatically detecting micro-objects in an illuminated image (e.g., a bright field image) is provided. The method may include: generating a plurality of pixel masks from the image for corresponding multiple micro-object characteristics; and identifying micro-objects in the image from at least one of the plurality of pixel masks. The method may also include obtaining a count of the identified micro-objects in the image. Generating a plurality of pixel masks may include processing pixel data from the image using a machine learning algorithm such as a convolutional neural network. Each pixel mask includes a set of pixel annotations, and each pixel annotation in the set of pixel annotations represents a probability that a corresponding pixel in the image represents a corresponding micro-object characteristic.
[0004] In another aspect, a method for detecting and characterizing micro-objects in a microfluidic device is provided. The method may include: receiving a first image and one or more second images of the same region of interest in the microfluidic device; preprocessing the first image and the one or more second images to reduce anomalies in the image data; transforming each of the one or more second images to optically align the second image with the first image; processing pixel data in the first image using a machine learning algorithm to detect micro-objects present in the region of interest, wherein detecting each micro-object includes identifying the boundaries of the micro-object; and detecting a signal within each boundary of each detected micro-object in each of the one or more second images. In some embodiments, the first image is an illuminated image, such as a bright field image, and each of the one or more second images is a non-illuminated image, such as a fluorescent image. In some embodiments, each of the one or more second images is a fluorescent image that captures a unique portion of the visible spectrum. In some embodiments, the one or more second images capture non-overlapping portions of the visible spectrum.
[0005] In another aspect, a machine-readable storage device is provided. In some embodiments, the storage device may store non-transitory machine-readable instructions, and execution of the instructions may cause a system including a computer to: store a first illumination image in a memory, the first illumination image may include one or more micro-objects; generate a plurality of pixel masks from the image for corresponding multiple micro-object characteristics in a first module; and obtain a micro-object count from at least one pixel mask in the plurality of pixel masks in a second module. The steps of generating and obtaining may be performed according to any of the methods disclosed herein. The first module and the second module may be identical to each other (i.e., there is a single module), or they may be separate, distinct modules. In some embodiments, a storage device may store non-transitory machine-readable instructions, and execution of the instructions may cause a system including a computer to: use an image element to acquire a first illuminated image (e.g., a bright field image) and one or more second non-illuminated images (e.g., fluorescent images) of the same region of interest in a microfluidic device; use an image preprocessing engine to preprocess the first image and the one or more second images to reduce anomalies in the image data; use an image alignment engine to transform each of the one or more second images to optically align the second image with the first image; use an image processing engine and a machine learning algorithm to process pixel data in the first image to detect micro-objects present in the region of interest; and use a detection engine to detect signals within the boundaries of each detected micro-object in each of the one or more second images. The image element, the image preprocessing engine, and the image alignment engine may be part of an image acquisition unit. Similarly, the image processing engine and the detection engine may be part of a micro-object detection and characterization unit. The image acquisition unit and the micro-object detection and characterization unit may be part of a single unit, or they may be separate units.
[0006] In another aspect, a method for repositioning a micro-object in a microfluidic device including a plurality of isolated docks is provided. The method may include: identifying a group of micro-objects (e.g., having one or more desired characteristics) disposed within the microfluidic device; calculating one or more trajectories, wherein each trajectory is a path connecting a micro-object in the group of micro-objects to an isolated dock in the plurality of isolated docks; selecting a trajectory in the one or more trajectories for one or more micro-objects in the group of micro-objects; and repositioning at least one micro-object in the one or more micro-objects having a selected trajectory by moving the micro-object along its selected trajectory. The step of identifying a group of micro-objects having one or more desired characteristics may be performed by any method disclosed herein.
[0007] In yet another aspect, a method for repositioning a micro-object in a microfluidic device is provided. The method may include: identifying a group of micro-objects (e.g., having one or more desired characteristics) disposed within a specified spatial region of the microfluidic device; calculating a set of vertices that divide the specified spatial region into sub-regions, each sub-region containing one or more micro-objects from the group of micro-objects; generating a first light cage for at least one micro-object from the group of micro-objects based on the calculated set of vertices; and moving the first light cage relative to the specified spatial region of the microfluidic device to reposition the at least one micro-object. The step of identifying a group of micro-objects having one or more desired characteristics may be performed by any method disclosed herein.
[0008] In another aspect, a method for detecting and characterizing micro-objects in a microfluidic device is provided. The method may include receiving a first image and one or more second images of an area of interest in a microfluidic device. The method may also include pre-processing the first image and the one or more second images to reduce anomalies in the image data; transforming each of the one or more second images to optically align the second image with the first image. The method may also include processing pixel data in the first image using a machine learning algorithm to detect micro-objects present in the area of interest, wherein detecting each micro-object includes identifying the boundaries of the micro-object. The method may also include detecting a signal within each boundary of each detected micro-object in each of the one or more second images.
[0009] In another aspect, a non-transitory computer-readable medium is provided, wherein a program for causing a system including a computer to perform a method for automatically detecting and characterizing micro-objects in a microfluidic device is stored. The method may include receiving a first image and one or more second images of an area of interest in a microfluidic device. The method may also include pre-processing the first image and each of the one or more second images to reduce anomalies in the image data; transforming each of the one or more second images to optically align the second image with the first image. The method may also include processing pixel data in the first image using a machine learning algorithm to detect micro-objects present in the area of interest, wherein detecting each micro-object includes identifying the boundaries of the micro-object. The method may also include detecting a signal within each boundary of each detected micro-object in each of the one or more second images.
[0010] In yet another aspect, a system for automatically detecting micro-objects in a microfluidic device is provided. The system may include an image acquisition unit, which may include: (a) an imaging element configured to capture a first image and one or more second images of an area of interest in the microfluidic device; (b) an image preprocessing engine configured to reduce anomalies in the image data; and (c) an alignment engine configured to transform the second image to optically align the second image with the first image. The system may also include a micro-object detection and characterization unit communicatively connected to the image acquisition unit. The micro-object detection and characterization unit may include: (a) an image processing engine configured to process pixel data in the first image using a machine learning algorithm to detect micro-objects present in the area of interest, wherein detecting micro-objects includes identifying the boundaries of each detected micro-object; and (b) a detection engine configured to detect a signal within each boundary of each detected micro-object in each of the one or more second images.
[0011] In another aspect, a computing device for characterizing and selecting micro-objects in a microfluidic device is provided. The computing device may include a display screen. The computing device may be configured to display a menu on the screen for selecting a first parameter selected from a provided parameter list for characterizing a group of detected micro-objects. The computing device may be configured to display a graph of a group of detected micro-objects on the screen based on a selected first parameter, wherein the provided parameter list is a finite list of parameters provided in the menu, and each parameter in the list may be selected to characterize a group of detected micro-objects based on associated parameters. The display screen may select a subgroup of a group of detected micro-objects based on at least one selected threshold value of the selected first parameter. The display screen may visually distinguish a subgroup that meets the at least one selected threshold value from the remaining micro-objects in a group of detected micro-objects to display a group of detected micro-objects.
[0012] Other aspects will become apparent from the following detailed description and the appended claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1A An example of a system for use with a microfluidic device and associated control instrumentation according to some embodiments of the present disclosure is shown.
[0014] Figure 1B and Figure 1C A microfluidic device according to some embodiments of the present disclosure is shown.
[0015] Figure 2A and Figure 2B An isolation dock according to some embodiments of the present disclosure is shown.
[0016] Figure 2C A detailed isolation dock is shown according to some embodiments of the present disclosure.
[0017] Figure 2D to Figure 2F An isolation dock according to some other embodiments of the present disclosure is shown.
[0018] Figure 2G A microfluidic device according to an embodiment of the present disclosure is shown.
[0019] Figure 2H Shown is a coated surface of a microfluidic device according to an embodiment of the present disclosure.
[0020] Figure 3A Specific examples of systems for use with microfluidic devices and associated control instrumentation according to some embodiments of the present disclosure are shown.
[0021] Figure 3B An imaging device according to some embodiments of the present disclosure is shown.
[0022] According to one embodiment of the present invention, Figure 4A , Figure 4B and Figure 4C The fence of micro-objects is depicted in parallel.
[0023] Figure 5 A block diagram of a computer system according to various embodiments is shown.
[0024] Figure 6A to Figure 6F The generation of an improved optical cage that can be used to separate micro-objects according to a specific embodiment of the present invention is shown.
[0025] Figure 7 Schematic diagram of a convolutional neural network according to various embodiments is shown.
[0026] Figure 8A to Figure 8C Schematic diagram illustrating a residual network, a downsampling block, and an upsampling block according to various embodiments.
[0027] FIG. 9A to FIG. 9D Portions of a more detailed schematic diagram of a convolutional neural network are shown according to various embodiments.
[0028] Fig.10 A flow chart of a method for automatically detecting micro-objects in an image according to various embodiments is shown.
[0029] Fig.11 A system for automatically detecting micro-objects in an image is shown according to various embodiments.
[0030] Fig.12A flow chart of a method for automatically detecting micro-objects in an image according to various embodiments is shown.
[0031] Fig.13 A system for automatically detecting micro-objects in an image is shown according to various embodiments.
[0032] Fig.14 Shown is a display screen for characterizing and selecting micro-objects according to various embodiments.
[0033] Fig.15 Shown is a display screen for characterizing and selecting micro-objects according to various embodiments.
[0034] Fig.16 Shown is a display screen for characterizing and selecting micro-objects according to various embodiments.
[0035] Fig.17 Shown is a display screen for characterizing and selecting micro-objects according to various embodiments.
[0036] Fig.18 Shown is a display screen for characterizing and selecting micro-objects according to various embodiments.
[0037] Fig.19 Shown is a display screen for characterizing and selecting micro-objects according to various embodiments.
[0038] Fig. 20 Shown is a display screen for characterizing and selecting micro-objects according to various embodiments.
[0039] Fig.21 Shown is a display screen for characterizing and selecting micro-objects according to various embodiments. Specific implementation plan
[0040] This specification describes exemplary embodiments and applications of the present disclosure. However, the present disclosure is not limited to these exemplary embodiments and applications, nor is it limited to the modes in which exemplary embodiments and applications are operated or described herein. In addition, the accompanying drawings may show simplified or partial views, and the size of the elements in the accompanying drawings may be exaggerated or disproportionate. In addition, due to the use of the term "on ... ", "attached to", "connected to", "coupled to" or similar words herein, an element (e.g., material, layer, substrate, etc.) can be "on another element", "attached to another element", "connected to another element" or "coupled to another element", regardless of whether the element is directly on the other element, attached to the other element, connected to the other element or coupled to the other element, or between the element and the other element, there are one or more spaced elements. In addition, unless the context otherwise provides, direction (e.g., on ..., under ..., top, bottom, side, up, down, directly below, directly above, above, below, horizontal, vertical, "x", "y", "z", etc.), if provided, is provided only by way of example and for ease of explanation and discussion rather than in a limiting manner. Where a list of elements is referred to (e.g., elements a, b, c), such reference is intended to include any one of the listed elements per se, any combination of less than all of the listed elements, and / or combinations of all of the listed elements. The section divisions in the specification are for ease of review only and do not limit any combination of the described elements.
[0041] Where the dimensions of a microfluidic feature are described as having a width or area, the dimensions are typically described relative to the x-axis and / or y-axis dimensions, both of which lie in a plane parallel to the substrate and / or lid of the microfluidic device. The height of a microfluidic feature may be described relative to the z-axis direction, which is perpendicular to the plane parallel to the substrate and / or lid of the microfluidic device. In some cases, the cross-sectional area of a microfluidic feature, such as a channel or passageway, may be referenced to the x-axis / z-axis, y-axis / z-axis, or x-axis / y-axis area.
[0042] As used herein, "substantially" means sufficient for the intended purpose. The term "substantially" therefore allows for minor, insignificant variations from an absolute or perfect state, dimension, measurement, result, etc., such as would be expected by one of ordinary skill in the art but which do not significantly affect overall performance. When used in relation to a numerical value or a parameter or characteristic that can be expressed as a numerical value, "substantially" means within ten percent.
[0043] The term "plurality" means more than one.
[0044] As used herein, the term "plurality" may be 2, 3, 4, 5, 6, 7, 8, 9, 10 or more.
[0045] As used herein: μm means micrometer, μm 3 refers to cubic micrometer, pL refers to picoliter, nL refers to nanoliter, and μL (or uL) refers to microliter.
[0046] As used herein, the term "positioned" includes "located" within its meaning.
[0047] As used herein, a "microfluidic device" or "microfluidic apparatus" is a device comprising one or more discrete microfluidic circuits configured to hold a fluid, each microfluidic circuit comprising fluidically interconnected circuit elements, including but not limited to one or more regions, one or more flow paths, one or more channels, one or more chambers and / or docks, and at least one port configured to allow fluid (and optionally, micro-objects suspended in the fluid) to flow into and / or out of the microfluidic apparatus. Typically, the microfluidic circuits of a microfluidic apparatus will include a flow region, which may include a microfluidic channel and at least one chamber, and will hold a volume of fluid less than about 1 mL, for example, less than about 750, 500, 250, 200, 150, 100, 75, 50, 25, 20, 15, 10, 9, 8, 7, 6, 5, 4, 3, or 2 μL. In some embodiments, the microfluidic circuit holds about 1-2, 1-3, 1-4, 1-5, 2-5, 2-8, 2-10, 2-12, 2-15, 2-20, 5-50, 10-50, 10-75, 10-100, 20-100, 20-150, 20-200, 50-200, 50-250, or 50-300 μL. The microfluidic circuit can be configured to have a first end fluidically connected to a first port (e.g., an inlet) in the microfluidic device and a second end fluidically connected to a second port (e.g., an outlet) in the microfluidic device.
[0048] As used herein, a “nanofluidic device” or “nanofluidic device” is a type of microfluidic device having a microfluidic circuit comprising at least one circuit element configured to hold a volume of fluid less than about 1 μL, for example, less than about 750, 500, 250, 200, 150, 100, 75, 50, 25, 20, 15, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1 nL, or less. The nanofluidic device can include a plurality of tubing elements (e.g., at least 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 25, 50, 75, 100, 150, 200, 250, 300, 400, 500, 600, 700, 800, 900, 1000, 1200, 2000, 2500, 3000, 3500, 4000, 4500, 5000, 6000, 7000, 8000, 9000, 10,000 or more). In some embodiments, one or more (e.g., all) of at least one tubing element are configured to hold a volume of fluid of approximately 100pL to 1nL, 100pL to 2nL, 100pL to 5nL, 250pL to 2nL, 250pL to 5nL, 250pL to 10nL, 500pL to 10nL, 500pL to 10nL, 750pL to 10nL, 750pL to 20nL, 750pL to 20nL, 1 to 20nL, 1 to 25nL, or 1 to 50nL. In other embodiments, one or more (e.g., all) of the at least one tubing element are configured to hold a volume of fluid of approximately 20nL to 200nL, 100 to 200nL, 100 to 300nL, 100 to 400nL, 100 to 500nL, 200 to 300nL, 200 to 400nL, 200 to 500nL, 200 to 600nL, 200 to 700nL, 250 to 400nL, 250 to 500nL, 250 to 600nL, or 250 to 750nL.
[0049] A microfluidic device may be referred to herein as a "microfluidic chip" or "chip", and a nanofluidic device may be referred to herein as a "nanofluidic chip" or "chip".
[0050] As used herein, "microfluidic channel" or "flow channel" refers to the flow area of a microfluidic device having a length significantly longer than both horizontal and vertical dimensions. For example, a flow channel can be at least 5 times the length of a horizontal or vertical dimension, such as at least 10 times the length, at least 25 times the length, at least 100 times the length, at least 200 times the length, at least 500 times the length, at least 1,000 times the length, at least 5,000 times the length or longer. In some embodiments, the length of the flow channel is in the range of about 50,000 microns to about 500,000 microns, including any value therebetween. In some embodiments, the horizontal dimension is about 100 microns to about 1000 microns (for example, about 150 to about 500 microns), and the vertical dimension is about 25 microns to about 200 microns (for example, about 40 to about 150 microns). It should be noted that flow channels can have a variety of different spatial configurations in microfluidic devices, and are therefore not limited to completely linear elements. For example, the flow channel can be or include one or more portions having the following configurations: curves, bends, spirals, slopes, declines, bifurcations (e.g., multiple different flow paths), and any combination thereof. In addition, the flow channel can have different cross-sectional areas along its path, widening and contracting to provide the desired fluid flow therein. The flow channel can include a valve, and the valve can be of any type known in the microfluidics field. Examples of microfluidic channels including valves are disclosed in U.S. Patents 6,408,878 and 9,227,200, each of which is incorporated herein by reference in its entirety.
[0051] As used herein, the term "obstacle" generally refers to a bump or similar type of structure that is large enough to partially (but not completely) hinder the movement of a target micro-object between two different areas or pipeline elements in a microfluidic device. The two different areas / pipeline elements can be, for example, a connection area and an isolation area of a microfluidic isolation dock.
[0052] As used herein, the term "constriction" generally refers to the narrowing of the width of a tubing element (or an interface between two tubing elements) in a microfluidic device. The constriction may be located at the interface between the isolation region and the connection region of the microfluidic isolation dock of the present disclosure, for example.
[0053] As used herein, the term "transparent" refers to a material that allows visible light to pass through without substantially altering the light as it passes through.
[0054] As used herein, the term "micro-object" generally refers to any microscopic object that can be isolated and / or manipulated according to the present disclosure. Non-limiting examples of micro-objects include: inanimate micro-objects, such as microparticles; micro-beads (e.g., polystyrene beads, Luminex TMBeads, etc.); magnetic beads; microrods; microfilaments; quantum dots, etc.; biological micro-objects, such as cells; biological organelles; vesicles or recombinants; synthetic vesicles; liposomes (e.g., synthesized or derived from membrane preparations); lipid nanorafts, etc.; or a combination of inanimate micro-objects and biological micro-objects (e.g., micro-beads attached to cells, micro-beads coated with liposomes, magnetic beads coated with liposomes, etc.). The beads can include covalently or non-covalently linked parts / molecules, such as fluorescent markers, proteins, carbohydrates, antigens, small molecule signaling parts, or other chemical / biological species that can be used in the assay. Lipid nanorafts have been described in, for example, Ritchie et al. (2009) "Reconstitution of Membrane Proteins in Phospholipid Bilayer Nanodiscs," Methods Enzymol., 464: 211-231.
[0055] As used herein, the term "cell" is used interchangeably with the term "biological cell". Non-limiting examples of biological cells include eukaryotic cells, plant cells, animal cells, such as mammalian cells, reptile cells, bird cells, fish cells, etc., prokaryotic cells, bacterial cells, fungal cells, protozoan cells, etc., cells dissociated from tissues, such as muscle, cartilage, fat, skin, liver, lung, neural tissue, etc., immune cells, such as T cells, B cells, natural killer cells, macrophages, etc., embryos (e.g., zygotes), oocytes, OVA, sperm cells, hybridomas, cultured cells, cells from cell lines, cancer cells, infected cells, transfected and / or transformed cells, reporter cells, etc. Mammalian cells can be, for example, from humans, mice, rats, horses, goats, sheep, cattle, primates, etc.
[0056] A colony of biological cells is a "clone" if all living cells in the colony that are capable of reproduction are daughter cells derived from a single parent cell. In certain embodiments, all daughter cells in a clonal colony are derived from a single parent cell no more than 10 divisions. In other embodiments, all daughter cells in a clonal colony are derived from a single parent cell no more than 14 divisions. In other embodiments, all daughter cells in a clonal colony are derived from a single parent cell no more than 17 divisions. In other embodiments, all daughter cells in a clonal colony are derived from a single parent cell no more than 20 divisions. The term "clonal cell" refers to cells of the same clonal colony.
[0057] As used herein, a "colony" of biological cells refers to 2 or more cells (e.g., about 2 to about 20, about 4 to about 40, about 6 to about 60, about 8 to about 80, about 10 to about 100, about 20 to about 200, about 40 to about 400, about 60 to about 600, about 80 to about 800, about 100 to about 1000, or greater than 1000 cells).
[0058] As used herein, the term "culturing cell(s)" refers to providing an environment, including fluid and gas components and optionally a surface, that provides the conditions necessary to keep the cells alive and / or proliferating.
[0059] As used herein, the term "expand" when referring to cells refers to an increase in the number of cells.
[0060] A "component" of a fluid medium is any chemical or biochemical molecule present in the medium, including solvent molecules, ions, small molecules, antibiotics, nucleotides and nucleosides, nucleic acids, amino acids, peptides, proteins, sugars, carbohydrates, lipids, fatty acids, cholesterol, metabolites, etc.
[0061] As used herein, a "capture moiety" is a chemical or biological species, function, or motif that provides a recognition site for a micro-object. A selected class of micro-objects can recognize a capture moiety generated in situ and can bind to or have an affinity for a capture moiety generated in situ. Non-limiting examples include antigens, antibodies, and cell surface binding motifs.
[0062] As used herein, a "flowable polymer" is a polymer monomer or macromer that is soluble or dispersed in a fluid medium (eg, a prepolymer solution). A flowable polymer can be input into a microfluidic flow region and flow with other components of the fluid medium therein.
[0063] As used herein, the term "photoinitiated polymer" (or monomeric molecule that can be used to produce a polymer) refers to a polymer that is capable of covalently crosslinking, forming specific covalent bonds, changing the regiochemistry around a rigidified chemical motif, or forming ion pairs that cause a change in physical state, thereby forming a polymer network when exposed to light. In some cases, a photoinitiated polymer may include polymer segments bound to one or more chemical moieties that are capable of covalently crosslinking, forming specific covalent bonds, changing the regiochemistry around a rigidified chemical motif, or forming ion pairs that cause a change in physical state. In some cases, a photoinitiated polymer may require a photoactivatable free radical initiator to initiate the formation of a polymer network (e.g., via polymerization of the polymer).
[0064] As used herein, "anti-immunoglobulin" refers to immunoglobulin (Ig) and includes polyclonal and monoclonal antibodies; primatized (e.g., humanized); mouse; mouse-human; mouse-primate; and chimeras; and may be an intact molecule, a fragment thereof (e.g., scFv, Fv, Fd, Fab, Fab' and F(ab')2 fragments), or a multimer or aggregate of intact molecules and / or fragments; and may occur in nature, or be produced, for example, by immunization, synthesis or genetic engineering. The term "antibody fragment" as used herein refers to a fragment derived from or associated with an antibody, which binds to an antigen and, in some embodiments, may be derivatized to exhibit structural features that facilitate clearance and absorption, for example, by the addition of galactose residues. This includes, for example, F(ab'), F(ab)'2, scFv, light chain variable region (VL), heavy chain variable region (VH) and combinations thereof.
[0065] As used herein with respect to a fluid medium, "diffuse" and "diffusion" refer to the thermodynamic movement of components of a fluid medium down a concentration gradient.
[0066] The phrase "flow of a medium" means the overall movement of a fluid medium primarily due to any mechanism other than diffusion. For example, the flow of a medium may involve the movement of a fluid medium from one point to another due to a pressure differential between the points. Such flow may include continuous, pulsed, periodic, random, intermittent, or reciprocating flow of a liquid, or any combination thereof. When one fluid medium flows into another fluid medium, turbulence and mixing of the media may result.
[0067] The phrase "substantially non-flowing" refers to a flow rate of a fluid medium whose average rate over time is less than the diffusion rate of components of a material (e.g., a target analyte) into or within the fluid medium. The diffusion rate of components of such a material may depend on, for example, temperature, the size of the components, and the strength of the interaction between the components and the fluid medium.
[0068] As used herein with respect to different regions within a microfluidic device, the phrase "fluidically connected" means that when the different regions are substantially filled with a fluid, such as a fluid medium, the fluid in each region is connected to form a single fluid body. This does not mean that the fluids (or fluid media) in the different regions are necessarily identical in composition. On the contrary, the fluids in different fluidly connected regions of a microfluidic device may have different compositions (e.g., different concentrations of solutes, such as proteins, carbohydrates, ions, or other molecules), which vary as the solutes flow through the device along their respective concentration gradients and / or fluid flow.
[0069] As used herein, "flow path" refers to one or more fluidically connected tubing elements (e.g., channels, regions, chambers, etc.) that define and experience the trajectory of a medium flow. Thus, a flow path is an example of a swept region of a microfluidic device. Other tubing elements (e.g., unswept regions) may be fluidically connected to the tubing elements that comprise the flow path, while experiencing the flow of the medium in the flow path.
[0070] As used herein, "isolating a micro-object" confines a micro-object to a defined area within a microfluidic device.
[0071] Microfluidic (or nanofluidic) devices may include "sweep" regions and "non-sweep" regions. As used herein, the "sweep" region includes one or more fluid interconnected tubing elements of a microfluidic circuit, each of which experiences the flow of a medium when the fluid flows through the microfluidic circuit. The tubing elements of a swept region may include, for example, regions, channels, and all or part of a chamber. As used herein, the "unswept" region includes one or more fluid interconnected tubing elements of a microfluidic circuit, each of which does not substantially experience the flow of a fluid when the fluid flows through the microfluidic circuit. The unswept region may be fluidically connected to the swept region, as long as the fluid connection is configured to diffuse but not substantially flow medium between the swept region and the unswept region. Therefore, the microfluidic device may be structured to substantially isolate the unswept region from the medium flow in the swept region, while substantially only realizing diffusion fluid communication between the swept region and the unswept region. For example, the flow channel of a microfluidic device is an example of a swept region, and the isolation region (described in further detail below) of a microfluidic device is an example of an unswept region.
[0072] As used herein, the term "non-illuminated", particularly with reference to an image, can refer to an illuminated image that is not illuminated in the same region of the spectrum being imaged, such as a fluorescent image. The term "non-illuminated" can also refer to imaging in the spectrum outside the visible spectrum, including, for example, infrared and ultraviolet light.
[0073] The ability of biological micro-objects (e.g., biological cells) to produce specific biological materials (e.g., proteins, such as antibodies) can be measured in such microfluidic devices. In the specific embodiment of the measurement, the sample material containing the biological micro-objects (e.g., cells) to be measured for the production of the target analyte can be loaded into the swept area of the microfluidic device. Some (e.g., mammalian cells, such as human cells) in the biological micro-objects can be selected for specific characteristics and arranged in the unswept area. The remaining sample material can then flow out of the swept area, and can be the analytical material flowing into the swept area. Because the selected biological micro-objects are in the unswept area, the selected biological micro-objects are not substantially affected by the outflow of the remaining sample material or the flow of the measuring material. The selected biological micro-objects can be allowed to produce an analyte of interest, which can diffuse from the unswept area into the swept area, wherein the analyte of interest can react with the analytical material to produce a local detectable reaction, each of which reaction can be associated with a specific unswept area. Any unswept area associated with the detected reaction can be analyzed to determine which (if any) of the biological micro-objects in the unswept area is a sufficient producer of the analyte of interest.
[0074] Microfluidic devices and systems for manipulating and observing such devices. Figure 1A An example of a microfluidic device 100 and system 150 that can be used to introduce, culture, and / or monitor micro-objects is shown. A perspective view of the microfluidic device 100 is shown with a portion of its cover 110 cut away to provide a partial view within the microfluidic device 100. The microfluidic device 100 generally includes a microfluidic circuit 120 that includes a flow path 106 through which a fluid medium 180 can flow, optionally carrying one or more micro-objects (not shown) into and / or through the microfluidic circuit 120. Although in Figure 1A A single microfluidic circuit 120 is illustrated in FIG. 1 , but a suitable microfluidic device may include a plurality (e.g., 2 or 3) of such microfluidic circuits. In any case, the microfluidic device 100 may be configured as a nanofluidic device. Figure 1A As shown, the microfluidic circuit 120 may include a plurality of microfluidic isolation docks 124, 126, 128, and 130, each of which may have one or more openings in fluid communication with the flow path 106. Figure 1A In some embodiments of the device, the isolation dock may have only a single opening in fluid communication with the flow path 106. As described further below, the microfluidic isolation dock includes various features and structures that have been optimized to retain micro-objects in a microfluidic device such as the microfluidic device 100, even when the medium 180 flows through the flow path 106. However, before turning to the foregoing, a brief description of the microfluidic device 100 and the system 150 is provided.
[0075] like Figure 1A As generally shown in FIG. 1 , microfluidic circuit 120 is defined by housing 102. Although housing 102 can be physically constructed in different configurations, Figure 1A In the example shown, the housing 102 is depicted as including a support structure 104 (e.g., a base), a microfluidic tubing structure 108, and a cover 110. The support structure 104, the microfluidic tubing structure 108, and the cover 110 can be attached to each other. For example, the microfluidic tubing structure 108 can be disposed on an inner surface 109 of the support structure 104, and the cover 110 can be disposed above the microfluidic tubing structure 108. The microfluidic tubing structure 108 can define elements of a microfluidic circuit 120 together with the support structure 104 and the cover 110.
[0076] like Figure 1A As shown, the support structure 104 can be located at the bottom and the cover 110 can be located on the top of the microfluidic line 120, or the support structure 104 and the cover 110 can be configured in other orientations. For example, the support structure 104 can be located at the top of the microfluidic line 120 and the cover 110 can be located at the bottom of the microfluidic line 120. In any case, there can be one or more ports 107, each of which includes a channel to enter or exit the housing 102. Examples of channels include valves, doors, through holes, etc. As shown, the port 107 is a through hole formed by a gap in the microfluidic line structure 108. However, the port 107 may be located in other components of the housing 102, such as the cover 110. Figure 1A 107, but the microfluidic circuit 120 may have two or more ports 107. For example, there may be a first port 107 that serves as an inlet for fluid entering the microfluidic circuit 120, and there may be a second port 107 that serves as an outlet for fluid leaving the microfluidic circuit 120. Whether a port 107 serves as an inlet or an outlet may depend on the direction in which the fluid flows through the flow path 106.
[0077] The support structure 104 may include one or more electrodes (not shown) and a substrate or multiple interconnected substrates. For example, the support structure 104 may include one or more semiconductor substrates, each of which is electrically connected to an electrode (e.g., all or a subset of the semiconductor substrates may be electrically connected to a single electrode). The support structure 104 may also include a printed circuit board assembly ("PCBA"). For example, the semiconductor substrates may be mounted on a PCBA.
[0078] The microfluidic circuit structure 108 may define circuit elements of the microfluidic circuit 120. Such circuit elements may include spaces or regions that may be fluidically interconnected when the microfluidic circuit 120 is filled with fluid, such as flow regions (which may include or may be one or more flow channels), chambers, docks, traps, etc. Figure 1A In the illustrated microfluidic circuit 120, the microfluidic circuit structure 108 includes a frame 114 and a microfluidic circuit material 116. The frame 114 may partially or completely surround the microfluidic circuit material 116. The frame 114 may be, for example, a relatively rigid structure that substantially surrounds the microfluidic circuit material 116. For example, the frame 114 may include a metal material.
[0079] The microfluidic tubing material 116 can be patterned with chambers or the like to define tubing elements and interconnections of the microfluidic tubing 120. The microfluidic tubing material 116 can include a flexible material, such as a flexible polymer (e.g., rubber, plastic, elastomer, silicone, polydimethylsiloxane ("PDMS"), etc.), which can be gas permeable. Other examples of materials that can constitute the microfluidic tubing material 116 include molded glass, etchable materials such as silicone (e.g., photopatternable silicone or "PPS"), photoresists (e.g., SU8), etc. In some embodiments, such a material, i.e., the microfluidic tubing material 116, can be rigid and / or substantially impermeable. In any case, the microfluidic tubing material 116 can be disposed on the support structure 104 and within the frame 114.
[0080] Cover 110 may be an integral part of frame 114 and / or microfluidic tubing material 116. Alternatively, Figure 1A As shown, the cover 110 can be a structurally distinct element. The cover 110 can include the same or different materials as the frame 114 and / or the microfluidic tubing material 116. Similarly, the support structure 104 can be a separate structure from the frame 114 or microfluidic tubing material 116 as shown, or an integral part of the frame 114 or microfluidic tubing material 116. Likewise, the frame 114 and the microfluidic tubing material 116 can be as shown. Figure 1A Individual structures shown or integral parts of the same structure.
[0081] In some embodiments, the cover 110 may include a rigid material. The rigid material may be glass or a material having similar properties. In some embodiments, the cover 110 may include a deformable material. The deformable material may be a polymer, such as PDMS. In some embodiments, the cover 110 may include a rigid and deformable material. For example, one or more portions of the cover 110 (e.g., one or more portions located above the isolation docks 124, 126, 128, 130) may include a deformable material that docks with the rigid material of the cover 110. In some embodiments, the cover 110 may also include one or more electrodes. One or more electrodes may include a conductive oxide, such as indium tin oxide (ITO), which may be coated on glass or similar insulating materials. Alternatively, one or more electrodes may be flexible electrodes, such as single-walled nanotubes, multi-walled nanotubes, nanowires, conductive nanoparticle clusters, or combinations thereof, embedded in a deformable material, such as a polymer (e.g., PDMS). Flexible electrodes that can be used in microfluidic devices have been described, for example, in US2012 / 0325665 (Chiou et al.), the contents of which are incorporated herein by reference. In some embodiments, the cover 110 can be modified (e.g., by adjusting all or part of the surface facing inwardly toward the microfluidic conduit 120) to support cell adhesion, activity and / or growth. The modification can include a coating of a synthetic or natural polymer. In some embodiments, the cover 110 and / or the support structure 104 are transparent to light. The cover 110 can also include at least one gas permeable material (e.g., PDMS or PPS).
[0082] Figure 1A Also shown is a system 150 for operating and controlling a microfluidic device such as the microfluidic device 100. The system 150 includes a power source 192, an imaging device (incorporated within the imaging module 164, the imaging device itself is not Figure 1A ) and a tilt device (part of the tilt module 166, wherein the tilt device itself is not shown in Figure 1A ).
[0083] The power supply 192 can provide power to the microfluidic device 100 and / or the tilting device 190, providing a bias voltage or current as needed. The power supply 192 can, for example, include one or more alternating current (AC) and / or direct current (DC) voltage or current sources. The imaging device 194 (part of the imaging module 164 discussed below) can include a device for capturing images of the interior of the microfluidic circuit 120, such as a digital camera. In some cases, the imaging device 194 also includes a detector with a fast frame rate and / or high sensitivity (e.g., for low-light applications). The imaging device 194 can also include a mechanism for directing stimulating radiation and / or light beams into the microfluidic circuit 120 and collecting radiation and / or light beams reflected or emitted from the microfluidic circuit 120 (or micro-objects contained therein). The emitted light beam can be in the visible spectrum and can, for example, include fluorescent emissions. The reflected light beam can include reflected emissions originating from an LED or a wide spectrum lamp, such as a mercury lamp (e.g., a high-pressure mercury lamp) or a xenon arc lamp. As described with respect to Figure 3B As discussed, the imaging device 194 may also include a microscope (or optical system), which may or may not include an eyepiece.
[0084] The system 150 also includes a tilting device 190 (part of the tilting module 166 described below) that is configured to rotate the microfluidic device 100 about one or more rotational axes. In some embodiments, the tilting device 190 is configured to support and / or hold the housing 102 including the microfluidic circuit 120 about at least one axis so that the microfluidic device 100 (and therefore the microfluidic circuit 120) can be maintained in a horizontal orientation (i.e., 0° relative to the x-axis and y-axis), a vertical orientation (i.e., 90 degrees relative to the x-axis and / or y-axis), or any orientation therebetween. The orientation of the microfluidic device 100 (and the microfluidic circuit 120) relative to an axis is referred to herein as the "tilted" orientation of the microfluidic device 100 (and the microfluidic circuit 120). For example, the tilt device 190 can tilt the microfluidic device 100 relative to the x-axis by 0.1°, 0.2°, 0.3°, 0.4°, 0.5°, 0.6°, 0.7°, 0.8°, 0.9, 1°, 2°, 3°, 4°, 5°, 10°, 15°, 20°, 25°, 30°, 35°, 40°, 45°, 50°, 55°, 60°, 65°, 70°, 75°, 80°, 90°, or any angle therebetween. The horizontal orientation (and therefore the x-axis and y-axis) is defined as being perpendicular to the vertical axis defined by gravity. The tilting device can also tilt the microfluidic device 100 (and the microfluidic circuit 120) relative to the x-axis and / or the y-axis by more than 90°, or tilt the microfluidic device 100 (and the microfluidic circuit 120) relative to the x-axis or the y-axis by 180°, so as to fully intertune the microfluidic device 100 (and the microfluidic circuit 120). Similarly, in some embodiments, the tilting device 190 tilts the microfluidic device 100 (and the microfluidic circuit 120) about a rotation axis defined by the flow path 106 or some other portion of the microfluidic circuit 120.
[0085] In some cases, the microfluidic device 100 is tilted into a vertical orientation so that the flow path 106 is positioned above or below one or more isolation docks. The term "above" as used herein means that the flow path 106 is positioned higher than one or more isolation docks on a vertical axis defined by gravity (i.e., objects in the isolation docks above the flow path 106 have a higher gravitational potential energy than objects in the flow path). The term "below" as used herein means that the flow path 106 is positioned lower than one or more isolation docks on a vertical axis defined by gravity (i.e., objects in the isolation docks below the flow path 106 have a lower gravitational potential energy than objects in the flow path).
[0086] In some cases, the tilting device 190 tilts the microfluidic device 100 about an axis parallel to the flow path 106. In addition, the microfluidic device 100 can be tilted to an angle less than 90° so that the flow path 106 is located above or below one or more isolation docks, but not directly above or below the isolation docks. In other cases, the tilting device 190 tilts the microfluidic device 100 about an axis perpendicular to the flow path 106. In other cases, the tilting device 190 tilts the microfluidic device 100 about an axis that is neither parallel nor perpendicular to the flow path 106.
[0087] The system 150 may also include a medium source 178. The medium source 178 (e.g., a container, a reservoir, etc.) may include multiple parts or containers, each part or container being used to hold a different fluid medium 180. Thus, the medium source 178 may be a device that is external to and separate from the microfluidic device 100, such as Figure 1A Alternatively, the media source 178 may be located in whole or in part within the housing 102 of the microfluidic device 100. For example, the media source 178 may include a reservoir that is part of the microfluidic device 100.
[0088] Figure 1A Also shown is a simplified block diagram depiction of an example of a control and monitoring device 152 that forms part of the system 150 and can be used in conjunction with the microfluidic device 100. As shown, an example of such a control and monitoring device 152 includes a main controller 154 that includes: a media module 160 for controlling a media source 178; a motion module 162 for controlling movement and / or selection of micro-objects (not shown) and / or media (e.g., droplets of media) in the microfluidic circuit 120; an imaging module 164 for controlling an imaging device 194 (e.g., a camera, a microscope, a light source, or any combination thereof for capturing images (e.g., digital images); and a tilt module 166 for controlling a tilt device 190. The control device 152 may also include other modules 168 for controlling, monitoring, or performing other functions with respect to the microfluidic device 100. As shown, the device 152 may also include a display device 170 and an input / output device 172.
[0089] The main controller 154 may include a control module 156 and a digital memory 158. The control module 156 may include, for example, a digital processor configured to operate according to machine executable instructions (e.g., software, firmware, source code, etc.) stored as non-temporary data or signals in the memory 158. Alternatively or additionally, the control module 156 may include hard-wired digital circuits and / or analog circuits. The media module 160, the motion module 162, the imaging module 164, the tilt module 166, and / or other modules 168 may be similarly configured. Therefore, the functions, process actions, actions, or steps of the processes performed by the microfluidic device 100 or any other microfluidic device described herein may be configured by any one or more of the main controller 154, the media module 160, the motion module 162, the imaging module 164, the tilt module 166, and / or other modules 168 configured as described above. Similarly, the main controller 154, media module 160, motion module 162, imaging module 164, tilt module 166 and / or other modules 168 can be communicatively coupled to send and receive data used in any of the functions, processes, process actions, actions, or steps described herein.
[0090] The media module 160 controls the media source 178. For example, the media module 160 can control the media source 178 to input the selected fluid media 180 into the housing 102 (e.g., through the inlet port 107). The media module 160 can also control the removal of the media from the housing 102 (e.g., through the outlet port (not shown)). Therefore, one or more media can be selectively input into and removed from the microfluidic circuit 120. The media module 160 can also control the flow of the fluid media 180 in the flow path 106 within the microfluidic circuit 120. For example, in some embodiments, the media module 160 prevents the flow of the media 180 in the flow path 106 and through the housing 102 before the tilt module 166 causes the tilt device 190 to tilt the microfluidic device 100 to the desired tilt angle.
[0091] The motion module 162 can be configured to control the selection, capture, and movement of micro-objects (not shown) in the microfluidic circuit 120. Figure 1B and 1C As described, the housing 102 may include dielectrophoresis (DEP), optoelectronic tweezers (OET), and / or optoelectronic wetting (OEW) structures ( Figure 1A ), and the motion module 162 can control the activation of electrodes and / or transistors (e.g., phototransistors) to select and move micro-objects (not shown) and / or droplets of medium (not shown) in the flow path 106 and / or the isolation docks 124, 126, 128, 130.
[0092] The imaging module 164 can control the imaging device 194. For example, the imaging module 164 can receive and process image data from the imaging device 194. The image data from the imaging device 194 can include any type of information captured by the imaging device 194 (e.g., the presence or absence of micro-objects, droplets of medium, accumulation of labels such as fluorescent markers, etc.). Using the information captured by the imaging device 194, the imaging module 164 can also calculate objects (e.g., micro-objects, droplets of medium) and / or the movement rate of such objects within the microfluidic device 100.
[0093] The tilt module 166 can control the tilting motion of the tilt device 190. Alternatively or additionally, the tilt module 166 can control the tilting rate and timing to optimize the transfer of the micro-objects to one or more isolation docks via gravity. The tilt module 166 is communicatively coupled to the imaging module 164 to receive data describing the movement of the micro-objects and / or media droplets in the microfluidic circuit 120. Using this data, the tilt module 166 can adjust the tilt of the microfluidic circuit 120 to adjust the rate at which the microfluids and / or media droplets move in the microfluidic circuit 120. The tilt module 166 can also use this data to iteratively adjust the position of the micro-objects and / or media droplets in the microfluidic circuit 120.
[0094] exist Figure 1A In the example shown, the microfluidic circuit 120 is shown to include a microfluidic channel 122 and isolation docks 124, 126, 128, 130. Each dock includes an opening leading to the channel 122, and the rest are closed, so that the dock can substantially isolate the micro-objects in the dock from the fluid medium 180 and / or the micro-objects in the flow path 106 of the channel 122 or other docks. The wall of the isolation dock extends from the inner surface 109 of the base to the inner surface of the cover 110 for closure. The opening of the dock to the microfluidic channel 122 is oriented to be at an angle with the fluid 106 of the fluid medium 180 so that the fluid 106 is not guided into the dock. The fluid can be tangent or orthogonal to the plane of the opening of the dock. In some cases, the docks 124, 126, 128, 130 are configured to physically support one or more micro-objects in the microfluidic circuit 120. As will be discussed and illustrated in detail below, the isolation docks of the present disclosure may include a variety of shapes, surfaces, and features that are optimized for use with DEP, OET, OEW, flowing fluids, and / or gravity.
[0095] Microfluidic circuit 120 can include any number of microfluidic isolation docks. Although five isolation docks are shown, microfluidic circuit 120 can have fewer or more isolation docks. As shown, microfluidic isolation docks 124, 126, 128 and 130 of microfluidic circuit 120 each include different features and shapes, which can provide one or more useful benefits for maintaining, separating, measuring or cultivating biological micro-objects. In some embodiments, microfluidic circuit 120 includes multiple identical microfluidic isolation docks.
[0096] exist Figure 1A In the illustrated embodiment, a single channel 122 and flow path 106 are shown. However, other embodiments may include a plurality of channels 122, each configured to include a flow path 106. The microfluidic circuit 120 also includes an inlet valve or port 107 in fluid communication with the flow path 106 and the fluid medium 180, whereby the fluid medium 180 can flow into the channel 122 via the inlet port 107. In some cases, the flow path 106 includes a single path. In some cases, the single path is arranged in a zigzag pattern, whereby the flow path 106 travels twice or more on the microfluidic device 100 in alternating directions.
[0097] In some cases, microfluidic circuit 120 includes multiple parallel channels 122 and flow paths 106, wherein the fluid medium 180 in each flow path 106 flows in the same direction. In some cases, the fluid medium in each flow path 106 flows in at least one of a forward or backward direction. In some cases, multiple isolation docks are configured (e.g., relative to channels 122) so that the isolation docks can be loaded with target micro-objects in parallel.
[0098] In some embodiments, the microfluidic circuit 120 also includes one or more micro-object traps 132. The traps 132 are typically formed in the walls that form the boundaries of the channel 122 and can be positioned opposite the openings of one or more of the microfluidic isolation docks 124, 126, 128, 130. In some embodiments, the traps 132 are configured to receive or trap a single micro-object from the flow path 106. In some embodiments, the traps 132 are configured to receive or trap a plurality of micro-objects from the flow path 106. In some cases, the traps 132 include a volume that is approximately equal to the volume of a single target micro-object.
[0099] The trap 132 may also include an opening configured to assist the target micro-object to flow into the trap 132. In some cases, the trap 132 includes an opening having a height and width approximately equal to the size of a single target micro-object, thereby preventing larger micro-objects from entering the micro-object trap. The trap 132 may also include other features configured to help keep the target micro-object within the trap 132. In some cases, the trap 132 is aligned with and located on the opposite side of the opening of the microfluidic isolation dock relative to the channel 122, so that when the microfluidic device 100 is tilted around an axis parallel to the microfluidic channel 122, the captured micro-object leaves the trap 132 with a trajectory that causes the micro-object to fall into the opening of the isolation dock. In some cases, the trap 132 includes a side channel 134 that is smaller than the target micro-object to facilitate flowing through the trap 132, thereby increasing the possibility of capturing micro-objects in the trap 132.
[0100] In some embodiments, dielectrophoresis (DEP) forces are applied to the fluid medium 180 (e.g., in the flow path and / or isolation dock) via one or more electrodes (not shown) to manipulate, transport, separate, and classify micro-objects located therein. For example, in some embodiments, DEP forces are applied to one or more portions of the microfluidic circuit 120 to transfer a single micro-object from the flow path 106 to a desired microfluidic isolation dock. In some embodiments, DEP forces are used to prevent micro-objects within an isolation dock (e.g., isolation dock 124, 126, 128, or 130) from shifting therefrom. In addition, in some embodiments, DEP forces are used to selectively remove micro-objects from an isolation dock previously collected according to embodiments of the present disclosure. In some embodiments, DEP forces include optoelectronic tweezers (OET) forces.
[0101] In other embodiments, photoelectrowetting (OEW) forces are applied to one or more locations in the support structure 104 (and / or lid 110) of the microfluidic device 100 (e.g., locations that help define flow paths and / or isolation docks) via one or more electrodes (not shown) to manipulate, transport, separate, and sort droplets located in the microfluidic circuit 120. For example, in some embodiments, OEW forces are applied to one or more locations in the support structure 104 (and / or lid 110) to transfer a single droplet from the flow path 106 to a desired microfluidic isolation dock. In some embodiments, OEW forces are used to prevent droplets within an isolation dock (e.g., isolation dock 124, 126, 128, or 130) from shifting therefrom. In addition, in some embodiments, OEW forces are used to selectively remove droplets from an isolation dock previously collected according to embodiments of the present disclosure.
[0102] In some embodiments, DEP forces and / or OEW forces are combined with other forces, such as flow and / or gravity, to manipulate, transport, separate, and sort micro-objects and / or droplets within the microfluidic circuit 120. For example, the housing 102 can be tilted (e.g., by tilting the device 190) to position the flow path 106 and the micro-objects therein above the microfluidic isolation dock, and gravity can transport the micro-objects and / or droplets into the dock. In some embodiments, the DEP force and / or OEW force can be applied before other forces. In other embodiments, the DEP force and / or OEW force can be applied after other forces. In other cases, the DEP force and / or OEW force can be applied simultaneously with other forces or in an alternating manner with other forces.
[0103] Figure 1B , Figure 1C and Figure 2A to Figure 2H Various embodiments of microfluidic devices that can be used to practice embodiments of the present disclosure are shown. Figure 1B An embodiment is depicted in which the microfluidic device 200 is configured as an optically actuated electrodynamic device. A variety of optically actuated electrodynamic devices are known in the art, including devices having an optoelectronic tweezers (OET) structure and devices having an optoelectronic electrowetting (OEW) structure. Examples of suitable OET structures are described in the following U.S. patent documents, each of which is incorporated herein by reference in its entirety: U.S. Patent No. RE44,711 (Wu et al.) (originally published as U.S. Patent No. 7,612,355); and U.S. Patent No. 7,956,339 (Ohta et al.). Examples of OEW structures are described in U.S. Patent No. 6,958,132 (Chiou et al.) and U.S. Patent Application Publication No. 2012 / 0024708 (Chiou et al.), both of which are incorporated herein by reference in their entirety. Yet another example of an optically actuated electric device includes a combined OET / OEW structure, examples of which are shown in U.S. Patent Publication Nos. 20150306598 (Khandros et al.) and 20150306599 (Khandros et al.) and their corresponding PCT publications WO2015 / 164846 and WO2015 / 164847, all of which are incorporated herein by reference in their entirety.
[0104] Examples of microfluidic devices with docks in which biological micro-objects can be placed, cultured and / or monitored have been described, for example, in US2014 / 0116881 (application number 14060117, filed on October 22, 2013), US2015 / 0151298 (application number 14520568, filed on October 22, 2014), and US2015 / 0165436 (application number 14521447, filed on October 22, 2014), each of which is incorporated herein by reference in its entirety. U.S. application numbers 14520568 and 14521447 also describe exemplary methods for analyzing secretions of cells cultured in microfluidic devices. Each of the foregoing applications further describes that the microfluidic device is configured to generate dielectrophoresis (DEP) forces, such as optoelectronic tweezers (OET), or is configured to provide optoelectronic wetting (OEW). For example, the optoelectronic tweezers device shown in FIG. 2 of US 2014 / 0116881 is an example of a device that may be utilized in embodiments of the present disclosure to select and move an individual biological micro-object or a group of biological micro-objects.
[0105] Microfluidic device power configuration. As described above, the control and monitoring equipment of the system may include a motion module for selecting and moving objects such as micro-objects or droplets in the microfluidic circuit of the microfluidic device. The microfluidic device may have various power configurations, depending on the type of object being moved and other considerations. For example, a dielectrophoresis (DEP) structure may be used to select and move micro-objects in a microfluidic circuit. Therefore, the support structure 104 and / or the cover 110 of the microfluidic device 100 may include a DEP structure for selectively inducing a DEP force on a micro-object in a fluid medium 180 in a microfluidic circuit 120, and thereby selecting, capturing and / or moving a separate micro-object or a group of micro-objects. Alternatively, the support structure 104 and / or the cover 110 of the microfluidic device 100 may include an electrowetting (EW) structure for selectively sensing an EW force on a droplet in a fluid medium 180 in a microfluidic circuit 120, and thereby selecting, capturing and / or moving a separate droplet or a group of droplets.
[0106] exist Figure 1B and Figure 1C An example of a microfluidic device 200 including a DEP structure is shown in FIG. Figure 1B and Figure 1CA side cross-sectional view and a top cross-sectional view of the housing 102 of a microfluidic device 200 having an open area / chamber 202 are shown, respectively, and it should be understood that the area / chamber 202 can be part of a fluid circuit element having a more detailed structure, such as a growth chamber, an isolation dock, a flow area, or a flow channel. In addition, the microfluidic device 200 may include other fluid circuit elements. For example, the microfluidic device 200 may include multiple growth chambers or isolation docks and / or one or more flow areas or flow channels, such as those described herein with respect to the microfluidic device 100. The DEP structure can be incorporated into any such fluid circuit element of the microfluidic device 200, or a selected portion. It should also be understood that any of the microfluidic device components and system components described above or below can be incorporated into the microfluidic device 200 and / or used in conjunction with the microfluidic device 200. For example, the system 150 including the control and monitoring device 152 described above may be used with a microfluidic device 200 that includes one or more of a media module 160 , a motion module 162 , an imaging module 164 , a tilt module 166 , and other modules 168 .
[0107] like Figure 1B As shown, the microfluidic device 200 includes a support structure 104 having a bottom electrode 204 and an electrode-activated substrate 206 covering the bottom electrode 204, and a cover 110 having a top electrode 210, the top electrode 210 being spaced apart from the bottom electrode 204. The top electrode 210 and the electrode-activated substrate 206 define opposing surfaces of the region / chamber 202. Thus, the medium 180 contained in the region / chamber 202 provides a resistive connection between the top electrode 210 and the electrode-activated substrate 206. A power source 212 is also shown, which is configured to be connected to the bottom electrode 204 and the top electrode 210 and to generate a bias voltage between the electrodes as needed for the generation of DEP forces in the region / chamber 202. The power source 212 may be, for example, an alternating current (AC) power source.
[0108] In certain embodiments, Figure 1B and Figure 1C The microfluidic device 200 shown in FIG. 2 may have an optically actuated DEP structure. Thus, changing the pattern 218 of light from the light source 216, which may be controlled by the motion module 162, may selectively activate and deactivate the changing pattern of the DEP electrode at the region 214 of the inner surface 208 of the electrode activation substrate 206. (Hereinafter, the region 214 of the microfluidic device having the DEP structure is referred to as the "DEP electrode region". Figure 1CAs shown, the light pattern 218 directed onto the inner surface 208 of the electrode active substrate 206 can illuminate the selected DEP electrode areas 214a (shown in white) in a pattern such as a square. The DEP electrode areas 214 that are not illuminated (cross-hatched) are referred to as "dark" DEP electrode areas 214 hereinafter. The relative electrical impedance through the DEP electrode active substrate 206 (i.e., from the bottom electrode 204 up to the inner surface 208 of the electrode active substrate 206 that interfaces with the medium 180 in the flow region 106) is greater than the relative electrical impedance through the medium 180 in the region / chamber 202 at each dark DEP electrode area 214 (i.e., from the inner surface 208 of the electrode active substrate 206 to the top electrode 210 of the cover 110). However, the illuminated DEP electrode areas 214a exhibit a reduced relative impedance of the entire electrode active substrate 206, which is less than the relative impedance of the entire medium 180 in the region / chamber 202 at each illuminated DEP electrode area 214a.
[0109] With the power source 212 activated, the aforementioned DEP structure generates an electric field gradient in the fluid medium 180 between the illuminated DEP electrode region 214a and the adjacent dark DEP electrode region 214, which in turn generates a local DEP force that attracts or repels nearby micro-objects (not shown) in the fluid medium 180. Thus, by changing the light pattern 218 projected from the light source 216 into the microfluidic device 200, DEP electrodes that attract or repel micro-objects in the fluid medium 180 can be selectively activated and deactivated at many different such DEP electrode regions 214 at the inner surface 208 of the region / chamber 202. Whether the DEP force attracts or repels nearby micro-objects can depend on parameters such as the frequency of the power source 212 and the dielectric properties of the medium 180 and / or the micro-objects (not shown).
[0110] exist Figure 1C The square pattern 220 of illuminated DEP electrode areas 214a shown in FIG. 2 is merely an example. Any pattern of DEP electrode areas 214 may be illuminated (and thereby activated) by a light pattern 218 projected into the microfluidic device 200, and the pattern of illuminated / activated DEP electrode areas 214 may be repeatedly changed by changing or moving the light pattern 218.
[0111] In some embodiments, the electrode activation substrate 206 may include or consist of a photoconductive material. In such an embodiment, the inner surface 208 of the electrode activation substrate 206 may be featureless. For example, the electrode activation substrate 206 may include or consist of a hydrogenated amorphous silicon (a-Si:H) layer. The a-Si:H may contain, for example, about 8% to 40% hydrogen (calculated as 100× the number of hydrogen atoms / the total number of hydrogen and silicon atoms). The a-Si:H layer may have a thickness of about 500 nm to about 2.0 microns. In such an embodiment, the DEP electrode region 214 may be created anywhere and in any pattern on the inner surface 208 of the electrode activation substrate 206 according to the light pattern 218. Therefore, the number and pattern of the DEP electrode regions 214 do not need to be fixed, but may correspond to the light pattern 218. Examples of microfluidic devices having DEP structures including a photoconductive layer as described above have been described, for example, in US Pat. No. RE44,711 (Wu et al.) (originally issued as US Pat. No. 7612355), the entire contents of which are incorporated herein by reference.
[0112] In other embodiments, the electrode-activated substrate 206 may include a substrate including a plurality of doped layers, electrically insulating layers (or regions), and conductive layers that form a semiconductor integrated circuit, such as is known in the semiconductor field. For example, the electrode-activated substrate 206 may include a plurality of phototransistors, including, for example, lateral bipolar phototransistors, each phototransistor corresponding to a DEP electrode region 214. Alternatively, the electrode-activated substrate 206 may include electrodes (e.g., conductive metal electrodes) controlled by a phototransistor switch, wherein each such electrode corresponds to a DEP electrode region 214. The electrode-activated substrate 206 may include a pattern of such phototransistors or phototransistor-controlled electrodes. The pattern may be, for example, an array of substantially square phototransistors or phototransistor-controlled electrodes arranged in rows and columns, such as Figure 2B. Alternatively, the pattern may be an array of substantially hexagonal phototransistors or phototransistor-controlled electrodes forming a hexagonal lattice. Regardless of the pattern, the circuit elements may form electrical connections between the DEP electrode regions 214 at the inner surface 208 of the electrode-activated substrate 206 and the bottom electrode 210, and those electrical connections (i.e., phototransistors or electrodes) may be selectively activated and deactivated by the light pattern 218. When not activated, each electrical connection may have a high impedance, such that at the corresponding DEP electrode region 214, the relative impedance of the entire electrode-activated substrate 206 (i.e., from the bottom electrode 204 to the inner surface 208 of the electrode-activated substrate 206 that interfaces with the medium 180 in the region / chamber 202) is greater than the relative impedance of the entire medium 180 (i.e., from the inner surface 208 of the electrode-activated substrate 206 to the top electrode 210 of the cover 110). However, when activated by light in the light pattern 218, the relative impedance of the entire electrode-activated substrate 206 is less than the relative impedance of the entire medium 180 at each illuminated DEP electrode region 214, thereby activating the DEP electrodes at the corresponding DEP electrode regions 214 as described above. Therefore, the DEP electrodes that attract or repel micro-objects (not shown) in the medium 180 can be selectively activated and deactivated at many different DEP electrode regions 214 at the inner surface 208 of the electrode-activated substrate 206 in the region / chamber 202 in a manner determined by the light pattern 218.
[0113] Examples of microfluidic devices having electrode-activated substrates containing phototransistors are described, for example, in U.S. Pat. No. 7,956,339 (Ohta et al.) (see, e.g. Fig.21 Examples of microfluidic devices having electrode-activated substrates including electrodes controlled by phototransistor switches are described, for example, in U.S. Patent Publication No. 2014 / 0124370 (Short et al.) (see, for example, devices 200, 400, 500, 600, and 900 shown throughout the figures and descriptions thereof), the entire contents of which are incorporated herein by reference.
[0114] In some embodiments of the DEP structured microfluidic device, the top electrode 210 is part of the first wall (or lid 110) of the housing 102, and the electrode activation substrate 206 and the bottom electrode 204 are part of the second wall (or support structure 104) of the housing 102. The region / chamber 202 can be between the first wall and the second wall. In other embodiments, the electrode 210 is part of the second wall (or support structure 104), and one or both of the electrode activation substrate 206 and / or the electrode 210 are part of the first wall (or lid 110). In addition, the light source 216 can alternatively be used to illuminate the housing 102 from below.
[0115] Using the DEP structure Figure 1B to Figure 1C In the microfluidic device 200 of the embodiment, the motion module 162 can select micro-objects (not shown) in the medium 180 in the region / chamber 202 by activating a first set of one or more DEP electrodes at a DEP electrode region 214A of the inner surface 208 of the substrate 206 by projecting a light pattern 218 into the microfluidic device 200 to activate electrodes in a pattern (e.g., a square pattern 220) that surrounds and captures the micro-objects. The motion module 162 can then move the captured micro-objects generated in situ by moving the light pattern 218 relative to the microfluidic device 200 to activate a second set of one or more DEP electrodes at the DEP electrode region 214. Alternatively, the microfluidic device 200 can be moved relative to the light pattern 218.
[0116] In other embodiments, the microfluidic device 200 can have a DEP structure that does not rely on photoactivation of DEP electrodes at the inner surface 208 of the electrode-activated substrate 206. For example, the electrode-activated substrate 206 may include selectively addressable and energizable electrodes located at a position opposite to a surface including at least one electrode (e.g., the cover 110). A switch (e.g., a transistor switch in a semiconductor substrate) can be selectively turned on and off to activate or deactivate the DEP electrode at the DEP electrode region 214, thereby generating a net DEP force on a micro-object (not shown) in the region / chamber 202 near the activated DEP electrode. Depending on characteristics such as the frequency and medium (not shown) of the power source 212 and / or the dielectric properties of the micro-object in the region / chamber 202, the DEP force can attract or repel nearby micro-objects. By selectively activating and deactivating a set of DEP electrodes (e.g., at a set of DEP electrode regions 214 forming a square pattern 220), one or more micro-objects in the region / chamber 202 can be captured and moved within the region / chamber 202. Figure 1A The motion module 162 in the can control such switches and thus activate and deactivate individual DEP electrodes to select, capture and move specific micro-objects (not shown) around the area / chamber 202. Microfluidic devices having DEP structures including selectively addressable and activatable electrodes are known in the art and have been described in, for example, U.S. Patents 6,294,063 (Becker et al.) and 6,276,776 (Medoro), the entire contents of which are incorporated herein by reference.
[0117] As another example, the microfluidic device 200 may have an electrowetting (EW) structure, which may replace the DEP structure or may be located in a portion of the microfluidic device 200 that is separate from the portion having the DEP structure. The EW structure may be a photoelectric electrowetting structure or an electrowetting on dielectric (EWOD) structure, both of which are known in the art. In some EW structures, the support structure 104 has an electrode activation substrate 206 sandwiched between a dielectric layer (not shown) and the bottom electrode 204. The dielectric layer may include a hydrophobic material and / or may be coated with a hydrophobic material, as described below. For a microfluidic device 200 having an EW structure, the inner surface 208 of the support structure 104 is the inner surface of the dielectric layer or its hydrophobic coating.
[0118] The dielectric layer (not shown) may include one or more oxide layers and may have a thickness of about 50 nm to about 250 nm (e.g., about 125 nm to about 175 nm). In certain embodiments, the dielectric layer may include an oxide layer, such as a metal oxide (e.g., aluminum oxide or hafnium oxide). In certain embodiments, the dielectric layer may include a dielectric material other than a metal oxide, such as silicon oxide or a nitride. Regardless of the exact composition and thickness, the dielectric layer may have an impedance of about 10 k ohms to about 50 k ohms.
[0119] In some embodiments, the surface of the dielectric layer facing the inner side of region / chamber 202 is coated with a hydrophobic material. The hydrophobic material may include, for example, fluorinated carbon molecules. Examples of fluorinated carbon molecules include perfluoropolymers, such as polytetrafluoroethylene (e.g., ) or poly(2,3-difluoromethylpentyl-perfluorotetrahydrofuran) (e.g., CYTOP TM ). The molecules constituting the hydrophobic material may be covalently bound to the surface of the dielectric layer. For example, the molecules of the hydrophobic material may be covalently bound to the surface of the dielectric layer through a linker such as a siloxane group, a phosphonic acid group, or a thiol group. Therefore, in some embodiments, the hydrophobic material may include an alkyl-terminated siloxane, an alkyl-terminated phosphonic acid, or an alkyl-terminated thiol. The alkyl group may be a long-chain hydrocarbon (e.g., a chain having at least 10 carbons, or at least 16, 18, 20, 22, or more carbons). Alternatively, a fluorinated (or perfluorinated) carbon chain may be used instead of the alkyl group. Therefore, for example, the hydrophobic material may include a fluoroalkyl-terminated siloxane, a fluoroalkyl-terminated phosphonic acid, or a fluoroalkyl-terminated thiol. In some embodiments, the hydrophobic coating has a thickness of about 10 nm to about 50 nm. In other embodiments, the hydrophobic coating has a thickness of less than 10 nm (e.g., less than 5 nm or about 1.5 to 3.0 nm).
[0120] In some embodiments, the cover 110 of the microfluidic device 200 having an electrowetting structure is also coated with a hydrophobic material (not shown). The hydrophobic material can be the same hydrophobic material used to coat the dielectric layer of the support structure 104, and the hydrophobic coating can have a thickness substantially the same as the thickness of the hydrophobic coating on the dielectric layer of the support structure 104. In addition, the cover 110 can include an electrode-activated substrate 206 sandwiched between the dielectric layer and the top electrode 210 in the manner of the support structure 104. The electrode-activated substrate 206 and the dielectric layer of the cover 110 can have the same composition and / or size as the electrode-activated substrate 206 and the dielectric layer of the support structure 104. Therefore, the microfluidic device 200 can have two electrowetting surfaces.
[0121] In some embodiments, the electrode-activated substrate 206 may include a photoconductive material, as described above. Thus, in certain embodiments, the electrode-activated substrate 206 may include or consist of a hydrogenated amorphous silicon layer (a-Si:H). The a-Si:H may contain, for example, about 8% to 40% hydrogen (calculated as 100× the number of hydrogen atoms / the total number of hydrogen and silicon atoms). The a-Si:H layer may have a thickness of about 500 nm to about 2.0 microns. Alternatively, as described above, the electrode-activated substrate 206 may include an electrode (e.g., a conductive metal electrode) controlled by a phototransistor switch. Microfluidic devices having photoelectrowetting structures are known in the art and / or may be constructed using electrode-activated substrates known in the art. For example, U.S. Patent No. 6,958,132 (Chiou et al.), the entire contents of which are incorporated herein by reference, discloses a photoelectrowetting structure having a photoconductive material such as a-Si:H, and the above-mentioned U.S. Patent Publication No. 2014 / 0124370 (Short et al.) discloses an electrode-activated substrate having electrodes controlled by a phototransistor switch.
[0122] The microfluidic device 200 may thus have an electro-photowetting structure, and the light pattern 218 may be used to activate a photoconductive EW region or a photo-responsive EW electrode in the electrode-activated substrate 206. Such an activated EW region or EW electrode of the electrode-activated substrate 206 may generate an electro-wetting force at the inner surface 208 (i.e., the inner surface covered with the dielectric layer or its hydrophobic coating) of the support structure 104. By changing the light pattern 218 on the electrode-activated substrate 206, a droplet (e.g., containing an aqueous medium, a solution, or a solvent) contacting the inner surface 208 of the support structure 104 may move through an immiscible fluid (e.g., an oil medium) present in the region / chamber 202.
[0123] In other embodiments, the microfluidic device 200 can have an EWOD structure, and the electrode activation substrate 206 can include selectively addressable and stimulable electrodes that do not rely on light for activation. Thus, the electrode activation substrate 206 can include a pattern of such electrowetting (EW) electrodes. The pattern can be, for example, an array of substantially square EW electrodes arranged in rows and columns, such as Figure 2B As shown. Alternatively, the pattern can be an array of substantially hexagonal EW electrodes forming a hexagonal lattice. Regardless of the pattern, the EW electrodes can be selectively activated (or deactivated) by an electrical switch (e.g., a transistor switch in a semiconductor substrate). By selectively activating and deactivating the EW electrodes in the electrode activation substrate 206, a droplet (not shown) contacting the inner surface 208 of the covering dielectric layer or its hydrophobic coating can move within the region / chamber 202. Figure 1A The motion module 162 in the can control such switches, thereby activating and deactivating individual EW electrodes to select and move specific droplets around the area / chamber 202. Microfluidic devices with EWOD structures having selectively addressable and activatable electrodes are known in the art and have been described, for example, in U.S. Pat. No. 8,685,344 (Sundarsan et al.), the entire contents of which are incorporated herein by reference.
[0124] Regardless of the configuration of the microfluidic device 200, the power supply 212 can be used to provide a potential (e.g., an AC voltage potential) for powering the circuits of the microfluidic device 200. The power supply 212 can be the same as the power supply 192 referenced in FIG. 1, or a component of the power supply 192 referenced in FIG. 1. The power supply 212 can be configured to provide an AC voltage and / or current to the top electrode 210 and the bottom electrode 204. For the AC voltage, the power supply 212 can provide a frequency range and an average or peak power (e.g., voltage or current) range sufficient to generate a sufficiently strong net DEP force (or electrowetting force) to capture and move each micro-object (not shown) in the region / chamber 202, and / or change the wetting properties of the inner surface 208 (i.e., the dielectric layer and / or the hydrophobic coating on the dielectric layer) of the support structure 104 in the region / chamber 202 as described above, also as described above. Such a frequency range and average or peak power range are known in the art. See, e.g., U.S. Pat. No. 6,958,132 (Chiou et al.), U.S. Pat. No. RE44,711 (Wu et al.) (originally published as U.S. Pat. No. 7,612,355), and U.S. Patent Application Publication Nos. US2014 / 0124370 (Short et al.), US2015 / 0306598 (Khandros et al.), and US2015 / 0306599 (Khandros et al.).
[0125] Isolation dock. Figure 2A to Figure 2CA non-limiting example of a universal isolation dock 224, 226, and 228 is shown within the illustrated microfluidic device 230. Each isolation dock 224, 226, and 228 may include an isolation structure 232 defining an isolation region 240 and a connection region 236 fluidically connecting the isolation region 240 to the channel 122. The connection region 236 may include a proximal opening 234 to the microfluidic channel 122 and a distal opening 238 to the isolation region 240. The connection region 236 may be configured such that the maximum penetration depth of the flow of a fluid medium (not shown) flowing from the channel 122 into the isolation docks 224, 226, 228 does not extend into the isolation region 240. Therefore, due to the connection region 236, micro-objects (not shown) or other materials (not shown) disposed in the isolation region 240 of the isolation docks 224, 226, 228 may be isolated therefrom and substantially unaffected by the flow of the medium 180 in the fluid channel 122.
[0126] Figure 2A to Figure 2C Each of the isolation docks 224, 226 and 228 has a single opening directly leading to the microfluidic channel 122. The opening of the isolation dock is open laterally from the microfluidic channel 122. The electrode activation substrate 206 covers both the microfluidic channel 122 and the isolation docks 224, 226 and 228. The upper surface of the electrode activation substrate 206 within the housing of the isolation dock forms the floor of the isolation dock, and the same level or substantially the same level of the upper surface of the electrode activation substrate 206 arranged in the channel 122 (or the flow area if the channel does not exist) forms the floor of the flow channel (or flow area) of the microfluidic device. The electrode activation substrate 206 can be featureless or can have an irregular or patterned surface that varies from its highest height to its lowest depression by less than about 3 microns, 2.5 microns, 2 microns, 1.5 microns, 1 micron, 0.9 microns, 0.5 microns, 0.4 microns, 0.2 microns, 0.1 microns or less. The variation in height across the upper surface of the substrate of the microfluidic channel 122 (or flow area) and the isolation dock can be less than about 3%, 2%, 1%, 0.9%, 0.8%, 0.5%, 0.3%, or 0.1% of the height of the wall of the isolation dock or the microfluidic device. Although the microfluidic device 200 is described in detail, this also applies to any of the microfluidic devices 100, 200, 230, 250, 280, 290, 300 described herein.
[0127] Thus, the microfluidic channel 122 can be an example of a swept area, and the isolated area 240 of the isolation docks 224, 226, 228 can be an example of an unswept area. As described above, the microfluidic channel 122 and the isolation docks 224, 226, 228 can be configured to accommodate one or more fluid media 180. Figure 2A to Figure 2BIn the example shown, port 222 is connected to microfluidic channel 122 and allows fluid medium 180 to be introduced into or removed from microfluidic device 230. Before introducing fluid medium 180, microfluidic device can be primed with a gas such as carbon dioxide gas. Once microfluidic device 230 includes fluid medium 180, the fluid 242 of the fluid medium 180 in microfluidic channel 122 can be selectively produced and stopped. For example, as shown in the figure, port 222 can be arranged at different positions (such as opposite ends) of microfluidic channel 122, and can lead to the fluid 242 of another port 222 used as outlet from a port 222 used as inlet to produce medium.
[0128] Figure 2C A detailed view of an example of an isolation dock 224 according to the present disclosure is shown. An example of a micro-object 246 is also shown.
[0129] As is known, the flow 242 of the fluid medium 180 through the proximal opening 234 of the isolation dock 224 in the microfluidic channel 122 may cause the secondary fluid 244 of the medium 180 to enter and / or flow out of the isolation dock 224. In order to isolate the micro-objects 246 in the isolation region 240 of the isolation dock 224 from the secondary fluid 244, the length L of the connection region 236 of the isolation dock 224 is con (ie, from the proximal opening 234 to the distal opening 238) should be greater than the penetration depth D of the secondary fluid 244 into the connection region 236. p The penetration depth D of the secondary fluid 244 p The flow rate 242 of the fluid medium 180 in the microfluidic channel 122 depends on various parameters related to the configuration of the microfluidic channel 122 and the proximal opening 234 of the connection area 236 leading to the microfluidic channel 122. For a given microfluidic device, the configuration of the microfluidic channel 122 and the opening 234 will be fixed, while the flow rate 242 of the fluid medium 180 in the microfluidic channel 122 will be variable. Therefore, for each isolation dock 224, the maximum flow rate V of the fluid medium 180 in the channel 122 can be identified. max , which ensures that the penetration depth D of the secondary fluid 244 p Not exceeding the length L of the connection area 236 con As long as the velocity of the fluid 242 of the fluid medium 180 in the microfluidic channel 122 does not exceed the maximum velocity V max, the resulting secondary fluid 244 can be confined to the microfluidic channel 122 and the connection area 236 and kept outside the isolation area 240. The fluid 242 of the medium 180 in the microfluidic channel 122 will therefore not draw the micro-object 246 out of the isolation area 240. On the contrary, regardless of the fluid 242 of the fluid medium 180 in the microfluidic channel 122, the micro-object 246 located in the isolation area 240 will remain in the isolation area 240.
[0130] Furthermore, as long as the velocity of the fluid 242 of the medium 180 in the microfluidic channel 122 does not exceed V max , the flow 242 of the fluid medium 180 in the microfluidic channel 122 will not move the confounding particles (eg, microparticles and / or nanoparticles) from the microfluidic channel 122 into the isolation region 240 of the isolation dock 224. con Greater than the maximum penetration depth D of the secondary fluid 244 p It is possible to prevent contamination of one isolation dock 224 with contaminant particles from the microfluidic channel 122 or another isolation dock (e.g., Figure 2D isolation docks 226, 228).
[0131] Because the connection area 236 of the microfluidic channel 122 and the isolation docks 224, 226, 228 can be affected by the fluid 242 of the medium 180 in the microfluidic channel 122, the microfluidic channel 122 and the connection area 236 can be considered as a swept (or flow) area of the microfluidic device 230. On the other hand, the isolation area 240 of the isolation docks 224, 226, 228 can be considered as an unswept (or non-flow) area. For example, components (not shown) in the first fluid medium 180 in the microfluidic channel 122 can be mixed with the second fluid medium 248 in the isolation area 240 substantially only by diffusion of components of the first medium 180 from the microfluidic channel 122 through the connection area 236 into the second fluid medium 248 in the isolation area 240. Similarly, components of the second medium 248 in the isolation region 240 (not shown) can be mixed with the first medium 180 in the microfluidic channel 122 substantially only by diffusion of components of the second medium 248 from the isolation region 240 through the connection region 236 into the first medium 180 in the microfluidic channel 122. In some embodiments, the degree of fluid medium exchange between the isolation region of the isolation dock and the flow region by diffusion is greater than about 90%, 91%, 92%, 93%, 94%, 95%, 96%, 97%, 98%, or greater than about 99%. The first medium 180 can be the same medium as the second medium 248 or a different medium. In addition, the first medium 180 and the second medium 248 can start out the same and then become different (e.g., by adjusting the passage conditions of the second medium 248 by one or more cells in the isolation region 240 or by changing the medium 180 flowing through the microfluidic channel 122).
[0132] As described above, the maximum penetration depth D of the secondary fluid 244 caused by the flow 242 of the fluid medium 180 in the microfluidic channel 122 is p The microfluidic channel 122 may depend on a number of parameters. Examples of such parameters include: the shape of the microfluidic channel 122 (e.g., the microfluidic channel may direct the medium into the connection region 236, transfer the medium away from the connection region 236, or direct the medium to the microfluidic channel 122 in a direction substantially perpendicular to the proximal opening 234 of the connection region 236); the width W of the microfluidic channel 122 at the proximal opening 234; ch (or cross-sectional area); and the width W of the connection region 236 at the proximal opening 234 con (or cross-sectional area); velocity V of the fluid 242 of the fluid medium 180 in the microfluidic channel 122; viscosity of the first medium 180 and / or the second medium 248, etc.
[0133] In some embodiments, the dimensions of the microfluidic channel 122 and the isolation docks 224, 226, 228 can be oriented relative to the vector of the flow 242 of the fluid medium 180 in the channel 122: the channel width W ch (or the cross-sectional area of the microfluidic channel 122) can be substantially perpendicular to the fluid 242 of the medium 180; the width W of the connection area 236 at the opening 234 con (or cross-sectional area) may be substantially parallel to the fluid 242 of the medium 180 in the microfluidic channel 122; and / or the length L of the connection region con The fluid 242 may be substantially perpendicular to the medium 180 in the channel 122. The foregoing are merely examples, and the relative positions of the microfluidic channel 122 and the isolation docks 224, 226, 228 may be in other orientations relative to each other.
[0134] like Figure 2C As shown, the width W of the connection area 236 con The width W of the connection region 236 at the distal opening 238 may be uniform from the proximal opening 234 to the distal opening 238. con The width W of the connection area 236 at the proximal opening 234 may be con Alternatively, the width W of the connection region 236 at the distal opening 238 is con It may be greater than the width W of the connection region 236 at the proximal opening 234. con .
[0135] like Figure 2C As shown, the width of the isolation region 240 at the distal opening 238 may be equal to the width W of the connection region 236 at the proximal opening 234. con Therefore, the width of the isolation region 240 at the distal opening 238 may be the width W of the connection region 236 at the proximal opening 234 herein. con Alternatively, the width of the isolation region 240 at the distal opening 238 may be greater or less than the width W of the connection region 236 at the proximal opening 234. con In addition, the distal opening 238 may be smaller than the proximal opening 234, and the width W of the connection region 236 may be con The connection region 236 may be narrowed between the proximal opening 234 and the distal opening 238. For example, the connection region 236 may be narrowed between the proximal opening and the distal opening using a variety of different geometries (e.g., chamfering, beveling the connection region). In addition, any portion or sub-portion of the connection region 236 may be narrowed (e.g., the portion of the connection region adjacent to the proximal opening 234).
[0136] Figure 2D to Figure 2FAnother exemplary embodiment of a microfluidic device 250 is depicted, the microfluidic device 250 comprising a microfluidic conduit 262 and a flow channel 264, which is Figure 1A The microfluidic device 250 also has a plurality of isolation docks 266, which are further variations of the isolation docks 124, 126, 128, 130, 224, 226, or 228 described above. In particular, it should be understood that Figure 2D to Figure 2F The isolation dock 266 of the device 250 shown in the figure can replace any of the isolation docks 124, 126, 128, 130, 224, 226 or 228 in the device 100. Likewise, the microfluidic device 250 is another variation of the microfluidic device 100, and can also have the same or different DEP structure as the above-mentioned microfluidic devices 100, 200, 230, 280, 290, 300, and any other microfluidic system components described herein.
[0137] Figure 2D to Figure 2F The microfluidic device 250 includes a support structure (in Figure 2D to Figure 2F Not visible in, but visible with Figure 1A The support structure 104 of the device 100 depicted in FIG. 1 is the same as or substantially similar to that of the device 100 depicted in FIG. 1 ), the microfluidic piping structure 256 and the cover (in Figure 2D-2F Not visible in the Figure 1A The microfluidic tubing structure 256 includes a frame 252 and a microfluidic tubing material 260, which can be the same as or substantially similar to the cover 122 of the device 100 depicted in FIG. Figure 1A The frame 114 and microfluidic tubing material 116 of the device 100 shown in FIG. 1 are the same or substantially similar. Figure 2D As shown, a microfluidic circuit 262 defined by a microfluidic circuit material 260 may include a plurality of channels 264 (two are shown, but there may be more), and a plurality of isolation docks 266 are fluidically connected to the plurality of channels 264 .
[0138] Each isolation dock 266 may include an isolation structure 272, an isolation region 270 within the isolation structure 272, and a connection region 268. From the proximal opening 274 at the microfluidic channel 264 to the distal opening 276 at the isolation structure 272, the connection region 268 fluidically connects the microfluidic channel 264 to the isolation region 270. Figure 2B and Figure 2C As described above, the flow 278 of the first fluid medium 254 in the channel 264 can create a secondary flow 282 of the first medium 254 that flows from the microfluidic channel 264 into and / or out of the corresponding connection area 268 of the isolation dock 266 .
[0139] like Figure 2EAs shown, the connection region 268 of each isolation dock 266 generally includes an area extending between a proximal opening 274 to the channel 264 and a distal opening 276 to the isolation structure 272. The length L of the connection region 268 is con can be greater than the maximum penetration depth D of the secondary fluid 282 p In this case, the secondary fluid 282 will extend into the connecting region 268 without being redirected toward the isolation region 270 (eg, Figure 2D Alternatively, Figure 2F As shown, the connection area 268 may have a maximum penetration depth D less than p Length L con , in which case the secondary fluid 282 will extend through the connection region 268 and be redirected toward the isolation region 270. In the latter case, the length L of the connection region 268 is c1 and L c2 The sum of the values is greater than the maximum penetration depth D p , so that the secondary fluid 282 will not extend into the isolation region 270. Regardless of the length L of the connection region 268 con Is it greater than the penetration depth D? p , or the length L of the connection area 268 c1 and L c2 Is the sum of greater than the penetration depth D p , channel 264 does not exceed the maximum rate V max The fluid 278 of the first medium 254 will produce a penetration depth D p The secondary flow of the micro-objects in the isolation area 270 of the isolation dock 266 (not shown but can be connected with Figure 2C The first medium 254 in the channel 264 (which is the same as or substantially similar to the micro-object 246 shown in FIG. 2 ) will not be drawn out of the isolation region 270 by the fluid 278 of the first medium 254 in the channel 264. The fluid 278 in the channel 264 also does not draw the mixed material (not shown) from the channel 264 into the isolation region 270 of the isolation dock 266. Therefore, diffusion is the only mechanism by which components in the first medium 254 in the microfluidic channel 264 can move from the microfluidic channel 264 to the second medium 258 in the isolation region 270 of the isolation dock 266. Likewise, diffusion is the only mechanism by which components in the second medium 258 in the isolation region 270 of the isolation dock 266 can move from the isolation region 270 to the first medium 254 in the channel 264. The first medium 254 can be the same medium as the second medium 258, or the first medium 254 can be a different medium from the second medium 258. Alternatively, the first medium 254 and the second medium 258 may be the same initially and then become different, for example, by conditioning the second medium by one or more cells in the isolation region 270 , or by changing the medium flowing through the microfluidic channel 264 .
[0140] like Figure 2E As shown, the width W of the microfluidic channel 264 in the microfluidic channel 264 is ch (i.e., transverse to the direction of flow of the fluid medium through the microfluidic channel, such as Figure 2D The width W of the proximal opening 274 may be substantially perpendicular to the width W of the proximal opening 274. con1 and is substantially parallel to the width W of the distal opening 276 con2 However, the width W of the proximal opening 274 is con1 and the width W of the distal opening 276 con2 For example, the width W of the proximal opening 274 is con1 The oriented axis (not shown) and the width W of the distal opening 276 con2 The angle between the other axis that is oriented may not be perpendicular and therefore not 90 degrees. Examples of alternatively oriented angles include angles of about 30 degrees to about 90 degrees, about 45 degrees to about 90 degrees, about 60 degrees to about 90 degrees, and the like.
[0141] In various embodiments of the isolation dock (e.g., 124, 126, 128, 130, 224, 226, 228, or 266), the isolation region (e.g., 240 or 270) is configured to contain a plurality of micro-objects. In other embodiments, the isolation region can be configured to contain only one, two, three, four, five, or similar relatively small numbers of micro-objects. Thus, the volume of the isolation region can be, for example, at least 1×10 6 , 2×10 6 , 4×10 6 , 6×10 6 Cubic microns or more.
[0142] In various embodiments of the isolation dock, the width W of the microfluidic channel (e.g., 122) at the proximal opening (e.g., 234) is ch The microfluidic channel (e.g., 122) may be about 50 to 1000 microns, 50 to 500 microns, 50-400 microns, 50 to 300 microns, 50 to 250 microns, 50 to 200 microns, 50 to 150 microns, 50 to 100 microns, 70 to 500 microns, 70 to 400 microns, 70 to 300 microns, 70 to 250 microns, 70 to 200 microns, 70 to 150 microns, 90 to 400 microns, 90 to 300 microns, 90 to 250 microns, 90 to 200 microns, 90 to 150 microns, 100 to 300 microns, 100 to 250 microns, 100 to 200 microns, 100 to 150 microns, or 100 to 120 microns. In some other embodiments, the width W of the microfluidic channel (e.g., 122) at the proximal opening (e.g., 234) is about 100 microns. chThe width W of the microfluidic channel 122 may be in the range of about 200 to 800 microns, 200 to 700 microns, or 200 to 600 microns. ch The width may be defined by any of the endpoints listed above. In addition, the Wch of the microfluidic channel 122 may be selected to be any of these widths in the region of the microfluidic channel excluding the proximal opening of the isolation dock.
[0143] In some embodiments, the height of the isolation dock is about 30 to about 200 microns or about 50 to about 150 microns. In some embodiments, the cross-sectional area of the isolation dock is about 1×10 4 ~3×10 6 Square micrometer, 2×10 4 ~2×10 6 Square micrometer, 4×10 4 ~1×10 6 Square micrometer, 2×10 4 ~5×10 5 Square micrometer, 2×10 4 ~1×10 5 square micrometers or approximately 2×10 5 ~2×10 6 Square micrometers.
[0144] In various embodiments of the isolation dock, the height H of the microfluidic channel (e.g., 122) at the proximal opening (e.g., 234) is ch It can be any of the following heights: 20-100 microns, 20-90 microns, 20-80 microns, 20-70 microns, 20-60 microns, 20-50 microns, 30-100 microns, 30-90 microns, 30-80 microns, 30-70 microns, 30-60 microns, 30-50 microns, 40-100 microns, 40-90 microns, 40-80 microns, 40-70 microns, 40-60 microns or 40-50 microns. The foregoing are examples only, and the height H of the microfluidic channel (e.g., 122) may be any of the following heights: 20-100 microns, 20-90 microns, 20-80 microns, 20-70 microns, 20-60 microns, 20-50 microns, 30-100 microns, 40-90 microns, 40-80 microns, 40-70 microns, 40-60 microns or 40-50 microns. The foregoing are examples only, and the height H of the microfluidic channel (e.g., 122) may be any of the following heights: ch The height H of the microfluidic channel 122 can be within any of the above-listed endpoints. ch Any of these heights may be chosen in the area of the microfluidic channel other than the proximal opening of the isolation dock.
[0145] In various embodiments of the isolation dock, the cross-sectional area of the microfluidic channel (e.g., 122) at the proximal opening (e.g., 234) can be about 500-50,000 square microns, 500-40,000 square microns, 500-30,000 square microns, 500-25,000 square microns, 500-20,000 square microns, 500-15,000 square microns, 500-10,000 square microns, 500-7,500 square microns, 500-5,000 square microns, 1,000-25,000 square microns, 1,000-20,000 square microns, 1,000-15 ,000 square microns, 1,000-10,000 square microns, 1,000-7,500 square microns, 1,000-5,000 square microns, 2,000-20,000 square microns, 2,000-15,000 square microns, 2,000-10,000 square microns, 2,000-7,500 square microns, 2,000-6,000 square microns, 3,000-20,000 square microns, 3,000-15,000 square microns, 3,000-10,000 square microns, 3,000-7,500 square microns, or 3,000-6,000 square microns. The foregoing are examples only, and the cross-sectional area of the microfluidic channel (e.g., 122) at the proximal opening (e.g., 234) can be any area within any of the endpoints listed above.
[0146] In various embodiments of the isolation dock, the length L of the connection area (e.g., 236) is con It may be about 1-600 microns, 5-550 microns, 10-500 microns, 15-400 microns, 20-300 microns, 20-500 microns, 40-400 microns, 60-300 microns, 80-200 microns or about 100-150 microns. The foregoing is only an example, and the length L of the connection region (e.g., 236) con Can be in any length within any of the endpoints listed above.
[0147] In various embodiments of the isolation dock, the width W of the connection region (eg, 236) at the proximal opening (eg, 234) is conThe thickness of the nanoparticles may be about 20-500 microns, 20-400 microns, 20-300 microns, 20-200 microns, 20-150 microns, 20-100 microns, 20-80 microns, 20-20 microns, 30-400 microns, 30-300 microns, 30-200 microns, 30-150 microns, 30-100 microns, 30-80 microns, 30-60 microns, 40-300 microns, 40- 200 microns, 40-150 microns, 40-100 microns, 40-80 microns, 40-60 microns, 50-250 microns, 50-200 microns, 50-150 microns, 50-100 microns, 50-80 microns, 60-200 microns, 60-150 microns, 60-100 microns, 60-80 microns, 70-150 microns, 70-100 microns or 80-100 microns. The foregoing is only an example, and the width W of the connection area (e.g., 236) at the proximal opening (e.g., 234) is con Can be different from the preceding examples (eg, any value within any of the aforementioned endpoints).
[0148] In various embodiments of the isolation dock, the width W of the connection region (eg, 236) at the proximal opening (eg, 234) is con The width W of the connection region (e.g., 236) at the proximal opening (e.g., 234) may be at least as large as the largest dimension of the micro-object (e.g., biological cell, which may be a T cell, a B cell, or a bead or an embryo) for the isolation dock. The foregoing is merely an example, and the width W of the connection region (e.g., 236) at the proximal opening (e.g., 234) may be at least as large as the largest dimension of the micro-object (e.g., biological cell, which may be a T cell, a B cell, or a bead or an embryo) for the isolation dock. con May vary from the preceding examples (eg, widths within any of the endpoints listed above).
[0149] In various embodiments of the isolation dock, the width W of the proximal opening of the connection region is pr The width W may be at least as large as the largest dimension of a micro-object (e.g., a biological micro-object such as a cell) for the isolation dock. pr It may be about 50 microns, about 60 microns, about 100 microns, about 200 microns, about 300 microns, or may be about 50-300 microns, about 50-200 microns, about 50-100 microns, about 75-150 microns, about 75-100 microns, or about 200-300 microns.
[0150] In various embodiments of the isolation dock, the length L of the connection area (e.g., 236) is con The width W of the connection area (eg, 236) at the proximal opening 234 conThe ratio L of the connection region 236 may be greater than or equal to any of the following ratios: 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0 or more. The foregoing is merely an example, and the length L of the connection region 236 is con The width W of the connection area 236 at the proximal opening 234 is con The ratio may be different from the aforementioned example.
[0151] In various embodiments of the microfluidic devices 100, 200, 23, 250, 280, 290, 300, V max It can be set to about 0.2, 0.5, 0.7, 1.0, 1.3, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, 5.5, 6.0, 6.7, 7.0, 7.5, 8.0, 8.5, 9.0, 10, 11, 12, 13, 14 or 15 microliters / second.
[0152] In various embodiments of a microfluidic device having an isolation dock, the volume of the isolation region (e.g., 240) of the isolation dock can be, for example, at least 5×10 5 , 8×10 5 , 1×10 6 , 2×10 6 , 4×10 6 , 6×10 6 , 8×10 6 , 1×10 7 , 5×10 7 , 1×10 8 , 5×10 8 , 8×10 8 In various embodiments of a microfluidic device having an isolation dock, the volume of the isolation dock can be about 5×10 5 , 6×10 5 , 8×10 5 , 1×106, 2×10 6 , 4×10 6 , 8×10 6 , 1×10 7 , 3×10 7 , 5×10 7 or about 8×10 7 In some other embodiments, the volume of the isolation dock can be about 1 nanoliter to about 50 nanoliters, 2 nanoliters to about 25 nanoliters, 2 nanoliters to about 20 nanoliters, about 2 nanoliters to about 15 nanoliters, or about 2 nanoliters to about 10 nanoliters.
[0153] In various embodiments, the microfluidic device has an isolation dock configured as in any of the embodiments discussed herein, wherein the microfluidic device has about 5 to about 10 isolation docks, about 10 to about 50 isolation docks, about 100 to about 500 isolation docks, about 200 to about 1000 isolation docks, about 500 to about 1500 isolation docks, about 1000 to about 2000 isolation docks, about 1000 to about 3500 isolation docks, about 3000 to about 7000 isolation docks, about 5000 to about 10,000 isolation docks, about 1,000 to about 15,000 isolation docks, or about 12,000 to about 20,000 isolation docks. The isolation docks need not all be the same size, and may include various configurations (e.g., different widths, different features within the isolation docks).
[0154] Figure 2G A microfluidic device 280 is shown according to one embodiment. Figure 2G The illustrated microfluidic device 280 is a stylized representation of the microfluidic device 100. In particular, the microfluidic device 280 and its component plumbing elements (eg, the channel 122 and the isolation dock 128) will have the dimensions discussed herein. Figure 2G The microfluidic circuit 120 shown in FIG. 2 has two ports 107, four different channels 122, and four different flow paths 106. The microfluidic device 280 also includes a plurality of isolation docks opening to each channel 122. Figure 2G In the microfluidic device shown, the isolation dock has a structure similar to that in Figure 2C The geometry of the dock shown in FIG. 1 and thus has both a connection area and an isolation area. Thus, the microfluidic circuit 120 includes a sweep area (eg, a maximum penetration depth D of the channel 122 and the secondary fluid 244). p The maximum penetration depth D of the secondary fluid 244 is not within the channel 122 and the portion of the connecting area 236 and the non-sweeping area (eg, the isolation area 240 and the connecting area 236 that are not within the secondary fluid 244). p part inside).
[0155] Figures 3A to 3B Various embodiments of a system 150 that can be used to operate and observe a microfluidic device (e.g., 100, 200, 230, 250, 280, 290, 300) according to the present disclosure are shown. Figure 3AAs shown in , the system 150 may include a structure ("nest") 300 configured to hold a microfluidic device 100 (not shown) or any other microfluidic device described herein. The nest 300 may include a socket 302 capable of engaging with a microfluidic device 320 (e.g., an optically actuated electrokinetic device 100) and providing an electrical connection from the power source 192 to the microfluidic device 320. The nest 300 may also include an integrated electrical signal generation subsystem 304. The electrical signal generation subsystem 304 may be configured to provide a bias voltage to the socket 302 such that when the socket 302 is held, a bias voltage is applied across a pair of electrodes in the microfluidic device 320. Thus, the electrical signal generation subsystem 304 may be part of the power source 192. The ability to apply a bias voltage to the microfluidic device 320 does not mean that the bias voltage will always be applied when the microfluidic device 320 is held by the socket 302. Instead, in most cases, the bias voltage will be applied intermittently, for example, only as needed to promote the generation of electrokinetic forces in the microfluidic device 320, such as dielectrophoresis or electrowetting.
[0156] like Figure 3A As shown, the nest 300 may include a printed circuit board assembly (PCBA) 322. The electrical signal generating subsystem 304 may be mounted on the PCBA 322 and electrically integrated into the PCBA 322. The exemplary support also includes a socket 302 mounted on the PCBA 322.
[0157] Typically, the electrical signal generating subsystem 304 includes a waveform generator (not shown). The electrical signal generating subsystem 304 may also include an oscilloscope (not shown) and / or a waveform amplification circuit (not shown) configured to amplify the waveform received from the waveform generator. The oscilloscope (if present) may be configured to measure the waveform supplied to the microfluidic device 320 held by the socket 302. In some embodiments, the oscilloscope measures the waveform at a position near the proximal end of the microfluidic device 320 (and away from the waveform generator), thereby ensuring higher accuracy when measuring the waveform actually applied to the device. The data obtained from the oscilloscope measurement can be provided to the waveform generator as feedback, for example, and the waveform generator can be configured to adjust its output based on such feedback. An example of a suitable combined waveform generator and oscilloscope is the Red Pitaya TM .
[0158] In some embodiments, the nest 300 also includes a controller 308, such as a microprocessor for sensing and / or controlling the electrical signal generating subsystem 304. Examples of suitable microprocessors include Arduino TM A microprocessor, such as an Arduino Nano TM The controller 308 may be used to perform functions and analyses, or may communicate with an external host controller 154 (in Figure 1A) to perform functions and analysis. Figure 3A In the illustrated embodiment, the controller 308 communicates with the main controller 154 via an interface 310 (eg, a plug or connector).
[0159] In some embodiments, the nest 300 may include an electrical signal generating subsystem 304 that includes a Red Pitaya TM A waveform generator / oscilloscope unit ("Red Pitaya unit") and a waveform amplification circuit that amplifies the waveform generated by the Red Pitaya unit and passes the amplified voltage to the microfluidic device 100. In some embodiments, the Red Pitaya unit is configured to measure the amplified voltage at the microfluidic device 320, and then adjust its own output voltage as needed so that the measured voltage at the microfluidic device 320 is the desired value. In some embodiments, the waveform amplification circuit can have a +6.5V to -6.5V power supply generated by a pair of DC-DC converters mounted on PCBA 322, resulting in a signal of up to 13Vpp at the microfluidic device 100.
[0160] like Figure 3A As shown, the support structure 300 may also include a thermal control subsystem 306. The thermal control subsystem 306 may be configured to regulate the temperature of the microfluidic device 320 held by the support structure 300 (e.g., a nest). For example, the thermal control subsystem 306 may include a Peltier thermoelectric device (not shown) and a cooling unit (not shown). The Peltier thermoelectric device may have a first surface configured to engage with at least one surface of the microfluidic device 320. The cooling unit may be, for example, a cooling block (not shown), such as a liquid-cooled aluminum block. A second surface of the Peltier thermoelectric device (e.g., a surface opposite the first surface) may be configured to engage with a surface of such a cooling block. The cooling block may be connected to a fluid path 314, which is configured to circulate a cooling fluid through the cooling block. In Figure 3A In the illustrated embodiment, the support structure 300 includes an inlet 316 and an outlet 318 to receive a cooling fluid from an external reservoir (not shown), introduce the cooling fluid into the fluid path 314 and pass through the cooling block, and then return the cooled fluid to the external reservoir. In some embodiments, the Peltier thermoelectric device, the cooling unit, and / or the fluid path 314 can be mounted on the housing 312 of the support structure 300. In some embodiments, the thermal control subsystem 306 is configured to adjust the temperature of the Peltier thermoelectric device so as to achieve the target temperature of the microfluidic device 320. For example, the temperature regulation of the Peltier thermoelectric device can be achieved by a Pololu TMThe thermal control subsystem 306 may include a feedback circuit, such as a temperature value provided by an analog circuit. Alternatively, the feedback circuit may be provided by a digital circuit.
[0161] In some embodiments, the nest 300 may include a thermal control subsystem 306 having a feedback circuit that is an analog voltage divider circuit (not shown) that includes a resistor (e.g., having a resistance of 1 kΩ + / - 0.1%, a temperature coefficient of + / - 0.02 ppm / C0) and an NTC thermistor (e.g., having a nominal resistance of 1 kΩ + / - 0.01%). In some cases, the thermal control subsystem 306 measures the voltage from the feedback circuit and then uses the calculated temperature value as an input to an onboard PID control loop algorithm. The output of the PID control loop algorithm may drive, for example, a Pololu TM Direction and pulse width modulated signal pins on a motor driver (not shown) to actuate the thermoelectric power source, thereby controlling the Peltier thermoelectric device.
[0162] The nest 300 may include a serial port 324 that allows the microprocessor of the controller 308 to communicate with the external host controller 154 via the interface 310 (not shown). In addition, the microprocessor of the controller 308 may communicate with the electrical signal generation subsystem 304 and the thermal control subsystem 306 (e.g., via a Plink tool (not shown)). Thus, via the combination of the controller 308, the interface 310, and the serial port 324, the electrical signal generation subsystem 304 and the thermal control subsystem 306 may communicate with the external host controller 154. In this manner, the host controller 154 may assist the electrical signal generation subsystem 304 by performing scaling calculations for output voltage regulation, among other things. A graphical user interface (GUI) (not shown) provided by a display device 170 coupled to the external host controller 154 may be configured to plot temperature and waveform data obtained from the thermal control subsystem 306 and the electrical signal generation subsystem 304, respectively. Alternatively or in addition, the GUI may allow for updating of the controller 308 , the thermal control subsystem 306 , and the electrical signal generation subsystem 304 .
[0163] As described above, system 150 may include imaging device 194. In some embodiments, imaging device 194 includes light modulation subsystem 330 (see Figure 3B). The light modulation subsystem 330 may include a digital mirror device (DMD) or a micro-gate array system (MSA), either of which may be configured to receive light from the light source 332 and emit a subset of the received light into the optical system of the microscope 350. Alternatively, the light modulation subsystem 330 may include a device that generates its own light (and therefore distributes without the need for the light source 332), such as an organic light emitting diode display (OLED), a liquid crystal on silicon (LCOS) device, a ferroelectric liquid crystal on silicon device (FLOS), or a transmissive liquid crystal display (LCD). The light modulation subsystem 330 may be, for example, a projector. Thus, the light modulation subsystem 330 is capable of emitting both structured light and unstructured light. In certain embodiments, the imaging module 164 and / or the motion module 162 of the system 150 may control the light modulation subsystem 330.
[0164] In some embodiments, the imaging device 194 also includes a microscope 350. In such embodiments, the nest 300 and the light modulation subsystem 330 can be separately configured to be mounted on the microscope 350. The microscope 350 can be, for example, a standard research-grade light microscope or a fluorescence microscope. Thus, the nest 300 can be configured to be mounted on the platform 344 of the microscope 350, and / or the light modulation subsystem 330 can be configured to be mounted on a port of the microscope 350. In other embodiments, the nest 300 and the light modulation subsystem 330 described herein can be an integrated component of the microscope 350.
[0165] In some embodiments, the microscope 350 may also include one or more detectors 348. In some embodiments, the detector 348 is controlled by the imaging module 164. The detector 348 may include an eyepiece, a charge coupled device (CCD), a camera (e.g., a digital camera), or any combination thereof. If there are at least two detectors 348, one detector may be, for example, a fast frame rate camera, and the other detector may be a high sensitivity camera. In addition, the microscope 350 may include an optical system configured to receive reflected and / or emitted light from the microfluidic device 320, and focus at least a portion of the reflected and / or emitted light on one or more detectors 348. The optical system of the microscope may also include different tube lenses (not shown) for different detectors so that the final magnification on each detector may be different.
[0166] In some embodiments, the imaging device 194 is configured to use at least two light sources. For example, a first light source 332 can be used to generate structured light (e.g., via a light modulation subsystem 330), and a second light source 334 can be used to provide unstructured light. The first light source 332 can generate structured light for optically actuated electrodynamic and / or fluorescence excitation, and the second light source 334 can be used to provide bright field illumination. In these embodiments, the motion module 164 can be used to control the first light source 332, and the imaging module 164 can be used to control the second light source 334. The optical system of the microscope 350 can be configured to (1) receive structured light from the light modulation subsystem 330 and focus the structured light on at least a first area in a microfluidic device (such as an optically actuated electrodynamic device) when the device is held by the nest 300, and (2) receive reflected and / or emitted light from the microfluidic device and focus at least a portion of such reflected and / or emitted light onto a detector 348. The optical system can also be configured to receive the unstructured light from the second light source and focus the unstructured light on at least a second region of the microfluidic device when the device is held by the nest 300. In some embodiments, the first region and the second region of the microfluidic device can be overlapping regions. For example, the first region can be a subset of the second region. In other embodiments, the second light source 334 can additionally or alternatively include a laser, which can have light of any suitable wavelength. Figure 3B The representation of the optical system shown in is only a schematic representation, and the optical system may include additional filters, notch filters, lenses, etc. When the second light source 334 includes one or more light sources for bright field and / or fluorescence excitation and laser illumination, the physical arrangement of the light source may be the same as Figure 3B 330, and the laser illumination may be introduced at any suitable physical location within the optical system. The schematic locations of light source 334 and light source 332 / light modulation subsystem 330 may also be interchanged.
[0167] exist Figure 3B 3, a first light source 332 is shown providing light to a light modulation subsystem 330, which provides structured light to an optical train of a microscope 350 of a system 355 (not shown). A second light source 334 is shown providing unstructured light to the optical train via a beam splitter 336. The structured light from the light modulation subsystem 330 and the unstructured light from the second light source 334 travel together from the beam splitter 336 through the optical train to the second beam splitter (or dichroic filter 338, depending on the light provided by the light modulation subsystem 330), where the light is reflected downwardly through an objective lens 336 to a sample plane 342. The reflected and / or emitted light from the sample plane 342 then passes through the objective lens 340, through the beam splitter and / or dichroic filter 338, back to the dichroic filter 346. Only a portion of the light that reaches the dichroic filter 346 passes through and reaches the detector 348.
[0168] In some embodiments, the second light source 334 emits blue light. With an appropriate dichroic filter 346, the blue light reflected from the sample plane 342 is able to pass through the dichroic filter 346 and reach the detector 348. In contrast, the structured light from the light modulation subsystem 330 is reflected from the sample plane 342, but does not pass through the dichroic filter 346. In this example, the dichroic filter 346 filters out visible light with wavelengths greater than 495nm. If the light emitted from the light modulation subsystem does not include any wavelengths shorter than 495nm, only the filtering of the light from the light modulation subsystem 330 is completed (as shown). In fact, if the light from the light modulation subsystem 330 includes wavelengths shorter than 495nm (e.g., blue light wavelengths), some of the light from the light modulation subsystem will pass through the filter 346 to the detector 348. In such an embodiment, the filter 346 is used to change the balance between the amount of light reaching the detector 348 from the first light source 332 and the second light source 334. This can be beneficial if the first light source 332 is significantly stronger than the second light source 334. In other embodiments, the second light source 334 can emit red light, and the dichroic filter 346 can filter out visible light other than red light (eg, visible light having a shorter wavelength than red light).
[0169] Coating solutions and coating agents. Without wishing to be limited by theory, the cultivation of biological micro-objects (e.g., biological cells) within a microfluidic device (e.g., a DEP-structured and / or EW-structured microfluidic device) can be facilitated (i.e., the biological micro-objects exhibit increased activity, greater amplification, and / or greater transplantability within the microfluidic device) when at least one or more inner surfaces of the microfluidic device have been conditioned or coated to present an organic and / or hydrophilic molecular layer that provides the primary interface between the microfluidic device and the biological micro-objects retained therein and is retained therein. In some embodiments, one or more inner surfaces of a microfluidic device (e.g., the inner surface of an electrode activation substrate of a DEP-structured microfluidic device, the cover of a microfluidic device, and / or the surface of a tubing material) can be treated or modified with a coating solution and / or coating agent to produce the desired organic and / or hydrophilic molecular layer.
[0170] The coating may be applied before or after the introduction of the biological micro-object, or may be introduced simultaneously with the biological micro-object. In some embodiments, the biological micro-object may be introduced into the microfluidic device in a fluid medium comprising one or more coating agents. In other embodiments, the inner surface of the microfluidic device (e.g., a microfluidic device with a DEP structure) is treated with a coating solution comprising a coating agent or a "primed" configured microfluidic device before the biological micro-object is introduced into the microfluidic device.
[0171] In some embodiments, at least one surface of the microfluidic device includes a coating material, which provides an organic and / or hydrophilic molecule layer suitable for the cultivation and / or amplification of biological micro-objects (e.g., providing a surface after adjustment as described below). In some embodiments, substantially all inner surfaces of the microfluidic device include a coating material. The inner surface of the coating can include a surface of a flow area (e.g., a channel), a chamber or an isolation dock or a combination thereof. In some embodiments, each of a plurality of isolation docks has at least one inner surface coated with a coating material. In other embodiments, each of a plurality of flow areas or channels has at least one inner surface coated with a coating material. In some embodiments, at least one inner surface of each of a plurality of isolation docks and a plurality of channels is coated with a coating material.
[0172] Coating Agents / Solutions. Any convenient coating agent / coating solution may be used, including, but not limited to, serum or serum factors, bovine serum albumin (BSA), polymers, detergents, enzymes, and any combination thereof.
[0173] Coating materials based on polymers. The at least one inner surface may include a coating material comprising a polymer. The polymer may be covalently or non-covalently bound to at least one surface (or may adhere non-specifically). The polymer may have a variety of structural motifs, such as those found in block polymers (and copolymers), star polymers (star copolymers), and grafted or comb-shaped polymers (graft copolymers), all of which may be applicable to methods disclosed herein.
[0174] Polymers can include polymers containing alkylene ether moieties. A variety of polymers containing alkylene ethers can be suitable for use in microfluidic devices as described herein. A non-limiting exemplary class of polymers containing alkylene ethers is an amphoteric nonionic block copolymer comprising blocks of polyethylene oxide (PEO) and polypropylene oxide (PPO) subunits having different ratios and positions within the polymer chain. The polymer (BASF) is a block copolymer of this type and is known in the art to be suitable for use in contact with living cells. The average molecular weight M of the polymer w It can be from about 2000Da to about 20kDa. In some embodiments, the PEO-PPO block copolymer can have a hydrophilic-lipophilic balance (HLB) greater than about 10 (e.g., 12 to 18). Polymers include L44, L64, P85 and F127 (including F127NF). Another class of alkylene ether-containing polymers is polyethylene glycol (PEG M w <100,000Da) or alternatively polyethylene oxide (PEO, M w>100,000). In some embodiments, PEG may have an M of about 1000 Da, 5000 Da, 10,000 Da, or 20,000 Da. w .
[0175] In other embodiments, the coating material may include a polymer containing a carboxylic acid moiety. The carboxylic acid subunit may be a subunit containing an alkyl, alkenyl or aromatic moiety of the subunit. A non-limiting example is polylactic acid (PLA). In other embodiments, the coating material may include a polymer containing a phosphate moiety, the polymer being at the end of the polymer backbone or from a side chain of the polymer backbone. In other embodiments, the coating material may include a polymer containing a sulfonic acid moiety. The sulfonic acid subunit may be a subunit containing an alkyl, alkenyl or aromatic moiety. A non-limiting example is polystyrene sulfonic acid (PSSA) or polyanethole sulfonic acid. In further embodiments, the coating material may include a polymer comprising an amine moiety. Polyamino polymers may include natural polyamine polymers or synthetic polyamine polymers. Examples of natural polyamines include spermine, spermidine and putrescine.
[0176] In other embodiments, the coating material can include a polymer containing a sugar moiety. In a non-limiting example, a polysaccharide such as xanthan gum or dextran can be suitable for forming a material that can reduce or prevent cell adhesion in a microfluidic device. For example, a dextran polymer having a size of about 3 kDa can be used to provide a coating material for a surface within a microfluidic device.
[0177] In other embodiments, the coating material can include a polymer containing a nucleotide portion, i.e., a nucleic acid, which can have a ribonucleotide portion or a deoxyribonucleotide portion, thereby providing a polyelectrolyte surface. The nucleic acid can contain only natural nucleotide portions or can contain non-natural nucleotide portions, which contain nucleobases, ribose or phosphate moieties analogs, such as 7-deazaadenine, pentose, methylphosphonic acid or thiophosphate moieties, but are not limited thereto.
[0178] In other embodiments, the coating material may include a polymer containing an amino acid moiety. The polymer containing an amino acid moiety may include a polymer containing natural amino acids or a polymer containing non-natural amino acids, any of which may include a peptide, a polypeptide or a protein. In a non-limiting example, the protein may be bovine serum albumin (BSA) and / or serum (or a combination of multiple different serums), which includes albumin and / or one or more other similar proteins as a coating agent. Serum may be from any convenient source, including but not limited to fetal bovine serum, sheep serum, goat serum, horse serum, etc. In certain embodiments, the BSA in the coating solution is present in a concentration of about 1 mg / mL to about 100 mg / mL, including 5 mg / mL, 10 mg / mL, 20 mg / mL, 30 mg / mL, 40 mg / mL, 50 mg / mL, 60 mg / mL, 70 mg / mL, 80 mg / mL, 90 mg / mL, or more or anywhere between the two. In certain embodiments, the serum in the coating solution may be present at a concentration of about 20% (v / v) to about 50% v / v, including 25%, 30%, 35%, 40%, 45% or more or between the two. In some embodiments, BSA may be present in the coating solution at 5 mg / mL as a coating solution, while in other embodiments, BSA may be present in a coating solution at 70 mg / mL as a coating agent. In certain embodiments, serum is present in the coating solution at 30% as a coating agent. In some embodiments, extracellular matrix (ECM) proteins may be provided in the coating material to optimize cell adhesion to promote cell growth. Cell matrix proteins that may be included in the coating material may include, but are not limited to, collagen, elastin, RGD-containing peptides (e.g., fibronectin) or laminin. In other embodiments, growth factors, cytokines, hormones or other cell signaling substances may be present in the coating material of the microfluidic device.
[0179] In some embodiments, the coating material may include a polymer containing more than one of an alkylene oxide moiety, a carboxylic acid moiety, a sulfonic acid moiety, a phosphate moiety, a sugar moiety, a nucleotide moiety, or an amino acid moiety. In other embodiments, the polymer-treated surface may include a mixture of multiple polymers, each polymer having an alkylene oxide moiety, a carboxylic acid moiety, a sulfonic acid moiety, a phosphate moiety, a sugar moiety, a nucleotide moiety, and / or an amino acid moiety, which may be incorporated into the coating material independently or simultaneously.
[0180] Covalently attached coating materials. In some embodiments, the at least one interior surface comprises covalently attached molecules that provide a layer of organic and / or hydrophilic molecules suitable for culturing / expansion of biological micro-objects within a microfluidic device, providing a treated surface for such cells.
[0181] The covalently linked molecules include linking groups, wherein the linking groups are covalently linked to one or more surfaces of the microfluidic device, as described below. The linking groups are also covalently linked to portions of an organic and / or hydrophilic molecular layer configured to provide a suitable culture / amplification of biological micro-objects.
[0182] In some embodiments, the covalently linked portion configured to provide an organic and / or hydrophilic molecular layer suitable for culture / expansion of biological micro-objects may include alkyl or fluoroalkyl (including perfluoroalkyl) portions; monosaccharides or polysaccharides (which may include but are not limited to dextran); alcohols (including but not limited to propargyl alcohol); polyols, including but not limited to polyvinyl alcohol; alkylene ethers, including but not limited to polyethylene glycol; polyelectrolytes (including but not limited to polyacrylic acid or polyvinylphosphonic acid); amino groups (including derivatives thereof, such as but not limited to alkylated amines, hydroxyalkylated amino groups, guanidinium salts, and heterocyclic groups containing non-aromatic nitrogen ring atoms, such as but not limited to morpholino or piperazinyl); carboxylic acids, including but not limited to propionic acid (which can provide a carboxylate anion surface); phosphonic acids, including but not limited to ethynylphosphonic acid (which can provide a phosphonate anion surface); sulfonate anions; carboxybetaines; sulfobetaines; aminosulfonic acids; or amino acids.
[0183] In various embodiments, the covalently linked portion configured to provide a layer of organic and / or hydrophilic molecules suitable for culture / amplification of biological micro-objects in a microfluidic device may include a non-polymeric portion, such as an alkyl portion, a substituted alkyl portion, such as a fluoroalkyl portion (including but not limited to a perfluoroalkyl portion), an amino acid portion, an alcohol portion, an amino portion, a carboxylic acid portion, a phosphonic acid portion, a sulfonic acid portion, an aminosulfonic acid portion, or a sugar portion. Alternatively, the covalently linked portion may include a polymer portion, which may be any of the above portions.
[0184] In some embodiments, the covalently linked alkyl moiety may include carbon atoms forming a straight chain (e.g., a straight chain of at least 10 carbons, or at least 14, 16, 18, 20, 22 or more carbons) and may be an unbranched alkyl moiety. In some embodiments, the alkyl group may include a substituted alkyl group (e.g., some of the carbons in the alkyl group may be fluorinated or perfluorinated). In some embodiments, the alkyl group may include a first segment, the first segment may include a perfluoroalkyl group, the first segment is connected to a second segment, the second segment includes an unsubstituted alkyl group, wherein the first segment and the second segment may be directly or indirectly connected (e.g., by means of an ether bond). The first segment of the alkyl group may be located at the distal end of the linking group, and the second segment of the alkyl group may be located at the proximal end of the linking group.
[0185] In other embodiments, the covalently linked moiety may include at least one amino acid, which may include more than one type of amino acid. Thus, the covalently linked moiety may include a peptide or protein. In some embodiments, the covalently linked moiety may include an amino acid that can provide a zwitterionic surface to support cell growth, activity, transplantability, or any combination thereof.
[0186] In other embodiments, the covalently linked moiety may include at least one alkylene oxide moiety, and may include any alkylene oxide polymer as described above. One class of useful alkylene ether-containing polymers is polyethylene glycol (PEG M w <100,000Da) or alternatively polyethylene oxide (PEO, M w >100,000). In some embodiments, PEG may have an M of about 1000 Da, 5000 Da, 10,000 Da, or 20,000 Da. w .
[0187] The covalently linked portion may include one or more carbohydrates. The covalently linked sugar may be a monosaccharide, a disaccharide or a polysaccharide. The covalently linked sugar may be modified to introduce a reactive pairing portion that allows coupling or refinement to attach to a surface. Exemplary reactive pairing portions may include aldehyde, alkyne or halogen moieties. Polysaccharides may be modified in a random manner, wherein each of the polysaccharide monomers may be modified or only a portion of the sugar monomers in the polysaccharide may be modified to provide a reactive pairing portion that may be directly or indirectly coupled to a surface. An example may include a dextran polysaccharide that may be indirectly coupled to a surface through an unbranched linking group.
[0188] The covalently linked moiety may include one or more amino groups. The amino group may be a substituted amine moiety, a guanidine moiety, a nitrogen-containing heterocyclic moiety, or a heteroaryl moiety. The amino-containing moiety may have a structure that allows pH modification of the environment within the microfluidic device and optionally within the isolation dock and / or flow region (e.g., channel).
[0189] The coating material providing the treated surface may contain only one covalently linked part, or may include a plurality of different types of covalently linked parts. For example, a fluoroalkyl-conditioned surface (including perfluoroalkyl) may have a plurality of covalently linked parts, all of which are identical, such as having the same linking group and covalently linked to the surface, the same total length and the same number of fluoromethylene units containing fluoroalkyl parts. Alternatively, the coating material may have a plurality of covalently linked parts attached to the surface. For example, the coating material may include molecules having covalently linked alkyl or fluoroalkyl parts with a specified number of methylene or methylene units, and may also include another group of molecules having a charged part covalently linked to an alkyl or fluoroalkyl chain with a greater number of methylene or methylene units, which may provide the ability to present a larger portion on the coating surface. In this case, the first group of molecules having different, lower spatially demanding end and main chain atoms can help functionalize the entire substrate surface, thereby preventing undesirable adhesion or contact of the substrate itself with silicon / silicon oxide, hafnium oxide or aluminum oxide. In another example, the covalently linked part can provide a zwitterionic surface that presents alternating charges in a random manner on the surface.
[0190] Surface properties after treatment. In addition to the composition of the treated surface, other factors such as the physical thickness of the hydrophobic material can affect the DEP force. Various factors can change the physical thickness of the treated surface, such as the way the treated surface is formed on the substrate (e.g., vapor deposition, liquid deposition, spin coating, dipping, and electrostatic coating). In some embodiments, the treated surface has a thickness of about 1 nm to about 10 nm; about 1 nm to about 7 nm; about 1 nm to about 5 nm; or any individual value therebetween. In other embodiments, the treated surface formed by the covalently linked portion may have a thickness of about 10 nm to about 50 nm. In various embodiments, the treated surface prepared as described herein has a thickness of less than 10 nm. In some embodiments, when covalently linked to the surface of a microfluidic device (e.g., the substrate surface of a DEP structure), the covalently linked portion of the treated surface may form a monolayer and may have a thickness of less than 10 nm (e.g., less than 5 nm, or about 1.5 to 3.0 nm). These values are opposite to the values of the surface prepared by spin coating, for example, it can generally have a thickness of about 30 nm. In some embodiments, the treated surface need not be a perfectly formed monolayer to be adequately used for operation within a DEP-structured microfluidic device.
[0191] In various embodiments, the coating material providing the treated surface of the microfluidic device can provide the desired electrical properties. Without wishing to be limited by theory, one factor affecting the robustness of a surface coated with a particular coating material is inherent charge capture. Different coating materials can capture electrons, which can lead to breakdown of the coating material. Defects in the coating material can increase charge capture and lead to further breakdown of the coating material. Similarly, different coating materials have different dielectric strengths (i.e., the minimum applied electric field that causes dielectric breakdown), which may affect charge capture. In certain embodiments, the coating material may have an overall structure (e.g., a densely packed monolayer structure) that reduces or limits the amount of charge capture.
[0192] In addition to its electrical properties, the treated surface may also have properties that are beneficial for use with biomolecules. For example, treated surfaces containing fluorinated (or perfluorinated) carbon chains can provide benefits in reducing the amount of surface fouling relative to alkyl-terminated chains. As used herein, surface fouling refers to the amount of indiscriminate material deposition on the surface of a microfluidic device, which may include permanent or semi-permanent deposition of biomaterials (such as proteins and their degradation products, acids and their respective degradation products, etc.).
[0193] Single or multi-part treated surfaces. The covalently attached coating material can be formed by reacting a molecule that already contains a portion configured to provide an organic and / or hydrophilic molecule layer suitable for culturing / amplifying a biological micro-object in a microfluidic device, as described below. Alternatively, the covalently attached coating material can be formed by coupling a two-part sequence configured to provide an organic and / or hydrophilic molecule layer suitable for culturing / amplifying a biological micro-object to a surface modification ligand that is itself already covalently attached to the surface.
[0194] Methods for preparing covalently attached coating materials. In some embodiments, the coating material covalently attached to a surface of a microfluidic device (e.g., at least one surface including an isolation dock and / or a flow area) has a structure of Formula 1 or Formula 2. When the coating material is introduced to the surface in one step, it has a structure of Formula 1, and when the coating material is introduced in a multi-step process, it has a structure of Formula 2.
[0195] The coating material may be covalently attached to the substrate of the DEP structure or EW structure. The substrate of the DEP structure or EW structure may include silicon, silicon oxide, aluminum oxide, or hafnium oxide. The oxide may be present as part of the native chemical structure of the substrate, or may be introduced as described below.
[0196] The coating material can be connected to the oxide via a linking group ("LG"), which can be a siloxy or phosphonate group formed by the reaction of a siloxane or phosphonic acid group with an oxide. The portion configured to provide an organic and / or hydrophilic molecular layer suitable for culturing / amplifying biological micro-objects in a microfluidic device can be any portion described herein. The linking group LG can be directly or indirectly connected to a portion configured to provide an organic and / or hydrophilic molecular layer suitable for culturing / amplifying biological micro-objects in a microfluidic device. When the linking group LG is directly connected to the portion, there is no optional linking group ("L") and n is 0. When the linking group LG is indirectly connected to the portion, the linking group L is present and n is 1. The linking group L can have a linear portion, wherein the main chain of the linear portion can include 1 to 200 non-hydrogen atoms selected from any combination of silicon, carbon, nitrogen, oxygen, sulfur and / or phosphorus atoms, which is limited by chemical bonding known in the art. It may be interrupted by any combination of one or more moieties, which may be selected from ether, amino, carbonyl, amido and / or phosphonate groups, arylene, heteroarylene or heterocyclic groups. In some embodiments, the backbone of the linking group L may include 10 to 20 atoms. In other embodiments, the backbone of the linking group L may include from about 5 atoms to about 200 atoms; from about 10 atoms to about 80 atoms; from about 10 atoms to about 50 atoms; or from about 10 atoms to about 40 atoms. In some embodiments, the backbone atoms are all carbon atoms.
[0197] In some embodiments, a portion configured to provide an organic and / or hydrophilic molecular layer suitable for culturing / expansion of biological micro-objects can be added to the surface of the substrate in a multi-step process and has a structure as shown above in Formula 2. The portion can be any portion described above.
[0198] In some embodiments, the coupling group CG represents a reactive moiety R x and the reactive pairing part R px (i.e., configured to react with the reactive portion R xThe coupling group CG can be located at the second end of the linking group L (i.e., the end close to the part of the organic and / or hydrophilic molecule layer configured to provide a biological micro-object suitable for culturing / amplifying the microfluidic device), which can include any combination of elements as described above. In some other embodiments, the coupling group CG can interrupt the main chain of the linking group L. When the coupling group CG is a triazole group, it can be a product produced by a coupling coupling reaction, and can be further substituted (e.g., a dibenzocyclooctenyl fused triazole group).
[0199] In some embodiments, the coating material (or surface modification ligand) is deposited on the inner surface of the microfluidic device using chemical vapor deposition. The vapor deposition process can be optionally improved, for example, by pre-cleaning the cover 110, the microfluidic tubing material 116, and / or the substrate (e.g., the inner surface 208 of the electrode activation substrate 206 of the substrate of the DEP structure, or the dielectric layer of the support structure 104 of the substrate of the EW structure), by exposure to a solvent bath, ultrasonic treatment, or a combination thereof. Alternatively or in addition, such pre-cleaning can include treating the cover 110, the microfluidic tubing material 116, and / or the substrate in an oxygen plasma cleaner, which can remove various impurities while introducing an oxidized surface (e.g., an oxide on the surface, which can be covalently modified as described herein). Alternatively, a liquid phase treatment, such as a mixture of hydrochloric acid and hydrogen peroxide or a mixture of sulfuric acid and hydrogen peroxide (e.g., a piranha solution, which can have a ratio of sulfuric acid to hydrogen peroxide of about 3:1 to about 7:1) can be used instead of an oxygen plasma cleaner.
[0200] In some embodiments, after the microfluidic device 200 has been assembled to form the housing 102 defining the microfluidic conduit 120, vapor deposition is used to coat the inner surface of the microfluidic device 200. Without wishing to be limited by theory, depositing such a coating material on the fully assembled microfluidic conduit 120 can be beneficial to prevent delamination caused by weakened bonding between the microfluidic conduit material 116 and the electrode activated substrate 206 dielectric layer and / or the cover 110. In an embodiment using a two-step method, the surface modification ligand can be introduced by vapor deposition as described above, and then the introduction portion is configured to provide an organic and / or hydrophilic molecule layer suitable for the culture / amplification of biological micro-objects. The subsequent reaction can be carried out by exposing the surface-modified microfluidic device to a suitable coupling agent in a solution.
[0201] Figure 2H A cross-sectional view of a microfluidic device 290 having an exemplary covalently attached coating material providing a treated surface is depicted. As shown, the coating material 298 (shown schematically) can include a monolayer of densely packed molecules covalently bonded to an inner surface 294 of a substrate 286 and an inner surface 292 of a cover 288 of the microfluidic device 290, the substrate 286 being a DEP substrate. The coating material 298 can be disposed on substantially all of the inner surfaces 294, 292 proximal to and inwardly of the housing 284 of the microfluidic device 290, including, in some embodiments and as described above, the surface of a microfluidic tubing material (not shown) used to define tubing elements and / or structures within the microfluidic device 290. In alternative embodiments, the coating material 298 can be disposed on only one or some of the inner surfaces of the microfluidic device 290.
[0202] exist Figure 2H In the illustrated embodiment, the coating material 298 can include a monolayer of organosiloxane molecules, each of which is covalently bound to the interior surfaces 292, 294 of the microfluidic device 290 via a siloxy linking group 296. Any of the above-described coating materials 298 (e.g., alkyl-terminated, fluoroalkyl-terminated moieties, PEG-terminated moieties, dextran-terminated moieties, or terminal moieties containing a positive or negative charge for the organosiloxy moiety) can be used, wherein the capping moiety is disposed at its end facing the housing (i.e., the portion of the monolayer of coating material 298 that is not bound to the interior surfaces 292, 294 and is proximal to the housing 284).
[0203] In other embodiments, the coating material 298 used to coat the inner surfaces 292, 294 of the microfluidic device 290 may include anionic, cationic, or zwitterionic moieties or any combination thereof. Without being limited by theory, by presenting cationic, anionic, and / or zwitterionic moieties at the inner surface of the housing 284 of the microfluidic conduit 120, the coating material 298 may form strong hydrogen bonds with water molecules, such that the resulting hydration water acts as a layer (or "barrier") that separates the biological micro-object from the interaction with non-biological molecules (e.g., silicon and / or silicon oxide of the substrate). In addition, in embodiments where the coating material 298 is used in conjunction with a coating agent, the anions, cations, and / or zwitterions of the coating material 298 may form ionic bonds with charged moieties of non-covalent coating agents (e.g., proteins in solution) present in the medium 180 (e.g., coating solution) in the housing 284.
[0204] In other embodiments, the coating material may include or be chemically modified to present a hydrophilic coating agent at its end facing the shell. In some embodiments, the coating material may include a polymer containing an alkylene ether, such as PEG. In some embodiments, the coating material may include a polysaccharide, such as dextran. As with the charged moieties discussed above (e.g., anions, cations, and zwitterionic moieties), the hydrophilic coating agent can form strong hydrogen bonds with water molecules, so that the resulting hydration water acts as a layer (or "barrier") that separates the interaction of the biological micro-object with non-biological molecules (e.g., silicon and / or silicon oxide of the substrate).
[0205] Further details of suitable coating treatments and finishes may be found in U.S. Patent Application No. 15 / 135,707, filed April 22, 2016, and incorporated herein by reference in its entirety.
[0206] Additional system components for culturing the activity of cells within the isolation dock of the microfluidic device. In order to promote the growth and / or expansion of the cell population, environmental conditions that are conducive to culturing functional cells can be provided by other components of the system. For example, such other components can provide nutrients, cell growth signal substances, pH adjustment, gas exchange, temperature control, and removal of waste products from cells.
[0207] Computer Systems
[0208] Figure 5 1 is a block diagram illustrating a computer system 1000 on which an embodiment or portion of an embodiment of the present teachings may be implemented. In various embodiments of the present teachings, the computer system 1000 may include a bus 1002 or other communication mechanism for transmitting information, and a processor 1004 coupled to the bus 1002 for processing information. In various embodiments, the computer system 1000 may also include a memory 1006, which may be a random access memory (RAM) or other dynamic storage device coupled to the bus 1002, for determining instructions to be executed by the processor 1004. The memory 1006 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processor 1004. In various embodiments, the computer system 1000 may also include a read-only memory (ROM) 1008 or other static storage device coupled to the bus 1002 for storing static information and instructions for the processor 1004. A storage device 1010, such as a magnetic disk or optical disk, may be provided and coupled to the bus 1002 for storing information and instructions.
[0209] In various embodiments, the computer system 1000 may be coupled to a display 1012, such as a cathode ray tube (CRT) or a liquid crystal display (LCD), via the bus 1002 for displaying information to a computer user. An input device 1014 including alphanumeric and other keys may be coupled to the bus 1002 for communicating information and command selections to the processor 1004. Another type of user input device is a cursor control 1016, such as a mouse, trackball, or cursor direction keys, for communicating direction information and command selections to the processor 1004 and for controlling cursor movement on the display 1012. The input device 1014 typically has two degrees of freedom in two axes, a first axis (i.e., x) and a second axis (i.e., y), which allows the device to specify a position in a plane. However, it should be understood that input devices 1014 that allow three-dimensional (x, y, and z) cursor movement are also contemplated herein.
[0210] According to certain embodiments of the present teachings, the computer system 1000 can provide results in response to the processor 1004 executing one or more sequences of one or more instructions contained in the memory 1006. Such instructions can be read into the memory 1006 from another computer-readable medium or a computer-readable storage medium (e.g., the storage device 1010). The execution of the sequence of instructions contained in the memory 1006 can cause the processor 1004 to perform the processes described herein. Alternatively, hard-wired circuits can be used instead of software instructions or in combination with software instructions to implement the present teachings. Therefore, the embodiments of the present teachings are not limited to any specific combination of hardware circuitry and software.
[0211] As used herein, the term "computer-readable medium" (e.g., data retention, data storage, etc.) or "computer-readable storage medium" refers to any medium that participates in providing instructions to processor 1004 for execution. Such media can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Examples of non-volatile media can include but are not limited to optical, solid-state, magnetic disks, such as storage device 1010. Examples of volatile media can include but are not limited to dynamic memory, such as memory 1006. Examples of transmission media can include but are not limited to coaxial cables, copper wire, and optical fiber, including the wires having bus 1002.
[0212] Common forms of computer readable media include, for example, floppy disks, floppy disk drives, hard disks, magnetic tape or any other magnetic medium, CD-ROMs, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM and EPROM, FLASH-EPROM, any other memory chip or cartridge, or any other tangible medium from which a computer can read.
[0213] In addition to computer-readable media, instructions or data may be provided as signals on a transmission medium included in a communication device or system to provide a sequence of one or more instructions for execution to the processor 1004 of the computer system 1000. For example, a communication device may include a transceiver with signals indicating instructions and data. The instructions and data are configured to cause one or more processors to implement the functions outlined herein. Representative examples of data communication transmission connections may include, but are not limited to: telephone modem connection, wide area network (WAN), local area network (LAN), infrared data connection, NFC connection, etc.
[0214] It should be understood that the methods described herein, including the flow charts, diagrams, and accompanying disclosure, may be implemented using the computer system 1000 as a standalone device or on a distributed network of shared computer processing resources, such as a cloud computing network.
[0215] It should also be understood that in certain embodiments, a machine-readable storage device for storing non-transitory machine-readable instructions for executing or performing the methods described herein is provided. The machine-readable instructions can control all aspects of image processing, convolutional neural network (CNN) flows (described in detail below), logic and memory modules, and micro-object detection and counting described in detail below. In addition, the machine-readable instructions can be initially loaded into the memory module or accessed via the cloud or via an API.
[0216] Automatic detection of micro-objects of interest. In one aspect, a method for automatically detecting micro-objects of interest in an illumination image is provided, the illumination image being such as a bright field image, and particularly a digital image (or an image that has been digitized). The micro-object of interest may be arranged in a microfluidic device. The micro-object of interest may be a cell, such as a mammalian cell (e.g., a hemocyte, a hybridoma, a cancer cell, a transformed cell, a gamete, an embryo, etc.). Alternatively, the micro-object of interest may be a bead, such as may be used in an assay (e.g., a microbead, a magnetic bead, etc.). The method may involve processing image data (i.e., data related to pixels in an image) using a machine learning algorithm. The machine learning algorithm may include a neural network, such as a convolutional neural network.
[0217] Image classification requires receiving an input image and outputting the class or probability of a class that best describes the image. This can be done using a computer system equipped with a processing engine that utilizes an algorithm to process the input image and output the results. Image detection can also utilize a similar processing engine, whereby the system receives an input image and uses an algorithm pre-programmed into the processing engine to identify objects of interest within the image with high accuracy.
[0218] Regarding the input image, the system typically orients the input image as an array of pixel values. These pixel values depend on the image resolution and size, corresponding to a digital array of (length) × (width) × (# of channels). The number of channels may also be referred to as depth. For example, the array may be L × W × red-green-blue color model (RBG values). RGB would be considered three channels, each representing one of the three colors in the RGB color model. For example, the system may typically characterize a 20 × 20 image with a representative array of 20 × 20 × 3 (for RGB), where each point in the array is assigned a value representing the pixel intensity (e.g., 0 to 255). Given this array of values, the processing engine can use its algorithm to process these values to output the probability of describing the image as a certain category (e.g., 0.80 for cells, 0.15 for cell walls, and 0.05 for no cells).
[0219] Convolutional neural networks (CNNs) typically implement a high-level form of image processing and classification / detection by first looking for low-level features such as edges and curves, and then advancing to more abstract (e.g., unique to the type of image being classified) concepts through a series of convolutional layers. CNNs can be implemented by passing an image through a series of convolutional, nonlinear, pooling (or downsampling, as will be discussed in more detail below), and fully connected layers and obtaining an output. Again, the output can be a single class or probability of a class that best describes the image or detects an object on the image.
[0220] Regarding the layers in a CNN, the first layer is typically a convolutional layer (Conv). This first layer will process a representative array of an image using a series of parameters. Instead of processing the image as a whole, a CNN will use filters (or neurons or kernels) to analyze a collection of image subsets. The subset will include the focal point in the array as well as the surrounding points. For example, the filter can examine a series of 5×5 areas (or regions) in a 32×32 image. These areas can be called receptive fields. Since the filter will typically have the same depth as the input, an image of size 32×32×3 will have a filter of the same depth (e.g., 5×5×3). The actual steps of convolving the filter using the above exemplary sizes include sliding the filter along the input image, multiplying the filter values with the original pixel values of the image to calculate element-wise multiplications, and summing these values to obtain a single number for the examined area of the image.
[0221] After this convolution step is completed, using 5×5×3 filters, an activation map (or filter map) with dimensions of 28×28×1 will be produced. For each additional layer used, the spatial dimensionality is better preserved, so that using two filters will result in an activation map of 28×28×2. Each filter will typically have a unique feature that it represents (e.g., color, edge, curve, etc.) that together represent the feature identifier required for the final image output. These filters, when used in combination, allow the CNN to process the image input to detect those features that are present at each pixel. Therefore, if the filter is used as a curve detector, the convolution of the filter along the image input will produce an array of numbers in the activation map that corresponds to a high likelihood of a curve (high sum element-wise multiplication), a low likelihood of a curve (low sum element-wise multiplication), or zero values, where the input volume at certain points will not activate the curve detector filter. Therefore, the greater the number of filters (also known as channels) in the Conv (convolution), the more depth (or data) is provided on the activation map, and therefore more information about the input will result in a more accurate output.
[0222] The balance with accuracy of CNNs is the processing time and power required to produce the results. In other words, the more filters (or channels) are used, the more time and processing power are required to perform the convolution. Therefore, the choice and number of filters (or channels) should be specifically chosen to meet the needs of the CNN method in order to produce an output that is as accurate as possible while taking into account the available time and power.
[0223] To further enable the CNN to detect more complex features, additional convolutions can be added to analyze the output from previous convolutions (i.e., activation maps). For example, if the first convolution looks for basic features such as curves or edges, the second convolution can look for more complex features, such as shapes, which may be a combination of individual features detected in earlier convolutional layers. By providing a series of convolutions, the CNN can detect higher and higher level features to ultimately reach the possibility of detecting a specific desired object. In addition, when convolutions are stacked on top of each other, analyzing the previous activation map output, each convolution in the stack naturally analyzes a larger and larger receptive field through the proportional reduction that occurs at each convolution level, allowing the CNN to respond to growing regions of pixel space when detecting objects of interest.
[0224] A CNN architecture typically consists of a set of processing blocks including at least one processing block for convolving an input volume (image) and at least one processing block for deconvolution (or transposed convolution). In addition, the processing blocks may include at least one pooling block and an unpooling block. Pooling blocks can be used to reduce an image by resolution to produce an output that can be used for convolution. This can help with computational efficiency (efficient time and power), which in turn can improve the actual performance of CNNs. Although pooling or upsampling blocks make filters smaller and computationally demanding, these blocks can coarsen the output (can result in loss of spatial information within the receptive field), reducing it by a specific ratio from the size of the input.
[0225] These coarse outputs can be reconstructed using an up-pooling block to produce an output volume with the same dimensionality as the input volume. The up-pooling block can be thought of as the reverse operation of the convolution block to return the activation output to the original input volume dimensionality.
[0226] However, the pooling block processing usually simply upscales the coarse output into a sparse activation map. To avoid this result, the deconvolution block densifies the sparse activation map to produce a final output volume that, after any further necessary processing, is closer in size and density to the input volume. As the reverse operation of the convolution block, the deconvolution block associates a single activation output point with multiple outputs to upscale and densify the resulting activation output, rather than reducing multiple array points in the receptive field to a single number.
[0227] It should be noted that pooling blocks can be used to downscale images and up-pooling blocks can be used to up-scale these scaled-down activation maps, while convolution and deconvolution blocks can be constructed to convolve / deconvolve and down-scale / up-scale without the need for separate pooling and up-pooling blocks.
[0228] Pooling and over-pooling processes can have drawbacks depending on the objects of interest detected in the image input. Since pooling typically scales down an image by looking at sub-image windows without overlapping the windows, there is a significant loss of spatial information as the downscaling occurs.
[0229] The processing block may include other layers wrapped with convolutional or deconvolutional layers. These may include, for example, rectified linear unit layers (ReLU) or exponential linear unit layers (ELU), which are activation functions that examine the output from the convolution in their processing blocks. The ReLU or ELU layers act as gate functions to advance only those values corresponding to positive detections of features of interest that are unique to the convolution.
[0230] Given the basic architecture, the CNN is then prepared for the training process to train its accuracy in image classification / detection (objects of interest). This involves a process called back propagation ("back propagation"), which uses a training data set or sample images for training the CNN so that it updates its parameters when it reaches an optimal or threshold accuracy. Back propagation involves a series of repeated steps (training iterations), which, depending on the parameters of the backtracking, will train the CNN slowly or quickly. The back propagation step typically includes a forward pass, a loss function, a backward pass, and a parameter (weight) update according to a given learning rate. The forward pass involves passing the training image through the CNN. The loss function is a measure of the error in the output. The backward pass determines the influencing factors of the loss function. Weight updates involve updating the parameters of the filter to move the CNN toward the optimal. The learning rate determines the extent of the weight update for each iteration to reach the optimal. If the learning rate is too low, the training may take too long and involve too much processing power. If the learning rate is too fast, each weight update may be too large to allow a given optimal or threshold to be accurately achieved.
[0231] The back-propagation process can lead to complications in training, leading to the need for lower learning rates and more specifically and carefully determined initial parameters at the beginning of training. One such complication is that when weight updates occur at the end of each iteration, changes to the parameters of the convolutions amplify the depth at which the network works, making it deeper. For example, if a CNN has multiple convolutions, which allow for higher-level feature analysis as discussed above, then parameter updates to the first convolution are multiplied at each subsequent convolution. The net effect is that a minimal change in parameters can have a large impact depending on the depth of a given CNN. This phenomenon is known as internal covariate shift.
[0232] The embodiments disclosed herein have several advantages over known CNNs. These advantages include, for example, providing CNNs that avoid the loss of spatial information inherent in pooling layers, reducing / minimizing the internal covariate shift inherent in the back-propagation process, and reducing the processing time and rate typically required in deep neural networks to achieve more complex feature detection.
[0233] As mentioned above, CNNs consist of multiple layers of receptive fields. These are clusters of "neurons" (or kernels) that process parts of the input image. The outputs of these clusters are then tiled so that their input regions overlap to get a better picture of the original image; this is repeated for each such layer. Tiling allows CNNs to tolerate translations of the input image. For example, CNNs have been described in Long et al., "Fully Convolutional Networks for Semantic Segmentation" CVPR 2015, and Noh et al., "Learning Deconvolutional Networks for Semantic Segmentation" ICCV 2015, the contents of each of which are incorporated herein by reference.
[0234] CNNs can include a combination of convolutional and fully connected layers, where point-wise nonlinearity is applied at the end or after each layer. Convolution operations on small input regions are introduced to reduce the number of free parameters and improve generalization. A major advantage of convolutional networks is the use of shared weights in convolutional layers, which means that the same filter (set of weights) is used for each pixel in the layer; this both reduces memory usage and improves performance.
[0235] In one embodiment, a CNN is formed by a stack of different layers that transform an input volume into an output volume (e.g., preserving class scores) through differentiable functions.
[0236] In this embodiment, embodiment convolutional layers are defined as empty, monoclonal, and polyclonal. The parameters of a layer may include a set of learnable filters that have a small receptive field but extend to the full depth of the input volume. During the forward pass, each filter is convolved over the width and height of the input volume, the dot product between the entries of the filter and the input is computed, and a two-dimensional activation map of that filter is produced. As a result, the network learns filters that activate when a certain type of feature is seen at a certain spatial location in the input.
[0237] The activation maps of all filters are stacked along the depth dimension to form the full output volume of the convolutional layer. Therefore, each entry in the output volume can also be interpreted as the output of a neuron that looks at a small area in the input and shares parameters with neurons in the same activation map.
[0238] In one embodiment, the spatial arrangement is based on hyperparameters that control the size of the output volume of the convolutional layer: such as depth, stride, and zero padding.
[0239] In one embodiment, the depth of the output volume controls the number of neurons in the layer that are connected to the same area of the input volume. All of these neurons will learn to activate different features in the input. For example, if the first convolutional layer takes the original image as input, different neurons along the depth dimension may activate in the presence of edges of various orientations or spots of color.
[0240] In one embodiment, the stride controls how the depth columns are distributed around the spatial dimensions (width and height). When the stride is 1, the depth columns of new neurons are assigned to spatial locations that are only 1 spatial unit apart. This leads to severe overlap of the receptive fields between columns and also leads to large output volumes. Conversely, if a higher stride is used, the receptive fields will overlap less and the resulting output volume will have a smaller size in space.
[0241] Sometimes it is convenient to pad the input with zeros on the boundaries of the input volume. The size of this zero padding is a third hyperparameter. Zero padding provides control over the spatial size of the output volume. In particular, it is sometimes desirable to preserve the spatial size of the input volume exactly.
[0242] In this implementation, a parameter sharing scheme is used in the convolutional layers to control the number of free parameters. It relies on a reasonable assumption: if a patch feature is useful to compute at a certain spatial location, it should also be useful to compute at a different location. In other words, a single 2D slice of depth is represented as a depth slice, and we constrain the neurons in each depth slice to use the same weights and biases.
[0243] Since all neurons in a single depth slice share the same parameterization, the forward pass in each depth slice of a convolutional layer can be computed as the convolution of the neuron’s weights with the input volume (hence the name: convolutional layer).
[0244] Therefore, the set of weights is often referred to as a filter that is convolved with the input. The result of this convolution is an activation map, and the set of activation maps for each different filter is stacked together along the depth dimension to produce the output volume. Parameter sharing contributes to the translation invariance of the CNN architecture.
[0245] In various embodiments, a neural network (or CNN) is provided, such as, for example Figure 7 8 and 9. FIG. 9A to FIG. 9D 8 and will be used for reference purposes only in describing this embodiment, as shown in FIG. FIG. 9A to FIG. 9D The captured CNN features can be compared with Figure 7 The network shown or various other embodiments described herein may be used in combination.
[0246] exist Figure 7 , the neural network 700 includes a first downsampling block 710, a second downsampling block 730, and a third downsampling block 750, which have associated first processing blocks 720, second processing blocks 740, and third processing blocks 760 (or residual network blocks). The first downsampling block 710 receives the input image 701. As illustrated, each downsampling block may be followed by its associated processing (or residual) block. As discussed in detail below, the processing (or residual) blocks may be single-branched or multi-branched.
[0247] A CNN can include multiple downsampling blocks (e.g. Figure 7 ), where each downsampling block may include a downsampling convolutional layer (Conv), a batch normalization (Norm) layer, and an activation layer including a gate function.
[0248] Figure 8B An example of a downsampling block is shown that receives an input 871 and provides an output 879, and includes a convolution 874 with a kernel size D×D, a batch normalization layer 876, and an activation layer 878. The activation layer can be, for example, an ELU or a ReLU. In various embodiments, the activation layer receives image data directly from the batch normalization layer, which receives image data directly from the downsampling convolution layer. The downsampling convolution layer can function to reduce the spatial resolution of the image data it receives. This will refer to FIG. 9A to FIG. 9D Discuss in more detail.
[0249] The processing block (or residual network block) can be a single branch processing block or a multi-branch processing block, where each branch processes the output from the previous downsampling block, and then combines the outputs of the two branches to produce a downsampled activation map for further downsampling or upsampling to the final output.
[0250] Fig. 8A The description is configured to downsample from an upstream block (not depicted, see Figure 8B 805 (e.g., in the form of an activation map). Block 800 includes a first branch 810 and a second branch 840. The first branch 810 includes: a first convolutional layer 815 (Conv) with an N×N kernel, a first batch normalization (norm) layer 820 that receives data from the first batch normalization layer 820, a first activation layer 825 (which may include or act as a gate function) that receives data from the first batch normalization layer 820, a second convolution 830 with an M×M kernel that receives data through the first activation layer 825, and a second batch normalization layer 835 that receives data from the second convolution 830. It should be noted that the kernels of the convolutions 815 (N×N) and 830 (M×M) can have the same size or can be different. As shown in FIG. Figures 9A to 9CAs shown (discussed below), in the residual network shown, the kernels from the serial convolutions are the same (3×3). In any case, it is generally preferred that the convolutions 815 / 830 have kernels larger than 1×1.
[0251] The second branch 840 includes a third convolution 845, a third batch normalization layer 850 that receives data from the third convolution 845, and a second activation layer 855 (which may include or act as a gate function) that receives data from the third batch normalization layer 850. The block 800 also includes a reorganization layer 860 that receives data from both the second batch normalization layer 835 and the data that passes through the second activation layer 855. Finally, the block 800 includes a block activation layer 862 that can be used as a gate function for the data received from the reorganization layer 860 before generating an output 864 from the block 800 for further processing. As described above, the activation layer can be, for example, an ELU or a ReLU. In various embodiments, the activation layer is an ELU.
[0252] exist Fig. 8A , the second branch 840 processes the image data received from the previous downsampling block to a lesser extent than the first branch 810. Specifically, the third convolution 845 of the second branch 840 uses a filter window (or dimension or kernel) of 1×1, while the first and second convolutions 815 / 830 of the first branch 810 use filter windows (or dimensions or kernels) of N×N and M×M, respectively, which, as described above, will typically be larger than 1×1. These filter windows can be adjusted as needed, taking into account factors such as image type, image quality, object type, object size, object shape, output requirements, time constraints, stride length (discussed below), and power / processing resources. For example, the first and second convolutions 815 / 830 can use a 3×3 filter window (or dimension) (see below) FIG. 9A to FIG. 9D , showing the filter window size).
[0253] Although Fig. 8A Both branches in may have convolutions with a stride of 1, but the strides may also be different. However, in order to allow the reorganization layer 860 to be effective, the product of the stride of the convolution 815 / 830 multiplied by the first branch 810 must be equal to the stride of the convolution 845 of the second branch 840. Again, strides will be discussed in more detail below.
[0254] Inserting a batch normalization layer before the activation step provides the advantage of helping to minimize internal covariate shift. By inserting a batch normalization layer in this way, and by extension, after the convolution, batch normalization can normalize the output of the convolution, thereby providing normalized data to the activation step, allowing for a more stable distribution of activations. By minimizing internal covariate shift during the back-propagation process, neural networks can be trained more aggressively via higher learning rates (the degree to which weights are updated), resulting in faster CNN learning without loss of efficiency and accuracy as the CNN works towards the optimal parameters for a given filter in the network.
[0255] Furthermore, adding a residual network with a branch that has minimally processed information (e.g., a 1×1 convolution branch) allows for easier learning during training. This minimally processed branch provides a more direct path to track the impact of earlier parameters on the final result. In fact, this branch serves the same purpose as the skip connection within a given residual network (discussed in more detail below), allowing some information to pass through the network unchanged so as not to lose spatial information that may be lost during downsampling.
[0256] Thus, in summary, the use of residual networks alone and in combination with batch normalization layers allows for easier and more efficient learning during training than neural networks known in the prior art. This advantage is achieved by, for example, maintaining more spatial information during downsampling and minimizing internal covariate shift. Minimizing the loss of spatial information is also achieved using strides (discussed in more detail below), which allow more overlap during downsampling with known methods such as pooling, as well as skip connections, which allow less processed information to be fed forward during the neural network process (in the downsampling step discussed above, and transferred to the upsampling step, as described below).
[0257] By using a multi-branch residual network, and in particular one of the branches using a 1×1 filter window (i.e., not downsampled), the neural network is allowed to further convolve the output data from the previous convolution while maintaining the same resolution to ensure that each pixel is combined for analysis as a single window, at the recombinant layer 860, with data from another branch (which may have undergone multiple convolutions at a larger kernel or filter size) to output quality image data (not downsampled from the previous convolution), in preparation for further downsampling.
[0258] Return to Figure 7 , the neural network 700 further includes a first upsampling block 770, a second upsampling block 780, and a third upsampling block 790, wherein the output 799 follows the third upsampling block 790. Each upsampling block may include a transposed convolution (or deconvolution) layer, an upsampling batch normalization layer, and an upsampling activation layer including a gate function.
[0259] Figure 8C An example of an upsampling block is shown which receives an input 881 and provides an output 889 and includes a transposed convolution 884 with a kernel size Z×Z, a batch normalization layer 886, and an activation layer 888. FIG. 9A to FIG. 9D These subcomponents are discussed in more detail. The transposed convolutional layer of each upsampling block can be configured to increase the spatial resolution of the image data it receives and thereby reconstruct the downsampled output. In addition, a recombination layer can also be included in one or more of the upsampling blocks, whereby image data from the upsampling batch normalization layer is merged with image data from a previous residual network block (via skip connections, discussed below).
[0260] Regarding the architecture of the neural network, the number of upsampling blocks can be configured to be equal to the number of downsampling blocks. In various embodiments, the neural network has n downsampling blocks, n residual network (or processing) blocks, n upsampling blocks, and n-1 upsampling blocks including a recombination layer (see Fig.9D ). As will be discussed in more detail below, as the spatial resolution decreases slightly during the downsampling process, one may wish to increase the spatial resolution at the same fractional rate. For example, if the spatial resolution is halved (halved) each time through a downsampling block (or a combined downsampling and residual network block), then it may be most efficient to double (twice) the spatial resolution back to the original image size. This may result in an equal number of downsampling and upsampling blocks.
[0261] For example, in Figure 7 In , each convolution reduces the spatial resolution of the image data by a factor of two, and each transposed convolution increases the spatial resolution of the image data by a factor of two. The reduction in spatial resolution can be achieved, for example, by sliding the convolution filter (or kernel) two pixels at a time. The two pixel slide is called the stride length. In the case of sliding two pixels at a time, the stride will be 2. By using a stride length of 2, the convolution can downsample the samples by halving the size of the activation map output from the convolution.
[0262] However, by striding rather than pooling as taught above, the loss of spatial information that may be inherent in pooling can be avoided. The filter size determines how much local information is introduced into the analysis of a single pixel to influence each pixel in the next layer in the network. Typically, the filter size is an odd number so as to be centered on the pixel of interest. For example, a 5×5 filter will examine the surrounding 24 pixels to analyze one central pixel of a given region. With pooling, a first region is examined to effectively determine a single value corresponding to the pixels in that first region. Once the filter moves to a second region, the pixels in the first region are no longer analyzed during the filter scan. This can lead to very misleading, rough, or inaccurate results, depending on the type of image analysis being performed (e.g., the type of object being detected).
[0263] On the other hand, using stride theory, once the first region (e.g., 5×5 region) is examined, and a two-pixel stride occurs for the second region (also 5×5), it is clear that the pixel is viewed more than once and considered in the decision of multiple pixels through overlap, while still allowing downsampling, because the final result of sampling with a two-pixel stride will result in an image output (activation map output) that is half the size of the previous one. Therefore, compared to pooling, less spatial information loss will occur with striding for downsampling. Factors used to determine the appropriate stride length include, for example, image type, image quality, object type, object size, object shape, output requirements, time constraints, and power / processing resources.
[0264] As illustrated, if the spatial resolution of input image 701 is X, downsampling block 710 may reduce the spatial resolution in half to X / 2, then to X / 4 through downsampling block 730, then to X / 8 through downsampling block 750. Upsampling block 770 may then double the X / 8 input to X / 4, block 780 to X / 2, and block 790 to X, or double to the original size at output 799. Figure 7 Visually represent the decrease in height of each down-sampling block and the increase in height of each up-sampling block.
[0265] As downsampling proceeds, the CNN can be designed to increase the complexity of the features it processes, moving from lower-level feature analysis to higher-level feature analysis. As previously discussed, to further enable the CNN to detect more complex features, additional convolutions can be added to analyze which outputs from previous convolutions (i.e., activation maps). For example, if the first convolution looks for basic features such as curves or edges, the second convolution can look for more complex features, such as shapes, which may be a combination of individual features detected in earlier convolutions. By providing a series of convolutions, the CNN can detect higher and higher levels of features to ultimately reach a specific desired object detection. In addition, as convolutions are stacked on top of each other, analyzing the previous activation map outputs, each convolution in the stack naturally analyzes a larger and larger receptive field through the proportional reduction that occurs at each convolution level, allowing the CNN to respond to growing regions of pixel space when detecting objects of interest.
[0266] exist Figure 7 , each convolution and processing block increases the channel depth by a factor of two, and each upsampling block reduces the channel depth by a factor of two, until the third upsampling block 790. As shown, at downsampling block 710 and processing block 720, 32 channels or filters are used. At downsampling block 730 and processing block 740, the number of channels is 64. Finally, downsampling block 750 and processing block 760 use 128 channels. In contrast, upsampling block 770 halves the channels back to 64, upsampling block 780 reduces it to 32, and upsampling block 790 to 3 (the importance of which will be discussed in more detail below). Figure 7 This increase and decrease in channel usage is generally represented where the width of each downsampling block increases and the width of each upsampling block decreases (except for the final block 790).
[0267] The rate of change of spatial resolution (original, X / 2, X / 4, X / 8, X / 4, X / 2, original) is almost inverse to the rate of channel depth (0, 32, 64, 128, 64, 32, 3), which is not necessary for the CNN architecture. However, the consistent change of spatial resolution with the number of channels advantageously allows the CNN to maximize the time, processing power and quality of the output 799 by offsetting the sequential increase of filter depth with the sequential decrease of input data (activation map dimension). In fact, as the processing demand on the CNN increases with the filter depth through each successive downsampling block, the CNN offsets it by reducing the image array input (activation map dimension) through each successive downsampling block to allow the CNN to analyze smaller inputs of greater depth. Accordingly, the reverse occurs to back up the upsampling blocks to the output 799.
[0268] The reconstruction of the image volume can also be aided by a form of skip architecture. For example, skip connections inserted in a neural network can project information from an earlier downsampling layer to a later upsampling layer so that this earlier, minimally processed information becomes part of the reconstruction process. Without the use of a skip architecture, some of the information captured in the initial convolutional layers (which can greatly help with reconstruction during upsampling) will have been lost during the downsampling process. In other words, this valuable information will have been downsampled to the point where it may become too abstract to use the information further. Using a skip architecture to provide this information from the main layer to the later upsampling layer allows the earlier information to be preserved and used for efficient upsampling.
[0269] In various embodiments, the neural network may include a first upsampling block having a reorganization layer that receives image data from a second residual network block (e.g., via a skip connection), a second upsampling block having a reorganization layer that receives image data from the first residual network block (e.g., via a skip connection), and a third upsampling block that does not include a reorganization layer.
[0270] exist Figure 7 , for example, a first skip connection 792 and a second skip connection 794 are provided. The first skip connection 792 feeds forward the output information from the processing block 720 to the recombinant layer of the upsampling block 780 at the X / 2 resolution, also after batch normalization at the X / 2 resolution (discussed below). Via this skip connection, the neural network provides earlier and minimally processed information at the same resolution as the corresponding upsampling block to allow for more accurate and efficient upsampling. The second skip connection 794 similarly feeds forward the output information at the X / 4 resolution from the processing block 740 to the recombinant layer of the upsampling block 770, also after batch normalization at the X / 4 resolution (discussed below).
[0271] As described above, CNN can be used for many purposes, including image classification and image detection (also including object detection within an image). Therefore, depending on the target of CNN, the output must answer the main questions raised to CNN. In various embodiments herein, CNN is used for image detection. In various embodiments, image detection can be used to detect an object of interest. In various embodiments, the object of interest can be a micro-object. In various embodiments, image detection can be used to classify micro-objects into at least one of a plurality of micro-object types. In various embodiments, the micro-object is a biological cell. In various embodiments, the biological cell is an immune cell, such as a T cell, a B cell, a NK cell, a macrophage, or a combination thereof. In various embodiments, the biological cell is a cell (such as a CHO cell) or a cancer cell from a cell strain. In various embodiments, the biological cell is an oocyte, a sperm, or an embryo.
[0272] about Figure 7The illustrated use of three channels in the upsampling block 790 of , in various embodiments, a system using a CNN obtains a micro-object count from an image input. The system can do this by annotating multiple pixels of the input image, each pixel annotation of the group representing the probability that the corresponding pixel in the image represents the corresponding micro-object characteristic. Based on this analysis, a micro-object count can be obtained. In various embodiments, the multiple micro-object characteristics include at least three micro-object characteristics. In various embodiments, multiple micro-object characteristics include at least a micro-object center, a micro-object edge, and a non-micro-object (or a cell center, a cell edge, and a non-cell). Figure 7 The upsampling block 790 shows these three micro-object characterizations through their three channel depths. Figure 7 The final upsampling block 790 provides the object representation required by the neural network 700 to determine accurate micro-object (e.g., cell) counts.
[0273] FIG. 9A to FIG. 9D A schematic diagram of a more detailed convolutional neural network (CNN) 900 is shown according to various embodiments. The schematic diagram incorporates many of the neural network principles discussed above, and for this reason, these principles will not be repeated in detail. However, it should be noted that while the principles may be similar, the parameters used in the various embodiments herein may all vary based on specific reasons as discussed above, including, for example, image type, image quality, object type, object size, object shape, output requirements, time constraints, and power / processing resources. Therefore, in FIG. 9A to FIG. 9D The parameters used in the schematic diagrams are examples only.
[0274] For orientation purposes, Fig.9A From left to right, a first downsampling block 910 is shown, followed by a first residual network block 920, according to various embodiments. Fig. 9B From left to right, the first residual network block 920 ( Fig.9A A second downsampling block 930 of received data is followed by a second residual network block 940. Fig. 9C From left to right are shown the second residual network block 940 ( Fig. 9B A third downsampling block 950 of received data is followed by a third residual network block 960. Fig.9D From left to right, a first upsampling block 970, a second upsampling block 980, and a third upsampling block 990 are shown. The first upsampling block 970 receives data from the third residual network block 960 ( Fig. 9C), and includes a first upsampling reorganization layer 976, whereby data from the batch normalization layer of the first upsampling block 970 is reorganized with data from the final ELU layer 948 of the second residual network block 940 fed forward via the second skip connection 994. Similarly, the second upsampling block 980 includes a second upsampling reorganization layer 986, whereby data from the batch normalization layer of the second upsampling block 980 is reorganized with data from the final ELU layer 928 of the first residual network block 920 fed forward via the first skip connection 992.
[0275] Return to reference Fig.9A , CNN 900 includes a first downsampling block 910 configured to receive an image input 901. The first downsampling block 910 includes a first convolution 912, a first batch normalization layer 914, and a first activation layer 916 (e.g., Fig.9A The first convolution 912 can have different parameters for kernel size and stride. Here, the kernel is 5×5 and the stride is two pixels. The output from layer 916 feeds the first residual network block 920, which includes a first branch 922 and a second branch 924. For a general discussion of the layout of the residual network, see Figure 8. In the first branch 922, the two convolutions have a kernel size of 3×3. Fig.9A Also shown is the start of the first skip connection 992 feeding the forward data, which as previously described is output after the first recombinant layer 926 and the first ELU 928. Also note that the downscaling at this stage of the CNN 900 is by a factor of 2 (downsampling to 1 / 2 spatial resolution) and that 32 channels of features are used at this stage.
[0276] refer to Fig. 9B , CNN900 also includes a second downsampling block 930, which includes a second convolution 932, a second batch normalization layer 934, and a second activation layer 936 (e.g., Fig. 9B 928). The second downsampling block 930 is configured to receive the output from the first ELU 928. The second convolution 932 may have different parameters for kernel size and stride. Here, the kernel is again 5×5 and the stride is again two pixels. The output from layer 936 feeds the second residual network block 940, which includes a third branch 942 and a fourth branch 944. For a general discussion of the layout of the residual network, see Figure 8. In the first branch 942, the two convolutions have a kernel size of 3×3. Fig. 9B Also shown is the start of the second skip connection 994 feeding the forward data, which is output after the second reassembly layer 946 and the second ELU 948 as described above. It should also be noted that this stage of the CNN 900 is relatively Fig.9AThe downscaling factor of the previous stage is 2 times (relative to the initial downsampling to 1 / 4 spatial resolution), and 64 channels of features are used in this stage.
[0277] refer to Fig. 9C , CNN 900 includes a third downsampling block 950, which includes a third convolution 952, a third batch normalization layer 954, and a third activation layer 956 (e.g., Fig. 9C 948). The third downsampling block 950 is configured to receive the output from the second ELU 948. The third convolution 952 can have different parameters for kernel size and stride. Here, the kernel is again 5×5 and the stride is again two pixels. The output from layer 956 feeds the third residual network block 960, which includes a fifth branch 962 and a sixth branch 964. For a general discussion of the layout of the residual network, see Figure 8. In the fifth branch 962, the two convolutions have a kernel size of 3×3. It should also be noted that the reduction factor of this stage of CNN900 is 2 times (downsampled to 1 / 8 spatial resolution) and that features of 128 channels are used at this stage.
[0278] refer to Fig.9D , CNN900 includes a first upsampling block 970, a second upsampling block 980, and a third upsampling block 990. The first upsampling block 970 includes a first upsampling convolution 972, a first upsampling batch normalization layer 974, a first upsampling reorganization layer 976, and a first upsampling activation layer 978 (e.g., ELU). The first upsampling reorganization layer 976 is configured to receive an input from a first skip connection 992, combine the input with an output from the first upsampling batch normalization layer 974, and feed the combined output to the first upsampling activation layer 978. As described above with reference to the downsampling convolution 912 / 932 / 952, the upsampling convolution layer may have different parameters for kernel size and stride. Here, for the first upsampling convolution 972, the kernel is 5×5 and the stride is two pixels. Also note that the upscaling factor of this stage of CNN900 is 2 (upsampling to 1 / 4 spatial resolution) relative to the output from the third residual network 960, and that 64 channels of features are used at this stage.
[0279] The second upsampling block 980 includes a second upsampling convolution 982, a second upsampling batch normalization layer 984, a second upsampling reorganization layer 986, and a second upsampling activation layer 988 (e.g., ELU). The second upsampling reorganization layer 986 is configured to receive input from the second skip connection 994, combine the input with the output from the second upsampling batch normalization layer 984, and feed the combined output to the second upsampling activation layer 988. As described above with reference to the downsampling convolution 912 / 932 / 952, the upsampling convolution layer may have different parameters for kernel size and stride. Here, for the second upsampling convolution 982, the kernel is 5×5 and the stride is two pixels. It is also noted that the magnification of this stage of CNN900 is 2 (upsampled to 1 / 2 spatial resolution) relative to the output from the third residual network 970, and features of 32 channels are used at this stage.
[0280] The third upsampling block 990 includes a third upsampling convolution 992, a third upsampling batch normalization layer 994, and a third upsampling activation layer 996 (e.g., ELU). Layer 996 produces an output 999 for CNN900. As discussed above with reference to downsampling convolutions 912 / 932 / 952, the upsampling convolution layer may have different parameters for kernel size and stride. Here, for the third upsampling convolution 992, the kernel is 5×5 and the stride is two pixels. It should also be noted that the magnification of this stage of CNN900 is 2 (upsampled to the initial spatial resolution) relative to the second upsampling block 980, and features of 3 channels are used at this stage.
[0281] As mentioned above about Figure 7 As discussed, in various embodiments, a system utilizing a CNN obtains micro-object counts from an image input. The system can do this by annotating multiple pixels of the input image, each pixel annotation of the group representing the probability that the corresponding pixel in the image represents the corresponding micro-object characteristic. Based on this analysis, a micro-object count can be obtained. In various embodiments, the multiple micro-object characteristics include at least three micro-object characteristics. In various embodiments, multiple micro-object characteristics include at least a micro-object center, a micro-object edge, and a non-micro-object (or a cell center, a cell edge, and a non-cell). Fig.9D The upsampling block 990 shows the three micro-object representations through their three channel depths. Fig.9D The final upsampling block 990 provides the object representation required by the neural network 900 to determine accurate micro-object (e.g., cell) counts.
[0282] According to various embodiments, systems and methods for automatically detecting micro-objects in images are disclosed. In various embodiments, the micro-object is a biological cell. In various embodiments, the biological cell is an immune cell, such as a T cell, a B cell, a NK cell, a macrophage, or a combination thereof. In various embodiments, the biological cell is a cell from a cell line (e.g., a CHO cell) or a cancer cell. In various embodiments, the biological cell is an oocyte, a sperm, or an embryo. In various embodiments, the biological cell is a bacterium. In various embodiments, the biological cell is a fungus, such as a yeast or a filamentous fungus. In various embodiments, prior to imaging and detection, the bacteria or fungus are exposed to a hypotonic solution, thereby swelling the cells and facilitating their detection.
[0283] Fig.10 is an exemplary flow chart illustrating a method for automatically detecting micro-objects in an image according to various embodiments. The exemplary flow chart may be used, for example, in Fig.11 As depicted herein, step 1110, which may be performed by imaging element 1206 of image acquisition unit 1202 of system 1200, includes receiving imaging data of a microfluidic device.
[0284] As depicted herein, step 1120 details exemplary workflow steps that may be implemented by the image preprocessing engine 1208 of the image acquisition unit 1202 of the system 1200. In step 1120, the method includes preprocessing the image data to reduce anomalies in the image data, such as noise and / or image distortion. An example of noise is a repeating pattern, such as may be associated with the interior surface of a microfluidic device.
[0285] As depicted herein, steps 1130 and 1140 detail exemplary workflow steps that may be implemented by a machine learning algorithm, such as a neural network 1210 of a micro-object detection unit 1204 of system 1200. At step 1130, the method includes processing pixel data in the imaging data using a neural network to annotate the pixel data according to a plurality of micro-object characteristics. At step 1140, the method includes outputting a probability value for each pixel in the pixel data. The output probability value may be in the form of a plurality of pixel masks, each mask corresponding to a micro-object characteristic from a plurality of micro-object characteristics. Each mask may include a set of pixel annotations (or a set of probability values) for an image of a particular micro-object characteristic associated with the mask.
[0286] As depicted herein, step 1150 details exemplary workflow steps that may be implemented by threshold engine 1212 of micro-object detection unit 1204 of system 1200. At step 1150, the method includes applying a threshold to determine which pixel probabilities at least meet a defined threshold.
[0287] As depicted herein, step 1160 details exemplary workflow steps that may be implemented by detection engine 1214 of micro-object detection unit 1204 of system 1200. At step 1160, the method includes determining a micro-object count based on the number of micro-objects identifiable after application of a threshold.
[0288] Fig.12 is an exemplary flow chart illustrating a method for detecting and characterizing micro-objects in a microfluidic device according to various embodiments. Fig.13 The exemplary flow chart is performed on the system 1400 of the present invention, as described in detail below. As shown herein, step 1310 that can be performed by the imaging element 1406 of the system 1400 includes receiving a first image and one or more second images of the same area of interest of the microfluidic device. According to various embodiments, the first image can be a bright field (or "illuminated") image, and each of the one or more second images can be a non-bright field (or non-illuminated) image, such as a fluorescent image, an infrared image, an ultraviolet image, etc. According to various embodiments, the first image can be a non-illuminated image (e.g., a fluorescent image, an infrared image, an ultraviolet image), and each of the one or more second images can be a non-illuminated image (e.g., a fluorescent image, an infrared image, an ultraviolet image). According to various embodiments, the first image can be a non-illuminated image (e.g., a fluorescent image, an infrared image, an ultraviolet image), and each of the one or more second images can be an illuminated or non-illuminated image (e.g., a fluorescent image, an infrared image, an ultraviolet image). Although not required, the imaging element 1406 can be part of the image acquisition unit of the system 1400.
[0289] In various embodiments, the time-lapse images of the region of interest of the microfluidic device can be collected in a selected time period. Collecting the time-lapse images can also include selecting a time-delay value. The time-delay value 1818 can be selected from a group including a time interval, a time delay, a total number of cycles, and a combination thereof. Time-lapse image analysis can be useful in many cases. For example, it can be useful for tracking the cell mobility associated with cells such as T cells. By using a time delay, the micro-object can be followed based on factors such as proximity, trajectory, and the change of any optional parameters (such as circularity, position, brightness, etc.) for characterizing the micro-object. In addition, an image sequence file can be maintained to capture the time-lapse image, which is configured to include timestamps, exposure times / sequences, illumination percentages, and other variables necessary for understanding the changes over time.
[0290] As depicted herein, step 1320 details exemplary workflow steps that may be implemented by the image preprocessing engine 1408 of the system 1400. In step 1320, the method includes preprocessing the image data to reduce anomalies, such as noise and / or image distortion, in the first image and each second image. Although not required, the preprocessing engine 1408 may be part of the image acquisition unit of the system 1400.
[0291] As depicted herein, step 1330 details exemplary workflow steps that may be implemented by the image alignment engine 1409 of the system 1400. In step 1330, the method includes transforming each second image to optically align the second image with the first image. Although not required, the image alignment element 1409 may be part of the image acquisition unit of the system 1400.
[0292] In various embodiments, each image (of the first image and one or more second images) can be associated with a specific parameter, such as a filter (or fluorescence) cube. In addition, each cube can have its own focal plane (or offset) along the z axis. If multiple images along the z axis are generated and analyzed, the derivative of z (d / dz) relative to the z stack of the image can be used to identify discontinuities, which can generally correspond to the edges of micro-objects (e.g., cells, organelles, etc.). In addition, the analyzed images can be falsely colored and layered or combined to generate a composite image. This may occur after image preprocessing and alignment.
[0293] As depicted herein, step 1340 details exemplary workflow steps that may be implemented by a machine learning algorithm (e.g., neural network 1410 (e.g., CNN)) of system 1400. At step 1340, the method includes processing pixel data in the first image using a neural network to detect micro-objects present in the region of interest. Detecting micro-objects present in the region of interest may include detecting corresponding boundaries of each detected micro-object. Although not required, the machine learning algorithm may be a micro-object detection unit (or micro-object detection and characterization unit—see Fig.13) 1404. In addition, in step 1340 or in a subsequent step after step 1350 discussed below, detecting the micro-objects present in the region of interest may include increasing or decreasing the detected boundary of at least one detected micro-object or each detected micro-object. The boundary can be increased or decreased by, for example, a fixed value (e.g., + / -0.5 micron, + / -1.0 micron, + / -1.5 micron, + / -2.0 micron, etc.). Alternatively, the boundary can be increased or decreased by, for example, a relative value (e.g., 10%, 20%, 30%, 40%, 50% or more of the diameter of the micro-object). Increasing the boundary can be used to ensure that all "positive" signals are captured in each image. Increasing the boundary can be used to eliminate all signals associated with the micro-object from the image (e.g., when calculating the background). Reducing the boundary can help ensure that the signal from the adjacent micro-object is not associated with the micro-object of interest.
[0294] As depicted herein, step 1350 details exemplary workflow steps that may be implemented by the detection engine 1414 of the system 1400. At step 1350, the method includes detecting a signal within a corresponding boundary of each detected micro-object in each of the one or more second images. The signal may be, for example, a fluorescent signal. Detecting the signal may include quantifying the amount of the signal. Although not required, the detection engine 1414 may be part of the detection and characterization unit 1404 of the system 1400.
[0295] As described herein, step 1360 details an exemplary workflow step that can be implemented by the data processing engine (not shown) of system 1400. In step 1360, the method optionally includes providing a visual display corresponding to the distribution of one or more characteristics of the detected micro-object. The visual display can be multidimensional (e.g., two-dimensional or three-dimensional). The visual display can identify a subset of micro-objects with the same characteristics. Alternatively or additionally, the visual display can allow user input to select a specific subgroup of micro-objects. Although not required, the data processing engine can be a part of the data processing unit (not shown) of system 1400.
[0296] Fig.12The method can be performed in a variety of different ways. The first image can be a bright field image. Each of the one or more second images can be a fluorescent image, a cold light image, an infrared image, an ultraviolet image, etc. Alternatively, the first image and each second image can be a fluorescent image, a cold light image, an infrared image, an ultraviolet image, etc. Each image can, for example, capture a signal emitted in a unique part of the electromagnetic spectrum. The unique parts of the electromagnetic spectrum can be overlapping or non-overlapping. The micro-object can be a cell. The method can also include receiving a plurality of second images (e.g., two, three, four, etc.), each of which can correspond to a unique characteristic (e.g., the expression of a specific biomolecule such as a cell surface protein). Preprocessing can include calculating a distortion correction for each received image. Distortion correction of an image can include, for example, using a lookup table, which is calculated by checking an array of points with known spacing between points. The transformation step can include, for example, scaling, shifting, rotating, or a combination thereof, of the second image (e.g., a fluorescent image) to align with the first image (e.g., a bright field image). Scaling can be linear or higher order as desired. Shifting can occur along the X-axis or the Y-axis. The neural network can be a convolutional neural network (discussed in detail above). Micro-object characteristics can include, for example, diameter, area, volume, circularity, brightness, brightness to background ratio, position, distance to the nearest neighbor micro-object, etc. of the micro-object (e.g., cell).
[0297] The grouping of micro-objects into subgroups based on fluorescence, luminescence and / or other characteristics (e.g., size, shape, distance to nearest neighbor, etc.) can be performed in a manner similar to fluorescence activated cell sorting (FACS), wherein the fluorescence signal is quantified and the forward scatter is optionally plotted in a two-dimensional graph, thereby allowing identification and selection of different subgroups. Typically, when multiple characteristics are involved, the graphical representation spans N dimensions, where N represents the number of characteristics involved (see above), whether multiple different fluorescence signals, or alternatively or additionally various characteristics such as those example characteristics provided above. In some cases, a two-dimensional graph can be used to represent N>2 dimensions, such as by using the presence or absence of shapes, colors, and symbols to represent additional information. For example, different shapes and colors can represent different expression levels of corresponding markers. The presence of a symbol can represent the presence or absence of a marker (at or below the background) at a threshold level (or greater) selected by the user. Alternatively, the presence of a symbol can represent the separation of the corresponding micro-object from its nearest neighbor's micro-object by a minimum distance selected by the user.
[0298] By identifying micro-objects (e.g., cells) and grouping these micro-objects into subgroups based on common characteristics, specific micro-objects can be selected and disposed accordingly. For example, for cells as described above, when grouped by characteristics, cells can be appropriately selected to move to corresponding docks (e.g., using dielectrophoretic forces) for further analysis. Such analysis can be performed on the system 150 itself, or can be exported offline using any available file format such as, for example, the flow cytometry standard (FCS) format for analysis on an application (such as, for example, FlowJo).
[0299] Fig.11 1 is a schematic diagram of a system for automatically detecting micro-objects in an image according to various embodiments. As depicted herein, system 1200 may include an image acquisition unit 1202, an image preprocessing engine 1208, a micro-object detection unit 1204, and an input / output device (I / O device) 1216 for outputting images and / or final counts of detected micro-objects. Fig.13 is another schematic diagram of a system for automatically detecting and characterizing micro-objects in an image, according to various embodiments. Fig.13 The system 1400 includes an image alignment engine 1409, but does not include Fig.11 The threshold engine provided in (threshold engine 1212 discussed below). The following discussion about system 1200 also applies to similar features (e.g., units, neural networks, imaging elements, and engines) in system 1400.
[0300] The I / O devices 1216 may be configured to include, for example, the associated display device 1012 and / or input device 1014 of the system 1000 (see Figure 5 ), which may be in the form of data (e.g., parameters, user requirements, etc.), transmitted to, for example, the image acquisition unit 1202, the image preprocessing engine 1208, the micro-object detection unit 1204, or a combination thereof. The I / O device 1216 may also be configured to communicate with the system 1000 via an associated display device 1012 and / or an input device 1014 (see Figure 5 ) receives user input, which may be in the form of data (e.g., parameters, user requirements, etc.) and transmitted to, for example, the image acquisition unit 1202, the image preprocessing engine 1208, the micro-object detection unit 1204, or a combination thereof. Alternatively or in combination, the input device 1014 (see Figure 5) can also be used to pass user input, parameters, etc. directly to, for example, the image acquisition unit 1202, the image preprocessing engine 1208, the micro-object detection unit 1204, or a combination thereof. In addition, the I / O device 1216 can be configured to display data or images received from, for example, the detection engine 1214 on the embedded display device 1012. The device 1216 can also be configured to transmit the data or images to the separate display 1012 for data or image display.
[0301] Image acquisition unit 1202 (for example but not limited to the above Figure 1A The imaging module 164 (as depicted in FIG. 1 ) may include an imaging element 1206 (such as, but not limited to, the imaging device 194 ). Alternatively, the unit 1202 may also be configured to include (or house) an image pre-processing engine 1208 .
[0302] The imaging element 1206 may be configured to capture one or more images (or image data). The images may be, for example, multiple chambers (e.g., an isolation dock) and / or surrounding structures (e.g., a channel) of a microfluidic device. The microfluidic device may include any of the various examples described herein (such as, but not limited to, Figure 1A to Figure 1C , Figure 2A to Figure 2B , Figure 2D and Figure 2G to Figure 2H 100, 200, 230, 250, 280, and 290 shown in the drawings). A microfluidic device may include a flow region, and a chamber or multiple chambers fluidically connected to the flow region, wherein each of the chambers may hold one or more micro-objects. As previously described, the chambers may be, for example, an isolation dock as required by the particular application in which they are used. It should be understood that the chambers may have any shape, size, or orientation. The flow region may be a single microfluidic channel, or a plurality of microfluidic flow channels (for example, but not limited to, as described above). Figure 1A and Figure 2A to Figure 2C Channel 122 shown, and above Figure 2D to Figure 2F 264) that provide a single flow path or multiple flow paths (such as but not limited to Figure 1A and Figure 2B 106). The flow region may be in fluid communication with a single chamber or multiple chambers. Alternatively, the flow region may be in fluid communication with a single chamber or multiple chambers via a reversible closure (e.g., a valve). The flow region may be configured to receive a material flow through an inlet as described above. The fluid of the material may include, for example, a fluid of a micro-object, an adhesive, or a reagent, or a fluid of a medium including a medium.
[0303] In various embodiments, the imaging component 1206 may also be configured to resize the captured image before sending it forward for further processing. For example, resizing may be accomplished by binning (e.g., four pixels to one).
[0304] In various embodiments, imaging element 1206 may also be configured to receive a first image and one or more second images of the same region of interest of a microfluidic device. Fig.13 The imaging element 1406 performs.
[0305] The image preprocessing engine 1208 may be configured to prepare the image for further analysis according to various embodiments. For example, if the captured image is merged before being received by the engine 1208, the engine 1208 may resize the image to full size to compensate for the merge. The engine 1208 may use, for example, linear interpolation between pixel values to resize. The engine 1208 may flip and / or rotate the image to a desired orientation as needed. The engine 1208 may apply a distortion correction step to the image using, for example, a lookup table calculated by examining an array of points with known spacing between points. The engine 1208 may be provided as part of the image acquisition unit 1202 (e.g., Fig.13 ) or may be an independent engine (such as the image preprocessing engine 1408 of the image acquisition unit 1402 in FIG. Fig.11 shown).
[0306] It should be noted that Fig.13 The system 1400 also includes an image alignment engine 1409 configured to transform each second image to optically align the second image with the first image. Alternatively, this function may be performed by the image preprocessing engine 1408, which has the following description Fig.11 The image pre-processing engine 1208 describes the characteristics.
[0307] In various embodiments, the engine 1208 can perform a graded brightness process across the image. For example, the engine 1208 can use a polynomial best fit correction, such as a quadratic or higher order polynomial best fit correction. Alternatively, a sine wave or exponential function can be used instead of a polynomial function. Leveling can be achieved by multiplying the image brightness by the scaled image, where the desired multiplier for the best fit function is determined during system calibration. The engine 1208 can also perform a radiometric correction to subtract background brightness from, for example, autofluorescence.
[0308] In various embodiments, sometimes a fluorescent image is needed to visualize cells that might otherwise appear semi-transparent (e.g., DAPI can be used to stain cell nuclei as a means to better detect / count certain cells.) In those cases, engine 1208 can scale, shift, and / or rotate the fluorescent image to align with the brightfield image, with the calibration being accomplished using a dot array.
[0309] In various embodiments, a Fourier transform can be used to reduce interference from a conductive silicon substrate on a microfluidic device. The Fourier transform allows for a frequency representation of an image, which helps identify artifacts and interference associated with a conductive silicon substrate, such as a phototransistor array. The Fourier transform of a time function is itself a complex-valued function of frequency, whose absolute value represents the amount of frequency present in the original function, and whose complex parameter is the phase shift of the fundamental sinusoid in that frequency. The Fourier transform is referred to as a frequency domain representation of the original signal. The term "Fourier transform" refers to both the frequency domain representation and the mathematical operation that relates the frequency domain representation to the time function. The Fourier transform is not limited to functions of time, but in order to have a unified language, the domain of the original function is often referred to as the time domain.
[0310] As will be discussed in more detail below, the micro-object of interest may have a similar hybrid morphology compared to the features of a microfluidic device (e.g., a phototransistor array). In addition, micro-objects such as cells may be relatively translucent compared to the various features of a microfluidic device. Therefore, it may be helpful to identify and remove unwanted features of a microfluidic device (e.g., a phototransistor array, a wall, or a pipeline element of a microfluidic device) before identifying the micro-object of interest. Fourier analysis may be used to remove, for example, transistor patterns before micro-object detection. This step may occur within engine 1208, or alternatively, in a post-processing step in the detection engine 1214 of the micro-object detection unit 1204 (described in more detail below).
[0311] In various embodiments, pre-processing the image can include using brightness normalization or contrast enhancement to reduce interference from the conductive silicon substrate on the microfluidic device.
[0312] In various embodiments, the engine 1208 may make a copy of the image pre-processed as described above and pass it to various "clients" 1220 (e.g., GUI, image processing, movie creation, image capture, memory / storage / server, etc.).
[0313] In various embodiments, a watershed algorithm can be used to fill in the cell boundaries on the original image input. The algorithm treats the image as a topographic map, with objects of interest as catchment basins and the edges of these objects as watershed lines around the basins. In this way, this image analysis method can more clearly define object boundaries (watershed lines) around objects (catchment basins).
[0314] Fig.11A micro-object detection unit 1204 of system 1200 may be communicatively connected to image acquisition unit 1202. In various embodiments, micro-object detection unit 1204 may include a neural network 1210, a threshold engine 1212, and a detection engine 1214. It should be appreciated that each component (e.g., engine, module, etc.) depicted as part of micro-object detection unit 1204 (described herein) may be implemented as hardware, firmware, software, or any combination thereof.
[0315] In various embodiments, micro-object detection unit 1204 may be implemented as an integrated instrument system component with image acquisition unit 1202. That is, micro-object detection unit 1204 and image acquisition unit 1202 may be housed in the same housing assembly and communicate via conventional device / component connection components (e.g., serial bus, optical cables, electrical cables, etc.).
[0316] In various embodiments, micro-object detection unit 1204 may be implemented as a separate computing device (e.g., a computer) communicatively connected to image acquisition unit 1202 via an optical, serial port, network, or modem connection. Fig.11 For example, the image processing unit may be connected via a LAN or WAN connection that allows the imaging data acquired by the image acquisition unit 1202 to be transferred to the micro-object detection unit 1204 for analysis.
[0317] In various embodiments, the functionality of micro-object detection unit 1204 may be implemented on a distributed network of shared computer processing resources (such as a cloud computing network) that is communicatively connected to image acquisition unit 1202 via a WAN (or equivalent) connection. For example, the functionality of micro-object detection unit 1204 may be partitioned into a distributed network such as AMAZON WEBSERVICES TM The process is implemented in one or more computing nodes on a cloud processing service.
[0318] The neural network 1210 may be designed and configured to receive image data input from the image preprocessing engine 1208, annotate pixel data in the image data according to a plurality of micro-object characteristics, and assign a probability value to each pixel in the pixel data based on the pixel annotation and outputting probability image data. The neural network 1210 may be a convolutional neural network and may have an architecture utilizing any combination of the above-described architecture examples (e.g., see Figure 7 , Figure 8 and FIG. 9A to FIG. 9D 1400 ). The neural network 1210 may be designed and configured to process pixel data in the first image to identify corresponding boundaries of micro-objects present in the region of interest, as done by the micro-object detection and characterization unit 1404 of the system 1400. As described above, the first image may be a bright field image.
[0319] Threshold engine 1212 can be designed and configured to receive output probability image data from neural network 1210 and apply given threshold value to determine which pixel probabilities at least meet the threshold value of definition.For example, in various embodiments, micro-object type can be one of cell center, cell boundary or non-cell type, and comprises micro-object characteristic, such as circular feature, size feature or both.The probability assigned to pixel annotation can be compared with specified threshold value, so as to further analyze or eliminate the pixel below threshold value.Threshold value can be user-defined, and if the probability of pixel annotation is lower than threshold value, pixel annotation can be reclassified into another type.The probability assigned usually indicates the confidence of pixel annotation.For example, probability can be assigned as follows: 0.15 for boundary, 0.8 for cell center, and 0.05 for non-cell.When analyzing pixel cluster, each pixel annotation can be used together with adjacent pixels to determine the possibility of correct identification.
[0320] Fig.11 The detection engine 1214 of the system 1200 can be designed and configured to receive the image output data, perform corrections for the threshold analysis in the threshold engine 1212, apply image post-processing techniques, and output micro-object counts. Fig.13 As provided by the detection engine 1414 of the system 1400, the detection engine 1214 can also be designed and configured to quantify the amount of signal (e.g., fluorescence, chemiluminescence, etc.) in the second image located within the corresponding boundaries of the detected micro-objects. Quantification of the signal can facilitate and improve the grouping of micro-objects into subgroups with the same characteristics.
[0321] Some examples of which are provided below, many post-processing techniques are contemplated. Engine 1214 can be configured to align a CAD model of an isolation dock (in a microfluidic device) to the actual image output to precisely find where the dock is located. In the case of a fluorescent image (depending on the cell type being detected), engine 1214 can be configured to remove background by subtraction, for example, by subtracting a corresponding image obtained from a blur (image) routine. Engine 1214 can also be configured to chop the image output into individual docks for micro-object counting. Engine 1214 can also apply a pixel mask that can remove any structure around the object of interest (e.g., a microfluidic device or dock wall). Finally, engine 1214 can determine micro-object counts based on objects that are identifiable after thresholding and post-processing. The counts and output images from engine 1214 can be transmitted to I / O device 1216, where it can, for example, be stored, transferred to memory storage, further analyzed and / or transmitted to client 1220 (see Fig.11 ).
[0322] According to various embodiments, image acquisition unit 1202 and micro-object detection unit 1204 may be integrated into a single physical unit. Alternatively, image acquisition unit 1202 and micro-object detection unit 1204 may be oriented detachably and provided as separate units, such that the units are still communicatively connected and able to exchange information.
[0323] Each component of the micro-object detection unit 1204 described above may be hardware or may be partially or entirely a software module.
[0324] According to various embodiments, a computing device for characterizing and selecting micro-objects in a microfluidic device is provided, wherein the computing device includes a display screen. The computing device can be configured to display a menu for selecting a first parameter (e.g., selected from a provided parameter list) on a screen for characterizing a group of micro-objects detected. The parameters that can be included in the parameter list are discussed below. The computing device can also be configured to display a graph of a group of micro-objects detected on a screen based on the selected first parameter.
[0325] According to various embodiments, the provided list of parameters may be a finite list of parameters provided within a menu, each parameter in the list being selectable to characterize a detected set of micro-objects based on the associated parameter.
[0326] According to various embodiments, the display screen can select a subgroup of the detected set of micro-objects based on at least one selected threshold of the selected first parameter, and can display the detected set of micro-objects by visually distinguishing the subgroup that meets the at least one selected threshold from the remaining micro-objects in the detected set of micro-objects.
[0327] As discussed in further detail below, when observing a population of objects, creating a set of filters may specifically enable separation of the population of objects into n different populations.
[0328] As referenced above Figure 5 As described above, the computing system (or device) 1000 may be provided to include an I / O device 1216. The I / O device 1216 may be configured to receive user input via an associated display device 1012 and / or input device 1014, which may be in the form of data (e.g., parameters, user requirements, etc.), which may be transmitted to the various units and engines discussed previously (see, e.g., Fig.12 Alternatively or in combination, the input device 1014 of the computer system 1000 may also be used to pass user input, parameters, etc. directly to the various units and engines discussed previously (see, e.g. Fig.12). In addition, the I / O device 1216 can be configured to display data or images received from, for example, the detection engine 1214 or other various units or engines discussed herein on the embedded display device 1012. The device 1216 can also be configured to transmit the data or images to a separate display 1012 for data or image display.
[0329] Reference now Fig.14 and 15 , as part of a computing device (e.g., computing system 1000), a display screen 1500 for characterizing and selecting micro-objects in a microfluidic device is provided. Display screen 1500 may be a graphical user interface (GUI). Screen 1500 displays a menu 1502 configured to select a first parameter 1503 selected from a provided parameter list for characterizing a set of detected micro-objects. The first parameter may be, for example, a maximum brightness (or "Delta Max brightness", which is a measure of brightness from which background brightness has been subtracted), such as Fig.14 and 15 As an example only, Fig.14 and 15 The display shows the observation of a set of objects with a known maximum brightness distribution across a fluorescent cube (in this example, configured to detect DAPI fluorophore) of a microfluidic device (e.g., chip). In addition, in this example, a threshold of 1506 was selected to further drill down into the micro-objects of interest. In this case, the brightness of the selected objects is above the maximum brightness difference of ~16000 (i.e., the maximum background brightness).
[0330] The provided parameter list may be a finite list of parameters provided, for example, within a menu, each parameter in the list being selectable to characterize a set of detected micro-objects based on the associated parameter. The provided parameter list may provide parameters selected from the group consisting of: Circularity, Centroid X Pixels, Centroid Y Pixels, Centroid X Microns, Centroid Y Microns, Centroid X Microns Pen Relative, Centroid Y Microns Pen Relative, Nearest Neighbor Microns, Diameter Microns, Volume Femtoliters, Background Area Microns, and Background Area Microns. ), average brightness, minimum brightness, maximum brightness, median brightness, background median brightness, median brightness difference (DeltaMedianBrightness), maximum brightness difference (DeltaMaxBrightness), logarithm of mean brightness (LogMeanBrightness), logarithm of maximum brightness (LogMaxBrightness), logarithm of median brightness (LogMedianBrightness), logarithm of maximum brightness difference (LogDeltaMaxBrightness), logarithm of median brightness difference CV (LogDeltaMedianBrightnessCV), background CV (BackgroundCV), logarithm of maximum brightness difference to background ratio (LogDeltaBrightnessMaxToBackgroundRatio), logarithm of sum of brightness differences (LogDeltaBrightnessSum), fluid channel number, field of view, cell count, number of cells per dock.
[0331] Circularity may refer to the circularity of a detected object, which may be quantified, for example, between 0 for a highly non-circular object and 1 for a perfectly circular object, and may be given by the formula 4*pi*area pixels / perimeter pixels 2 to define.
[0332] The X centroid in pixels may refer to the centroid of the object along the x-axis, which may be defined in pixels.
[0333] The Y centroid in pixels may refer to the centroid of the target along the y-axis, which may be defined in pixels. In various embodiments, the y-coordinate on the map may be opposite to the y-coordinate on the associated image, so the target position may be reversed along the y-axis.
[0334] X-centroid in microns may refer to the center of mass of the target along the x-axis, which may be defined in microns.
[0335] Y center of mass in microns may refer to the center of mass of the target along the y-axis, which may be defined in microns.
[0336] The dock-relative amount of the X centroid in microns may refer to the position of the center of a micro-object along the x-axis of the isolation dock in which the micro-object is located, as measured in microns relative to a selected origin within (or at an edge or corner of) the isolation dock.
[0337] The dock-relative amount of the Y centroid in microns may refer to the position of the center of the micro-object along the y-axis of the isolation dock in which the micro-object is located, as measured in microns relative to a selected origin within (or at an edge or corner of) the isolation dock.
[0338] The nearest neighbor in microns may refer to the number of microns to the nearest detected object.
[0339] The diameter in microns may refer to the target diameter of the effective measurement in microns as calculated from the area.
[0340] The volume in femtoliters may refer to an estimate of the volume of a micro-object, which may depend mainly on the shape of the micro-object and its diameter (or major and minor axes if the micro-object has an ellipsoidal shape).
[0341] Background Area in Microns may refer to the area used in background calculations, defined in microns. This may be referred to as the "donut" area around the selected object, excluding dock walls and nearby objects detected. Background Area in Pixels is the same parameter, except defined in pixels instead of microns.
[0342] The average brightness may refer to the average brightness of an area within the detected object boundary.
[0343] The minimum brightness may refer to the minimum brightness of an area within the detected boundary of the object.
[0344] The maximum brightness may refer to the maximum brightness of an area within the detected boundary of the target.
[0345] The median brightness may refer to the median brightness of an area within the detected object boundary.
[0346] The background median brightness may refer to the median brightness of the background area surrounding the detected target.
[0347] The median brightness difference may refer to the difference between the median brightness of the detected target and the median brightness of the surrounding background area.
[0348] The maximum brightness difference may refer to the difference between the maximum brightness of the detected target and the median brightness of the surrounding background area.
[0349] The logarithmic value of the average brightness may refer to a logarithmic (base 10) value of the average brightness of the detected target.
[0350] The logarithmic value of the maximum brightness may refer to a logarithmic (base 10) value of the maximum brightness of the detected target.
[0351] The logarithm value of the median brightness may refer to a logarithm (base 10) value of the median brightness of the detected targets.
[0352] The logarithm value of the maximum brightness difference may refer to a logarithm (based on 10) value of the difference between the maximum brightness of the detected target and the median brightness of the surrounding background area.
[0353] The logarithm value of the median brightness difference refers to the logarithm (based on 10) value of the difference between the median brightness of the detected target and the median brightness of the surrounding background area.
[0354] CV stands for coefficient of variation, which may represent the ratio of the standard deviation of target brightness to the median target brightness.
[0355] Background CV represents the coefficient of variation of the background area, which can be expressed as the ratio of the standard deviation of the background brightness to the median background brightness.
[0356] The logarithm value of the maximum brightness difference to background ratio may refer to the logarithm (base 10) value of the difference between the maximum target and median background brightness divided by the median background brightness.
[0357] The logarithm value of the sum of brightness differences may refer to the logarithm (base 10) value of the difference between the mean target and median background brightness multiplied by the area (which may be in pixels).
[0358] The fluid channel number may refer to the fluid channel number of the channel in which the target is found. This number may be 0 to 4, but may also be a different value, such as -1, if the number is not found in the chip definition file.
[0359] The field of view may refer to the portion of the microfluidic chip that can be observed by the imaging system at a single point in time. This may depend primarily on the position of the imaging system relative to the microfluidic chip and the power of the objective lens used by the imaging system. This parameter may allow characterization of cells on a per field of view basis.
[0360] Cell counting can refer to counting the number of cells detected by an imaging system; cells can be counted on a per-field-of-view basis or across the entire microfluidic chip.
[0361] The number of cells per dock can refer to the count of the number of cells detected in each isolation dock.
[0362] Screen 1500 may also display a graph 1504 that may visually represent a group of detected micro-objects based on the selected first parameter. The display screen may select a subgroup of the group of detected micro-objects based on at least one selected threshold 1506 for the selected first parameter, and may display the group of detected micro-objects by visually distinguishing the subgroup that satisfies the at least one selected threshold from the remaining micro-objects in the group of detected micro-objects.
[0363] The selection can be Fig.14 occurs on the graph 1504 provided in FIG. 1504 , where the threshold 1506 is Fig.15 , which is represented as a vertical line, distinguishes the first subgroup 1508 of micro-objects from the second subgroup 1510 of micro-objects, either of which can be considered as a subgroup that meets the set threshold. Fig.15 , the second subgroup 1510 is a subgroup that satisfies the threshold (233 of 1538 micro-objects). Therefore, the threshold may include an upper threshold. Alternatively, the threshold may include a lower threshold, or both a lower threshold and an upper threshold.
[0364] exist Fig.15 , subgroups 1508 and 1510 are distinguished by color, which may be grayscale or any color from the visual spectrum. Visual distinction may take many other forms, including, for example, the size of the data points on the graph (e.g., the size of the data points for one subgroup is greater than the size of the data points for another subgroup) and the sign of the data points (e.g., "x" for one subgroup and "*" for another subgroup), either of which may be combined with color to increase distinctiveness.
[0365] According to various embodiments, the display screen may enable a slidable selector for threshold selection. The slidable selector may be provided, for example, as Fig.15 Alternatively or additionally, the display screen may enable a point selector for threshold selection. Alternatively or additionally, the display screen may enable user-entered values for threshold selection. Alternatively or additionally, and as Fig.21 As shown in (described in detail below), the display can enable region selection, thereby selecting a region of the image within which are micro-objects that meet a threshold. Such region selection features can be in the form of circles, squares, and any other conceivable shapes required to define a region of interest.
[0366] According to various embodiments, and as Fig.16 and 17 As shown in , the menu 1602 displayed on the screen 1600 may also be configured to select a second parameter 1604 selected from a provided parameter list for characterizing a set of detected micro-objects also characterized by the first parameter 1606. Fig.16 and 17 In the case of the example shown in , the second parameter is the logarithm of the maximum brightness under the fluorescence cube (in the example, configured to detect the CY5 fluorophore) (or the "logarithm of the maximum brightness difference", which is a brightness measure where the background brightness has been subtracted and the resulting value is converted to a logarithmic value, which can be based on 10, e, or any other suitable logarithmic base). The second parameter can be added as a filter under the first parameter (in this case, the maximum brightness difference under the cube configured to detect the DAPI fluorophore) to analyze micro-objects under both parameters.
[0367] According to various embodiments, and as Fig.16 As shown, the menu 1602 displayed on the screen 1600 can be associated with a graph that displays a characterization of micro-objects based on the selected first parameter 1606 and second parameter 1604. For example, the display screen 1600 can be further enabled to display a graph of a subgroup 1608 of detected micro-objects that meet at least one threshold of the first parameter 1606 and are characterized by the second parameter 1604.
[0368] According to various embodiments, and as Fig.17 As shown, the display screen may also select a subset 1610 of the detected subpopulation 1608 of micro-objects based on at least one selected threshold 1612 for the selected second parameter 1604 . Fig.17 Such at least one threshold is shown as having both an upper threshold and a lower threshold, but as discussed herein, either threshold may be used alone.
[0369] According to various embodiments, the computing device is further configured to accept user instructions and / or display instructions on a screen for repositioning a group of detected micro-objects, a subgroup of a group of detected micro-objects, or one of the subsets of the subgroups. For example, the repositioning can be performed as described elsewhere herein.
[0370] According to various embodiments, and as Fig.18 and 19As shown, the computing device 1000 may also be configured to display an imaging menu 1802 on the screen 1800 for selecting an imaging parameter 1804 selected from the provided imaging parameter list for imaging at least a portion of the microfluidic device. The computing device may also be configured to display an imaging menu on the screen for selecting a plurality of imaging parameters 1804 / 1806 / 1808 selected from the provided imaging parameter list for imaging at least a portion of the microfluidic device. Fig.18 and 19 Three different imaging parameter selections are shown, but the number of imaging parameters is accordingly not limited. In various embodiments, the number of imaging parameters may range from 1 to 5, 1 to 10, etc.
[0371] According to various embodiments, the imaging parameters may include a filter or "cube" type. The cube type is selected based on the portion of the electromagnetic spectrum that passes through the cube and is detected. The cube type may include bright field (e.g., unfiltered, sampling the entire visible spectrum; Fig.18 The imaging parameters may also include at least one sub-parameter 1814. The at least one sub-parameter 1814 or multiple sub-parameters may be selected from the group consisting of: illumination percentage, exposure time milliseconds, user offset micrometers (across the z-axis), and combinations thereof.
[0372] According to various embodiments, the computing device may also be configured to display an algorithm selector 1810 on the screen for selecting an algorithm from the provided list for analyzing the image obtained by each selected imaging parameter (e.g., Fig.18 and 19 1804 / 1806 / 1808) acquired by the imaging parameters, and detecting a set of micro-objects. Thus, each imaging parameter may be provided with a field for selecting the algorithm to be applied for that particular parameter. In various embodiments, a "master" algorithm may be used on a set of images in one of the imaging parameters (e.g., a brightfield cube), and thus the positions of micro-objects detected in the master set of images may be used to analyze a set of images of other selected imaging parameters (e.g., a fluorescent cube).
[0373] According to various embodiments, and as Fig.19As shown, the displayed imaging menu is also configured to provide a time delay selector 1816. The time delay selector enables the selection of a time delay value for imaging at least a portion of the microfluidic device within a selected time period. The time delay selector 1816 can also include a selector for a time delay value 1818. The time delay value 1818 can be selected from a group including a time interval, a time delay, a total number of cycles, and a combination thereof. In many cases, time delay image analysis can be useful. For example, it is useful for tracking the cell mobility associated with cells such as, for example, T cells. Using time delay, micro-objects can be followed based on changes such as proximity, trajectory, and any selectable parameters (such as circularity, position, brightness, etc.) for characterizing micro-objects. In addition, an image sequence file can be maintained to capture time delays, which can include timestamps, exposure times / sequences, illumination percentages, and other variables required for understanding changes over time.
[0374] Reference now Fig. 20 , a display 2000 is shown. As illustrated, the computing device can be configured to display at least one image 2020 of each individual micro-object detected on screen 2010. Micro-objects in a microfluidic device (or cells in a microfluidic chip) can be identified in the image using colored symbols (overlaid on the micro-object or cell, or surrounding micro-objects or cells) or using a false color display of a selected micro-object or cell. In various embodiments, the number of images displayed for each detected micro-object is equal to the number of selected imaging parameters. For example, referring to Fig.17 and 18 , select three imaging parameters (i.e., OEP, DAPI, and CY5). Fig. 20 The display provided in shows that for each micro-object that advances vertically on screen 2010, three images are displayed, corresponding to OEP, DAPI and CY5 cubes respectively. Therefore, each image can be associated with a specific parameter (in this case, the filter cube). In addition, each cube can have its own focal plane along the z axis, or a series of images along the z axis can be acquired. If multiple images are acquired along the z axis, the derivative of z (d / dz) relative to the z stack of the image can be used to identify discontinuities, which can generally correspond to the edges of micro-objects (e.g., cells, organelles, etc.). In addition, the analyzed images can be falsely colored and layered or combined to generate a composite image. This can occur after image preprocessing and alignment.
[0375] Reference now Fig.21, a display 2100 is shown. As illustrated, the computing device can be configured to display on screen 2110 a graph of micro-objects (e.g., cells) stained with two different fluorescent markers, one of which is represented on the x-axis and the other on the y-axis. The figure also illustrates three different groups / types of cells, each of which can be selected generally by region selection or by the thresholding step discussed above. In either case, a region in the figure can be selected within which the micro-objects meet the threshold or are of interest. This region selection feature 2120 can be in the form of a circle, a square, and any other conceivable shape required to define a region of interest. In Fig.21 In this study, cells with high levels of labeling using PE2 have been selected.
[0376] Automatic detection of micro-objects. A method for automatically detecting micro-objects of interest in an image is provided. The micro-object of interest may have a similar, mixed morphology compared to one or more other features in the image. For example, in some cases, the detection of micro-objects arranged in a microfluidic device may be complicated by features of the microfluidic device having a morphology similar to that of the micro-object of interest. For example, in a case where a cell has a diameter of 10 microns, it may be difficult to distinguish the cell from a phototransistor array having a 10 micron pitch in two dimensions (i.e., each phototransistor has a size of 10 microns × 10 microns). In addition, micro-objects such as cells may be relatively translucent compared to various features of the microfluidic device. Therefore, it may be helpful to identify and remove unwanted features of a microfluidic device (e.g., a phototransistor array, a wall, or a pipeline element of a microfluidic device) before identifying the micro-object of interest.
[0377] In some embodiments, a single pixel may correspond to an area substantially smaller than the cross-sectional area of the micro-object of interest in the microfluidic device. For example, the micro-object may have a cross-sectional area of about 80 square microns, and a pixel may correspond to an area of about 2 square microns. In such an embodiment, one or more pixel clusters will be needed to cover the cross-sectional area of the micro-object (e.g., in the aforementioned example, 40 pixels will be taken up to cover the cross-sectional area of the micro-object, or 24 pixels will be taken up to cover the cross-sectional area of the circumference of the micro-object).
[0378] The analysis of pixel clusters may also include multiple other features in addition to the area and circumference of the pixel clusters. Pixel clusters may be analyzed based on global morphology (i.e., the size and shape of a collection of one or more pixel clusters), local morphology (i.e., the size and shape of individual pixel clusters), positive and negative light intensity values Li, and other features based on a combination of these elements (e.g., light intensity as a function of size). Various methods may be used to analyze a group of pixel clusters including traditional machine learning techniques, in which the above features are calculated for a collection of images of micro-objects and used to train a classifier to identify micro-objects of interest in new images based on the same features.
[0379] Micro-object identification (discussed in more detail below) can also be used in combination with using force, such as OET or DEP force to manipulate or reposition micro-object.In some embodiments, the micro-object identified in the position of a specific pipeline element (such as passage or isolation dock) or microfluidic pipeline can be moved to (that is, repositioned in) the position of another type of pipeline element or microfluidic pipeline.For example, micro-object can be identified in the channel in the microfluidic pipeline and repositioned in the isolation dock in the microfluidic pipeline (referred to as "fence" to micro-object in this article).On the contrary, the micro-object identified in the isolation dock in the microfluidic pipeline can be moved to the channel in the microfluidic pipeline.Alternately, one or more micro-objects can be identified in an isolation dock and repositioned in an empty isolation dock (referred to as "re-fence" to micro-object in this article).According to the embodiment, various mechanisms including OET and DEP force can be used to move micro-object.Similarly, micro-object can be repositioned sequentially (that is, one micro-object at a time), in parallel or any combination thereof (for example, sequentially repositioning the group of multiple cells in parallel).
[0380] In the case where the micro-object is relocated from the passage to a separate isolation dock (or relocated to another isolation dock from a separate isolation dock), different algorithms can be used to distribute the micro-object to an empty isolation dock. In some embodiments, the algorithm will be used to distribute the micro-object to an empty isolation dock so that the distance between the micro-object and the dock (that is, the track or path that the micro-object must travel during relocation) is minimized. In these embodiments, the use of the force (e.g., OET or DEP force) for moving the micro-object is also minimized because the micro-object only needs to travel the minimum distance to be relocated in the empty isolation dock.
[0381] In these embodiments, the local micro-object density in the channel (that is, the number of micro-objects in the specific spatial region of the channel) can be used to determine the appropriate algorithm for assigning the specific micro-objects in the channel to the empty isolation dock. The local micro-object density can be calculated in a variety of ways. In some embodiments, the local micro-object density can be calculated based on a fixed size region (e.g., 200 square microns, or a region of 100 microns of channel length, and extending the width of the channel) or using a method with various size regions. In other embodiments, the local micro-object density can be calculated based on the distance between the cluster of identified micro-objects or the identified micro-objects. The local micro-object density can also be calculated by subdividing the channel into a grid or using a "sliding window" technique to calculate the density of the overlapping region of the channel.
[0382] If the local micro-object density is higher than a threshold T1 density, the micro-objects may be assigned to the nearest empty isolation dock so that the distance between the micro-objects and the isolation dock is minimized. If the local micro-object density is lower than a specific threshold T1 density, the empty isolation dock may be assigned to the micro-object closest to the empty isolation dock so that the distance between the micro-objects and the isolation dock is minimized. In some cases, the local T1 density may be calculated based on the number of empty docks and the density of micro-objects within a channel within a predetermined neighborhood region.
[0383] Different methods for calculating the distance between micro-object and empty isolation dock (that is, the track that micro-object or path needs to move during fence) can be used to assign specific micro-object to empty isolation dock.In some embodiments, the distance between micro-object and potential isolation dock can be calculated based on the optimal track using OET and / or DEP force only.In some cases, the optimal track using OET or DEP force involves a combination of orthogonal motion paths (for example, only along the combination of different movements of y-axis and x-axis) to move micro-object.In other cases, the distance can be based on the shortest possible path between micro-object and isolation dock without constraint (that is, micro-object can travel along any path to arrive isolation dock).In most embodiments, using the same track determined by the algorithm for calculating distance (track), micro-object will be relocated (that is, by "fence" or "re-fence").
[0384] Similarly, when a large amount of micro-objects are assigned to the isolation dock (or vice versa), different algorithms can be used to calculate the optimal allocation of micro-objects to the dock (or vice versa). These algorithms can use different calculation methods that determine that the micro-objects are assigned to the isolation dock, and the method minimizes the total distance (that is, the length of track) that needs to move the micro-objects, so that the micro-objects are relocated to the isolation dock. For example, the algorithm can use the sum of the lengths of all tracks as a heuristic to minimize the distance that the micro-objects need to travel. In some embodiments, constraints such as the maximum distance that the micro-objects move during relocation can be introduced into the calculation of the optimal allocation. Various combined algorithms can be used to calculate the optimal allocation between micro-objects and the isolation dock. Suitable algorithms include: for example greedy algorithms, nonlinear optimization, heuristic-based algorithms and constrained search. Other similar algorithms are known in the art.
[0385] Once the optimal allocation and trajectory are calculated for the micro-object, forces such as OET and / or DEP can be used to move the micro-object to its assigned dock. The micro-object can be repositioned using a pattern of light (e.g., a "light cage") that surrounds the micro-object and subjects the micro-object to OET and / or DEP forces or by using a rod or similar structure to apply OET and / or DEP forces to the micro-object. Typically, a light cage is a structure that substantially surrounds the micro-object (e.g., a square, circular, or polygonal shape). However, in some cases, the light cage may include a rift or opening so that the micro-object is not completely enclosed.
[0386] As mentioned above, in most embodiments, micro-object will be moved according to the distance (track) of the best distribution for calculating micro-object to dock.According to embodiment, micro-object can move sequentially or in parallel (for example, sequentially move cell group in parallel).In the embodiment that micro-object is moved in parallel, the algorithm for calculating optimal distribution or track can compare tracks and ensure that micro-object does not conflict when they are moved in parallel by modifying the track of micro-object to dock and distribution.In specific embodiments, algorithm can distribute " exchange " to dock by micro-object when identifying potential collision.In this embodiment, when the optimal track of the first micro-object intersects with the optimal track of the second micro-object, the optimal track of the first micro-object is assigned to the second micro-object, and the optimal track of the second micro-object is assigned to the first micro-object.In another specific embodiment, the algorithm delays the relocation of the first micro-object, until the time when the first micro-object and the second micro-object can move along its corresponding track when not conflicting.
[0387] In some cases, the density of micro-objects may be so high that the micro-objects need to be separated from each other and repositioned (i.e., "fenced" or "re-fenced") before the micro-objects are assigned to the isolation dock. For example, the density of micro-objects may be so high that the micro-objects cannot be fenced using OET and / or DEP forces, because the light cage used to reposition the object using OET and / or DEP forces cannot be used on a single micro-object without interfering with other micro-objects. In the case where it is important to minimize the amount of OET and / or DEP forces applied to the micro-objects, this interference is particularly worrying. For example, the micro-objects may be affected by the byproducts of OET and / or DEP forces or OET forces (e.g., electrolysis associated with OET and / or DEP forces). In these cases, the information generated during micro-object identification (e.g., radius, center of mass, perimeter and position of micro-objects) can be used to move the micro-objects so that the micro-objects can be fenced or re-fenced without interfering with other cells (referred to as "separated" micro-objects in this article).
[0388] In order to identify instances where micro-objects need to be separated before fencing, the local micro-object density can be calculated based on the defined spatial region and compared with the second threshold T2 density Density can be compared. Alternatively, the distance between micro-objects (e.g., the distance between the center of mass of the micro-object, the distance between the perimeter of the micro-object) can be calculated and used to determine whether to separate the micro-objects. However, as can be appreciated, in some cases, the distance between the micro-objects may be too small to identify the micro-objects as separate micro-objects and micro-objects. In these cases, the micro-objects can be re-identified after repositioning (i.e., "fences") the micro-objects to ensure that each isolation dock contains a single micro-object.
[0389] In some embodiments, light box is used for separating micro-objects before or during fence (or re-enclosure).When forming light box (or light cage), a division algorithm can be used to calculate a group of vertices, and this group of vertices isolates each identified micro-object (for example, the part of passage or isolation dock) in the spatial region of microfluidic device from other micro-objects in the same spatial region.But, as will be appreciated by those skilled in the art, this group of vertices can be drawn so that only the subset of the micro-object in the spatial region of microfluidic device is separated from other micro-objects.For example, this group of vertices can only separate the subset of the microfluid in the spatial region, because they are very close to other microfluids and need to reposition.
[0390] In a particular embodiment, the centroid of each micro-object is used to calculate the triangulation of the point set. The triangulation of the point set produces a set of triangles connecting the centroids of the micro-objects. The Voronoi diagram is then calculated based on the torus of the triangles calculated using the triangulation of the point set. The Voronoi diagram is a set of vertices that divides a spatial region into a set of sub-regions so that the distance between the set of vertices and the centroid of the micro-object is maximized. Other methods of calculating a set of vertices that partition each cell with other cells in the spatial region are known in the art.
[0391] Once the set of vertices has been calculated, the set of vertices can be used in conjunction with OET and / or DEP forces to move the micro-object. Figure 6A to Figure 6F Micro-object separation according to various embodiments of the present invention is shown. Fig. 6A The triangulation of a set of points of a set of micro-objects in a specified spatial region and the corresponding Voronoi diagram are shown. Figure 6B The corresponding Voronoi diagram of the triangulation without the point set is shown. Figure 6C Shown is a light cage commonly used to move micro-objects overlaid on a Voronoi diagram. Fig.6D It is shown that by calculating Figure 6C Improved light cage generated by the intersection between the typical light cage and the Voronoi diagram. Fig. 6E The separation of micro-objects in close proximity to each other using the improved optical cage is shown. Fig. 6F Separated micro-objects are shown.
[0392] In one embodiment, one or more light cages are generated by generating a plurality of light bars linking a subset of vertices of the set of vertices, wherein the subset of vertices includes (or consists of) vertices that are closest to and surround each micro-object to be moved. For example, Figure 6B Any of the polygonal shapes shown in can be used to define a light cage around a micro-object. In some embodiments, the light cage formed in this manner can be collapsed to separate the micro-object within the light cage from other micro-objects and / or light cages in a specified spatial region. In other embodiments, the light cage can be collapsed by forming a polygonal shape (see Figure 6C ) and generate a light cage resulting from the intersection of the standard light cage design and the polygonal shape to define the light cage, such as Fig.6D As shown. In this example, the intersection of the vertices and the light cage is defined as the area where the light cages do not intersect or overlap, thereby allowing the "standard" light cage to be redrawn so that it does not interfere with other micro-objects. Regardless of the formation method, once formed, the light cage can be used to separate micro-objects by repositioning the micro-objects by moving the micro-objects away from each other. In some cases, the modified light cage can be redrawn while the micro-objects are repositioned, so that the original light cage is drawn when the micro-objects are in the final position.
[0393] In various embodiments, non-standard (or "improved") light cages can be used to reposition micro-objects. According to an embodiment, the improved light cages for two adjacent micro-objects are used to reposition micro-objects before or after calculating and selecting tracks and assigning to the isolation dock for each micro-object. In some embodiments, the improved light cages are used to reposition micro-objects iteratively or sequentially. In addition, the improved light cages can be used to place micro-objects in the isolation docks assigned to them. In some embodiments, the perimeter of the closest spatial region or the micro-objects that are closest together in space can be repositioned or fenced before repositioning or fenced other micro-objects.
[0394] Figure 4A , Figure 4B and Figure 4C A micro-object enclosure using a light box is shown. Figure 4A , a biological cell within a channel of a microfluidic circuit is shown immediately after identifying the cell and assigning the cell to a dock. The black box surrounding the cell illustrates the output of the cell recognition algorithm, i.e., the identification of the cell indicated by the box surrounding the cell. The white box surrounding the black box is a light cage for the OET force to relocate the cell. Finally, the black line connecting the box surrounding the cell to the isolation dock illustrates the optimal trajectory calculated when assigning the cell to the isolation dock. Figure 4B The same cell is shown at a later time point, where the light cage has moved along its chosen trajectory. Figure 4C The same cell is shown at a third time point, where the light cage has moved almost completely along its chosen trajectory to position the cell in the isolation dock.
[0395] When moving micro-objects, OET and / or DEP can be gradually accelerated to move the speed of cells, so as to "increase" the motion of micro-objects and ensure that micro-objects will not be lost from their light cages. For example, in a specific embodiment, the initial velocity of micro-objects can be gradually accelerated to a higher travel velocity from a low initial velocity. This gradual acceleration can be applied in the case where micro-objects are automatically repositioned (e.g., marking, repositioning and output) and in the case where micro-objects are manually repositioned (e.g., manually selecting and moving cells). Similarly, when micro-objects arrive at the end of their trajectory and are in their final position, high travel velocity can "drop" to zero at the final velocity.
[0396] Method of the present invention can be used for automatically detecting micro-objects in all types of microfluidic devices. In certain embodiments, microfluidic devices may include flow areas (or flow channels) and one or more chambers (or isolation docks). Alternatively or additionally, microfluidic devices may be electric devices, such as optically actuated electric devices, or may include regions configured for electrodynamics. If the transistors in the array have regions similar to the cross-sectional area of the micro-object being detected, electric devices, particularly electric devices with transistor arrays (e.g., phototransistors) can provide particularly complex backgrounds. Methods described herein are particularly effective when detecting micro-objects arranged in such devices.
[0397] In certain embodiments, the present invention also provides a machine-readable storage device for storing non-transitory machine-readable instructions for performing any of the methods described herein. The machine-readable instructions can control an imaging device for obtaining an image and / or a processor (e.g., in a computing device) for aligning an image, generating a differential image, and / or analyzing a differential image.
[0398] The methodologies described herein may be implemented by various means depending on the application. For example, the methods may be implemented in hardware, firmware, software, or any combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein, or a combination thereof.
[0399] In various embodiments, the methods of the present teachings may be implemented as firmware and / or software programs and applications written in conventional programming languages such as C, C++, etc. If implemented as firmware and / or software, they may be implemented on a non-transitory computer-readable medium in which a program for causing a computer to perform the above methods is stored. It should be understood that the methods may be implemented in a computer such as Figure 5 The various engines described herein are provided on the computer system 1000 of the computer system, whereby the processor 1004 will perform the analysis and determination provided by these engines, subject to instructions provided by any one or a combination of the memory components 1006 / 1008 / 1010 and user input provided via the input device 1014.
[0400] Although the present teaching is described in conjunction with various embodiments, it is not intended that the present teaching be limited to these embodiments. On the contrary, as will be appreciated by those skilled in the art, the present teaching includes various substitutions, modifications and equivalents.
[0401] In addition, when describing various embodiments, the specification may have presented the method and / or process as a specific order of steps. However, insofar as the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the specific order of steps described. As will be understood by those of ordinary skill in the art, other orders of steps are also possible. Therefore, the specific order of the steps set forth in the specification should not be interpreted as a limitation to the claims. In addition, the claims for the method and / or process should not be limited to the execution of its steps in the order written, and those skilled in the art can easily understand that the order can be changed and still remain within the spirit and scope of the various embodiments.
[0402] The embodiments described herein may be practiced with other computer system configurations, including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. The embodiments may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a network.
[0403] It should also be understood that the embodiments described herein may employ various computer-implemented operations involving data stored in a computer system. These operations are those operations that require physical manipulation of physical quantities. Although not required, these quantities are typically in the form of electrical or magnetic signals that are stored, transmitted, combined, compared, and otherwise manipulated. In addition, the manipulations performed are typically referenced to terms such as "generating," "identifying," "determining," or "comparing."
[0404] Any operation that forms part of the embodiments described herein is a useful machine operation. The embodiments described herein also relate to devices or apparatus for performing these operations. The systems and methods described herein may be specially constructed for the desired purpose, or it may be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines may be used with computer programs written according to the teachings of this article, or more specialized devices may be more conveniently constructed to perform the desired operations.
[0405] Certain embodiments may also be embodied as computer readable code on a computer readable medium. A computer readable medium is any data storage device that can store data and then be read by a computer system. Examples of computer readable media include hard drives, network attached storage (NAS), read-only memory, random access memory, CD-ROM, CD-R, CD-RW, magnetic tapes and other optical, flash memory and non-optical data storage devices. Computer readable media may also be distributed on network coupled computer systems to store and execute computer readable code in a distributed manner.
[0406] Although specific embodiments and applications of the present invention have been described in this specification, these embodiments and applications are merely exemplary and many variations are possible.
[0407] Description of some embodiments of the invention.
[0408] 1. A method for automatically detecting micro-objects in an illuminated image (e.g., a bright field image), the method comprising: generating a plurality of pixel masks from the image for corresponding plurality of micro-object properties, wherein generating the plurality of pixel masks comprises: processing pixel data from the image using a machine learning algorithm, and wherein each pixel mask comprises a set of pixel annotations, each pixel annotation in the set of pixel annotations representing a probability that a corresponding pixel in the image represents the corresponding micro-object property; and obtaining a micro-object count from at least one pixel mask in the plurality of pixel masks.
[0409] 2. A method according to embodiment 1, wherein the micro-object count is obtained from a combination of pixel masks among the plurality of pixel masks.
[0410] 3. A method according to embodiment 1 or 2, wherein the multiple micro-object characteristics include at least three micro-object characteristics.
[0411] 4. A method according to embodiment 1 or 2, wherein the plurality of micro-object characteristics include at least: (i) the center of the micro-object; (ii) the edge of the micro-object; and (iii) non-micro-objects.
[0412] 5. A method according to embodiment 4, wherein obtaining the micro-object count includes obtaining the micro-object count from the pixel mask corresponding to the central characteristic of the micro-object, or a combination of pixel masks including the pixel mask corresponding to the central characteristic of the micro-object.
[0413] 6. A method according to any one of embodiments 1 to 5, wherein the machine learning algorithm comprises a neural network (e.g., a convolutional neural network).
[0414] 7. A method according to embodiment 6, wherein the neural network includes multiple downsampling blocks (for example, at least 2, 3, 4, etc. downsampling blocks), each downsampling block includes a first downsampling convolution layer, a first batch normalization layer and a first ELU layer including a gate function, and wherein each first downsampling convolution layer reduces the spatial resolution of the image data it receives.
[0415] 8. A method according to embodiment 7, wherein one or more (e.g., each) of the downsampling blocks consists of (or essentially consists of) the first downsampling convolutional layer, the first batch normalization layer, and the first ELU layer, wherein the first ELU layer receives image data directly from the first batch normalization layer, and wherein the first batch normalization layer receives image data directly from the first downsampling convolutional layer.
[0416] 9. A method according to embodiments 7 or 8, wherein each downsampling convolutional layer reduces the spatial resolution of the image data it receives by a factor of two (e.g., by sliding the convolution filter (or kernel) two pixels at a time).
[0417] 10. A method according to any one of embodiments 7 to 9, wherein each of the first downsampling convolutional layers comprises a 5×5 convolutional filter.
[0418] 11. A method according to any one of embodiments 7 to 10, wherein one or more (e.g., each) of the multiple downsampling blocks is followed by a residual network block with a branch structure.
[0419] 12. A method according to embodiment 11, wherein the branch structure of the residual network block includes a first branch and a second branch, and wherein the first branch processes the image data received from the previous downsampling block to a lesser extent than the second branch.
[0420] 13. A method according to embodiment 12, wherein the first branch of the residual network block includes a second convolutional layer, a second batch normalization layer, and a second ELU layer including a gate function.
[0421] 14. A method according to embodiment 13, wherein the first branch of the residual network block consists of (or essentially consists of) the second convolutional layer, the second batch normalization layer and the second ELU layer, wherein the second ELU layer receives image data directly from the second batch normalization layer, and wherein the second batch normalization layer receives image data directly from the second convolutional layer.
[0422] 15. A method according to embodiment 13 or 14, wherein the second convolutional layer includes a 1×1 convolutional filter.
[0423] 16. A method according to any one of embodiments 11 to 15, wherein the second branch of the residual network block includes two or more processing units, wherein each processing unit includes a convolutional layer and a batch normalization layer.
[0424] 17. A method according to embodiment 16, wherein the second branch of the residual network block consists of (or essentially consists of) a third convolutional layer, a third batch normalization layer, a third ELU layer inclu...
Claims
1. A method for detecting and characterizing micro-objects in a microfluidic device, the method comprising: receiving a first image and one or more second images of a region of interest in the microfluidic device, the first image comprising an illuminated image and the one or more second images comprising one or more non-illuminated images; transforming each of the one or more second images to optically align the one or more second images with the first image; processing pixel data in the first image using a machine learning algorithm to detect micro-objects present in the region of interest, wherein detecting each micro-object includes identifying a boundary of the micro-object; as well as A signal is detected within each boundary of each detected micro-object in each of the one or more second images, wherein each detected signal is associated with an agent that specifically binds to a biomolecule comprised by the one or more detected micro-objects. 2 . The method of claim 1 , wherein at least one of the one or more second images is a fluorescence image, and wherein the signal detected in the at least one second image is a fluorescence signal. 3 . The method of claim 1 , wherein each of the one or more second images is a fluorescence image, and wherein the signal detected in each of the one or more second images is a fluorescence signal. The method of claim 2 , wherein each fluorescence image represents fluorescence signals from non-overlapping portions of the visible spectrum. The method of claim 2 , wherein each fluorescent image represents a fluorescent signal from a unique portion of the visible spectrum.
6. A method according to any one of claims 1 to 5, wherein processing pixel data in the first image to detect micro-objects present in the region of interest includes using a machine learning algorithm to generate multiple pixel masks from the first image for corresponding multiple micro-object characteristics, wherein each pixel mask includes a set of pixel annotations, each pixel annotation in the set of pixel annotations represents the probability that the corresponding pixel in the image represents the corresponding micro-object characteristic. The method of claim 6 , wherein detecting the micro-object comprises using a combination of pixel masks from among the plurality of pixel masks.
8. The method according to claim 6, wherein the plurality of micro-object characteristics at least include: (i) micro-object center; (ii) micro-object edge; and (iii) non-micro objects.
9. The method of claim 8, wherein detecting the micro-object is based on a pixel mask corresponding to a micro-object center characteristic or a combination of pixel masks including a pixel mask corresponding to a micro-object center characteristic.
10. The method of claim 6, wherein the machine learning algorithm comprises a neural network.
11. A method according to any one of claims 1 to 5, wherein there are at least two second images.
12. The method according to any one of claims 1 to 5, further comprising: The detected micro-objects are grouped into subgroups of micro-objects having one or more common characteristics.
13. The method of claim 12, wherein the detected micro-objects are grouped into subgroups based on their proximity in an n-dimensional space, wherein each of the n dimensions is a measurable characteristic of the micro-objects.
14. The method according to any one of claims 1 to 5, further comprising: A visual display is provided that represents a distribution of at least one characteristic of the detected micro-objects.
15. The method of claim 14, wherein the visual display is a two-dimensional graph representing at least two characteristics of the detected micro-objects.
16. The method of claim 14, wherein the visual display is a three-dimensional graph representing at least three characteristics of the detected micro-objects.
17. The method of claim 14, further comprising providing a user interface that allows a user to select a subgroup of detected micro-objects and allows the user to provide instructions for relocating the selected subgroup.
18. The method according to any one of claims 1 to 5, further comprising increasing or decreasing the identified boundary of the micro-object.
19. The method according to any one of claims 1 to 5, wherein the micro-objects are imaged within a microfluidic device, and wherein pre-processing comprises subtracting a repetitive pattern generated by at least one component of the microfluidic device during imaging.
20. The method of claim 19, wherein the pre-processing comprises applying a Fourier transform to the image to identify the repeating pattern.
21. The method of claim 19, wherein the at least one component of the microfluidic device is a substrate surface.
22. The method of claim 19, wherein the at least one component of the microfluidic device is a substrate surface comprising an array of phototransistors.
23. The method according to any one of claims 1 to 5, wherein the micro-objects are living biological cells.
24. The method according to claim 23, wherein the living biological cell is: an immune cell; a cell from a cell line or a cancer cell; or an oocyte, a sperm or an embryo.
25. A system for automatically detecting micro-objects in a microfluidic device, comprising: An image acquisition unit, comprising: an imaging element configured to capture a first image and one or more second images of a region of interest in the microfluidic device, the first image comprising an illuminated image and the one or more second images comprising one or more non-illuminated images; and an alignment engine configured to transform the second image to optically align the one or more second images with the first image, and A micro-object detection and characterization unit communicatively connected to the image acquisition unit, comprising: an image processing engine configured to process pixel data in the first image using a machine learning algorithm to detect micro-objects present in the region of interest, wherein detecting the micro-objects includes identifying a boundary of each detected micro-object; and A detection engine configured to detect a signal within each boundary of each detected micro-object in each of the one or more second images, wherein each detected signal is associated with an agent that specifically binds to a biomolecule included in the one or more detected micro-objects.
26. The system of claim 25, further comprising: A user interface, wherein the user interface is configured to allow a user to select a subgroup of detected micro-objects and optionally allow a user to provide instructions for relocating the selected subgroup.
