Computer-implemented method for selecting a subset of images from a database containing a plurality of initial images from a microscopy sample - Patent Application 20070122997

Abstract

The computer-implemented method includes providing a database (2) containing a plurality of initial images (11), each of the initial images (11) being an image of a different portion of a microscopy sample (10) containing a plurality of objects (31, 32), and selecting a subset of the plurality of initial images (11), the selection being based on determined dimensions of microscopic features (31, 32, 33) in at least one of the initial images (45).

Description

[Technical Field]

[0001] The present invention relates to a computer-implemented method for selecting a subset of a plurality of identifying initial images from a plurality of initial images of a microscopy sample.

[0002] Optical microscopy is a powerful single-cell technique that enables quantitative spatial information at subcellular resolution. However, unlike flow cytometry and single-cell sequencing techniques, microscopy has problems achieving high-quality population-wide sample characterization while maintaining high resolution, resulting in a compromise between resolution and population context. Understanding the contextual relevance of acquired data is particularly challenging for high-resolution live imaging applications, where the field of view limits cell population analysis.

[0003] For example, in studies of cell migration and infection, it is difficult to capture rare or common events of interest with high accuracy and at high resolution and / or magnification.

[0004] There are also the additional challenges of reducing human bias, increasing reproducibility, and placing single-cell characteristics in the context of the sample population when interpreting microscopy data, all of which can result in reduced data fidelity.

[0005] For example, in most high-resolution microscopy applications, images are typically acquired only from selected points, lacking population context and risking bias, especially since data selection is often left to the human operator. An even more stringent requirement is ensuring relevant data is acquired during live imaging, such as when recording high-resolution cell migration data or capturing host-pathogen interactions.

[0006] Integrating image analysis, including machine learning classification algorithms, into data selection has improved efficiency and reduced overall bias in image acquisition. These solutions, called event-driven feedback microscopy or intelligent microscopy, enable high-throughput, targeted imaging of cells of interest in high-spatial-temporal settings. Such decisions are typically based on predetermined criteria of image characteristics through image segmentation and are not adaptive to the data distribution of a particular sample population.

[0007] According to a first aspect of the present invention, a. providing a database including a plurality of initial images, each initial image being an image of a different portion of a microscopy sample including a plurality of objects; b. selecting a subset of the plurality of initial images, the selection being based on the determined dimensions of the microscopic features in at least one of the initial images.

[0008] Preferably, the method comprises: c. selecting a portion of the microscopy sample corresponding to one of the images of the subset; d. Capturing a first image of the portion or subsection of the portion.

[0009] Typically, the method comprises: e. capturing a second image of the portion or subsection; A second image is captured at a time interval after the capture of the first image.

[0010] Typically, multiple additional images may be captured, each at a different time interval after the capture of the second image, which may be substantially equal to the time interval between the capture of the first image and the capture of the second image, or may be substantially a multiple of the time interval.

[0011] The time interval can be 10 seconds to 30 minutes, preferably 1 minute to 30 minutes, and more preferably 5 minutes to 30 minutes. For example, the time interval can be 10 minutes or 20 minutes.

[0012] Preferably, the first image, and any second or further images, are captured at at least one of (i) a higher resolution than the initial image, and (ii) a higher magnification than the initial image. If they are captured at a higher resolution, the higher resolution may be at least one of a higher spatial resolution and a higher temporal resolution.

[0013] The determined dimensions of the microscopic features may be pre-determined or may be user-selected.

[0014] Preferably, the determined dimensions of the microscopic feature include at least one of (i) a dimension of an object, and (ii) a distance between two objects.

[0015] In one example of the present invention, the microscopic features may be two objects, such as two adjacent objects. The two objects may be determined objects. The determined objects may be pre-determined or user-selected.

[0016] Typically, the determined dimension may include a range having at least one of a lower limit and an upper limit. Preferably, when the determined dimension is a separation between two objects, the determined dimension has an upper limit. Typically, the selected subset includes initial images in which two adjacent objects have a separation that is equal to or less than the upper limit.

[0017] In one example of the present invention, the microscopic feature can be two objects with a separation below an upper limit or threshold. Typically, the separation is 25 μm or less, more preferably 10 μm or less, and even more preferably less than 5 μm. Most preferably, the separation is 4 μm or less, and can be 3.7 μm or less. Preferably, the separation is at least 1 μm.

[0018] Preferably, the initial image covers substantially the entire field of view of the microscopy sample in at least one plane, such as a plane substantially perpendicular to the optical axis of the microscope.

[0019] Preferably, the initial image includes multiple images of each field of view, each of the multiple images of the field of view captured at a different initial time interval. The multiple images for each field of view may include images for multiple different channels captured at each initial time interval. Typically, the number of multiple images for each field of view may be 2 to 10 per channel. However, it is possible that there may be more than 10 images per channel for each field of view.

[0020] The initial time interval can be 10 seconds to 30 minutes, preferably 1 minute to 30 minutes, and more preferably 5 minutes to 30 minutes. For example, the initial time interval can be 10 minutes or 20 minutes.

[0021] Typically, the initial image covers the microscopy sample in at least one of the sample XY plane and the sample Z axis, the XY plane being defined as the plane substantially perpendicular to the optical axis of the microscope.

[0022] Preferably, the method further includes controlling a microscope having an image capture device to capture a plurality of initial images and creating a database including the plurality of initial images.

[0023] Typically, at least one of the objects is a biological object, such as at least one of a cell and a non-cellular organism. Any of the objects may be a pathogen, such as a virus, bacterium, parasite, or fungus.

[0024] The selection in step b may also be based on at least one additional parameter, which may be an image property or a property of an object in the image selected from intensity, signal-to-noise ratio, density, shape, size, and contrast.

[0025] Preferably, the method is a method for identifying interactions between biological objects in a microscopy sample, such as interactions between cells and pathogens.

[0026] According to a second aspect of the present invention there is provided an apparatus for implementing the method according to the first aspect, which may typically comprise storage means for storing a database and a processor coupled to the database for selecting a subset.

[0027] Typically, where the method further comprises capturing a first, second or further image, the apparatus is configured to create a separate database of the further captured images and store the other database in the storage means or another storage means.

[0028] If the method further includes controlling a microscope, the apparatus may further include a microscope having an image capture device, the microscope and the image capture device coupled to a computer allowing the computer to control the image capture device and the microscope and to receive images captured from the image capture device.

[0029] According to a third aspect of the present invention there is provided a method of identifying interactions in a microscopy sample, comprising the steps of: a. mounting a sample containing at least two objects on a microscope; b. capturing a plurality of initial images of different portions of the specimen on an image capture device; c. using a processor to analyze the initial image to identify a portion of the sample that includes two objects having a separation below a threshold.

[0030] Typically, the initial images are all captured at the same magnification and resolution.

[0031] Preferably, the method of the third aspect comprises: d. capturing a first image of the portion; A processor analyzes the first image of the portion to determine whether the two biological objects are or have interacted with each other.

[0032] The first image may be captured at a higher magnification and / or a higher resolution than the initial image, which may be a higher spatial resolution and / or a higher temporal resolution.

[0033] Typically, the method of the third aspect comprises: e. capturing a second image of the portion; A second image is captured at a time interval after the capture of the first image, and a processor compares the first image of the portion with the second image of the portion to determine whether the two biological objects are interacting or have interacted with each other.

[0034] Typically, multiple further images are captured, each of the multiple further images being captured at a different time interval after the capture of the second image.

[0035] The different time interval may be substantially equal to the time interval between the capture of the first image and the capture of the second image, or may be substantially a multiple of the time interval.

[0036] Preferably, at least one of the objects is a cell. For example, at least one of the biological objects may be a eukaryotic cell or a microorganism, such as a bacterium, algae, virus, or protozoan. However, at least one of the objects may be a non-biological object, such as a latex particle.

[0037] In one example of the present invention, one of the objects is a microorganism. For example, the organism can be a pathogen such as a virus, bacterium, parasite, or fungus.

[0038] Preferably, the analysis further comprises analyzing at least one additional parameter of the two biological objects, which may be selected from intensity, signal-to-noise ratio, density, shape, size, and contrast.

[0039] Typically, the selection of the subset is also based on at least one additional parameter.

[0040] The method may further include identifying a location of the microscopic feature in each of the images of the subset and identifying a portion of the sample containing the microscopic feature using the location in the images. Identifying the location may include identifying coordinates of the microscopic feature in the images.

[0041] Preferably, steps b and c of the third aspect are repeated until a portion of the sample has one of two biological objects identified as having a separation below the threshold or identified as having a separation below the upper limit.

[0042] Typically, the method further includes storing the captured image in a memory device.

[0043] An example of a computer-implemented method for selecting a subset of a plurality of initial images of a microscopy sample will now be described with reference to the accompanying drawings. [Brief explanation of the drawings]

[0044] [Figure 1] FIG. 1 is a schematic diagram illustrating the process of data-driven microscopy. [Figure 2] FIG. 2 is a block diagram of an example of hardware for implementing the process of FIG. 1. [Figure 3] FIG. 1 is a schematic diagram illustrating the process of data-driven microscopy for the identification of interactions of biological objects. [Figure 4] A series of five time-lapse images obtained during the data-independent acquisition phase of the process depicted in Figure 3 is shown. [Figure 5a]Time-lapse images of the different interactions identified are shown in Figure 4 . [Figure 5b] Time-lapse images of the different interactions identified are shown in Figure 4 . [Figure 5c] Time-lapse images of the different interactions identified are shown in Figure 4 . [Figure 6a] Figure 5 shows different interactions imaged using a data-driven acquisition phase. [Figure 6b] Figure 5 shows different interactions imaged using a data-driven acquisition phase. [Figure 6c] Figure 5 shows different interactions imaged using a data-driven acquisition phase. [Figure 7] A series of 12 time-lapse images of the interaction shown in Figure 6a is shown.

[0045] As used herein, the term "dimension" is intended to refer to a particular type of measurable extent, such as the length, width, depth or height of an object, or the distance (or separation) between two objects.

[0046] Data-driven microscopy is an approach for automated, targeted image acquisition of relevant data, built around two imaging strategies interconnected through a shared server database. The data-driven microscopy process is illustrated schematically in Figure 1, and a block diagram of suitable hardware is shown in Figure 2. The first imaging strategy is data-independent acquisition (DIA) 1. The goal of DIA 1 is to capture and characterize the complete sample population in real time. DIA 1 provides complete population characterization at the single-cell level of a biological sample 10 mounted on a microscope 21. The microscope 21 is also equipped with a suitable camera 22 capable of capturing images of the sample 10. The combination of the microscope 21 and camera 22 is commonly known as a digital optical microscope. The microscope 21 includes a motorized stage 25, an objective lens 26, an eyepiece 27, and control electronics 28. The control electronics 28 is connected to the camera 22, the motorized stage 28, and the main microscope body 29. The samples 10 were placed in a suitable sample holder such as a μ-slide 8-well glass bottom slide (Ibidi) 9 .

[0047] The microscope 21 may be an inverted Nikon® Ti2-E widefield fluorescence microscope used with a Nikon® Plan Apo λ10x0.45 numerical aperture (NA) objective and a Perfect Focus System (PFS) for maintaining focus over time. The camera 22 may be a Nikon® DS-Qi2 CMOS camera. Sample imaging was automated using a Nikon® TI-S-ER motorized stage with JOBS (NIS322 element extension, Nikon®) and encoders to generate stage positions covering the sample area. The same system was also used with a Nikon® CFI SR Plan Apo IR 60x AC WI / 1.27 NA objective with a software-driven TI2-N-WID water immersion dispenser. Typically, a 10x0.45 NA objective is used for DIA, and a 60x AC WI / 1.27 NA objective is used for DDA. However, it is possible that the same objective lens can be used for both DIA and DDA. Alternatively or additionally, the temporal and / or spatial resolution can be changed between DIA and DDA, with typically higher spatial and / or temporal resolution being used for DDA.

[0048] Single-cell data captured and generated from the sample 10 using the microscope 21 and camera 22 is continuously output from the control electronics to a computer server 23 and stored on a storage device 24 within the server 23 in the DIA database 2. The storage device is typically in the form of a hard disk drive (HDD) or solid-state drive (SSD). The server may be connected in close proximity to the microscope 21 and camera 22, for example, via an Ethernet, wireless connection, or USB cable. Alternatively, the server 23 may be located in a remote location (such as in the cloud) and connected, for example, via the Internet.

[0049] From this database 2, a user can define cells of interest by searching for and filtering (gating) features or criteria 15, either post-acquisition or in real time. Thus, the DIA database 2 enables real-time analysis of images 11 acquired by the microscope 21 and camera 22, as well as filtering and single-cell targeting 12 of the data in the DIA database 2 to generate targeting criteria 15. Alternatively, the targeting criteria 15 can be predefined. Examples of features or criteria 15 in a biological sample that can be used for filtering include dimensions of biological objects present in the sample, such as cells, bacteria, or other pathogens, other dimensions such as separation between different biological objects, and size of the biological objects.

[0050] The predefined or generated targeting criteria 15 are then fed back to the microscope 21 for a second imaging strategy, data-dependent acquisition (DDA) 3. DDA 3 performs targeted, high-fidelity imaging of the biological object or event 13 of interest. Fidelity can be increased by increasing one or more of spatial resolution, temporal resolution, and magnification, or by imaging with a different modality. DDA 3 can be performed after DIA 1, where gating is performed in real time using stored stage coordinates corresponding to each image captured and stored in database 2, or using gates from a separate DIA experiment.

[0051] The high fidelity data from DDA 3 is then stored in high fidelity database 4, which is interconnected to DIA database 2. This brings the high fidelity data in high fidelity database 4 in the context of the entire sample population. The interconnection of the DIA and DDA databases allows the high fidelity data to be placed in the context of the entire sample.

[0052] Accessing both databases2,4 allows users to further explore the data in its relative context and discover new insights5 that may motivate further experimentation. Taken together, establishing a synergistic relationship between the two imaging strategies and placing biological objects (or features) of interest in their population context can increase the fidelity and relevance of the data.

[0053] For example, the process described above allows a user to determine whether a particular biological feature or event is an anomaly within a biological sample or a general characteristic of the sample.

[0054] Thus, as described above in connection with Figure 1, the DDM process allows for automated, high-fidelity sampling of targeted, multi-labeled subpopulations.

[0055] 1 and 2, the DDM process can be used to identify and capture events such as interactions between biological objects within the biological sample 10. Such interactions can be, for example, host-pathogen interactions (such as bacteria-cell interactions) or cell-cell interactions.

[0056] When imaging live host-pathogen interactions, a major challenge is predicting where in the sample such events are likely to occur, especially since many of the interactions are rare and may be time-dependent. To validate DDM's ability to capture live cells and rare events, a data-driven approach was developed to target live single-cell interactions between cells and bacteria, which is illustrated in Figure 3.

[0057] HeLa cells expressing mScarlet-LifeAct and the bacterial pathogen Yersinia pseudotuberculosis expressing GFP were used. Through DIA, sample populations of HeLa cells (population = 16,988 cells) and Yersinia pseudotuberculosis (population = 990 bacteria) and their interactions (number of interactions = 120) were monitored over time using time-lapse imaging, capturing images at 10, 30, 50, 70, and 90 minutes, respectively (see Figure 4).

[0058] In this experiment, there were approximately 20 different fields of view for sample 10. Images of each field were captured on each of four different microscope channels for each time interval. Therefore, approximately 400 images were captured for DIA Database 2 using DIA35.

[0059] A live cell imaging system was set up for HeLa mScarlet-LifeAct cells 31 and the bacterial pathogen Yersinia pseudotuberculosis 32. Using DIA 35, interactions in the form of adhesion between 990 bacteria 32 and 16,988 cells 31 were monitored using time-lapse imaging, capturing images at 20-minute time intervals (see Figure 4), and 120 interactions were identified. Once identified, these 120 interactions were targeted for DDA 36. Using DIA 35, interactions were revealed to be sparse across the sample. Interactions were categorized according to the minimum distance x between each bacterium 32 and its nearest neighboring cell 31. The minimum distance x was determined based on thresholding the Euclidean distance between each bacterium and the nearest segmented actin signal. Three potential interactions 33 are identified in the final image 45 (time = 90 min) of Figure 4.

[0060] For the purposes of this experiment, a potential interaction 33 between a cell 31 and a bacterium 32 was determined to exist if the minimum distance x was 3.7 μm or less over at least 20 minutes or for at least three of the time-lapse captured images 41, 42, 43, 44, and 45 in Figure 4.

[0061] Figures 5a-c show four time-lapse images 51, 52, and 53 of each of the three bacteria-cell interactions 33 from Figure 4, where bacteria 32 adhered to cells 31 and were imaged using DIA (10x magnification, scale bar 25 µm). In each of Figures 5a-c, time-lapse images for each of images 51, 52, and 53 were captured every 10 minutes. Upon meeting a predetermined number of interactions 33 in DIA, the bacteria-cell interaction was automatically targeted for high-fidelity live cell imaging in DDA. This was based on two independent experiments in which 120 fields of view were selected.

[0062] Figures 6a-6c show exemplary images 61, 62, and 63, respectively, of the same three bacteria-cell interactions 33 shown in Figures 5a-5c, targeted and imaged in DDA 36 (60x magnification, scale bar 10 μm). Image 61 corresponds to image 51, image 62 corresponds to image 52, and image 63 corresponds to image 53.

[0063] Yersinia pseudotuberculosis disrupts host cell function by interfering with the formation of actin filaments through the injection of multiple toxins through its type III secretion system. Figure 7 shows 12 time-lapse images of the interaction 33 shown in image 61 of Figure 6a. In Figure 7, each image was captured at a time interval of 20 minutes after the previous image. In Figure 7, "104" (t = 104 min) refers to the time in minutes after image acquisition began. Each subsequent image capture is at 20-minute intervals after t = 104 min, up to 220 min, as shown along the top of Figure 7.

[0064] In Figure 7, at each time interval, the top image shows cells 31, the middle image shows bacteria 32, and the bottom image shows both cells 31 and bacteria 32. These images reveal distinct effects on host cell 31 actin morphology and dynamics by bacteria 32. Data are based on two independent experiments capturing 120 independent time-lapse acquisitions of host cell-bacteria interactions from 120 corresponding fields of view.

[0065] When selecting interactions for further investigation using DDA, the selection of interactions may be based at least in part on at least one additional parameter of the features and / or biological objects, such as intensity, signal-to-noise ratio, density, shape, size, and contrast.

[0066] In the experiments described above using DDM, we captured all events and achieved a 100% hit rate.

[0067] In comparison, traditional approaches for obtaining high-resolution cell-bacteria interactions based on manual monitoring would achieve an estimated hit rate of only 1.4% in the same experiment, based on the proportion of interacting bacteria and cells in the sample. Additionally, sample population data cannot be obtained with standard approaches, making it impossible to assess overall relevance.

[0068] Figures 6a-c show three examples from DDA. Figure 7 shows representative results of bacteria-cell interactions over time. This demonstrates the adaptability and capability of DDM for acquiring population-wide data and subsequent targeting of time-dependent and rare events for live cell imaging over time.

[0069] DDM implements a data-centric approach to image acquisition. The initial scan, which collects overview data using DIA1, also leads to one of the approach's most obvious advantages, as it provides coordinates and basic features of interest within the sample. This basic sample overview data enables real-time analysis of population context based on objective data rather than the microscopist's subjective experience.

[0070] Additionally, the population data can be filtered (gated) on additional channels to determine what data to collect. For example, the additional filtering can be based on image or object properties such as intensity, signal-to-noise ratio, density, shape, size, and contrast.

[0071] DDM also inherently provides information about what constitutes a representative subject in a sample population: cells can be considered representative when they are placed in the context of the population feature distribution.

[0072] In live interaction studies such as those described above and illustrated in Figures 3-7, using DDM leads to a dramatic increase in hit rate for high-resolution data collection compared to traditional approaches (from 1.4% to 100% in the current example). There is also the valuable addition of population-wide context compared to other feedback microscopy solutions. DDM uses the best aspects of each modality and controls acquisition in an automated and efficient manner, resulting in large, context-aware datasets with high fidelity.

[0073] DDM enables significant post-acquisition benefits compared to the current state-of-the-art. To increase introspection and reproducibility, DDM inherently logs every action performed and the state of a running experiment, providing clear status updates to the user. DDM essentially provides a collective fingerprint for each experiment, making it less prone to human error and bias.

[0074] In this study, as a proof of principle, we established a DDM framework on a Nikon® digital microscope. The framework is compatible with any digital microscope, provided that the control software can send images to a server or call an external program to achieve this. In summary, DDM provides a useful framework for more robust and unbiased acquisition of high-fidelity microscopy data.

[0075] In the examples described above, the sample is a biological sample, but interactions between biological objects or entities are being analyzed or investigated. The sample can also be a microscopy sample, including non-biological objects and interactions between non-biological objects in the sample being investigated. For example, the non-biological objects can be beads or particles, such as latex particles, coated with chemicals or biological substances, such as proteins. The present invention can be used to investigate interactions between beads or particles with these coatings, for example, interactions between two beads or particles coated with different proteins.

Claims

1. 1. A computer-implemented method comprising: a. providing a database including a plurality of initial images, each initial image being an image of a different portion of a microscopy sample including a plurality of objects; b. selecting a subset of the plurality of the initial images, the selection being based on a determined size of a microscopic feature in at least one of the initial images.

2. c. selecting a portion of the microscopy sample corresponding to the subset of images; d. capturing a first image of said portion or a subsection of said portion.

3. e. capturing a second image of the portion or subsection; The method of claim 2 , wherein the second image is captured at a time interval after the capture of the first image.

4. The method of claim 3 , wherein a plurality of additional images are captured, each of the plurality of additional images being captured at a different time interval after the capture of the second image.

5. The method of claim 4 , wherein the different time interval is substantially equal to or substantially a multiple of the time interval between the capture of the first image and the capture of the second image.

6. 6. The method of any one of claims 2 to 5, wherein the first image, and any second or further images, are captured at at least one of: (i) a higher resolution than the initial image; and (ii) a higher magnification than the initial image.

7. The method of claim 6 , wherein the higher resolution can be at least one of a higher spatial resolution and a higher temporal resolution.

8. 10. A method according to any one of the preceding claims, wherein the microscopic features may be selected during selection of the subset or may be selected before selection of the subset.

9. 10. The method of any one of the preceding claims, wherein the dimension of the microscopic feature comprises at least one of: (i) a dimension of an object; and (ii) a distance between two objects.

10. 10. A method according to any one of the preceding claims, wherein the dimension comprises a range having at least one of a lower limit and an upper limit.

11. 10. A method according to any one of the preceding claims, wherein the initial image covers substantially all of the microscopy sample.

12. 10. The method according to any one of the preceding claims, wherein the initial image covers the sample in at least one of an XY plane of the sample and a Z axis of the sample.

13. 10. The method of any one of the preceding claims, further comprising controlling a microscope having an image capture device to capture the plurality of initial images and creating the database comprising the plurality of initial images.

14. 10. The method of any one of the preceding claims, wherein at least one of the objects is a eukaryotic cell or a microorganism, and typically, when one of the biological objects is a microorganism, the microorganism is a pathogen such as a virus, bacterium, parasite, or fungus.

15. 10. The method according to any one of the preceding claims, wherein the selection of the subset in step b is also based on at least one additional parameter, the at least one additional parameter being selected from intensity, signal-to-noise ratio, density, shape, size, and contrast.

16. 10. The method of any one of the preceding claims, wherein the method is a method for identifying interactions between objects in the sample, such as interactions between cells and pathogens.

17. 10. A data processing apparatus comprising means for carrying out a method according to any one of the preceding claims.

18. A computer program comprising instructions, which when said program is executed by a computer, cause said computer to carry out a method according to any one of claims 1 to 16.

19. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 16.

Images