How to save images acquired using single molecule localization microscopy

Abstract

The present invention relates to a method for summing images acquired using single molecule localization microscopy, the method comprising the steps of acquiring an image of a sample comprising a plurality of single fluorescent molecules using single molecule localization microscopy, detecting a plurality of spots in the image, each spot representing one of the plurality of single fluorescent molecules, identifying one or more regions of interest in the image comprising the detected plurality of spots, identifying metadata defining each of the one or more local regions of interest, creating one or more image files, the one or more image files including image data for each of the one or more local regions of interest and the identified metadata defining each of the one or more regions of interest, the image data being part of the image, and saving the one or more image data.

Description

[Technical field]

[0001] The present invention relates to a method and apparatus for storing images acquired using single molecule localization microscopy. [Background technology]

[0002] Single Molecule Localization Microscopy (SMLM) is based on delayed stochastic emission of molecules to achieve emission of single fluorescent molecules. A sample containing fluorescent molecules is illuminated with an illumination intensity adjusted such that, on average, only one molecule is activated (i.e., emits light) within the focal volume of the observation microscope each time. In particular, each fluorescent molecule is labeled with an SMLM fluorophore, which reversibly transitions (i.e., blinks) between on and off emission states when illuminated. Images acquired by SMLM show image spots called point spread functions (PSFs), each spot representing the image of a fluorescent molecule. From the PSF, it is possible to determine the position of the molecule by identifying the center of the spot using an algorithm. This position can be determined with a certain accuracy, which can be much higher than the diffraction-limited resolution of the microscope. This stochastic process is repeated for tens of thousands of images to obtain a final super-resolution image with high spatial resolution (up to 10 nm).

[0003] Tens of thousands of images are needed to reconstruct a final super-resolution image with a size of tens to hundreds of gigabytes, thus requiring large storage areas (e.g., large storage servers) to store the SMLM dataset, making it difficult to use, for example, cloud storage services. Summary of the Invention [Means for solving the problem]

[0004] These problems are solved or alleviated by the claimed apparatus and methods.

[0005] Approximately 10% of each pixel of tens of thousands of acquired images are used to reconstruct the super-resolution image. In particular, most biological samples have a heterogeneous density of fluorescent molecules, so some areas of the sample may be completely devoid of fluorescent molecules. Therefore, in the present invention, the region of interest includes only areas that contain fluorescent molecules. Furthermore, because single fluorescent molecules blink upon illumination, areas of the acquired image may not show a PSF because some of the fluorescent molecules are "off" at the time of image acquisition.

[0006] Rather than storing all the pixels of each image, which would result in a large data set for each SMLM experiment, the present invention relates to identifying a region of interest in each image acquired during a SMLM experiment, the region of interest containing the PSF, from which it is possible to store only pixels that correspond to the region of interest, thus excluding regions that do not show fluorescent molecules.

[0007] Embodiments of the present invention will now be described with reference to the accompanying drawings as follows: [Brief description of the drawings]

[0008] [Figure 1a] 1 is a schematic diagram of an apparatus for storing images acquired using single molecule localization microscopy, according to certain examples. [Figure 1b] 1 is a schematic diagram of an apparatus for storing images acquired using single molecule localization microscopy, according to certain examples. [Diagram 2] 1 shows an example of an image captured by a device according to an example. [Diagram 3] FIG. 1 is a flow diagram of a method for storing images acquired using a single molecule localization microscope, according to an example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] 1a and 1b are schematic diagrams of an apparatus 100 for storing images of a sample 102 acquired using single molecule localization microscopy (SMLM) according to two examples.

[0010] The sample 102 may be fabricated, for example, of cells (such as neurons) or tissue samples that contain fluorescently labeled molecules. For example, the fluorescent molecules may be pre-labeled with SMLM fluorophores, such as photoswitchable, photoactivatable, photoconvertible, spontaneously blinking, or time-blinking fluorophores. Once labeled, the fluorescent molecules reversibly transition (i.e., blink) between on and off emission states upon illumination. Generally, samples that contain fluorescent molecules are heterogeneous in their density of the fluorescent molecules. Thus, some areas of the sample 102 may have a high density of the fluorescent molecules, while other areas may have a low density of the fluorescent molecules. Additionally or alternatively, other areas may be completely devoid of fluorescent molecules.

[0011] As shown in Fig. 1a, the apparatus 100 includes an image acquisition unit 104, a detection unit 108, an identification unit 110, a calculation unit 114, and a file unit 118. Each of the detection unit 108, the identification unit 110, the calculation unit 114, and the file unit 118 may be implemented using hardware, software, and / or a combination thereof. For example, the hardware devices may be implemented using processing circuits, such as, but not limited to, a processor, a central processing unit (CPU), a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a system-on-chip (SoC), a programmable logic unit, a microprocessor, or any other device capable of responding to and executing instructions in a predetermined manner. The software may include computer programs, program codes, instructions, or any combination thereof, for individually or collectively instructing or configuring the hardware devices to operate in a desired manner. The computer program and / or program code may include one or more hardware devices, such as programs or computer readable instructions, software components, software modules, data files, data structures, and / or the like that may be implemented by one or more of the aforementioned hardware devices. When the hardware device is a computer processing device (e.g., a CPU, controller, ALU, digital signal processor, microcomputer, microprocessor, etc.), the computer processing device may be configured to execute the program code by performing arithmetic, logic, and input / output operations according to the program code. Each unit may also include one or more storage devices. The one or more storage devices may be tangible or non-transitory computer readable storage media, such as random access memory (RAM), read only memory (ROM), permanent mass storage device (such as a disk drive), solid state (e.g., NAND flash) device, and / or any other similar data storage mechanism that can store and record data.The one or more storage devices may be configured to store computer programs, program codes, instructions, or any combination thereof, for one or more operating systems and / or for implementing the exemplary embodiments described herein. The computer programs, program codes, instructions, or any combination thereof may also be loaded from another computer-readable storage medium into one or more storage devices and / or one or more computer processing devices using a drive mechanism. Such another computer-readable storage medium may include a Universal Serial Bus (USB) flash drive, a memory stick, a Blu-ray / DVD / CD-ROM drive, a memory card, and / or other similar computer-readable storage medium.

[0012] Alternatively, two or more of the detection unit 108, the identification unit 110, the calculation unit 114, and the file unit 118 may be implemented by the same hardware and / or software. For example, the detection unit 108, the identification unit 110, the calculation unit 114, and the file unit 118 may all be implemented by a single computer storage medium that can provide each unit with instructions to perform the functions described below.

[0013] The image acquisition unit 104 is configured to acquire an image 106 of the sample 102 while the sample 102 is being illuminated. The acquisition unit 104 can be, for example, a camera, such as an sCMOS camera. To illuminate the sample 102, a scanning unit (not shown) including a laser emitting at a first wavelength that corresponds to the excitation wavelength of the fluorophore used to label the molecules contained in the sample 102. For example, the laser can be a diode-pumped solid-state laser emitting at 638 nm. The scanning unit can be, for example, a 638 nm laser, as described in Mau et al. 1 The system may be the Adaptive Scanning for Tunable Excitation Regions (ASTER) system disclosed in

[0014] The image 106 acquired while being illuminated is acquired by the image acquisition unit 104 at a second wavelength corresponding to the emission wavelength of the fluorophore used to label the molecules of the sample 102. The first wavelength and the second wavelength may be the same. Figures 1a and 1b show an example of an image 106 showing spots, or point spread functions (PSFs), each PSF representing a single fluorescent molecule. The density of the PSFs of the image 106 is not uniform, in this example, because the sample 102 shows a heterogeneous density of fluorescent molecules. In particular, the sample 102, and thus the image 106, shows high density regions (i.e. regions showing a high spot density), low density regions (i.e. regions showing a low spot density), medium density regions (i.e. regions showing an intermediate spot density), and regions without fluorescent molecules (i.e. dark regions).

[0015] The detection unit 108 is configured to receive the image 106 from the image acquisition unit 104 and detect a number of spots in the image 106, each spot representing one fluorescent single molecule contained in the sample 102. The detection unit 108 may be implemented using an algorithm that analyzes the pixels of the image 106 and detects contrast between the pixels, i.e. changes in the intensity of the pixels. In particular, the algorithm detects high-contrast contours, which represent the edges of each spot. The detection unit 108 may then define a mapping of the spots on the image 106, which indicates the location of each spot based on the detected contrast. In one example, Bourg et al. 2 is used to detect a number of spots in the image 106. In particular, each spot can be modeled as a dipolar emitter emitting in the far field. Additionally, each spot has a non-propagating near-field component, which is proportional to the refractive index of the surroundings n m It depends on the refractive index n g >n m When there is an interface (such as a glass slide) where m is the medium and g is the glass interface, the transmitted light follows the Snell-Descartes law of refraction. The refracted light is at a critical angle θ c =arcsin(n m / n g) and is called under-critical angle fluorescence (UAF). However, the distance d between the fluorescent molecule and the interface increases with the fluorescence wavelength λ em For shorter distances, SAF (supercritical angle fluorescence) emission is also observed. The non-propagating near-field components at the interfaces around the spot have a refractive index of n g >n m The critical angle θ c The SAF emission is detectable for fluorophores in the cell medium near the coverslip. UAF photons N UAF The number of SAF photons N SAF The number of N SAF and N UAF The SAF ratio, ρ SAF =N SAF / N UAF By calculating:

[0016] The identification unit 110 is configured to define local regions of interest 120a, 120b that include spots in the image 106. In particular, the identification unit 110 receives a mapping of the spots from the detection unit 108. Based on the locations of the spots, the identification unit 110 defines local regions of interest 120a, 120b that are optimized to include all of the areas of the image 106 that include PSFs, while excluding areas that do not include PSFs. For example, as shown in FIG. 1a, the local regions of interest 120a are ellipses that are defined such that the entire surface of the local region of interest 120a is as small as possible. Alternatively, as shown in FIG. 1b, the local regions of interest 120b may be polygons. For example, the polygons may be defined such that each polygon includes as much of the PSF as possible and as small as possible of the surface of the areas that do not include PSFs. In another example, the local region of interest 120b may include at least one ellipse and one polygon. In other examples, the local region of interest 120b may include only one PSF. Note that the regions of interest do not cover all of the image 106. In particular, the surface (i.e., number of pixels) covered by all of the regions of interest 120a, 120b is smaller than the surface (i.e., number of pixels) of the entire image 106.

[0017] The computation unit 114 is configured to determine metadata defining each of the regions of interest 120a, 120b. For example, the metadata may define the shape (e.g., ellipse or polygon) of each local region of interest 120a, 120b and / or coordinates defining the position of the local region of interest 120a, 120b in the image 106. For example, the computation unit 114 may define axes x, y corresponding to two of the edges of the image 106 on which the local region of interest 120a, 120b is defined. Considering the example of FIG. 1a, the computation unit 114 may define coordinates in the xy plane corresponding to the center, vertices, and co-vertices of each ellipse 120a. Alternatively, considering the example of FIG. 1b, the computation unit 114 may define coordinates in the xy plane corresponding to each vertex of each polygon 120b.

[0018] The file unit 118 creates an image file 122 that includes image data of the local regions of interest 120a, 120b and the identified metadata. In particular, the image data is a portion of the image that includes the regions of interest 120a, 120b. In other words, each image file 122 includes a portion of the initial image 106, each portion including the regions of interest 10a, 120b. For example, as can be seen in FIG. 1a, the file unit 118 creates 19 image files 122, each image file 122 including image data for each of the 19 ellipses 120a defined by the identification unit 110. Alternatively, as can be seen in FIG. 1b, the file unit 118 creates four image files 122, each image file 122 including image data for each of the four polygons defined by the identification unit 110. The sum of the surface of all image data is smaller than the surface of the image 106. Typically, in single molecule localization microscopy, about 10% of the pixels of the image 106 represent the region of interest.

[0019] Furthermore, the file unit 118 stores the image files 122. For example, the file unit 118 stores the image files 122 including the image data of the local regions of interest 120a, 120b and their corresponding metadata in a file, which may be compressed. Thus, for each image 106, a smaller image file can be stored than the initial image 106, since the image file includes a portion of the image (and not the entire image). As a result, less storage space can be used to store the image files compared to the storage space required to store the initial image 106. The files can be stored in a local storage area and / or in a cloud storage service.

[0020] Typically, in SMLM, tens of thousands of images are used to obtain a final super-resolution image with high spatial resolution. Thus, the process performed by the device 100 described above may be repeated multiple times. For example, the process may be performed thousands or tens of thousands of times. Each time the process performed by the device 100 is repeated, another image is obtained by the image acquisition unit 104, another PSF is detected in the other image by the detection unit 108, another metadata defining another local region of interest is identified by the calculation unit 114, and another image file including image data and metadata of the other local region of interest is created and stored by the file unit 118. The file unit 118 may create a final file, which includes all image files (i.e., the image file 122 and the other image file). The final file may be compressed and / or stored in a local storage area and / or in a cloud storage service. The final file may be stored in a database, and the metadata may be used to search for a particular region of interest 120a, 120b in the database.

[0021] Additionally, the process performed by the above-mentioned device 100 may be repeated at least three times, and a first image 206a, a second image 206b, and a third image 206b may be acquired by the image acquisition unit 104, and the identification unit 110 may identify the background of the second image 206b. However, as shown in FIG. 2, the first image 206a, the second image 206b, and the third image 206b do not represent the same PSF because they are acquired at different times. In particular, since the fluorescent molecules in the sample 102 blink, they do not emit light at the same time, and the images acquired at different times will show different emitting fluorescent molecules. Therefore, the identification unit 110 may identify the background of the second image 206b by comparing the second image 206b with the image acquired before (i.e., the first image 206a) and the image acquired after (i.e., the third image 206c). In particular, the background corresponds to the area of ​​the image that does not contain a PSF. However, because the molecules do not emit light simultaneously, dark regions, i.e., regions that do not show fluorescent molecules, may contain PSF at other times. Thus, several images must be considered to subtract the background. For example, the identification unit 110 may combine the regions of interest 120a, 120b in each of the first image 206a, the second image 206b, and the third image 206b to identify the background that corresponds to the remaining regions of the second image 206b. Furthermore, the file unit 118 may create an image file that includes the image data of the second image 206b with the background subtracted and does not include the background. The file unit 118 may save the image file. As mentioned above, tens of thousands of images are used in SMLM to obtain a final super-resolution image with high spatial resolution. Thus, tens of thousands of images may be used to remove the background of an image obtained in an SMLM experiment. In one example, Hoogendoom et al. 3 As described in, median spatiotemporal filtering can be used for background removal of images acquired in SMLM experiments.

[0022] 3 is a flow diagram illustrating a method 300 for storing an image 106 acquired using single molecule localization microscopy. For example, the method 300 may be performed by the apparatus 100 described above with respect to FIGS. 1a and 1b.

[0023] In block 302, an image 106 of a sample 102 containing a plurality of single fluorescent molecules is acquired using single molecule localization microscopy. In one example, the image 106 is acquired using a camera that is part of an Adaptable Scanning for Tunable Excitation Regions (ASTER) system.

[0024] At block 304, a plurality of spots are detected in the image 106, each spot representing one fluorescent single molecule of the plurality of fluorescent single molecules. In one example, the spots are detected using an algorithm that analyzes contrast in the image 106.

[0025] At block 306, one or more regions of interest 120a, 120b are identified that include the detected spots. For example, the regions of interest 120a, 120b may have a shape that is polygonal and / or elliptical. In some examples, the regions of interest 120a, 120b may include only one spot.

[0026] Metadata defining each of the one or more local regions of interest is identified at block 308. For example, the metadata may include coordinates of an ellipse center and vertices and co-vertices for local region of interest 120a having an elliptical shape, and the metadata includes coordinates of polygon vertices for local region of interest 120b having a polygonal shape.

[0027] At block 310, one or more image files 122 are created, each of the one or more image files 122 including image data for each of the one or more local regions of interest 120a, 120 and the identified metadata defining each of the one or more local regions of interest 120a, 120. The image data is part of the image 106.

[0028] At block 312, the one or more image files are saved. For example, the image files 122 may be saved to a cloud storage service.

[0029] Since tens of thousands of images are required to reconstruct a final super-resolution image in an SMLM experiment, the method 300 may be repeated multiple times. Each time the method is repeated, another image 106 of the sample 102 is acquired, another number of spots in the another image 106 are detected, one or more local regions of interest 120a, 120b containing the another number of detected spots are detected, another metadata defining each of the other one or more local regions of interest 120a, 120b and one or more other image files 122 are created. Each of the other one or more image files includes another image data for each of the other one or more local regions of interest and another metadata defining each of the other one or more local regions of interest. The final file including the image data for each of the one or more image files and the other image data for the other one or more image files is stored, for example, to a cloud storage service.

[0030] In one example, the method 300 may be performed three times. In this example, a first image 206a, a second image 206b, and a third image 206c may be acquired in block 302, and the above steps are repeated for the first image 206a, the second image 206b, and the third image 206c. Each of the first image 206a, the second image 206b, and the third image 206c shows a background, which represents an area of ​​the image that does not include a local region of interest. The background of the second image may be identified based on the regions of interest 120a, 120b identified for each of the first image 206a, the second image 206b, and the third image 206c. The background of the second image 206b may be subtracted to create a background-free image that includes image data of the second image 206b, but not including the background.

[0031] Although the present invention has been illustrated and described in detail using preferred embodiments, the present invention is not limited to the disclosed examples. Those skilled in the art can imagine other variations without departing from the scope of protection of the claimed invention. For example, the examples of local regions of interest are illustrated as ellipses and polygons, but other shapes can also be used. In another example, the shape of the local region of interest is circular and the metadata can define the radius of the circle. Furthermore, Bourg et al. 2 Although the PSF was detected using the method described in, various methods can be used to detect the spots and define the region of interest. For example, the region of interest can be manually selected by the user. Furthermore, although these examples are described with respect to the ASTER system acquisition system, other systems can be used to scan the sample and acquire the images.

[0032] References 1. Mau et.al,Fast widefield scan provides tunable and uniform illumination optimizing super-resolution microscopy on large fields,Nature Communications,vol.12,3077,2021. 2. Bourg et.al,Direct optical nanoscopy with axially localized detection,Nature Photonics,vol,9,587,2015. 3. Hoogendoorn et.al,The fidelity of stochastic single-molecule super-resolution reconstructions critically depends upon robust background estimation,Scientific Reports,4,3854,2014.

Claims

1. A method for storing an image acquired using single molecule localization microscopy, comprising: a) acquiring an image of a sample containing a plurality of single fluorescent molecules using single molecule localization microscopy; b) detecting a plurality of spots in the image, each spot representing one fluorescent single molecule of the plurality of fluorescent single molecules; c) identifying one or more regions of interest in the image that contain the detected spots; and d) identifying metadata defining each of said one or more local regions of interest; e) creating one or more image files, the one or more image files including image data for each of the one or more local regions of interest and the identified metadata defining each of the one or more regions of interest, the image data being part of the image; f) storing said one or more image data; The method includes:

2. the method is repeated a number of times, and during each iteration of the method, the method repeats steps a) through e); g) creating one or more further image files, each of the one or more further image files including further image data for each of the one or more further regions of interest and further metadata defining the one or more further regions of interest; h) creating a final file including the image data for each of the one or more image files and the other image data for the one or more further image files; The method of claim 1 , comprising:

3. The method is performed at least three times, in which in step a) a first image, a second image, and a third image are acquired, and steps b) to h) are performed on the first image, the second image, and the third image, respectively; i) identifying a background of the second image based on the one or more further regions of interest identified in each of the first image, the second image, and the third image; j) subtracting the background of the second image; i) creating a background-less image file containing image data of the second image, without the background; The method of claim 2 , comprising:

4. 4. The method of claim 1, wherein each of the one or more local regions has a shape that is a polygonal shape and / or an elliptical shape, and the metadata defines the shape and position of the one or more local regions within the image.

5. An apparatus for storing images acquired using single molecule localization microscopy, an image acquisition unit configured to acquire an image of a sample including a plurality of single fluorescent molecules; a detection unit configured to detect a plurality of spots in the image, each spot representing one fluorescent single molecule of the plurality of fluorescent single molecules; an identification unit configured to identify one or more local regions of interest that include the detected plurality of spots; a computing unit configured to identify metadata defining each of the one or more regions of interest; a file unit configured to create one or more image files and to store said one or more image files; Including, The one or more image files include image data for each of the one or more local regions of interest and the identified metadata defining each of the one or more local regions of interest, the image data being part of the image.

6. The image acquisition unit is configured to acquire further images of the sample, and for each of the acquired further images, the file unit: creating one or more further image files, each of the one or more further image files including further image data for each of the one or more further regions of interest and further metadata defining each of the one or more further local regions of interest; Creating a final file that includes the image data for each of the one or more image files and the other image data for the one or more other image files. The apparatus of claim 5 , configured to:

7. the image acquisition unit is configured to acquire a first image, a second image, and a third image; the identification unit is configured to identify a background of the second image based on the one or more other regions of interest identified for each of the first image, the second image, and the third image; the identification unit is configured to subtract the background of the second image; The apparatus of claim 6 , wherein the file unit is configured to create a background-less image file that includes image data of the second image, without the background.

8. 8. A method according to any one of claims 5 to 7, wherein each of the one or more local regions has a shape that is a polygonal shape and / or an elliptical shape, and the metadata defines the shape and position of the one or more local regions within the image.

Images