Systems and methods for image processing and adaptive control in microscope systems

Abstract

The present disclosure is directed to a method for hardware accelerated processing in a microscope system. The disclosed method includes receiving an image from an imaging module and computationally processing the image. The computational processing includes determining a detection set of candidate emitter pixels, and further determining a localized set of coordinates for each candidate emitter pixel. The localized set of coordinates is then provided to a control module for facilitating adaptive control of the microscope system.

Description

[0001] SYSTEMS AND METHODS FOR IMAGE PROCESSING AND ADAPTIVE CONTROL IN MICROSCOPE SYSTEMS

[0002] PRIORITY

[0003] This application claims the benefit under 35 U.S.C. § 119(e) of U.S. Provisional Patent Application No. 63 / 549,226, filed 02 February 2024, which is incorporated herein by reference.

[0004] TECHNICAL FIELD

[0005] The present disclosure is directed to facilitating adaptive control, improved imaging and illumination conditions, as well as rapid and efficient data processing and storage for high throughput microscope systems.

[0006] BACKGROUND

[0007] The movement of proteins, within the crowded environment of living cells are profoundly influenced by interactions with their surroundings. Single molecule tracking (SMT) is one method for capturing protein movement as a reporter of activity. In SMT, a protein (e.g., a fluorescent protein) of interest is imaged at high spatiotemporal resolution to track its movement in a complex system, e.g., a live cell. The information embedded in these tracks has been used to investigate diverse cellular phenomena including protein-protein interactions, e.g., interactions mediating signal transduction, inter-organelle communication, nuclear organization, and transcription regulation. Adaptive feedback control and / or suitable data pipelining in SMT applications pose significant technical challenges for implementation in microscopy systems.

[0008] SUMMARY OF THE INVENTION

[0009] In certain aspects, the compositions, systems, and methods described herein relate to systems and methods for image processing and / or adaptive control relating to microscope systems.

[0010] In particular embodiments, a method for hardware accelerated processing in a microscope system is disclosed, the method including receiving, by a hardware acceleration module of the microscope system, a first image of a sequence of images from an imaging module, each image respectively including a digital representation of a corresponding spatially distributed set of emitters located at a sample plane of a microscope module; computationally processing, by the hardware acceleration module, the first image, including: determining, based on performing one or more local filtering operations on the first image, a detection set of candidate emitter pixels corresponding to the respective set of emitters; and determining, based on performing a localizing computation for each candidate emitter pixel of the detection set, a localized set of coordinates corresponding to respective locations of the set of emitters, each of the localized set of coordinates having a sub-pixel resolution; providing, by the hardware acceleration module, the localized set of coordinates corresponding to the set of emitters of the first image to a control module of the microscope system; and receiving, by the hardware acceleration module, a second image of the sequence of images, wherein a processing time interval between processing the first image and processing the second image is less than or equal to a sampling time interval of the imaging module, thereby permitting adaptive control for single molecule tracking by the microscope system.

[0011] In particular embodiments, which may combine the features of some or all of the above embodiments, a computing module of the microscope system includes one or more computing processors or cores, and wherein the computing module includes the hardware acceleration module. In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module includes a Field Programmable Gate Array (FPGA). In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module includes a Graphics Processing Unit (GPU). In particular embodiments, which may combine the features of some or all of the above embodiments, computing module includes a Central Processing Unit (CPU).

[0012] In particular embodiments, which may combine the features of some or all of the above embodiments, a method for hardware accelerated processing in a microscope system further comprises determining a signal-to-noise ratio for each emitter of the set of emitters; and providing one or more of the determined signal-to-noise ratios to the control module of the microscope system.

[0013] In particular embodiments, which may combine the features of some or all of the above embodiments, computationally processing the first image further includes, prior to determining the localized set of coordinates: extracting, by the hardware acceleration module, a set of image fragments associated with the detection set, each image fragment including a plurality of pixels proximal to a respective candidate emitter pixel; and determining, by the hardware acceleration module, a set of filter values for each image fragment associated with the detection set, wherein each set of filter values is associated with a corresponding set of filters and includes a reconstructible representation of the corresponding image fragment. In particular embodiments, which may combine the features of some or all of the above embodiments, one or more of the sets of filters associated with the detection set comprise orthogonal filters. In particular embodiments, which may combine the features of some or all of the above embodiments, the localized set of coordinates is determined based on applying a feed-forward neural network to the set of image fragments of the detection set, the feed-forward neural network trained based on one or more features of the set of filters. In particular embodiments, which may combine the features of some or all of the above embodiments, the localized set of coordinates is determined based on applying a feed-forward neural network to the set of filter values associated with the respective image fragment.

[0014] In particular embodiments, which may combine the features of some or all of the above embodiments, computationally processing the first image further comprises extracting, by the hardware acceleration module, a set of image fragments associated with the detection set, each image fragment comprising a plurality of pixels proximal to a respective candidate emitter pixel; and performing, by the hardware acceleration module, an iterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates.

[0015] In particular embodiments, which may combine the features of some or all of the above embodiments, computationally processing the first image further comprises extracting, by the hardware acceleration module, a set of image fragments associated with the detection set, each image fragment comprising a plurality of pixels proximal to a respective candidate emitter pixel; and performing, by the hardware acceleration module, a non-iterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates.

[0016] In particular embodiments, which may combine the features of some or all of the above embodiments, the at least a part of each image fragment comprises a plurality of discretized positions, and the non-iterative optimization of the cost metric comprises non- iteratively optimizing an associated value that is evaluated at every position of the plurality of discretized positions.

[0017] In particular embodiments, which may combine the features of some or all of the above embodiments, optimizing the cost metric is associated with optimizing a position of a localization kernel. In particular embodiments, which may combine the features of some or all of the above embodiments, the localization kernel comprises a Gaussian kernel.

[0018] In particular embodiments, which may combine the features of some or all of the above embodiments, determining the detection set of candidate emitter pixels further includes: determining, based on applying a first one-dimensional filter along a first dimension of the first image and applying a second one-dimensional filter along a second dimension of the first image, a filtering set of candidate emitter pixels corresponding to the respective set of emitters; extracting, based on determining the filtering set, a corresponding set of image fragments, each image fragment including a plurality of pixels proximal to a respective candidate emitter pixel of the filtering set, the plurality of pixels of each image fragment distributed along the first dimension and the second dimension of the first image; and determining, based on applying one or more two-dimensional filters along the first dimension and the second dimension of each image fragment, the detection set of candidate emitter pixels corresponding to the respective set of emitters, wherein determining the filtering set of candidate emitter pixels prior to application of the one or more two- dimensional filters and prior to performing the localizing computation facilitates low latency computational processing by the hardware acceleration module, thereby permitting adaptive feedback control of the microscope system for single molecule tracking.

[0019] In particular embodiments, which may combine the features of some or all of the above embodiments, the localized set of coordinates is determined based on a radial symmetry of each emitter of the set of emitters. In particular embodiments, which may combine the features of some or all of the above embodiments, applying the first onedimensional filter along the first dimension of the first image is simultaneously performed with applying the second one-dimensional filter along the second dimension of the first image. In particular embodiments, which may combine the features of some or all of the above embodiments, a plurality of the one or more two-dimensional filters are simultaneously applied along each image fragment during the computational processing by the hardware acceleration module. In particular embodiments, which may combine the features of some or all of the above embodiments, the first dimension is orthogonal to the second dimension.

[0020] In particular embodiments, which may combine the features of some or all of the above embodiments, a method further includes pre-conditioning, prior to determining the detection set of candidate emitter pixels, the first image received by the hardware acceleration module. In particular embodiments, which may combine the features of some or all of the above embodiments, the pre-conditioning includes removing, by the hardware acceleration module, at least a part of fixed pattern noise associated with the first image. In particular embodiments, which may combine the features of some or all of the above embodiments, the pre-conditioning includes normalizing, by the hardware acceleration module, a respective pixel value associated with each pixel of the first image. In particular embodiments, which may combine the features of some or all of the above embodiments, the pre-conditioning includes performing, by the hardware acceleration module, a denoising operation on the first image. In particular embodiments, which may combine the features of some or all of the above embodiments, pre-conditioning includes performing, by the hardware acceleration module and based on a reference image, a background subtraction operation on the first image.

[0021] In particular embodiments, which may combine the features of some or all of the above embodiments, a method further includes filtering, prior to determining the localized set of coordinates, the detection set of candidate emitter pixels based on one or more filtering criteria. In particular embodiments, which may combine the features of some or all of the above embodiments, the filtering criteria includes a threshold value of a log likelihood ratio. In particular embodiments, which may combine the features of some or all of the above embodiments, the filtering criteria includes a threshold value of a signal-to-noise ratio. In particular embodiments, which may combine the features of some or all of the above embodiments, the filtering criteria include one or more peak searching criteria. In particular embodiments, which may combine the features of some or all of the above embodiments, memory buffering is used to facilitate consistency of a mean processing time interval of the hardware acceleration module.

[0022] In particular embodiments, which may combine the features of some or all of the above embodiments, a method further includes performing an adaptive focusing operation by the control module of the microscope system. In particular embodiments, which may combine the features of some or all of the above embodiments, a method further included: determining a focusing score generated by an autofocus module of the imaging module; and operating, by the control module and based on optimizing the focusing score, a focusing mechanism of the microscope module to obtain or maintain focus on at least a portion of the set of emitters located at the sample plane, for single molecule tracking. In particular embodiments, which may combine the features of some or all of the above embodiments, operating the focusing mechanism is associated with changing a distance between an objective and a stage of the microscope module, the stage associated with receiving a sample including emitters. In particular embodiments, which may combine the features of some or all of the above embodiments, a method further includes performing an adaptive imaging operation by the control module of the microscope system. In particular embodiments, which may combine the features of some or all of the above embodiments, a method further includes providing one or more signals to a laser engine module as an image exposure feedback operation, the one or more signals configured to modify an operational set point of a laser output to facilitate live-cell fluorescence imaging by the microscope system.

[0023] In particular embodiments, which may combine the features of some or all of the above embodiments, a method further includes providing one or more respective signals to each of a first laser configured for illuminating the sample plane and a second laser configured for controlling an emission of at least a portion of the set of emitters. In particular embodiments, which may combine the features of some or all of the above embodiments, a method further includes providing one or more signals to the imaging module as a field of view feedback operation, the one or more signals configured to modify a field of view associated with acquiring the sequence of images. In particular embodiments, which may combine the features of some or all of the above embodiments, the field of view is modified based on operating a scanning optical component or a translating optical component of the microscope system. In particular embodiments, which may combine the features of some or all of the above embodiments, the field of view is modified based on operating a galvo mirror of the microscope system. In particular embodiments, which may combine the features of some or all of the above embodiments, the field of view is modified based on altering one or more sensor parameters of the imaging module.

[0024] In particular embodiments, which may combine the features of some or all of the above embodiments, a method further includes providing one or more signals to the imaging module as a frame rate feedback operation, the one or more signals configured to modify a frame rate or an exposure time of acquiring the sequence of images. In particular embodiments, which may combine the features of some or all of the above embodiments, a method further includes, prior to or concurrent with modifying the frame rate, providing a signal to a laser engine module to modify an operational set point of a laser output. In particular embodiments, which may combine the features of some or all of the above embodiments, a method further includes performing an adaptive screening operation by the control module of the microscope system.

[0025] In particular embodiments, which may combine the features of some or all of the above embodiments, a method of computationally processing image data associated with a microscope system is disclosed, the processing performed by a computing module including one or more processors, the method including: receiving, by the computing module, a first image of a sequence of images, the first image including a digital representation of a corresponding spatially distributed set of emitters located at a sample plane of a microscope module; determining, by the computing module, a detection set of candidate emitter pixels corresponding to the respective set of emitters; extracting, by the computing module, a set of image fragments associated with the detection set, each image fragment including a plurality of pixels proximal to a respective candidate emitter pixel; determining, by the computing module, a set of filter values for each image fragment associated with the detection set, wherein each set of filter values is associated with a corresponding set of filters and includes a reconstructible representation of the corresponding image fragment; and providing, by the computing module, one or more of the sets of filter values associated with the first image for storage in a data storage module associated with the microscope system.

[0026] In particular embodiments, which may combine the features of some or all of the above embodiments, one or more of the sets of filters associated with the detection set comprise orthogonal filters. In particular embodiments, which may combine the features of some or all of the above embodiments, a method further includes determining, by the computing module, a localized set of coordinates based on determining each set of filter values, the localized set of coordinates corresponding to respective locations of the set of emitters, each of the localized set of coordinates having a sub-pixel resolution.

[0027] In particular embodiments, which may combine the features of some or all of the above embodiments, a method further includes performing, by the computing module, an iterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates. In particular embodiments, which may combine the features of some or all of the above embodiments, a method further includes performing, by the computing module, a non-iterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates. In particular embodiments, which may combine the features of some or all of the above embodiments, the at least a part of each image fragment comprises a plurality of discretized positions, and the non-iterative optimization of the cost metric comprises non- iteratively optimizing an associated value that is evaluated at every position of the plurality of discretized positions.

[0028] In particular embodiments, which may combine the features of some or all of the above embodiments, optimizing the cost metric is associated with optimizing a position of a localization kernel. In particular embodiments, which may combine the features of some or all of the above embodiments, the localization kernel comprises a Gaussian kernel.

[0029] In particular embodiments, which may combine the features of some or all of the above embodiments, a method further includes providing, by the computing module, the localized set of coordinates to a control module of the microscope system to facilitate performing one or more adaptive feedback operations. In particular embodiments, which may combine the features of some or all of the above embodiments, the localized set of coordinates is determined based on applying a feed-forward neural network to the set of filter values associated with the respective image fragment. In particular embodiments, which may combine the features of some or all of the above embodiments, the localized set of coordinates is determined based on applying a feed-forward neural network to the set of image fragments of the detection set, the feed-forward neural network trained based on one or more features of the set of filters. In particular embodiments, which may combine the features of some or all of the above embodiments, the localized set of coordinates is determined based on applying a feed-forward neural network to the set of filter values associated with the respective image fragment. In particular embodiments, which may combine the features of some or all of the above embodiments, a method further includes receiving, by the computing module, a second image of the sequence of images, wherein a time interval for computationally processing the first image prior to receiving the second image is less than or equal to a sampling time interval of an imaging module of the microscope system, thereby permitting adaptive control for single molecule tracking by the microscope system.

[0030] In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module includes a plurality of computing processors or cores. In particular embodiments, which may combine the features of some or all of the above embodiments, the plurality of computing processors or cores are heterogeneous. In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module includes a hardware accelerated processing module. In particular embodiments, which may combine the features of some or all of the above embodiments, the hardware accelerated processing module includes a Field Programmable Gate Array (FPGA).

[0031] In particular embodiments, which may combine the features of some or all of the above embodiments, a microscope system includes: a microscope module including a light source optically connected for illuminating a sample plane; and an imaging module including a detector device and configured to acquire a sequence of images, each image respectively including a digital representation of a spatially distributed set of emitters located at the sample plane during a corresponding sampling time interval; a computing module including a hardware acceleration module configured to: receive a first image of the sequence of images from the imaging module; determine a detection set of candidate emitter pixels corresponding to the respective set of emitters based on performing one or more local filtering operations on the first image; determine a localized set of coordinates corresponding to the respective set of emitters based on performing a localizing computation for each candidate emitter pixel of the detection set, each of the localized set of coordinates having a sub-pixel resolution; and receive a second image of the sequence of images from the imaging module; and a control module configured to receive the localized set of coordinates from the hardware acceleration module and perform an adaptive feedback operation associated with the microscope system, wherein a processing time interval between determining the respective localized sets of coordinates corresponding to the first image and the second image is less than or equal to the sampling time interval of the imaging module, thereby permitting adaptive control for single molecule tracking by the microscope system.

[0032] In particular embodiments, which may combine the features of some or all of the above embodiments, a microscope system further includes a computing module including one or more computing processors or cores. In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module includes a Field Programmable Gate Array (FPGA). In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module includes a Graphics Processing Unit (GPU). In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module includes a Central Processing Unit (CPU).

[0033] In particular embodiments, which may combine the features of some or all of the above embodiments, the hardware acceleration module in a microscope system is further configured to determine a signal-to-noise ratio for each emitter of the set of emitters; and provide one or more of the determined signal-to-noise ratios to the control module of the microscope system.

[0034] In particular embodiments, which may combine the features of some or all of the above embodiments, wherein prior to determining the localized set of coordinates, the hardware acceleration module is further configured to: extract a set of image fragments associated with the detection set, each image fragment including a plurality of pixels proximal to a respective candidate emitter pixel; and determine a set of filter values for each image fragment associated with the detection set, wherein each set of filter values is associated with a corresponding set of filters and includes a reconstructible representation of the corresponding image fragment. In particular embodiments, which may combine the features of some or all of the above embodiments, one or more of the sets of filters associated with the detection set comprise orthogonal filters. In particular embodiments, which may combine the features of some or all of the above embodiments, the localized set of coordinates is determined based on applying a feed-forward neural network to the set of filter values associated with the respective image fragments. In particular embodiments, which may combine the features of some or all of the above embodiments, the localized set of coordinates is determined based on applying a feed-forward neural network to the set of image fragments of the detection set, the feed-forward neural network trained based on one or more features of the set of filters. In particular embodiments, which may combine the features of some or all of the above embodiments, wherein the localized set of coordinates is determined based on applying a feed-forward neural network to the set of filter values associated with the respective image fragment.

[0035] In particular embodiments, which may combine the features of some or all of the above embodiments, the hardware acceleration module is further configured to extract a set of image fragments associated with the detection set, each image fragment comprising a plurality of pixels proximal to a respective candidate emitter pixel; and perform an iterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates. In particular embodiments, which may combine the features of some or all of the above embodiments, the hardware acceleration module is further configured to extract a set of image fragments associated with the detection set, each image fragment comprising a plurality of pixels proximal to a respective candidate emitter pixel; and perform a noniterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates.

[0036] In particular embodiments, which may combine the features of some or all of the above embodiments, the at least a part of each image fragment comprises a plurality of discretized positions, and the non-iterative optimization of the cost metric comprises non- iteratively optimizing an associated value that is evaluated at every position of the plurality of discretized positions. In particular embodiments, which may combine the features of some or all of the above embodiments, optimizing the cost metric is associated with optimizing a position of a localization kernel. In particular embodiments, which may combine the features of some or all of the above embodiments, the localization kernel comprises a Gaussian kernel.

[0037] In particular embodiments, which may combine the features of some or all of the above embodiments, wherein, for determining the detection set of candidate emitter pixels, the hardware acceleration module is further configured to: determine a filtering set of candidate emitter pixels corresponding to the respective set of emitters based on applying a first one-dimensional filter along a first dimension of the first image and applying a second one-dimensional filter along a second dimension of the first image; extract a corresponding set of image fragments based on determining the filtering set, each image fragment including a plurality of pixels proximal to a respective candidate emitter pixel of the filtering set, the plurality of pixels of each image fragment distributed along the first dimension and the second dimension of the first image; and determine the detection set of candidate emitter pixels corresponding to the respective set of emitters based on applying one or more two- dimensional filters along the first dimension and the second dimension of each image fragment, wherein determining the filtering set of candidate emitter pixels prior to application of the one or more two-dimensional filters and prior to performing the localizing computation facilitates low latency computational processing by the hardware acceleration module, thereby permitting adaptive feedback control of the microscope system for single molecule tracking. In particular embodiments, which may combine the features of some or all of the above embodiments, the localized set of coordinates is determined based on a radial symmetry of each emitter of the set of emitters. In particular embodiments, which may combine the features of some or all of the above embodiments, applying the first onedimensional filter along the first dimension of the first image is simultaneously performed with applying the second one-dimensional filter along the second dimension of the first image. In particular embodiments, which may combine the features of some or all of the above embodiments, a plurality of the one or more two-dimensional filters are simultaneously applied along each image fragment during the computational processing by the hardware acceleration module. In particular embodiments, which may combine the features of some or all of the above embodiments, prior to determining the detection set of candidate emitter pixels, the hardware acceleration module is further configured to precondition the first image. In particular embodiments, which may combine the features of some or all of the above embodiments, the pre-conditioning includes removing, by the hardware acceleration module, at least a part of fixed pattern noise associated with the first image. In particular embodiments, which may combine the features of some or all of the above embodiments, the pre-conditioning includes normalizing, by the hardware acceleration module, a respective pixel value associated with each pixel of the first image. In particular embodiments, which may combine the features of some or all of the above embodiments, the pre-conditioning includes performing, by the hardware acceleration module, a de-noising operation on the first image.

[0038] In particular embodiments, which may combine the features of some or all of the above embodiments, the pre-conditioning includes performing, by the hardware acceleration module, a background subtraction operation on the first image. In particular embodiments, which may combine the features of some or all of the above embodiments, wherein prior to determining the localized set of coordinates, the hardware acceleration module is further configured to filter the detection set of candidate emitter pixels based on one or more filtering criteria. In particular embodiments, which may combine the features of some or all of the above embodiments, the filtering criteria includes a threshold value of a log likelihood ratio. In particular embodiments, which may combine the features of some or all of the above embodiments, the filtering criteria includes a threshold value of a signal- to-noise ratio. In particular embodiments, which may combine the features of some or all of the above embodiments, the filtering criteria include one or more peak searching criteria. In particular embodiments, which may combine the features of some or all of the above embodiments, a microscope system further includes a memory buffer to facilitate consistency of a mean processing time interval of the hardware acceleration module.

[0039] In particular embodiments, which may combine the features of some or all of the above embodiments, the adaptive feedback operation performed by the control module includes an adaptive focusing operation. In particular embodiments, which may combine the features of some or all of the above embodiments, the adaptive focusing operation includes: determining a focusing score generated by an autofocus module of the imaging module; and operating, by the control module and based on optimizing the focusing score, a focusing mechanism of the microscope module to obtain or maintain focus on at least a portion of the set of emitters located at the sample plane, for single molecule tracking. In particular embodiments, which may combine the features of some or all of the above embodiments, operating the focusing mechanism is associated with changing a distance between an objective and a stage of the microscope module, the stage associated with receiving a sample including emitters. In particular embodiments, which may combine the features of some or all of the above embodiments, the adaptive feedback operation performed by the control module includes an adaptive imaging operation.

[0040] In particular embodiments, which may combine the features of some or all of the above embodiments, the adaptive imaging operation further includes providing one or more signals to a laser engine module as an image exposure feedback operation, the one or more signals configured to modify an operational set point of a laser output to facilitate live-cell fluorescence imaging by the microscope system. In particular embodiments, which may combine the features of some or all of the above embodiments, the adaptive imaging operation further includes providing one or more respective signals to each of a first laser configured for illuminating the sample plane and a second laser configured for controlling an emission of at least a portion of the set of emitters. In particular embodiments, which may combine the features of some or all of the above embodiments, the adaptive imaging operation further includes providing one or more signals to the imaging module as a field of view feedback operation, the one or more signals configured to modify a field of view associated with acquiring the sequence of images.

[0041] In particular embodiments, which may combine the features of some or all of the above embodiments, the field of view is modified based on operating a scanning optical component or a translating optical component of the microscope system. In particular embodiments, which may combine the features of some or all of the above embodiments, the field of view is modified based on operating a galvo mirror of the microscope system. In particular embodiments, which may combine the features of some or all of the above embodiments, the field of view is modified based on altering one or more sensor parameters of the imaging module.

[0042] In particular embodiments, which may combine the features of some or all of the above embodiments, the adaptive imaging operation further includes providing one or more signals to the imaging module as a frame rate feedback operation, the one or more signals configured to modify a frame rate or an exposure time of acquiring the sequence of images. In particular embodiments, which may combine the features of some or all of the above embodiments, the adaptive imaging operation further includes, prior to or concurrent with modifying the frame rate, providing a signal to a laser engine module to modify an operational set point of a laser output. In particular embodiments, which may combine the features of some or all of the above embodiments, the adaptive feedback operation performed by the control module includes an adaptive imaging operation.

[0043] In particular embodiments, which may combine the features of some or all of the above embodiments, a microscope system includes: a microscope module including: a light source optically connected for illuminating a sample plane; and an imaging module including a detector device and configured to acquire a sequence of images, each image respectively including a digital representation of a spatially distributed set of emitters located at the sample plane during a corresponding sampling time interval; a computing module configured to: receive a first image of the sequence of images from the imaging module; determine a detection set of candidate emitter pixels corresponding to the respective set of emitters; extract a set of image fragments associated with the detection set, each image fragment including a plurality of pixels proximal to a respective candidate emitter pixel; and determine a set of filter values for each image fragment associated with the detection set, wherein each set of filter values is associated with a corresponding set of filters and includes a reconstructible representation of the corresponding image fragment; and transmit one or more of the sets of filter values associated with the first image to one or more of a storage module or a control module. In particular embodiments, which may combine the features of some or all of the above embodiments, one or more of the sets of filters associated with the detection set comprise orthogonal filters. In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module is further configured to determine a localized set of coordinates based on determining each set of filter values, the localized set of coordinates corresponding to respective locations of the set of emitters, each of the localized set of coordinates having a sub-pixel resolution.

[0044] In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module is further configured to provide the localized set of coordinates to a control module of the microscope system to facilitate performing one or more adaptive feedback operations. In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module is further configured to determine a signal-to-noise ratio for each emitter of the set of emitters; and provide one or more of the determined signal-to-noise ratios to one or more of the storage module or the control module. In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module is further configured to determine the localized set of coordinates based on applying a feed-forward neural network to the set of filter values associated with the respective image fragment.

[0045] In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module is configured to determine the localized set of coordinates based on applying a feed-forward neural network to the set of image fragments of the detection set, the feed-forward neural network trained based on one or more features of the set of filters.

[0046] In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module is further configured to perform an iterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates. In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module is further configured to perform a non-iterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates.

[0047] In particular embodiments, which may combine the features of some or all of the above embodiments, the at least a part of each image fragment comprises a plurality of discretized positions, and the non-iterative optimization of the cost metric comprises non- iteratively optimizing an associated value that is evaluated at every position of the plurality of discretized positions. In particular embodiments, which may combine the features of some or all of the above embodiments, optimizing the cost metric is associated with optimizing a position of a localization kernel. In particular embodiments, which may combine the features of some or all of the above embodiments, the localization kernel comprises a Gaussian kernel.

[0048] In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module is configured to determine the localized set of coordinates based on applying a feed-forward neural network to the set of filter values associated with the respective image fragment. In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module is configured to receive a second image of the sequence of images, wherein a time interval for computationally processing the first image prior to receiving the second image is less than or equal to a sampling time interval of an imaging module of the microscope system, thereby permitting adaptive control for single molecule tracking by the microscope system.

[0049] In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module includes a plurality of computing processors or cores. In particular embodiments, which may combine the features of some or all of the above embodiments, the plurality of computing processors or cores are heterogeneous. In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module includes a hardware accelerated processing module. In particular embodiments, which may combine the features of some or all of the above embodiments, the hardware accelerated processing module includes a Field Programmable Gate Array (FPGA).

[0050] In particular embodiments, which may combine the features of some or all of the above embodiments, a computing module for a microscope system is disclosed, the computing module including a hardware acceleration module configured to: receive a first image of a sequence of images from an imaging module, each image of the sequence of images respectively including a digital representation of a spatially distributed set of emitters located at a sample plane of the microscope system during a corresponding sampling time interval; determine a detection set of candidate emitter pixels corresponding to the respective set of emitters based on performing one or more local filtering operations on the first image; determine a localized set of coordinates corresponding to the respective set of emitters based on performing a localizing computation for each candidate emitter pixel of the detection set, each of the localized set of coordinates having a sub-pixel resolution; and receive a second image of the sequence of images from the imaging module, wherein a processing time interval between determining the respective localized sets of coordinates corresponding to the first image and the second image is less than or equal to the sampling time interval of the imaging module, thereby permitting adaptive control for single molecule tracking by the microscope system.

[0051] In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module includes one or more computing processors or cores.

[0052] In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module includes a Field Programmable Gate Array (FPGA). In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module includes a Graphics Processing Unit (GPU). In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module includes a Central Processing Unit (CPU).

[0053] In particular embodiments, which may combine the features of some or all of the above embodiments, the hardware acceleration module is further configured to: determine a signal-to-noise ratio for each emitter of the set of emitters; and provide one or more of the determined signal-to-noise ratios to a control module of the microscope system.

[0054] In particular embodiments, which may combine the features of some or all of the above embodiments, prior to determining the localized set of coordinates, the hardware acceleration module is further configured to: extract a set of image fragments associated with the detection set, each image fragment including a plurality of pixels proximal to a respective candidate emitter pixel; and determine a set of filter values for each image fragment associated with the detection set, wherein each set of filter values is associated with a corresponding set of filters and includes a reconstructible representation of the corresponding image fragment. In particular embodiments, which may combine the features of some or all of the above embodiments, one or more of the sets of filters associated with the detection set comprise orthogonal filters.

[0055] In particular embodiments, which may combine the features of some or all of the above embodiments, the localized set of coordinates is determined based on applying a feedforward neural network to the set of filter values associated with the respective image fragments. In particular embodiments, which may combine the features of some or all of the above embodiments, the localized set of coordinates is determined based on applying a feed- forward neural network to the set of image fragments of the detection set, the feed-forward neural network trained based on one or more features of the set of filters. In particular embodiments, which may combine the features of some or all of the above embodiments, the localized set of coordinates is determined based on applying a feed-forward neural network to the set of filter values associated with the respective image fragment.

[0056] In particular embodiments, which may combine the features of some or all of the above embodiments, the hardware acceleration module is further configured to extract a set of image fragments associated with the detection set, each image fragment comprising a plurality of pixels proximal to a respective candidate emitter pixel; and perform an iterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates.

[0057] In particular embodiments, which may combine the features of some or all of the above embodiments, the hardware acceleration module is further configured to extract a set of image fragments associated with the detection set, each image fragment comprising a plurality of pixels proximal to a respective candidate emitter pixel; and perform a noniterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates.

[0058] In particular embodiments, which may combine the features of some or all of the above embodiments, the at least a part of each image fragment comprises a plurality of discretized positions, and the non-iterative optimization of the cost metric comprises non- iteratively optimizing an associated value that is evaluated at every position of the plurality of discretized positions. In particular embodiments, which may combine the features of some or all of the above embodiments, optimizing the cost metric is associated with optimizing a position of a localization kernel. In particular embodiments, which may combine the features of some or all of the above embodiments, the localization kernel comprises a Gaussian kernel.

[0059] In particular embodiments, which may combine the features of some or all of the above embodiments, the hardware acceleration module is further configured to: determine a filtering set of candidate emitter pixels corresponding to the respective set of emitters based on applying a first one-dimensional filter along a first dimension of the first image and applying a second one-dimensional filter along a second dimension of the first image; extract a corresponding set of image fragments based on determining the filtering set, each image fragment including a plurality of pixels proximal to a respective candidate emitter pixel of the filtering set, the plurality of pixels of each image fragment distributed along the first dimension and the second dimension of the first image; and determine the detection set of candidate emitter pixels corresponding to the respective set of emitters based on applying one or more two-dimensional filters along the first dimension and the second dimension of each image fragment, wherein determining the filtering set of candidate emitter pixels prior to application of the one or more two-dimensional filters and prior to performing the localizing computation facilitates low latency computational processing by the hardware acceleration module, thereby permitting adaptive feedback control of the microscope system for single molecule tracking.

[0060] In particular embodiments, which may combine the features of some or all of the above embodiments, the localized set of coordinates is determined based on a radial symmetry of each emitter of the set of emitters. In particular embodiments, which may combine the features of some or all of the above embodiments, applying the first onedimensional filter along the first dimension of the first image is simultaneously performed with applying the second one-dimensional filter along the second dimension of the first image. In particular embodiments, which may combine the features of some or all of the above embodiments, a plurality of the one or more two-dimensional filters are simultaneously applied along each image fragment during the computational processing by the hardware acceleration module.

[0061] In particular embodiments, which may combine the features of some or all of the above embodiments, prior to determining the detection set of candidate emitter pixels, the hardware acceleration module is further configured to pre-condition the first image. In particular embodiments, which may combine the features of some or all of the above embodiments, the pre-conditioning includes removing, by the hardware acceleration module, at least a part of fixed pattern noise associated with the first image. In particular embodiments, which may combine the features of some or all of the above embodiments, the pre-conditioning includes normalizing, by the hardware acceleration module, a respective pixel value associated with each pixel of the first image.

[0062] In particular embodiments, which may combine the features of some or all of the above embodiments, the pre-conditioning includes performing, by the hardware acceleration module, a de-noising operation on the first image. In particular embodiments, which may combine the features of some or all of the above embodiments, the pre- conditioning includes performing, by the hardware acceleration module, a background subtraction operation on the first image.

[0063] In particular embodiments, which may combine the features of some or all of the above embodiments, prior to determining the localized set of coordinates, the hardware acceleration module is further configured to filter the detection set of candidate emitter pixels based on one or more filtering criteria. In particular embodiments, which may combine the features of some or all of the above embodiments, the filtering criteria includes a threshold value of a log likelihood ratio. In particular embodiments, which may combine the features of some or all of the above embodiments, the filtering criteria includes a threshold value of a signal-to-noise ratio. In particular embodiments, which may combine the features of some or all of the above embodiments, the filtering criteria include one or more peak searching criteria.

[0064] In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module further includes a memory buffer to facilitate consistency of a mean processing time interval of the hardware acceleration module.

[0065] In particular embodiments, which may combine the features of some or all of the above embodiments, a computing module for a microscope system is disclosed, the computing module configured to: receive a first image of a sequence of images from an imaging module, each image of the sequence of images respectively including a digital representation of a spatially distributed set of emitters located at a sample plane of the microscope system during a corresponding sampling time interval; determine a detection set of candidate emitter pixels corresponding to the respective set of emitters; extract a set of image fragments associated with the detection set, each image fragment including a plurality of pixels proximal to a respective candidate emitter pixel; determine a set of filter values for each image fragment associated with the detection set, wherein each set of filter values is associated with a corresponding set of filters and includes a reconstructible representation of the corresponding image fragment; and transmit one or more of the sets of filter values associated with the first image to one or more of a storage module or a control module. In particular embodiments, which may combine the features of some or all of the above embodiments, one or more of the sets of filters associated with the detection set comprise orthogonal filters.

[0066] In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module is further configured to determine a localized set of coordinates based on determining each set of filter values, the localized set of coordinates corresponding to respective locations of the set of emitters, each of the localized set of coordinates having a sub-pixel resolution. In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module is further configured to provide the localized set of coordinates to a control module of the microscope system to facilitate performing one or more adaptive feedback operations.

[0067] In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module is further configured to: determine a signal-to- noise ratio for each emitter of the set of emitters; and provide one or more of the determined signal-to-noise ratios to one or more of the storage module or the control module. In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module is configured to determine the localized set of coordinates based on applying a feed-forward neural network to the set of filter values associated with the respective image fragment. In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module is configured to determine the localized set of coordinates based on applying a feed-forward neural network to the set of image fragments of the detection set, the feed-forward neural network trained based on one or more features of the set of filters. In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module is configured to determine the localized set of coordinates based on applying a feed-forward neural network to the set of filter values associated with the respective image fragment.

[0068] In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module is further configured to perform an iterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates. In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module is further configured to perform a non-iterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates.

[0069] In particular embodiments, which may combine the features of some or all of the above embodiments, the at least a part of each image fragment comprises a plurality of discretized positions, and the non-iterative optimization of the cost metric comprises non- iteratively optimizing an associated value that is evaluated at every position of the plurality of discretized positions. In particular embodiments, which may combine the features of some or all of the above embodiments, optimizing the cost metric is associated with optimizing a position of a localization kernel. In particular embodiments, which may combine the features of some or all of the above embodiments, the localization kernel comprises a Gaussian kernel.

[0070] In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module is configured to receive a second image of the sequence of images, wherein a time interval for computationally processing the first image prior to receiving the second image is less than or equal to a sampling time interval of the imaging module of the microscope system, thereby permitting adaptive control for single molecule tracking by the microscope system. In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module includes a plurality of computing processors or cores. In particular embodiments, which may combine the features of some or all of the above embodiments, the plurality of computing processors or cores are heterogeneous. In particular embodiments, which may combine the features of some or all of the above embodiments, the computing module includes a hardware accelerated processing module. In particular embodiments, which may combine the features of some or all of the above embodiments, the hardware accelerated processing module includes a Field Programmable Gate Array (FPGA).

[0071] BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The figures provided herein may be illustrated schematically rather than literally or precisely; components and aspects of the figures may also not necessarily be to scale. Moreover, while like reference numerals may designate corresponding parts throughout the different views in many cases, like parts may not always be provided with like reference numerals in each view.

[0073] FIG. 1 illustrates a schematic view of an exemplary microscope system, according to particular embodiments of the present disclosure.

[0074] FIG. 2 illustrates a schematic view of an exemplary microscope module of a microscope system, according to particular embodiments.

[0075] FIG. 3 provides a schematic representation of a sample plate, according to particular embodiments.

[0076] FIG. 4A illustrates a schematic view of an exemplary set of pixels associated with a detection process, according to particular embodiments.

[0077] FIG. 4B illustrates a schematic view of sub-pixel localization, according to particular embodiments.

[0078] FIG. 4C illustrates a schematic view of sub-pixel localization based on non-iterative optimization, according to particular embodiments.

[0079] FIG. 5A illustrates a schematic of a processing timeline of exemplary image processing steps, according to particular embodiments.

[0080] FIG. 5B illustrates a schematic of an exemplary method for hardware accelerated processing in a microscope system, according to particular embodiments.

[0081] FIG. 6A illustrates a visual depiction and respective variances of an exemplary set of orthogonal filters, according to particular embodiments.

[0082] FIG. 6B illustrates a schematic process flow for detection and localization, according to particular embodiments.

[0083] FIG. 7 illustrates a set of image pairs depicting image reconstruction based on sparse representation, according to particular embodiments.

[0084] FIG. 8A illustrates schematic process flow steps for neural network-based inference and exemplary architecture, according to particular embodiments.

[0085] FIG. 8B illustrates a schematic neural network training procedure depicting exemplary unlabeled images, training sets, and localization errors, according to particular embodiments. FIG. 8C illustrates comparison results of neural network-based subpixel localization procedures, according to particular embodiments.

[0086] FIG. 8D illustrates detection performance results comparing hardware-accelerated and software-based implementations, according to particular embodiments.

[0087] FIG. 9 illustrates an example computer-implemented environment in connection with the subject matter described herein.

[0088] FIG. 10 is a diagram illustrating a sample computing device architecture for implementing various aspects described herein.

[0089] FIG. 11 illustrates a schematic architecture of a Field Programmable Gate Array (FPGA), according to particular embodiments.

[0090] FIG. 12 illustrates a schematic parallelized detection process, according to particular embodiments.

[0091] DETAILED DESCRIPTION

[0092] For purposes of clarity of disclosure and not by way of limitation, the detailed description is divided into the following subsections:

[0093] 1. Definitions

[0094] 2. Microscope System Overview

[0095] 2.1 Microscope Module

[0096] 2.1.1 Light Source

[0097] 2.1.2 Microscope Apparatus

[0098] 2.1.3 Sample

[0099] 2.1.4 Imaging Module

[0100] 2.2 Computing Module

[0101] 2.2.1 Image Processing Methods

[0102] 2.2.2 Computer Hardware and Software Overview

[0103] 2.3 Storage Module

[0104] 2.4 Microscope and Platform Control Module

[0105] 3. Examples

[0106] 3.1 Example 1

[0107] 3.2 Example 2

[0108] 3.3 Example 3

[0109] 3.4 Example 4 3.5 Example 5

[0110] 1. Definitions

[0111] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. In case of conflict, the present document, including definitions, will control. Exemplary methods and materials are described below, although methods and materials similar or equivalent to those described herein can be used in practice or testing of the presently disclosed subject matter. All publications, patent applications, patents and other references mentioned herein are incorporated by reference in their entirety. The materials, methods, and examples disclosed herein are illustrative only and not intended to be limiting.

[0112] The terms “comprise(s),” “include(s),” “having,” “has,” “can,” “contain(s),” and variants thereof, as used herein, are intended to be open-ended transitional phrases, terms, or words that do not preclude the possibility of additional acts or structures. The singular forms “a,” “an” and “the” include plural references unless the context clearly dictates otherwise. The present disclosure also contemplates other instances “comprising,” “consisting of’, and “consisting essentially of,” the instances or elements presented herein, whether explicitly set forth or not. The use of “or” can include “and,” unless contextually precluded or otherwise mentioned.

[0113] For the recitation of numeric ranges herein, each intervening number within the range is explicitly contemplated with the same degree of precision. For example, for the range of 6-9, the numbers 7 and 8 are contemplated in addition to 6 and 9, and for the range 6.0-7.0, the number 6.0, 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7, 6.8, 6.9, and 7.0 are explicitly contemplated.

[0114] As used herein, the term “about” or “approximately” means within an acceptable error range for the particular value as determined by one of ordinary skill in the art, which will depend in part on how the value is measured or determined, i.e., the limitations of the measurement system. For example, “about” can mean within 3 or more than 3 standard deviations, per the practice in the art. Alternatively, “about” can mean a range of up to 20%, preferably up to 10%, more preferably up to 5%, and more preferably still up to 1% of a given value. Alternatively, particularly with respect to biological systems or processes, the term can mean within an order of magnitude, preferably within 5-fold, and more preferably within 2-fold, of a value. 2. Microscope System Overview

[0115] FIG. 1 illustrates a schematic view of an exemplary microscope system, according to particular embodiments of the present disclosure.

[0116] In particular embodiments, microscope system 100 may comprise a microscope module 110 and a control module 120 for microscope and platform control. As will be further described herein, in particular embodiments, microscope module 110 may be configured to illuminate a sample provided in a sample plane. In particular embodiments, an imaging module of microscope module 110 may be configured to capture one or more images based on light received from an illuminated sample plane. In particular embodiments, an imaging module of microscope module 110 may be configured to capture a movie, such as comprising a sequence of images, based on light received from an illuminated sample plane. In particular embodiments, data associated with and / or obtained from the operation of microscope module 110 may be transferred to a storage module 130 for storage. As used herein, data may comprise image data and / or movies obtained by microscope module 110. By way of example and not limitation, one or more images and / or movies captured by an imaging module of microscope module 110 may be transferred to storage module 130 for storage.

[0117] In particular embodiments, microscope and platform control module 120 may provide control of parameters and / or operation of microscope module 110. In particular embodiments, microscope system 100 may separately or additionally comprise a computing module 140. In particular embodiments, computing module 140 may be configured for processing data received from microscope module 110. Separately or additionally, in particular embodiments, computing module 140 may be configured to control data flow to and from storage module 130. Separately or additionally, in particular embodiments, computing module 140 may be configured to interface with microscope and platform control module 120. By way of example and not limitation, based on receiving and processing images from microscope module 110, computing module 140 may be configured to transmit particular unprocessed images and / or processed information to storage module 130. Separately or additionally, in particular embodiments, computing module 140 may be configured to operate microscope and platform control module 120 to vary specific operational parameters associated with microscope module 110, and / or otherwise adaptively control microscope module 110. By way of example and not limitation, as will be described further, computing module 140 may be configured to adaptively vary particular operational parameters of microscope module 110 based on computationally processing a sequence of images received from an imaging module of microscope module 110.

[0118] 2.1 Microscope Module

[0119] FIG. 2 illustrates a schematic view of an exemplary microscope module, according to particular embodiments. In particular embodiments, a microscope module 110 of a microscopy system may comprise a light source 210, microscopy apparatus 220, sample 230, and / or an imaging module 240. By way of example and not limitation, light source 210 may comprise a laser source. By way of example and not limitation, microscopy apparatus 220 may comprise an optical light path and / or an optical train configured to control, modify, and / or convey light from light source 210 to sample 230. In particular embodiments, microscopy apparatus 220 may be separately or additionally used to control, modify, and / or convey light from sample 230 to imaging module 240.

[0120] 2.1.1 Light Source

[0121] In particular embodiments, the system can comprise a light source 210 configured to emit light. The light source 210, in certain implementations of the microscope module disclosed herein, can be configured to emit light of a single wavelength. In certain implementations of the microscope module disclosed herein, the light source 210 can be configured to emit light of two, three, four, five, or more individual wavelengths. In certain implementations, the wavelength(s) of light emitted by the light source are predetermined. For example, but not by way of limitation, the wavelength(s) can be predetermined such that the emitted light elicits fluorescence emission when illuminating a sample, e.g., a sample comprising a fluorescent protein. In certain non-limiting implementations, the light source 210 can be configured to comprise three lasers, such as with nominal central wavelengths 405 nm, 560 nm, 640 nm, that could vary within absorption band of the fluorophores used. In certain instances the 405 nm wavelength is used to excite Hoechst dye. In certain instances, a 560 nm wavelength is used to excite dyes (e.g., JF549) attached to HaloTag.

[0122] In certain non -limiting implementations, light source 210 can be used to catalyze photochemical reactions. For example, but not by way of limitation, the wavelength(s) and illumination intensities can be such that cleavage of a chemical bond occurs. As an additional example, but not by way of limitation, the wavelength(s) and illumination intensities may induce the adoption of a non-radiative dark state (i.e., “photobleached molecule”). As an additional example, but not by way of limitation, the wavelength(s) and illumination intensities may induce radiative or non-radiative energy transfer between fluorophores within the sample.

[0123] In certain implementations of exemplary microscope modules described herein, light source 210 can be configured to deliver a predetermined amount of power to the back focal plane of an objective. In certain implementations of exemplary microscope modules described herein, light source 210 can be configured to emit pulsed light. For example, but not by way of limitation, light source 210 can be configured to emit stroboscopic pulsed light. In certain implementations of exemplary microscope modules described herein, the light source 210 can be configured to emit pulsed light in synchrony with the start of image acquisition by imaging module 240. In certain, non-limiting implementations, light source 210 can pulse at specific time intervals, such as depending on the number of frames per second being captured. For example, but not by way of limitation, if 100 Frames Per Second (FPS) are being captured by a detector of imaging module 240, the laser may be ON for 9 ms and OFF for 1 ms. In contrast, by way of non-limiting example, in 200 FPS mode, the laser may be ON for 4 ms and OFF for 1 ms. In particular embodiments, the emission of light by light source 210 and the direction of that light to exemplary microscope modules disclosed herein, can be facilitated using a single mode fiber. In particular embodiments, one or more control parameters of light source 210 are controllable by control module 120 in a stand-alone mode, and / or by control module 120 in communication with computing module 140.

[0124] 2.1.2 Microscope Apparatus

[0125] In particular embodiments, a light source 210, such as a fluorescence excitation light source, may provide a light beam to a one or more optical elements or assemblies of microscopy apparatus 220, which may be configured to provide a light beam to further optical elements and assemblies. In particular embodiments, the light beam may be optically provided to an objective, and further used to illuminate a sample plane. In particular embodiments, a light beam may be collimated by particular one or more optical elements within the microscopy apparatus 220.

[0126] In particular embodiments, one or more scanning element may be configured to selectively scan the light beam. As non-limiting examples, a scanning element may comprise one or more galvo mirrors, one-dimensional galvo mirrors, polygonic mirrors and / or scanners, micro mirrors, microelectromechanical (MEMS) scanning mirrors, optical modulators, and / or piezo elements. Additionally, or alternatively, optical elements or assemblies of microscopy apparatus 220 configured to translate or otherwise modify a light beam may comprise one or more actuators and / or motors, and / or may be computer- controlled, such as by control module 120.

[0127] In certain embodiments, one or more optical forming elements may be provided in microscopy apparatus 220 and configured to produce one or more light beams of particular shape(s). In particular embodiments, it is possible to enable the location of one or more scanning elements prior to, or upstream of, particular additional optical elements that may encountered by a light beam, i.e., before corresponding additional operations or manipulations are performed on the light beam.

[0128] In particular embodiments, an optical forming element may produce a linear (or linelike) light beam. In particular embodiments, an optical relay may be separately or additionally configured to collimate, manipulate, filter, clip, stretch, modulate, modify, reshape, condition, and / or redirect parts or all of the light beam, such as a linear light beam. In particular embodiments, an optical relay may be separately or additionally configured to focus the light beam (e.g., a collimated and / or linear light beam) at a back focal plane of an objective. In certain non-limiting embodiments, an optical relay may comprise one or more lenses, such as achromatic lenses, doublets, slits, adjustment mechanisms, and / or other optical elements and assemblies suitable for the above disclosed purposes and functions.

[0129] According to specific embodiments, one or more additional optical elements or assemblies, such as a dichroic mirror, may be used to selectively route the light beam en route to the objective, and / or from the objective (for e.g., to imaging module 240). In certain, non-limiting implementations, an objective inclines the light beam, and / or may direct the beam on the sample plane to be analyzed. As has been previously discussed in parts herein, in particular embodiments, a scanning element, such as galvo mirror, may selectively scan a light beam (for e.g., a collimated, linear, and / or uniform light beam) such that an illuminating light beam provided at a sample plane may correspondingly scan, traverse, or translate in the sample plane.

[0130] In particular applications related to Single Molecule Tracking (SMT), particular embodiments of microscopy apparatus 220 disclosed herein enable single molecule tracking in live cells with minimal background noise, correspondingly high signal to noise ratio (SNR), and large field of view (FOV), while preserving photon budgets for low light fluorescence imaging.

[0131] 2.1.3 Sample

[0132] FIG. 3 provides a schematic representation of a sample plate 305 comprising a plurality of wells 310, in which samples can be prepared and analyzed, according to particular embodiments. In particular embodiments, a suitable part of microscopy apparatus 220 (e.g., an objective) can illuminate a sample plane. In particular embodiments, the same part, or a different part of microscopy apparatus 220 may be configured to convey light from the sample plane to an imaging module 240.

[0133] FIG. 3 also provides a schematic representation of components of a sample, e.g., a cell 315 and fluorescent target proteins 320 within the cell. It should be noted that FIG. 3 is not intended to convey scale, e.g., each sample present in a well 310 can comprise thousands of cells and each cell can comprise numerous fluorescent target proteins.

[0134] FIG. 3 also schematically illustrates the ability of sample handling systems of the present disclosure to add additional reagents 325 to samples. Such reagent addition can be handled by robotic manipulations, such as, but not limited to, the translation of robotic fluid handling systems relative to the individual wells 310 of the sample plate 305, the translation of the sample plate 305 itself, or combinations of both. In certain implementations of the microscope module 110, the sample plate 305 may be maintained in a temperature- controlled environment through an environmental control area 330.

[0135] In particular embodiments, fluorescence imaging methods disclosed herein may comprise the use of photoactivatable or photoswitchable dyes. By way of example and not limitation, microscope system 100 may comprise using multiple laser sources, such as different and / or separate lasers. By way of example and not limitation, one or more first lasers may be used for illuminating the sample plane, and one or more second lasers may be used for controlling densities and / or rates of emission of particular sets of emitters.

[0136] 2.1.4 Imaging Module

[0137] In certain, non-limiting implementations of the image acquisition systems of the present disclosure, the same objective as used to direct light to a sample plane, or a different objective, may be used to focus and / or direct the light from the sample, such as fluorescence emitted by the sample in response to the illumination provided by the light source, to one or more detector devices of an imaging module 240. In certain, non-limiting implementations, the objective-focused fluorescence emission may pass through one or more emission filters, for e.g., a bandpass emission filter matched to the spectrum of the fluorophore under observation, and collected by a detector device.

[0138] In certain, non-limiting implementations, the objective-focused fluorescence emission may be directed to an optical tube or relay prior to collection by the detector device. For example, but not by way of limitation, such an optical tube or relay can comprise one or more lenses (e.g., a tube lens), and / or one or more additional optical elements (e.g., an element configured to reject additional scattered light, prior to collection by a detector device). In certain, non-limiting implementations, the objective-focused fluorescence emission may be directed through one or more dichroic mirrors, such as to split the emission over multiple regions of a detector device of imaging module 240. In certain, non-limiting implementations, the objective-focused fluorescence emission may be directed through another dichroic mirror to split the emission over multiple detectors of imaging module 240. In particular embodiments, a rolling shutter operation, and / or one or more physical slits, pinholes, or equivalent optical elements or assemblies may be used for providing confocality.

[0139] In certain non-limiting implementations of microscope module 110, a detector device may be configured to synchronize detection and / or activation of photosensitive segments with the translation of a light beam across the sample plane. For example, but not by way of limitation, a detector device can comprise a CCD image sensor or a CMOS image sensor, for e.g., a back illuminated CMOS image sensor (such as the Hamamatsu Fusion BT). By way of example and not limitation, a detector device can include one or more of a single photon avalanche detector (e.g., SPAD), an avalanche photo diode detector (e.g., APD) array, a photo diode, and / or other suitable discrete devices. In particular embodiments, selective activation of photosensitive segments synchronized with scanning the light beam across sample plane can provide a confocal effect of selectively filtering out incoming light from out-of-focus planes.

[0140] In certain implementations of microscope module 110, a detector can be operated such that, for each field of view (FOV) captured, a series of frames are collected. For example, but not by way of limitation, 1-20,000 frames, 1-15,000 frames, 1-10,000 frames, 1-5,000 frames, 1-1,000 frames, 2-500 frames, 5-250 frames, 10-200 frames, 100-200 frames, or 200 frames are collected per field of view. In particular embodiments, frames collected may be single molecule tracking (SMT) frames. By way of example and not limitation, in certain implementations, an image sensor can be configured to acquire frames at a frame rate of between 0.5 and 1000 Hz, or in certain implementations, at 100 Hz. For example, but not by way of limitation, certain cellular SMT implementations can be performed at 100 Hz.

[0141] In particular embodiments, such as corresponding to single molecule tracking (SMT) applications, disclosure aspects that are contemplated and disclosed herein may simultaneously require one or more of a high frame rate (e.g., 100-200 FPS) image acquisition, high spatial resolution, and / or high localization precision (e.g., sub-pixel scale). Separately or additionally adaptive feedback and / or control in applications disclosed herein can require image and other processing to occur within extremely short tolerances or margins for lag or delay, such as within 1-2 frames at high frame rates of image acquisition (e.g., 100 FPS). As described herein, adaptive feedback and / or control may be used to perform adaptive imaging, adaptive screening, adaptive focus, adaptive illumination, and / or other suitable adaptive functions.

[0142] In certain, non-limiting implementations of microscope module 110 of the present disclosure, a detector device can be configured to transmit a signal with each frame to trigger other elements of microscope module 110. For example, but not by way of limitation, the detector device may trigger the illumination from light source 210 so as to collect fluorescence emission associated with stroboscopic laser pulses. For example, but not by way of limitation, such fluorescence emission collection is associated with 10 to 100 ms frames and / or a 2 ms stroboscopic laser pulse.

[0143] In certain implementations, imaging module 240 can be configured to acquire one or more predetermined image dimensions per frame. In certain implementations, the image dimensions acquired, or capable of being acquired, may differ based on the frame rate employed.

[0144] By way of example and not limitation, at 100 FPS frame rate may correspond to image dimensions of a 2304 x 1728 pixel, which can correspond to 248.832 x 186.624 microns in the sample plane, depending on the magnification ratio in use (e.g., a 60X objective). By way of example and not limitation, in contrast, at 200 FPS, the corresponding image dimensions may be 2304 x 768 pixels, which may correspond to 248.832 x 82.944 microns in the sample plane, depending on the magnification ratio in use. In certain implementations, microscope module 110 can be configured to perform predetermined sweep rates at predetermined frame rates. For example, but not by way of limitation, at 100 FPS, a sweep rate can be 186.624 microns / 9 ms, which can be equivalent to 20.8 microns / ms at the sample plane, or 2.08 cm / s. In contrast, by way of continuing non-limiting example, at 200 FPS the sweep rate can be 82.94 microns / 4 ms, which can be equivalent to 20.7 microns / ms in the sample plane, or 2.07 cm / s. In particular embodiments, a sweep rate may be provided by suitably configuring and / or controlling a scanning element of microscopy apparatus 220.

[0145] In certain implementations, one or more detector devices of imaging module 240 may be used to collect fluorescence emission at a multiple wavelengths. For example, but not by way of limitation, fluorescence emission of additional fluorophores can be collected at the same frame rate or different frame rates for the same fields of view to provide downstream registration of SMT tracks to other cellular components, e.g., nuclei. Additional channels of the detector device can be used as desired to expand the number of simultaneously captured fluorescence emissions for the same fields of view to provide downstream registration of SMT tracks to other cellular components, e.g., nuclei.

[0146] 2.2 Computing Module

[0147] In particular embodiments, computing module 140 may be configured to process images received from imaging module 240. In particular embodiments, computing module 140 may be configured to provide hardware accelerated in-line image processing. By way of example and not limitation, providing in-line processing, and / or optimizing a processing pipeline for bandwidth and / or low latency computation, can enable and / or facilitate adaptive control of imaging, experiments, and / or screening methods.

[0148] It will be appreciated that while particular computing architectures, hardware, software, storage aspects, structures, interfaces, and / or other related aspects may be disclosed herein for performing the particular functions and operations, these are disclosed as non-limiting illustrative examples for providing a better understanding. This disclosure fully contemplates any suitable computing architectures, hardware, software, and / or structures, distributed among any suitable system and / or subsystem organization using any suitable connections and interfaces, for performing the disclosed features, functions, and operations. By way of example and not limitation, one or more image processing outputs may be used for adaptive feedback control of microscope system 100. By way of example and not limitation, as will be described further herein, computing module 140 may be configured to permit and / or facilitate sufficiently adaptive variation of microscope operational parameters to enable use of microscope system 100 for particularly demanding applications. By way of example and not limitation, microscope system 100 may be used for tracking single particles, such as for single molecule tracking (SMT). By way of example and not limitation, microscope system 100 may be used for imaging and / or tracking live cells.

[0149] Separately or additionally, computing module 140 may be configured in particular embodiments for facilitating efficient and / or rapid compression of data (e.g., images) acquired and / or generated by microscope system 100. By way of example and not limitation, many benefits of compression may accrue, and / or be best realized, by performing compression early and / or upstream in the processing pipeline. For example, downstream networking and / or storage needs can be greatly reduced, with a lighter data load. By way of example and not limitation, microscope systems 100, for example as may be used in high throughput applications such as drug screening and discovery, may generate large volumes of data. In particular embodiments, computing module 140 may be configured to provide compression of data acquired and / or generated by microscope system 100 to reduce downstream data processing, storage, and / or communication requirements. Accordingly, in particular embodiments, it may be beneficial to perform one or more compression-related processes as early in the process (i.e., as far upstream in the data pipeline) as feasible. Separately or additionally, data compression that may be provided by computing module 140 may accelerate or otherwise facilitate sufficiently rapid data processing and control methodologies. By way of example and not limitation, real-time or near real-time processing of image data may accordingly permit or facilitate adaptive feedback control of microscope system 100.

[0150] 2.2.1 Image Processing Methods

[0151] In particular embodiments, computing module 140 may be configured to receive a sequence of images from imaging module 240. In particular embodiments, each image in a sequence of images may be a digital representation of light received by imaging module 240 from an illuminated sample plane of microscope module 110. By way of example and not limitation, images may be received by computing module 140 as raw image data from a detector or image sensor of imaging module 240, or as data that may have been pre- processed or otherwise processed, such as by imaging module 240. In particular embodiments, each image received by computing module 140 may represent a spatial distribution of fluorescent emitters located at the sample plane and imaged during a specific time interval by imaging module 240.

[0152] In particular embodiments, computing module 140 may be configured to perform one or more image processing steps, transforms (e.g., Fourier transforms), and / or filtering processes on a received image. In particular embodiments, computing module 140 may be configured to perform one or more detection steps, for example, to isolate and / or cluster particular subarrays or fragments of an image that may contain candidate pixels associated with emitter images. In particular embodiments, computing module 140 may be configured to perform one or more localization steps, for example, to spatially locate emitter positions with sub-pixel resolution. In particular embodiments, localization computing steps may be performed only on image fragments or subarrays previously isolated by detection.

[0153] In particular embodiments, spatial filtering can comprise applying a filter (and / or mask, kernel, or window) to pixels of an image. In particular embodiments, filtering may comprise sliding a filter over pixels, and / or sets of pixels, of an image. In particular embodiments, filtering may be performed by sliding a filter along one or two dimensions of the image pixels. In particular embodiments, filtering may involve computing an inner product of filter and a pixel’s value (e.g., intensity value). In particular embodiments, such as based on the form of a filter used, during a local filtering operation, neighboring pixels of a target filtered pixel may influence the output or result of filtering. By way of example and not limitation, filtering may be used for de-noising, enhancing contrast and / or Signal- to-Noise ratio, and / or removing image artifacts.

[0154] In particular embodiments, convolution methods may be used for spatial filtering. In particular embodiments, separable convolution methods may be used to separate the filter (and / or mask, kernel, or window) into smaller and / or separate filters. By way of example and not limitation, filters used may comprise Gaussian, normal (rectangular), and / or normal (squared) filters or kernels. In particular embodiments, separable filtering may provide benefits of naturally dividing image processing operations into multiple processing tasks, some or all of which processing may be optionally parallelized, such as to accelerate processing, reduce bandwidth requirements, and / or achieve low latency in processing. In particular embodiments, relative to a corresponding non-separable filtering operation, separable filtering may alternatively or additionally provide benefits of reducing an overall number of processing tasks to be performed for a given filtered output. By way of example and not limitation, for a 11x11 kernel, separable horizontal and vertical passes for each point can correspond to two sets of 11-point dot product computations, instead of a sum of 11x11 = 121 multiplications.

[0155] In particular embodiments, for images having multiple channels, particular methods described herein with respect to filtering, such as spatial filtering, may be suitably extended to other situations, such as to a depth dimension based on the number of channels.

[0156] In particular embodiments, computing module 140 may be configured to perform one or more pre-conditioning and / or pre-processing operations on an image received therein. Additionally or alternatively, in particular embodiments, one or more preconditioning and / or pre-processing operations may be performed by imaging module 240 on an acquired image prior to transmission. By way of example and not limitation, preconditioning may comprise de-noising, such as removing at least a part of one or more of fixed pattern noise, artifacts (e.g., striping artifacts), and / or false signals. By way of example and not limitation, pre-conditioning may comprise steps to improve a signal-to-noise ratio of information within an image, and / or performing a background subtraction operation. By way of example and not limitation, pre-conditioning may comprise normalizing pixel values in an image, such as by gain normalization.

[0157] In particular embodiments, a pre-conditioning operation can comprise quantifying row-wise and / or column-wise imaging noise characteristics by a neural network trained to detect such noise. By way of example and not limitation, convolutional layers and / or recurrent layers may be used to construct a suitable neural network for noise estimation and / or denoising. In particular embodiments, feature extraction and cascaded convolutions may be used to learn noise patterns to generate noise maps, and fusion network(s) may be used to produce noise estimation based on the noise maps. By way of example and not limitation, row-wise and / or column-wise imaging noise may be quantified using a neural network, which may be followed by then subtracting a corresponding row-wise and / or column-wise offset in particular embodiments.

[0158] FIG. 4A illustrates a schematic view of an exemplary set of pixels associated with a detection process, according to particular embodiments. In particular embodiments, an objective of a detection process may be to identify pixels, and / or groups of pixels, that may be correlated with images of emitters in the corresponding image. By way of example and not limitation, for an exemplary 6x6 subarray or image fragment 405 illustrated in FIG. 4A, a detection process may be used to determine that a particular pixel 410 is associated with or contains, and / or is most likely to contain relative to neighboring pixels, an image corresponding to a particle of interest, such as an emitter.

[0159] In particular embodiments, filtering processes performed on a received image can comprise one or more detection processes. In particular embodiments, a detection process may comprise one or more processing steps to identify candidate emitter pixels in an image, such as may correspond to a respective set of emitters. In particular embodiments, candidate emitter pixels may be detected based on processing, scanning, and / or filtering each image for particular characteristics, such as pixel intensity, distributions and / or shapes of neighboring pixel intensities. In particular embodiments, computing module 140 may be configured to perform one or more local filtering operations on an image during a detection process.

[0160] In particular embodiments, a detection process may comprise, or may be preceded by, one or more coarse convolutions, such as a one-dimensional convolution. By way of example and not limitation, a 5-point one-dimensional kernel may be used for a coarse pair- wise operation, e.g., by row. In particular embodiments, a moving background noise averaging operation may be performed preceding or concurrent with detection, for e.g., by row.

[0161] In particular embodiments, for performing a detection process, such as to determine a detection set of candidate emitter pixels, computing module 140 may be configured to apply a first one-dimensional filter along a first dimension of an image, and a second onedimensional filter along a second dimension of the image. In particular embodiments, computing module 140 may be configured to use one-dimensional and / or two-dimensional convolution for filtering the image for a detection process. In particular embodiments, computing module 140 may be configured to use separable convolution steps.

[0162] By way of example and not limitation, a detection process performed by computing module 140 may comprise multiple filter passes, such as a horizontal pass and a vertical pass (or other terms denoting the axes of a two-dimensional arrangement of pixels). By way of example and not limitation, each pass may comprise using an 11 -point kernel for processing 64 pixels in parallel. In particular embodiments, with reference to above nonlimiting example, detection may be performed using the full 11x11 2D kernel to determine one or more best pixel locations of candidate emitters. In particular embodiments, coarse threshold detection process may comprise a pulse phase discriminator computation. In particular embodiments, pulse phase discriminator steps may be performed in conjunction with or following coarse convolutions and / or with noise floor estimation steps. In particular embodiments, a pulse phase discriminator process may comprise sliding a pulse discriminator filter on candidate pixels, such as on onedimensional or two-dimensional filtered output. By way of example and not limitation, when the pulse is centered on a candidate pixel, the discriminator filter output can be set to go to zero. By way of example and not limitation, the discriminator filter output can be set to have opposite signs on either side of a candidate pixel, thereby providing a high sensitivity coarse threshold detection method to rapidly identify candidate pixels.

[0163] In particular embodiments, computing module 140 may be configured to extract an image fragment, or subarray, in the neighborhood or proximity of each detected candidate emitter pixel, for e.g., based on a coarse threshold detect map. By way of example and not limitation, each image fragment may comprise multiple pixels of a square or rectangular image portion including, around, and / or surrounding one or more candidate emitter pixels, i.e., containing and / or bounding a Region of Interest (ROI). By way of example and not limitation, each image fragment can comprise a square fragment of 16x16 pixels containing a Region of Interest (ROI).

[0164] In particular embodiments, computing module 140 may be configured to perform one or more fine-level image processing operations and / or spatial filtering operations at the image fragment or sub-array level. By way of example and not limitation, an image square operation may be performed for each 16x16 pixel image fragment at the sub-array level to create a power subarray. By way of example and not limitation, separately or additionally, fine-level spatial filtering operations, such as using one or more separable 5x5 arrays of central pixels, may be performed for each 16x16 pixel image fragment, i.e., at the sub-array level. In particular embodiments, a plurality of separable convolutions, such as fine-level separable convolutions performed on an image fragment or sub-array level corresponding to a Region of Interest (ROI), may be performed in parallel, so as to facilitate parallelization and / or low latency processing.

[0165] In particular embodiments, prior to performing a subsequent localization process, computing module 140 may be configured to performing a filtering operation for determining and / or finalizing a detection set of candidate emitter pixels in an image, such as based on one or more filtering criteria. In particular embodiments, a filtering operation may be separately or additionally performed on a first set of detected candidate emitter pixels, for example, to eliminate false positive detections.

[0166] By way of example and not limitation, filtering criteria may comprise one or more suitable metrics associated with identifying emitters. By way of example and not limitation, filtering criteria may comprise a threshold value of a likelihood metric and / or ratio, such as a log likelihood ratio (LLR). In particular embodiments, a likelihood metric and / or ratio used herein may comprise computing a statistical likelihood of an emitter (e.g., a point spread function corresponding to an emitter) being located within a particular image fragment, and / or at a particular pixel location. By way of example and not limitation, a likelihood metric may compute a relative probability of an image fragment or sub-window containing a spot (e.g., a Gaussian spot) of particular intensity value relative to containing only background noise (e.g., Gaussian-distributed background noise). In particular embodiments, a likelihood ratio may be scaled logarithmically.

[0167] By way of example and not limitation, filtering criteria may comprise a threshold value of a signal-to-noise ratio. By way of example and not limitation, filtering criteria may comprise one or more peak searching criteria. By way of example and not limitation, a peak detection filtering step may eliminate one or more emitter candidates that do not exceed a threshold, such as a log likelihood ratio (LLR) threshold.

[0168] In particular embodiments, computing module 140 may be configured to perform one or more localization processes. By way of example and not limitation, a localization process may be performed to obtain a more precise location of an emitter represented on an image, for example, to a sub-pixel resolution. FIG. 4B illustrates a schematic view of subpixel localization, according to particular embodiments. By way of example and not limitation, an image fragment 450, which may correspond in particular embodiments to the same image segment considered during a detection step (for e.g., image fragment 405), may contain and / or otherwise be associated with an particle image 455 that may be located to sub-pixel resolution based on performing a localization process.

[0169] In particular embodiments, computing module 140 may be configured to perform a localization computation on each image fragment extracted and associated with detected candidate emitter pixels. In particular embodiments, a localization computation may be based on a radial symmetry of each emitter, and / or of a point spread function associated with each emitter. In particular embodiments, particular methods of localization may be analytically calculable, for example, without using iterative methods, to determine a sub- pixel location of a particle image, such as an emitter image. By way of example and not limitation, one or more gradients of pixel intensity values may be analytically computed for radial symmetry-based localization of a particle image, such as an emitter image. By way of example and not limitation, an intersection location of a plurality of pixel intensity gradients can correspond to a center and / or centroid location of a particle image, such as an emitter image. In particular embodiments, localization may be used to locate a center and / or centroid of a particle image, such as an emitter image, or a point spread function thereof. In particular embodiments, gradient computations may be accompanied by other processing steps, such as smoothing of gradient slopes of pixel intensity. In particular embodiments, alternative or additional methods for localization may be used, such as based on Gaussian fitting, nonlinear fitting, maximum likelihood estimators, weighting methods, and / or other geometric methods to locate centers and / or centroids.

[0170] As discussed herein, in particular embodiments, one or more image fragments extracted during processing by computing module 140 may be associated with detected candidate emitter pixels. In particular embodiments, a localization computation performed by computing module 140 to provide a localized sub-pixel coordinate within a corresponding image fragment may comprise solving one or more optimization functions over at least a part of that image fragment. In particular embodiments, solving an optimization function may be performed using a suitable iterative search algorithm. In particular embodiments, solving an optimization function may be performed using a suitable non-iterative algorithm.

[0171] In particular embodiments, using a non-iterative algorithm for performing a localization computation may include computationally traversing every possible position of a search space comprising a set of discretized spatial positions, for e.g., to compute a respective local value of an associated function. By way of example and not limitation, a search space comprising a set of discretized positions may be located within an image fragment corresponding to one or more detected candidate emitters, and may be accordingly evaluated to identify one or more respective localized positions of the detected candidate emitters. In particular embodiments, optimization using a non-iterative algorithm may comprise non-iteratively optimizing an associated value, such as a cost metric, that is evaluated at every position of the set of discretized positions. In some contexts, an algorithm or method comprising evaluation over every possible location in a given search space can be considered to employ a brute-force approach. In particular embodiments, a search area for localizing a sub-pixel location for a candidate emitter by performing a localization computation may be selected to be smaller than an image fragment associated with a detection of the candidate emitter. In particular embodiments, a search area for localizing a sub-pixel location may be associated with a single detected candidate emitter. In particular embodiments, a grid resolution for a search area for localization may be determined prior to performing the localization computation, for e.g., based on predetermined criteria, heuristics, and / or suitable computational steps to minimize a localization error. In particular embodiments, one or more computational parameters for performing localization may be modified based on other factors, e.g., image noise levels, point spread function characteristics, and / or subject motion blur due to positional change of detected candidate emitter(s) with time.

[0172] In particular embodiments, a least-squares cost metric may be utilized for optimization by computing module 140 to identify localized sub-pixel positions of candidate emitters. In particular embodiments, a cost metric may be operated with a localization kernel to perform a localization computation. By way of example and not limitation, the localization kernel may comprise a Gaussian kernel. In particular embodiments, the localization kernel may be suitably sized based on the image processing pipeline. By way of example and not limitation, a localization kernel may be selected to be narrower than an image fragment and / or a search area associated with a detection of a candidate emitter.

[0173] In particular embodiments, leveraging spatial separability of a localization kernel and / or suitably casting one or more optimization functions can facilitate parallel computation, such as for low latency in-line processing by microscope system 100 (e.g., using an FPGA and / or GPU based implementation). In particular embodiments, using a noniterative optimization approach for localization can facilitate low latency in-line processing in particular embodiments, e.g., based on suitable casting of computing steps performed by computing module 140 in terms of additions, multiplications, and / or argmax operations as non-limiting examples of mathematical operations that may be well suited to such processing. By way of example and not limitation, a non-iterative optimization problem may be expressed in terms of an argmax of a matrix quadratic function.

[0174] Particular approaches to localization may introduce biases or other errors to localization result comprising a sub-pixel location of a candidate emitter. By way of example and not limitation, particular approaches may include a systematic localizing tendency toward spatially central positions of associated search areas and / or image fragments. By way of example and not limitation, sources of undue sensitivity of localization results to input parameters, boundary conditions, noise, and / or other variabilities may be linked to particular limitations of training data sets, non-representative kernels or functional forms, and / or unsuitable assumptions that are inherent in or imposed on particular localization methods. In particular embodiments, using a non-iterative optimization approach for localization, such as discussed herein, can facilitate reducing or eliminating such sensitivities based on employing an operational basis of computationally traversing every possible position of a candidate set of discretized spatial positions prior to and for the purpose of obtaining a localization result.

[0175] FIG. 4C illustrates a schematic view of sub-pixel localization based on non-iterative optimization, according to particular embodiments.

[0176] As discussed herein, a localization algorithm can be cast in the form of a leastsquares metric in particular embodiments, for e.g., as a matching operation to an assumed point spread function (PSF). By way of example and not limitation, this can be mathematically expressed as: x,y = argmax 5 gx,y[i,j]a[i,j], x,y i where a[i,j] is the value of the region of interest (ROI) around the detected emitter, at pixel index zj; gx yis the PSF kernel associated with the emitter location and x,y is the estimated emitter location. In particular embodiments, such as in some applications having high signal-to-noise, a maximization problem such as the above example may be solved using an iterative algorithm. In particular embodiments, it may be separately or additionally possible to optimize, e.g., maximize, a suitable localization metric by exhaustive evaluation at all values of interest, for e.g., to identify the largest value.

[0177] In particular embodiments, such as live cell single molecule tracking (SMT), noniterative optimization may be cast in a computationally competitive framework to mitigate many of the drawbacks associated with iterative approaches. By way of example and not limitation, a non-iterative approach may partly or wholly avoid the use of complex mathematical operations, the need to define stopping criteria, and / or challenges associated with convergence to local maxima.

[0178] In particular embodiments, signal levels may be generally or consistently low relative to other applications, which can limit the achievable localization precision. By way of example and not limitation, single molecule tracking (SMT) applications, such as considered herein, may be associated with low signal levels in particular embodiments. As a non-limiting example, relatively low signal levels inherent in an application may render a fairly coarse non-iterative search comparably accurate to an iterative approach.

[0179] As further discussed herein, a modeled PSF may be mathematically separable in particular embodiments. By way of example and not limitation, this can be mathematically expressed as: gx,y[i>j = Px [i]qy[ / ]

[0180] By way of example and not limitation, a Gaussian kernel may be taken as an example of a separable PSF. Following the above non-limiting example, substituting both the separability condition and the symmetry associated with an isotropic Gaussian into the maximization problem stated above gives: x,y = argmax x,y

[0181] By way of example and not limitation, the above equation demonstrates that the localization metric can be cast as matrix multiplications, e.g., in the case of a separable kernel. FIG. 4C schematically illustrates such an operation as a non-limiting example of non-iterative optimization cast as matrix multiplications. The non-limiting example of FIG. 4C comprises 25 possible offsets considered in each dimension, within the range of ±1.5 pixels from a corresponding detection pixel. That is, in the example of FIG. 4C, D represents the localization metric evaluated over a central 3x3 pixel window in an image fragment corresponding to detection of a candidate emitter. In particular embodiments, an operation such as described above and illustrated in FIG. 4C can be significantly more computationally efficient than naive evaluation of the full 2D localization metric. In particular embodiments, such reduced computational burden can allow, for example, low latency and / or in-line processing, such as in high throughput single molecule tracking (SMT) applications.

[0182] It will be appreciated that one or more references to images, image intensity distributions and the like may additionally or alternatively be associated with point spread functions of corresponding point sources and / or point objects (e.g., emitters in the sample plane). By way of example and not limitation, such point spread functions may be received by imaging module 240 and / or as digital image representations by computing module 140.

[0183] It will also be appreciated that particular image processing operations and steps are described to provide a better understanding. Any suitable image processing methods and / or operations may be used herein additionally or alternatively with particular described steps. By way of example and not limitation, a step comprising fast Fourier transforms (FFTs) computations may be performed prior to fast convolutions and / or coarse filtering operations in particular embodiments. By way of example and not limitation, one or more suitable iterative and / or non-iterative algorithms may be used to perform sub-pixel localization. By way of example and not limitation, it will be understood that particular forms of algorithms and / or image process steps described herein are merely provided as illustrative examples and are not a requirement, nor are they intended to otherwise limit the scope of this disclosure. In particular embodiments, one or more methods, algorithms, and / or process steps may be omitted, added, or suitably combined.

[0184] By way of example and not limitation, particular steps and / or adaptations disclosed herein can facilitate or permit hardware acceleration of image processing steps, so as to permit adaptive control of microscope system 100. By way of example and not limitation, separable convolution operations may be performed in parallel on a suitable hardware architecture of computing module 140 (e.g., comprising a Field Programmable Gate Array (FPGA) and / or a Graphics Processing Unit (GPU)). By way of example and not limitation, separable convolution operations can significantly reduce the total number of computational operations required to perform a detection process on an image.

[0185] In particular embodiments, based on memory size and / or bandwidth limitations, particular parts of computing module 140 may not be able to buffer entire images. Accordingly, it may become necessary to perform multiple read / operations between various forms of memory associated with computing module 140. In particular embodiments, separable and / or parallelizable computations, such as described herein, can reduce latency as well as input / output and / or memory bandwidth requirements, such as by overlapping particular processing steps or portions of steps (for e.g., horizontal and vertical processing in an image). In particular embodiments, for further acceleration processing time and / or reducing latency, particular operations and / or computational steps may be optimized. By way of example and not limitation, optimizing for bandwidth and / or low latency can comprise reducing or eliminating particular computationally expensive operations (e.g., divisions, square roots). By way of example and not limitation, optimizing for bandwidth and / or low latency can comprise replacing computationally complex and / or expensive steps with reasonable approximations, such as may be calculable using fewer and / or less expensive computational steps. By way of example and not limitation, optimizing for bandwidth and / or low latency can comprise organizing, pipelining, substitutions, and / or preferred use of mathematically equivalents (or approximately equivalent) operations more suitable to native processing and memory characteristics of hardware used for low latency in-line processing (e.g., FPGA and / or GPU) in microscope system 100.

[0186] FIG. 5A illustrates a schematic of a processing timeline of exemplary image processing steps, according to particular embodiments. In particular embodiments, computing module 140 may be configured to process a pipeline as illustrated by performing a plurality of first pass image processing steps simultaneously, and / or with large overlaps in time. By way of example and not limitation, a first pass of image processing steps may comprise coarse and / or sparse assay image processing steps. By way of example and not limitation, a first coarse image processing step 510 (e.g., a convolution operation along one axis of an image) and a second coarse image processing step 520 (e.g., a convolution operation along another axis of the image) may be performed partially or wholly in parallel. By way of example and not limitation, a reference timescale 575 may be 3 ms, or 6 ms in particular embodiments.

[0187] In particular embodiments, a plurality of second pass image processing and / or filtering operations (e.g., 530, 540, and / or 550) may be performed partially or wholly in parallel. By way of example and not limitation, a set of fine filtering operations may be based on the output of optional previously completed coarse filtering operations. By way of example and not limitation, fine filtering operations may be performed on image fragments or sub-arrays of an overall image. By way of example and not limitation, one or more fine filtering operations may comprise using filtering sub-arrays and / or rectangular filters. By way of example and not limitation, one or more fine filtering operations may comprise using a Gaussian kernel, and / or an input-squared kernel. By way of example and not limitation, a fine filtering operation may comprise using a rectangular filter for squaring pixel values of each pixel of an image and / or image fragment.

[0188] In particular embodiments, a threshold filtering step 560, such as a log likelihood ratio threshold step, may be performed at least partially overlapping in time with a previous pass of image processing. In particular embodiments, a localization computation step 570 may be performed at least partially overlapping in time with a previous pass of image processing (e.g., 530, 540, 550, and / or 560), wherein on the results isolated by detection steps are processed for localization. FIG. 5B illustrates a schematic of an exemplary method for hardware accelerated processing in a microscope system, according to particular embodiments. By way of example and not limitation, corresponding to an exemplary step 580, a method for hardware accelerated processing in a microscope system may comprise receiving, by a hardware acceleration module of the microscope system, a first image of a sequence of images from an imaging module, each image respectively comprising a digital representation of a corresponding spatially distributed set of emitters located at a sample plane of a microscope module. By way of example and not limitation, corresponding to an exemplary subsequent step 582, a method for hardware accelerated processing in a microscope system may comprise computationally processing, by the hardware acceleration module, the first image. In particular embodiments, exemplary step 582 may comprise one or more of determining, based on performing one or more local filtering operations on the first image, a detection set of candidate emitter pixels corresponding to the respective set of emitters; and determining, based on performing a localizing computation for each candidate emitter pixel of the detection set, a localized set of coordinates corresponding to respective locations of the set of emitters, each of the localized set of coordinates having a sub-pixel resolution.

[0189] By way of example and not limitation, corresponding to an exemplary subsequent step 584, a method for hardware accelerated processing in a microscope system may comprise providing, by the hardware acceleration module, the localized set of coordinates corresponding to the set of emitters of the first image to a control module of the microscope system.

[0190] By way of example and not limitation, corresponding to an exemplary subsequent step 588, a method for hardware accelerated processing in a microscope system may comprise receiving and computationally processing, by the hardware acceleration module, a second image of the sequence of images, wherein a processing time interval between processing the first image and processing the second image is less than or equal to a sampling time interval of the imaging module. By way of example and not limitation, such a processing time interval between processing the first image and the second image may enable, facilitate, and / or permit adaptive control for single molecule tracking by the microscope system, in particular embodiments.

[0191] It will be appreciated that while particular imaging processing pipelines may be outlined comprising particular steps (for e.g., in FIGs. 5A and / or 5B) to provide an understanding, any suitable process, sequence of processes, and / or processing steps may be separately or additionally performed or substituted therein.

[0192] In particular embodiments, additional or alternative methods may be used for sparse representation, extraction, storage, and / or recreation of information relating to images and / or point spread functions of particles, such as emitters. By way of example and not limitation, methods based on sparse representation referred to herein as spot fingerprinting may be used by computing module 140, such as to determine a set of filter values and / or coefficient values for each image fragment extracted and associated with detected candidate emitter pixels. In particular embodiments, each spot on an image may be represented as a linear combination of particular modes. In particular embodiments, such modes may comprise orthogonal modes. Accordingly, in particular embodiments, an orthogonal set of filters may be constructed to form an orthonormal basis for representing and / or retaining information about an imaged spot or particle (e.g., an emitter). In particular embodiments, the orthonormal basis may be selected to maximize the amount of variance retained from each spot. By way of example and not limitation, principal component analysis (PCA) may be used to construct a suitable orthogonal set of filters. In particular embodiments, each image fragment of interest may be projected on to a suitable orthogonal basis, such as an orthonormal basis, and the resultant coefficients may be stored as a sparse representation of that image fragment. In particular embodiments, a suitable set of filters may correspond to mathematical bases. In particular embodiments, filter values associated with a set of filters comprise the respective projection coefficients associated with projection against corresponding bases.

[0193] In particular embodiments, a trade-off between capturing higher order details of imaged spots and data sparsity for representing the spots may be obtained by selecting a first ‘x’ number of filters (i.e., a set size) from the principal components ordered by decreasing eigenvalue. In particular embodiments, the size of each set of orthogonal filters (i.e., ‘x’) used for representing each spot may be 8-12. In particular embodiments, the size of each set of orthogonal filters (i.e., ‘x’) used for representing each spot may lie between 5 and 25.

[0194] FIG. 6A illustrates a visual depiction of an exemplary set of orthogonal filters comprising the first twenty -two eigenvalues (modes 0-21) of a spot representation, according to particular embodiments, by way of non-limiting example. By way of example and not limitation, percentages, or eigenvalues, depicted in FIG. 6A indicate the fraction of variance in spots from the training set captured by the corresponding filter, according to particular embodiments.

[0195] In particular embodiments, according to linear superposition principles, a reasonable approximation of an imaged spot (e.g., corresponding to an emitter) may be formed based on reconstructing and / or recreating a representation of the spot and / or the corresponding image fragment. By way of example and not limitation, a reconstructible representation may be formed in particular embodiments by combining (for e.g., by simple addition) each filter value or component for the full set of filters selected to represent the imaged spot. In particular embodiments, reconstruction may be suitably performed based on a combination of only a subset of a set of filters, wherein the subset may be sufficient to form a reasonable approximation of an imaged spot and / or fragment. As discussed, for providing a reasonable approximation, a trade-off may be determined and controlled between capturing additional features and details (e.g., using and retaining more filters to represent each spot) and data sparsity, for e.g., using and / or retaining fewer filters to represent each spot.

[0196] As discussed above, in particular embodiments, each filter value may be associated with a corresponding reference set of filters. By way of example and not limitation, a reference set of filters may comprise an orthogonal set of filters. In particular embodiments, a reference set of filters may separately or additionally comprise non-orthogonal filters. In particular embodiments, filter values associated with one or more non-orthogonal filters may be processed to determine filter values corresponding to an orthogonal basis of filters. In particular embodiments, each filter value of a set of filter values obtained for a particular image fragment may be considered to be the respective constituent or component of the corresponding reference filter identified in that particular image fragment.

[0197] FIG. 6B illustrates a schematic view process flow for detection and localization, according to particular embodiments. In particular embodiments, the process flow illustrated therein may be applied to discuss methods based on sparse representation, such as spot fingerprinting. In particular embodiments, an image 650 comprising a plurality of spots, such as emitter images among other artifacts, may be processed by computing module 140. In particular embodiments, computing module 140 may use detection methods, such as discussed herein, for isolating candidate emitter spots in image fragments or subarrays (e.g., spot sub-windows 660). In particular embodiments, computing module 140 may be configured to perform a localization operation 670 to locate particles of interest (e.g., emitters) with sub-pixel resolution. In particular embodiments, a spot table 680 may be generated by computing module 140. In particular embodiments, spot table 680 may identify and list spots (e.g., images and / or point spread functions corresponding emitters), and one or more attributes associated with each spot. By way of example and not limitation, a framewise location of each spot may be recorded, along with spot location with sub-pixel resolution. By way of example and not limitation, other attributes, such as signal-to-noise ratio (SNR), and / or related to tracking information, may be separately or additionally computed and / or recorded. In particular embodiments, a signal-to-noise ratio (SNR) may be calculated corresponding to one or more images, spots, emitters, and / or for larger or spatially aggregated image portions. By way of example and not limitation, SNR calculations may be performed for each emitter of an imaged set of emitters. In particular embodiments, SNR calculated for one or more spots, emitters, and / or aggregated zones may be output for further computation, and / or to control module 120 for implementing adaptive control. By way of example and not limitation, SNR output for one or more spots, emitters, and / or aggregated zones may be used for modifying laser illumination power, and / or adaptively modifying integration time and / or sampling frequency.

[0198] As discussed herein, in particular embodiments, images of spots may be sparsely represented, and / or recreated as desired. A level of detail (e.g., higher order features) may be determined based on a number of orthogonal filters used for representation. FIG. 7 illustrates a set of image pairs 700 depicting image reconstruction based on sparse representation, according to particular embodiments. As depicted by way of example and not limitation, each image pair (e.g., 710-1) comprises an original image (i.e., digital representation as captured on an input image) of a spot on the left side (for e.g., 715-1), and a corresponding reconstructed image of the spot on the right side (for e.g., 720-1), wherein reconstruction of the reconstructed image is based using methods for sparse representation of the original image, as described herein. An R2value (e.g., 725-1) is provided for each pair of images to illustrate a fractional variance between the respective original and their corresponding reconstructed images. A fractional variance metric (e.g., 730-1) is also included in parentheses to indicate a fraction of variance that is not due to shot noise (i.e., 1- mean / variance).

[0199] As described herein, in particular embodiments, localization may be performed by computing module 140 based on radial symmetry and / or pixel intensity gradient calculations. In particular embodiments, localization may be performed by computing module 140 fitting and / or spatially perturbing a particle and / or pixel intensity distribution within an image fragment relative to corresponding idealized and / or known distributions, and / or point spread functions.

[0200] In particular embodiments, localization may be performed by computing module 140 based on applying a suitable neural network, such as a feed-forward neural network (e.g., a convolutional neural network (CNN)), to the image fragments extracted based on the detection set of candidate emitter pixels. In particular embodiments, the neural network may be trained based on one or more features of the set of reference filters (e.g., set of orthogonal filters).

[0201] By way of example and not limitation, neural networks may be trained and used to detect in-focus images, out-of-focus images, and / or degrees of defocus of out-of-focus images, such as by computing module 140. By way of example and not limitation, neural networks may be used for subpixel localization, such as by computing module 140.

[0202] FIG. 8A illustrates schematic process flow steps for neural network-based inference and exemplary architecture, according to particular embodiments. In particular embodiments, a detection method and / or algorithm, such as described herein, may be used to identify image fragments containing and / or bounding Regions of Interest (ROIs). In particular embodiments, Regions of Interest (ROIs) may be detected and extracted from a portion or an entirety of larger images, which may be acquired using imaging module 240. FIG. 8A illustrates an l lxl l-pixel image fragment 810 bounding an exemplary Region of Interest (ROI) corresponding to a portion of exemplary image 805, by way of non-limiting example.

[0203] In particular embodiments, one or more Regions of Interest (ROIs) may be projected into a spot fingerprint basis. As described herein and by way of non-limiting example, a spot fingerprint basis may comprise an orthonormal basis for spot representation. In particular embodiments, a spot fingerprint basis may be identified using principal component analysis (PCA) on a large collection of experimentally observed spots and / or images of emitters, such as from single molecule tracking (SMT) experiments. In particular embodiments, one or more projections may be fed into a suitable neural network.

[0204] By way of example and not limitation, FIG. 8A depicts a 20-dimensional spot fingerprint 815 corresponding to a projection of image fragment 810 into a spot fingerprint basis. By way of continuing non-limiting example, spot fingerprint 815 may be fed into a neural network 820 (e.g., a feed-forward neural network) for predicting an output 825. By way of example and not limitation, image fragment 810 may be linearly projected onto spot fingerprint 815, such as by a matmul operation. By way of example and not limitation, spot fingerprint 815 may be linearly projected onto neural network 820, such as by a matmul operation. By way of example and not limitation, spot fingerprint 815 may comprise a set of filter values corresponding to image fragment 810.

[0205] In particular embodiments, a spot fingerprint may be comprised of filter values, and / or coefficients. By way of example and not limitation, filter values and / or coefficients corresponding to an image fragment may be obtained by projecting the respective image fragment on to a suitable orthogonal basis.

[0206] In particular embodiments, the filter values may permit a linear combination of particular modes, which can be orthogonal modes. In particular embodiments, a predicted output of the neural network can comprise the predicted localized subpixel-level coordinates (such as x and y coordinates) of one or more emitters contained within the corresponding image fragment. In particular embodiments, a predicted output of the neural network may separately or additionally comprise estimates of error magnitudes in the predicted coordinates.

[0207] It will be appreciated that while FIG. 8A depicts particular neural network process flows and / or architectures, such as layers, activation functions, and / or dimensional changes, as non-limiting illustrative examples to provide a better understanding, any suitable neural network process flows and / or architectures for implementing the disclosed functions and operations may be additionally or alternatively used, and are fully contemplated by this disclosure.

[0208] FIG. 8B illustrates a schematic neural network training procedure, according to particular embodiments. By way of example and not limitation, such a training procedure may be used to train a neural network model for subpixel localization. In particular embodiments, training images and / or movies may be experimentally acquired from unlabeled movies and / or images 830, such as from cellular Single Molecule Tracking (SMT) experiments. By way of example and not limitation, such unlabeled images 830 may provide realistic backgrounds, representative camera noise, cellular autofluorescence, and / or typical measurement clutter suitable for neural network training. In particular embodiments, unlabeled experimental images 830 may then be augmented and / or otherwise modified to create synthetic or modified training images 835. By way of example and not limitation, optical and / or dynamic simulations may be performed, and simulated emitters may be introduced to unlabeled experimental images 830 to provide modified training images 835. In particular embodiments, additional simulated noise, such as Gaussian noise, may be separately or additionally added for producing modified training images 835. In particular embodiments, simulated emitters may incorporate realistic effects, such as defocus, motion blur, overlapping emitters, and / or shot noise.

[0209] In particular embodiments, Regions of Interest (ROIs) (e.g., contained or bounded by image fragments, such as 11x11 pixel image fragments, or 16x16 pixel image fragments, by way of non-limiting examples) may be excised around each simulated emitter, such as by using a center-of-mass position during a simulated laser or illumination pulse as a truth label. Accordingly, in particular embodiments, training and test sets 840 can comprise large collections of ROIs with simulated spots at different conditions, along with ground truth coordinates (e.g., x and y coordinates), as further described in the Examples section herein. By way of example and not limitation, different conditions can comprise different lens Numerical Apertures, different particle and emitter densities, and different illumination or brightness levels.

[0210] In particular embodiments, a loss, such as an L2 loss, may be used to reward the model for one or more objectives. In particular embodiments, the model may be rewarded for correctly predicting the location of the particle, and / or for making realistic estimate(s) of its own error.

[0211] In particular embodiments, an L2 loss may be scaled by an estimated localization error. By way of example and not limitation, surface plots 845 and 850 depict distributions of predicted localization errors. By way of example and not limitation, 845 illustrates a low localization error (high confidence) comparison of a predicted position relative to a corresponding truth-labeled position. By way of example and not limitation, 850 illustrates a high localization error (low confidence) comparison of a predicted position relative to a corresponding truth-labeled position.

[0212] 2.2.2 Computing Hardware and Software Overview

[0213] FIG. 9 illustrates an example computer-implemented environment 900 where imaging module 240 of microscope module 110 can interface and interact with a computing architecture to perform the various methods and / or algorithms described herein. By way of example and not limitation, as shown in FIG. 9, the imaging module 240 can interface with one or more clients 950 (e.g., clients via a web application having a graphical user interface). The one or more clients 950 can interface with one or more servers 920 accessible through the network(s) 930. The one or more clients 950 can host a frame grabber that captures images from a camera (e.g., movies). Those images can be temporarily stored on the one or more clients 950 and periodically transferred to the one or more servers 920 for remote storage via network 930. The one or more servers 920 can also contain or have access to one or more data stores 940 for storing data collected and / or extracted from a sample by imaging module 240. In some variations, the network 930 may include or interface with one or more network storage arrays 960 for storing data such as the captured images (e.g., movies).

[0214] FIG. 10 illustrates an exemplary schematic computing module architecture 1000 of an exemplary computing module 140 for implementing various aspects described herein, according to particular embodiments of the present disclosure. In some embodiments, the sample computing device architecture can be that of client(s) 950 and / or of server(s) 920 and some components described in relation to diagram 1000 may be optional for the client(s) 950 and / or servers(s) 920. A bus 1004 can serve as the information highway interconnecting the other illustrated components of the hardware.

[0215] It should be appreciated that not every part or component of computing module architecture 1000, such as illustrated in FIG. 10 by way of non-limiting example, may be present in every embodiment. By way of example and not limitation, particular embodiments of computing module 140 may comprise one or more CPUs 1008, but no FPGA 1014, or GPU 1012. As another non-limiting example, particular embodiments of computing module 140 may comprise an FPGA 1014 only, which may be optionally connected to other modules or devices, such as via a system bus 1004, and / or other dedicated interface(s). Further, figures provided herein such as FIG. 10 are illustrative and not intended to be exhaustive; any embodiments contemplated herein can include and / or interface with any suitable modules, components, and / or combinations thereof, as may be understood by one skilled in these arts, and / or that may not be specifically illustrated or described herein.

[0216] In particular embodiments, computing module 140 may comprise one or more computing processors or cores. In particular embodiments, computing module 140 may comprise a plurality of processing units, and / or heterogeneous processing units. By way of example and not limitation, computing module 140 may comprise one or more Central Processing Units (CPUs) 1008, one or more Graphics Processing Units (GPUs) 1012, one or more Field Programmable Gate Arrays (FPGAs) 1014, or any combination thereof. By way of example and not limitation, a processing system 1008 labeled CPU (central processing unit) (e.g., one or more computer processors / data processors at a given computer or at multiple computers), can perform calculations and logic operations required to execute a program. Optionally or additionally, a processing system 1012 labeled GPU (graphics processing unit) (e.g., one or more computer processors / data processors at a given computer or at multiple computers), can perform calculations and logic operations required to execute a program. A non-transitory processor-readable storage medium, such as read only memory (ROM) 1016 and random access memory (RAM) 1020, can be in communication with the processing system 1008 and / or processing system 1012 and can include one or more programming instructions for the operations specified here. In particular embodiments, memory buffering may be used to facilitate consistency of a mean processing time interval associated with processing by computing module 140, such as inline processing of images by hardware acceleration to facilitate adaptive feedback. Optionally, program instructions can be stored on a non-transitory computer-readable storage medium such as a magnetic disk, optical disc, recordable memory device, flash memory, solid state memory, and / or other physical storage medium.

[0217] In particular embodiments, a disk controller 1048 can interface with one or more optional removable storage 1056 or local storage 1052 to the system bus 1004. The removable storage 1056 can be external or internal disk drives, or solid state drives, or external hard drives. The local storage 1052 can be internal hard drives and / or memory. As indicated previously, these various examples of removable storage 1056, local storage 1052, and disk controllers 1048 are optional devices. The system bus 1004 can also include at least one communications interface 1024 to allow for communication with external devices either physically connected to the computing system or available externally through a wired or wireless network such as cloud storage and remote services. In some cases, the at least one communications interface 1024 includes or otherwise comprises a network interface.

[0218] In particular embodiments, structural and / or functional aspects of storage module 130, data stores 940, network storage arrays 960, local storage 1052, and / or removable storage 1056 may be interchangeable, shared and / or combined. As with other aspects described herein, not every disclosed possibility of hardware and / or software need be present and / or required in each embodiment for performing the functions and operations described herein.

[0219] In some variations, such as for client(s) 950, to provide for interaction with a user, the subject matter described herein can be implemented on a computing device having a display device 1044 (e.g., LCD (liquid crystal display) or LED (light-emitting diode) monitor) for displaying information obtained from the bus 1004 via a display interface 1040 to the user and an input device 1032 such as keyboard and / or a pointing device (e.g., a mouse or a trackball) and / or a touchscreen by which the user can provide input to the computer. Other kinds of input devices 1032 can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback by way of a microphone 1036, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input. The input device 1032 and the microphone 1036 can be coupled to and convey information via the bus 1004 by way of an input device interface 1028. By way of example, input device 1032 may be an imaging module 240 configured with abilities to capture a sequence of images as described herein. A frame grabber 1058 can capture or grab individual frames from analog or digital data encapsulating the sequence of images obtained from the bus 1004. Frame grabber 1058 may include memory that can store individual or multiple frames. Frame grabber 1058 can also provide individual or multiple frames to bus 1004 for further storage on, for example, local storage 1052 and / or removable storage 1056. Other computing devices, such as dedicated servers, can omit one or more of the components described in connection with FIG. 10.

[0220] In particular embodiments, suitable architectures of computing module 140 may be configured to facilitate hardware accelerated processing, such as disclosed herein. By way of example and not limitation, hardware accelerated processing may be used to separately or additionally facilitate adaptive control for single molecule tracking by the microscope system 100.

[0221] In particular embodiments, one or more Field Programmable Gate Arrays (FPGA) 1014, and / or one or more Graphic Processing Units (GPU) 1012, may be used to accelerate and / or facilitate particular functions and aspects disclosed herein. By way of example and not limitation, FIG. 11 illustrates a schematic architecture of a Field Programmable Gate Array (FPGA), according to particular embodiments. In particular embodiments, FPGA 1014 may comprise one or more memory handling subcircuits 1110. In particular embodiments, FPGA 1014 may comprise one or more detector subcircuits 1120, which may be configured to perform, accelerate, and / or otherwise facilitate detection functions, such as disclosed in detail herein. In particular embodiments, FPGA 1014 may comprise one or more localizer subcircuits 1130, which may be configured to perform, accelerate, and / or otherwise facilitate sub-pixel localization functions, such as disclosed in detail herein. In particular embodiments, and as illustrated in FIG. 11 by way of non-limiting example, FPGA 1014 may be configured to suitably interface with other modules or parts of microscope system 100, such as other parts of computing module 140, storage module 130, and / or control module 120.

[0222] FIG. 12 illustrates a schematic parallelized detection process, according to particular embodiments. By way of example and not limitation, a parallelizing process such as depicted in FIG. 12 may be used to facilitate hardware acceleration (e.g., by an FPGA 1014). As schematically depicted in FIG. 12 by way of non-limiting example, for hardware accelerated processing of an input image 1210 based on a 32-pixel wide memory, an exemplary approach may use a minimum parallelization of 64 pixels. Accordingly, as indicated by ref. 1220 for this non-limiting example, 64 pixels of image 1210 may be processed (e.g., by applying suitable filtering and / or convolution methods, such as disclosed herein) in each of two memory reads. By way of continuing non-limiting example, results of 64-pixel horizontal processing, depicted in FIG. 12 as two blocks of “HlDx32,” may be buffered on chip, which can enable vertical processing (denoted as “VlDx64”) at a minimum latency in particular embodiments. By way of continuing non-limiting example, filtering operations and / or criteria, such as a log likelihood ratio (LLR) computation indicated by ref. 1230 in FIG. 12, may be then pipelined to match a throughput of processing performed through step 1220 (e.g., 2D convolution). In particular embodiments, by way of continuing non-limiting example, other processing steps 1240 may include pipelining and / or parallelizing threshold processing steps, such as a peak search filtering to isolate particular results from the previous LLR computation for subsequent localization processing.

[0223] It should be appreciated that while FIG. 11 depicts specific modules, configurations and architecture of an FPGA 1014 by way of non-limiting example to provide a better understanding, any suitable configuration(s) and architecture(s) of FPGA 1014, and / or other suitable computing devices such as other GPUs 1012, may be used to perform the functions and aspects disclosed herein, and are fully contemplated herein. It should also be appreciated that while FIG. 12 depicts particular processes, and sequences of processes, for implementing hardware acceleration of particular methods disclosed herein as a nonlimiting example to provide understanding, this disclosure fully contemplates any suitable methods, processes, and / or sequences of processes to implement and / or facilitate hardware acceleration of methods and systems disclosed herein. By way of example and not limitation, hardware acceleration may be implemented (e.g., using FPGA 1014 and / or other suitable parts of computing module 140) to perform sub-pixel localization using neural network methods, such as disclosed herein.

[0224] One or more aspects or features of the subj ect matter described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) computer hardware, firmware, software, and / or combinations thereof. These various aspects or features can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device. The programmable system or computing system may include clients and servers. A client and server may be generally remote from each other, and / or may interact through a communication network. The relationship of client and server may arise by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0225] These computer programs, which can also be referred to as programs, software, software applications, applications, components, or code, include machine instructions for a programmable processor, and can be implemented in a high-level procedural language, an object-oriented programming language, a functional programming language, a logical programming language, and / or in assembly / machine language. As used herein, the term “machine-readable medium” refers to any computer program product, apparatus and / or device, such as for example solid state memory, magnetic disks, optical discs, memory, and Programmable Logic Devices (PLDs), used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor. The machine-readable medium can store such machine instructions non-transitorily, such as for example as would a non-transient solid-state memory or a magnetic hard drive or any equivalent storage medium. The machine-readable medium can alternatively or additionally store such machine instructions in a transient manner, such as for example as would a processor cache or other random access memory associated with one or more physical processor cores.

[0226] 2.3 Storage Module

[0227] In particular embodiments, storage module 130 may comprise one or more storage devices and / or systems, such as based on magnetic, optical, and / or solid state storage media. In particular embodiments, part or all of storage module 130 may be located away and / or off-site from other parts of microscope system 100. By way of example and not limitation, part or all of storage module 130 may comprise a networked storage system, and / or a cloudbased storage system, which may interface with control module 120 and / or computing module 140, or other parts of microscope system 100. In particular embodiments, storage module 130 may be connected to portions of microscope system 100 via one or more network connections, such as wired and / or wireless connections. In particular embodiments, computing module 140 may comprise part or all of storage module 130. In particular embodiments, parts or all of storage module 130, functionally and / or structurally, may be shared with, connected to, or interchangeable with network storage array 960, local storage 1052, and / or removable storage 1056.

[0228] 2.4 Microscope and Platform Control Module

[0229] In particular embodiments, control module 120 may comprise one or more actuators, drives, and / or motors. Control module 120 may comprise one or more controllers and control systems, which may be hardware-based, software-based, or based on a combination of hardware and software. By way of example and not limitation, control module 120 may comprise hardwired, programmable, and / or re-programmable circuits and / or other devices for controlling particular aspects of microscope system 100. By way of example and not limitation, control module 120 may comprise code (for e.g., based on a Lab VIEW routine or a C program) designed to monitor and / or control particular aspects of microscope system 100.

[0230] In particular embodiments, control module 120 may be provided with dedicated input / output devices for human interfacing, or control module 120 may share some or all input / output devices with another part of microscope system 100. In particular embodiments, control module 120 may comprise one or more hardware buses, ports, and / or interfaces, such as serial and / or parallel ports, PCI Express interfaces, SCSI interfaces, and the like. In particular embodiments, some or all of the hardware buses, ports, and / or interfaces of control module 120 may overlap and / or be otherwise shared with another part of microscope system 100, such as computing module 140.

[0231] In particular embodiments, control module 120 may be configured as a stand-alone module of microscope system 100. In particular embodiments, one or more other portions of microscope system 100 may comprise part or all of control module 120. By way of example and not limitation, computing module 140 may comprise part or all of control module 120 in particular embodiments. In particular embodiments, hardware, software, and / or other resources may be shared between control module 120 and one or more other parts of microscope system 100.

[0232] In particular embodiments, control module 120 may be configured to perform adaptive control and / or operations. By way of example and not limitation, control module 120 may be configured to perform adaptive control and / or operations based on receiving one or more signals, instructions, and / or specific data from computing module 140. By way of example and not limitation, control module 120 may receive localized coordinates corresponding to emitters from computing module 140.

[0233] In particular embodiments, control module 120 may be configured to perform adaptive focusing, such as based on receiving one or more signals, instructions, and / or suitable data from computing module 140. In particular embodiments, performing an adaptive focusing operation may include controlling operation of a focusing mechanism of microscope module 110. By way of example and not limitation, a focusing mechanism may be adaptively operated based on control module 120 receiving a recurring and updated set of localized emitter coordinates from computing module 140, such as may be required to obtain or maintain focus on at least a portion of the set of emitters located at the sample plane, and / or for single molecule tracking.

[0234] In particular embodiments, operating the focusing mechanism may be associated with changing a distance between an objective and a stage of microscope module 110, such as by operating an associated stepper motor. By way of example and not limitation, the stage may be associated with receiving and / or holding a sample comprising emitters.

[0235] In particular embodiments, performing an adaptive focusing operation may comprise determining or receiving a focusing-related metric, such as a focusing score. In particular embodiments, a focusing score or metric may be generated by imaging module 240, such as by a focus and / or autofocus module configured and provided therein. In particular embodiments, additional or integral to one or more imaging sensors, imaging module 240 may comprise an autofocus sensor, and / or other autofocus optical element suitable for monitoring a state of focus, and / or for providing a focusing score or metric. In particular embodiments, one or more images or image streams from a suitable image sensor of imaging module 240 may be processed (for e.g., by imaging module 240 and / or computing module 140) to determine a state of focus on a sample plane of interest based on detecting contrast. In particular embodiments, imaging module 240 may comprise a phasedetect module for determine a state of focus, and / or generating a focusing score or metric. In particular embodiments, two or more of imaging module 240, control module 120, and / or computing module 140 may suitably cooperate to perform an autofocus operation.

[0236] In particular embodiments, processing time, bandwidth, and / or latency requirements for computing module 140 to process a sequence of images for enabling and / or facilitating an adaptive focusing operation may be demanding in particular applications. By way of example and not limitation, in a single molecule tracking application with frames acquired at 100 FPS, providing and updating localized emitter positions for adaptive adjustment of focus to continue following and tracking particular partici e(s) of interest can be required to occur within a few frames (e.g., 1-3 frames).

[0237] In particular embodiments, control module 120 may be configured to perform an adaptive imaging operation, such as based on receiving one or more signals, instructions, and / or specific data (e.g., localized emitter coordinates) from computing module 140. By way of example and not limitation, an adaptive imaging operation may comprise operating light source 210 of microscope module 110, such as a laser engine module. In particular embodiments, such operation of light source 210 may be performed in a feedback loop, so as to provide a response at a sample plane and accordingly in an image exposure level, a signal-to-noise ratio, and / or other metric sensed and / or determined by computing module 140 and / or imaging module 240. In particular embodiments comprising such an image exposure feedback process, light source 210 may be operated to modify an operational laser output set point, for example, to facilitate live-cell fluorescence imaging by microscope system 100.

[0238] In particular embodiments, control module 120 may be configured to perform adaptive imaging control and / or operations, such as by modifying and / or otherwise controlling a field of view (FOV) associated with illuminating and / or acquiring images of a sample (e.g., comprising emitters) at a sample plane.

[0239] In particular embodiments, control module 120 may be configured to perform adaptive imaging control and / or operations, such as by modifying and / or otherwise controlling a frame rate associated with acquiring images of a sample (e.g., comprising emitters) at a sample plane, separately or additionally, in particular embodiments, a time interval between frames may be adaptively varied (for e.g., asynchronously, without requiring a constant frame rate). In particular embodiments, control module 120 may be configured to implement image acquisition at a longest exposure time allowable or otherwise considered to be suitable for a given frame rate (e.g., corresponding to 1 / frame rate). In particular embodiments, corresponding to particular situations such as an excessively bright image intensity relative to desired parameters, control module 120 may implement a relative reduction of an exposure time, while maintaining a given frame rate in some cases.

[0240] In particular embodiments, processing time, bandwidth, and / or latency requirements for computing module 140 to process a sequence of images for enabling and / or facilitating an adaptive imaging operation may be demanding in particular applications. By way of example and not limitation, in a single molecule tracking application with frames nominally acquired at 100 FPS, providing and updating localized emitter positions for adaptive imaging adjustment to continue following and tracking particular particle(s) of interest can be required to occur within a few frames (e.g., 1-3 frames). By way of example and not limitation, a change in sampling frame rate may be rapidly required to capture temporal details of rapid biological events that may occur infrequently. By way of continuing example, a change in frame rate may force, and / or otherwise necessitate, a change of imaging dimensions, field of view (FOV) of capture (e.g., based on bandwidth requirements and / or for spatial detail capturing requirements), and / or exposure parameters (e.g., to maintain acceptable image signal-to-noise ratio (SNR) values). By way of example and not limitation, such as in cases of live cell imaging and / or for managing fluorophore photon budgets against experimental dynamics and image signal-to-noise ratio, close adaptive control of a light source (e.g., laser output parameters) can be important to accomplish within a few frames (for e.g., at a sampling rate of 100 FPS, or 200 FPS).

[0241] In particular embodiments, adaptive control and / or operations can be optimized to minimize photodamage from sample illumination by light source 210. In particular embodiments, adaptive control and / or operations can be optimized to achieve a specific and / or optimal protein and / or emitter counts. In particular embodiments, adaptive control and / or operations can be used to optimizing the density of protein (e.g., emitters), such as through the adjustment of one or more additional or secondary illumination wavelength(s) by light source 210.

[0242] In other examples, a need for changing imaging parameters may not be as stringent as some adaptive examples described herein. By way of example and not limitation, during scenarios involving slower dynamics such as once per well, once per field of view, a few times per change of experiment, and / or during a kinetics study, a sampling frame rate may be modified with less frequently based on lower need for updating. Accordingly, in particular embodiments, computing module 140 may not employ a hardware accelerated module for image and other processing and / or to drive feedback control in every scenario. In particular embodiments, certain aspects of image and other processing may be deferred for post-processing, such as when large bandwidth and / or high compute is needed, and / or if adaptive control is not as critical.

[0243] In particular embodiments, control module 120 may be configured to perform adaptive control and / or operations, such as to trigger and / or prioritize processing and / or storage of particular images and / or sequences of images. By way of example and not limitation, based on image and other processing by computing module 140 based on methods described herein, computing module 140 may be used to discriminate, detect, and / or determine particular events of interest to prioritize storage options (e.g., by storage module 130). By way of example and not limitation, computing module 140 may be configured to detect temporally infrequent event(s) of interest, wherein images acquired during a dormant period may be discarded (or infrequently saved), whereas upon detection of an event of interest, computing module 140 may trigger saving a more information-rich data stream. In particular embodiments, adaptive storage may be suitably combined with other adaptive operations (e.g., adaptive sampling frame rate control).

[0244] In particular embodiments, control module 120 may be configured to perform an adaptive screening operation. By way of example and not limitation, screening experiments can be adaptively adjusted based on prior well results. By way of example and not limitation, quality control measures can be tightly integrated, and / or imaging or screening runs can be adaptively adjusted based on quality and / or yield metrics. By way of example and not limitation, event-driven experimental sequences like multiple compound sequences may be adaptively controlled. By way of example and not limitation, as discussed relating to adaptive storage, screening efficiency can be increased, such as by selectively storing and / or processing image sequences and / or videos that meet particular threshold screening metrics, such as molecule counts.

[0245] By way of example and not limitation, adaptive screening can comprise modulation and / or introduction of perturbing factors (e.g., external perturbations), such as adaptively introducing one or more compounds, electrical inputs, and / or genetic inputs. In particular embodiments, adaptive screening may comprise changing an experimental design, such as to optimize collection of data by finer sampling resolution and / or by adjusting field of view (FOV). By way of example and not limitation, adaptive screening controls may be activated when particular phenomena are occurring and / or particular results are available, and / or to avoid the collection and / or storage of data not related to phenomena and / or results of interest.

[0246] 3. Examples

[0247] 3.1 Example 1

[0248] As described herein, FIG. 7 illustrates a set of image pairs 700 depicting image reconstruction. The left side image (e.g., 715-1) of each image pair (e.g., 710-1 correspondingly) in the example of FIG. 7 was based on a single molecule tracking (SMT) movie captured by an electron multiplying charge-coupled device (EMCCD) sensor of imaging module 240. The right side image (e.g., 720-1) of each image pair (e.g., 710-1 correspondingly) was generated by processing by computing module 140 based on sparse representation (spot fingerprinting) methods described herein. As previously discussed, an R2value (e.g., 725-1) is provided for each pair of images to illustrate a fractional variance between the respective original and their corresponding reconstructed images. More specifically, as depicted in this non-limiting example, the R2values correspond to respective fractions of variance in each original spot retained in reconstruction. A fractional variance metric (e.g., 730-1) is also included in parentheses to indicate an estimated fraction of variance that is not due to shot noise (i.e., 1- mean / variance), under a shot noise-limited regime.

[0249] 3.2 Example 2 As described herein, FIG. 8B illustrates a schematic neural network training procedure, according to particular embodiments. The following non-limiting example of training set generation is further provided for illustration.

[0250] To generate ground truth emitters for the exemplary training set, a simple paraxial imaging system was simulated using a scalar diffraction-based approximation of the system’s point spread function (PSF). First, the paths of Brownian emitters were simulated at a fine temporal grain («1 ps) via the Euler-Maruyama method. For each integration time, the corresponding fragment of the emitter’s path was convolved with a three-dimensional paraxial point spread function at the desired numerical aperture (NA), generating defocus and motion blur. Simulated photons from these smeared PSFs were then sampled as a Poisson process, yielding simulated shot noise. Finally, these spots were added to a real unlabeled Single Molecule Tracking (SMT) movie, along with extra Gaussian noise. In this non-limiting exemplary training set generation process, the simulation depth was substantially greater than the imaging system’s depth of field. Accordingly, most simulated emitters were out-of-focus. As a result, training was performed only on emitters that were within 300 nm of the focus; both detected and undetected emitters were included in the training set. The true position of the emitter was taken to be its mean position during a corresponding integration time. More specifically, the target for inference was taken to be the mean position of each emitter during the corresponding integration time.

[0251] In this non-limiting exemplary training set generation process, the simulations incorporated a wide range of numerical apertures (NAs), densities, and spot brightness values, for a total training set exceeding 10 million spots, and a test set exceeding 1 million spots.

[0252] 3.3 Example 3

[0253] As provided herein, FIG. 8B illustrates a schematic neural network training procedure, according to particular embodiments. The following non-limiting example of training procedure is further provided for illustration.

[0254] In this non-limiting exemplary training procedure, training and test sets for training an exemplary neural network for subpixel localization comprised a collection of 11x11 pixel image fragments containing or bounding corresponding Regions of Interest (ROIs), along with accompanying (x, y) ground truth coordinates. Spots were normalized to mean zero, L2 norm 1, and projected into a 20-dimensional spot fingerprint basis. The exemplary non- limiting neural network illustrated was provided the goal to predict the (x, y) coordinates from the fingerprint, along with the error in the (x, y) coordinates. This corresponded to the following loss function:

[0255] In the above equation, the three outputs of the network were xpred, ypred, and Tpred, and is the softplus function. This further corresponded to Gaussian-distributed errors around the true position with ID root mean square deviation (RMSD) of (J = ^ L / S(Tpred. Training was performed from Glorot-initialized weights with the Adam algorithm, using a learning rate of 0.001. Training was stopped arbitrarily when the loss stopped decreasing.

[0256] Example 4

[0257] FIG. 8C illustrates results of neural network-based subpixel localization procedures, such as described herein, according to particular embodiments. Ref. 855 illustrates the accuracy of neural network modeling approach considered and disclosed herein. Each illustrated test dataset was comprised of simulated emitters on different kinds of backgrounds and in different spatial distributions. A broken line provided in Ref. 855 corresponds to the corresponding exemplary image size (0.1083 pm) of a single pixel after magnification. Ref. 860 illustrates estimated localization precision of different methods on real Single Molecule Tracking (SMT) data using H2B-HaloTag in U2OS cells, which were labeled with chloroalkylated JF549. Localization precision was estimated using a root negative jump covariance approach. Quantiles were calculated over movie-level means.

[0258] The control or comparison sets illustrated comprised: (a) Nearest pixel, wherein each spot was assigned to the center of the pixel in which it was detected, i.e., without subpixel localization; (b) Random 1x1, wherein each spot was assigned to a random location in the pixel in which it was detected; (c) Random 3x3, wherein each spot was assigned to a random location in the pixel in which it was detected or adjacent pixels; (d) Radial symmetry, or RS, based on a known closed-form estimator based on radial symmetry; (e) LS Int. Gauss, or LS, based on an iterative Levenberg-Marquardt fit of the observed image fragment to a 2D integrated Gaussian model; (f) nnv22, an embodiment of a non-fingerprint based neural network method for sub-pixel localization considered herein; (g) nnv24, an embodiment of a fingerprint-based neural network method for sub-pixel localization considered herein. Exemplary neural network architectures corresponding to nnv22 and nnv24 are illustrated and further described herein corresponding to FIG. 8A.

[0259] 3.5 Example 5

[0260] FIG. 8D illustrates detection performance results comparing fixed-point hardware- accelerated implementations and floating-point software-based implementations, according to particular embodiments. The fixed-point hardware acceleration results depicted herein were implemented on Field-Programmable Gate Array (FPGA) hardware comprising millions of gates. Results comparing the hardware and software implementations were drawn from validation studies covering over 10,000 test cases, and are illustrated herein based on three exemplary variations (plots I-III) in image compound types and noise estimates.

[0261] Each plot (I-III) of FIG. 8D depicts two sets of data, Recall data 865 and False Positive data 870. Values of Recall data 865 are associated with an implementation correctly identifying spots that do exist. Recall data 865 are comparable with each other on the y-axes within and across plots, with higher values indicating better performance. Values of False Positive data 870 are associated with an implementation erroneously identifying spots that do not exist. False Positive data 870 are comparable with each other on the y-axes within and across plots, with lower values indicating better performance. The x-axis of each plot (I-III) indicates variation of a detection threshold. In these examples, the detection threshold is expressed in units of a log likelihood ratio (LLR), and is indicative of how well each set of measurements aligns with a statistical model based on a representative point spread function (PSF).

[0262] The results depicted in FIG. 8D show close agreement between the fixed-point hardware-accelerated implementations and floating-point software-based implementations in each case. Specifically, each plot of Recall data 865 and of False Positive data 870 associated with each plot (I-III) respectively comprise data corresponding to both hardware and software implementations. Based on the relatively close agreement in performance of the hardware and software implementations, these data nearly overlap on the plots such that the hardware and software implementations may be difficult to visually differentiate at the plotted scales. An exemplary detailed view of False Positive data 870 provided as an inset in plot II schematically depicts software-based False Positive data 875 closely tracking hardware acceleration-based False Positive data 880. An exemplary detailed view of Recall data 865 provided as an inset in plot III schematically depicts software-based Recall data 885 closely tracking hardware acceleration-based Recall data 890.

[0263] Although the presently disclosed subject matter and its advantages have been described in detail, it should be understood that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the present disclosure. Moreover, the scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, and composition of matter, means, methods and steps described in the specification. As one of ordinary skill in the art will readily appreciate from the presently disclosed subject matter, processes, machines, manufacture, compositions of matter, means, methods, or steps, presently existing or later to be developed that perform substantially the same function or achieve substantially the same result as the corresponding embodiments described herein can be utilized according to the presently disclosed subject matter. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or steps.

[0264] Various patents, patent applications, publications, product descriptions, protocols, and sequence accession numbers are cited throughout this application, the contents of which are incorporated by reference in their entirety for all purposes.

Claims

CLAIMS:What is claimed is:

1. A method for hardware accelerated processing in a microscope system, the method comprising: receiving, by a hardware acceleration module of the microscope system, a first image of a sequence of images from an imaging module, each image respectively comprising a digital representation of a corresponding spatially distributed set of emitters located at a sample plane of a microscope module; computationally processing, by the hardware acceleration module, the first image, comprising: determining, based on performing one or more local filtering operations on the first image, a detection set of candidate emitter pixels corresponding to the respective set of emitters; and determining, based on performing a localizing computation for each candidate emitter pixel of the detection set, a localized set of coordinates corresponding to respective locations of the set of emitters, each of the localized set of coordinates having a sub-pixel resolution; providing, by the hardware acceleration module, the localized set of coordinates corresponding to the set of emitters of the first image to a control module of the microscope system; and receiving and computationally processing, by the hardware acceleration module, a second image of the sequence of images, wherein a processing time interval between processing the first image and processing the second image is less than or equal to a sampling time interval of the imaging module, thereby permitting adaptive control for single molecule tracking by the microscope system.

2. The method of claim 1, wherein a computing module of the microscope system comprises one or more computing processors or cores, and wherein the computing module comprises the hardware acceleration module.

3. The method of claim 2, wherein the computing module comprises a Field Programmable Gate Array (FPGA).

4. The method of claim 2, wherein the computing module comprises a GraphicsProcessing Unit (GPU).

5. The method of claim 2, wherein the computing module comprises a Central Processing Unit (CPU).

6. The method of claim 1, further comprising: determining a signal-to-noise ratio for each emitter of the set of emitters; and providing one or more of the determined signal-to-noise ratios to the control module of the microscope system.

7. The method of claim 1, wherein computationally processing the first image further comprises, prior to determining the localized set of coordinates: extracting, by the hardware acceleration module, a set of image fragments associated with the detection set, each image fragment comprising a plurality of pixels proximal to a respective candidate emitter pixel; and determining, by the hardware acceleration module, a set of filter values for each image fragment associated with the detection set, wherein each set of filter values is associated with a corresponding set of filters and comprises a reconstructible representation of the corresponding image fragment.

8. The method of claim 7, wherein one or more of the sets of filters associated with the detection set comprise orthogonal filters.

9. The method of claim 7, wherein the localized set of coordinates is determined based on applying a feed-forward neural network to the set of image fragments of the detection set, the feed-forward neural network trained based on one or more features of the set of filters.

10. The method of claim 1, wherein computationally processing the first image further comprises:extracting, by the hardware acceleration module, a set of image fragments associated with the detection set, each image fragment comprising a plurality of pixels proximal to a respective candidate emitter pixel; and performing, by the hardware acceleration module, an iterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates.

11. The method of claim 1, wherein computationally processing the first image further comprises: extracting, by the hardware acceleration module, a set of image fragments associated with the detection set, each image fragment comprising a plurality of pixels proximal to a respective candidate emitter pixel; and performing, by the hardware acceleration module, a non-iterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates.

12. The method of claim 11, wherein the at least a part of each image fragment comprises a plurality of discretized positions, and wherein the non-iterative optimization of the cost metric comprises non-iteratively optimizing an associated value that is evaluated at every position of the plurality of discretized positions.

13. The method of any of claims 10 to 12, wherein optimizing the cost metric is associated with optimizing a position of a localization kernel.

14. The method of claim 13, wherein the localization kernel comprises a Gaussian kernel.

15. The method of claim 1, wherein determining the detection set of candidate emitter pixels further comprises: determining, based on applying a first one-dimensional filter along a first dimension of the first image and applying a second one-dimensional filter along a second dimension of the first image, a filtering set of candidate emitter pixels corresponding to the respective set of emitters;extracting, based on determining the filtering set, a corresponding set of image fragments, each image fragment comprising a plurality of pixels proximal to a respective candidate emitter pixel of the filtering set, the plurality of pixels of each image fragment distributed along the first dimension and the second dimension of the first image; and determining, based on applying one or more two-dimensional filters along the first dimension and the second dimension of each image fragment, the detection set of candidate emitter pixels corresponding to the respective set of emitters, wherein determining the filtering set of candidate emitter pixels prior to application of the one or more two-dimensional filters and prior to performing the localizing computation facilitates low latency computational processing by the hardware acceleration module, thereby permitting adaptive feedback control of the microscope system for single molecule tracking.

16. The method of claim 1, wherein the localized set of coordinates is determined based on a radial symmetry of each emitter of the set of emitters.

17. The method of claim 15, wherein applying the first one-dimensional filter along the first dimension of the first image is simultaneously performed with applying the second one-dimensional filter along the second dimension of the first image.

18. The method of claim 15, wherein a plurality of the one or more two- dimensional filters are simultaneously applied along each image fragment during the computational processing by the hardware acceleration module.

19. The method of claim 15, wherein the first dimension is orthogonal to the second dimension.

20. The method of claim 1, further comprising pre-conditioning, prior to determining the detection set of candidate emitter pixels, the first image received by the hardware acceleration module.

21. The method of claim 20, wherein the pre-conditioning comprises removing, by the hardware acceleration module, at least a part of fixed pattern noise associated with the first image.

22. The method of claim 20, wherein the pre-conditioning comprises normalizing, by the hardware acceleration module, a respective pixel value associated with each pixel of the first image.

23. The method of claim 20, wherein the pre-conditioning comprises performing, by the hardware acceleration module, a de-noising operation on the first image.

24. The method of claim 20, wherein pre-conditioning comprises performing, by the hardware acceleration module and based on a reference image, a background subtraction operation on the first image.

25. The method of claim 1, further comprising filtering, prior to determining the localized set of coordinates, the detection set of candidate emitter pixels based on one or more filtering criteria.

26. The method of claim 25, wherein the filtering criteria comprises a threshold value of a log likelihood ratio.

27. The method of claim 25, wherein the filtering criteria comprises a threshold value of a signal-to-noise ratio.

28. The method of claim 25, wherein the filtering criteria comprise one or more peak searching criteria.

29. The method of claim 1, wherein memory buffering is used to facilitate consistency of a mean processing time interval of the hardware acceleration module.

30. The method of claim 1, further comprising performing an adaptive focusing operation by the control module of the microscope system.

31. The method of claim 30, further comprising: determining a focusing score generated by an autofocus module of the imaging module; and operating, by the control module and based on optimizing the focusing score, a focusing mechanism of the microscope module to obtain or maintain focus on at least a portion of the set of emitters located at the sample plane, for single molecule tracking.

32. The method of claim 31, wherein operating the focusing mechanism is associated with changing a distance between an objective and a stage of the microscope module, the stage associated with receiving a sample comprising emitters.

33. The method of claim 1, further comprising performing an adaptive imaging operation by the control module of the microscope system.

34. The method of claim 33, further comprising providing one or more signals to a laser engine module as an image exposure feedback operation, the one or more signals configured to modify an operational set point of a laser output to facilitate live-cell fluorescence imaging by the microscope system.

35. The method of claim 34, comprising providing one or more respective signals to each of a first laser configured for illuminating the sample plane and a second laser configured for controlling an emission of at least a portion of the set of emitters.

36. The method of claim 33, further comprising providing one or more signals to the imaging module as a field of view feedback operation, the one or more signals configured to modify a field of view associated with acquiring the sequence of images.

37. The method of claim 36, wherein the field of view is modified based on operating a scanning optical component or a translating optical component of the microscope system.

38. The method of claim 36, wherein the field of view is modified based on operating a galvo mirror of the microscope system.

39. The method of claim 36, wherein the field of view is modified based on altering one or more sensor parameters of the imaging module.

40. The method of claim 33, further comprising providing one or more signals to the imaging module as a frame rate feedback operation, the one or more signals configured to modify a frame rate or an exposure time of acquiring the sequence of images.

41. The method of claim 40, further comprising, prior to or concurrent with modifying the frame rate, providing a signal to a laser engine module to modify an operational set point of a laser output.

42. The method of claim 1, further comprising performing an adaptive screening operation by the control module of the microscope system.

43. A method of computationally processing image data associated with a microscope system, the processing performed by a computing module comprising one or more processors, the method comprising: receiving, by the computing module, a first image of a sequence of images, the first image comprising a digital representation of a corresponding spatially distributed set of emitters located at a sample plane of a microscope module; determining, by the computing module, a detection set of candidate emitter pixels corresponding to the respective set of emitters; extracting, by the computing module, a set of image fragments associated with the detection set, each image fragment comprising a plurality of pixels proximal to a respective candidate emitter pixel; determining, by the computing module, a set of filter values for each image fragment associated with the detection set, wherein each set of filter values is associated with a corresponding set of filters and comprises a reconstructible representation of the corresponding image fragment; andproviding, by the computing module, one or more of the sets of filter values associated with the first image for storage in a data storage module associated with the microscope system.

44. The method of claim 43, wherein one or more of the sets of filters associated with the detection set comprise orthogonal filters.

45. The method of claim 43, further comprising determining, by the computing module, a localized set of coordinates based on determining each set of filter values, the localized set of coordinates corresponding to respective locations of the set of emitters, each of the localized set of coordinates having a sub-pixel resolution.

46. The method of claim 45, further comprising performing, by the computing module, an iterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates.

47. The method of claim 45, further comprising performing, by the computing module, a non-iterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates.

48. The method of claim 47, wherein the at least a part of each image fragment comprises a plurality of discretized positions, and wherein the non-iterative optimization of the cost metric comprises non-iteratively optimizing an associated value that is evaluated at every position of the plurality of discretized positions.

49. The method of any of claims 46 to 48, wherein optimizing the cost metric is associated with optimizing a position of a localization kernel.

50. The method of claim 49, wherein the localization kernel comprises aGaussian kernel.

51. The method of claim 45, further comprising providing, by the computing module, the localized set of coordinates to a control module of the microscope system to facilitate performing one or more adaptive feedback operations.

52. The method of claim 45, wherein the localized set of coordinates is determined based on applying a feed-forward neural network to the set of filter values associated with the respective image fragment.

53. The method of claim 45, wherein the localized set of coordinates is determined based on applying a feed-forward neural network to the set of image fragments of the detection set, the feed-forward neural network trained based on one or more features of the set of filters.

54. The method of claim 43, further comprising receiving, by the computing module, a second image of the sequence of images, wherein a time interval for computationally processing the first image prior to receiving the second image is less than or equal to a sampling time interval of an imaging module of the microscope system, thereby permitting adaptive control for single molecule tracking by the microscope system.

55. The method of claim 43, wherein the computing module comprises a plurality of computing processors or cores.

56. The method of claim 55, wherein the plurality of computing processors or cores are heterogeneous.

57. The method of claim 43, wherein the computing module comprises a hardware accelerated processing module.

58. The method of claim 57, wherein the hardware accelerated processing module comprises a Field Programmable Gate Array (FPGA).

59. A microscope system comprising: a microscope module comprising:a light source optically connected for illuminating a sample plane; and an imaging module comprising a detector device and configured to acquire a sequence of images, each image respectively comprising a digital representation of a spatially distributed set of emitters located at the sample plane during a corresponding sampling time interval; a computing module including a hardware acceleration module configured to: receive a first image of the sequence of images from the imaging module; determine a detection set of candidate emitter pixels corresponding to the respective set of emitters based on performing one or more local filtering operations on the first image; determine a localized set of coordinates corresponding to the respective set of emitters based on performing a localizing computation for each candidate emitter pixel of the detection set, each of the localized set of coordinates having a sub-pixel resolution; and receive a second image of the sequence of images from the imaging module; and a control module configured to receive the localized set of coordinates from the hardware acceleration module and perform an adaptive feedback operation associated with the microscope system, wherein a processing time interval between determining the respective localized sets of coordinates corresponding to the first image and the second image is less than or equal to the sampling time interval of the imaging module, thereby permitting adaptive control for single molecule tracking by the microscope system.

60. The microscope system of claim 59, wherein the computing module comprises one or more computing processors or cores.

61. The microscope system of claim 60, wherein the computing module comprises a Field Programmable Gate Array (FPGA).

62. The microscope system of claim 60, wherein the computing module comprises a Graphics Processing Unit (GPU).

63. The microscope system of claim 60, wherein the computing module comprises a Central Processing Unit (CPU).

64. The microscope system of claim 59, wherein the hardware acceleration module is further configured to: determine a signal-to-noise ratio for each emitter of the set of emitters; and provide one or more of the determined signal-to-noise ratios to the control module of the microscope system.

65. The microscope system of claim 59, wherein, prior to determining the localized set of coordinates, the hardware acceleration module is further configured to: extract a set of image fragments associated with the detection set, each image fragment comprising a plurality of pixels proximal to a respective candidate emitter pixel; and determine a set of filter values for each image fragment associated with the detection set, wherein each set of filter values is associated with a corresponding set of filters and comprises a reconstructible representation of the corresponding image fragment.

66. The microscope system of claim 65, wherein one or more of the sets of filters associated with the detection set comprise orthogonal filters.

67. The microscope system of claim 65, wherein the localized set of coordinates is determined based on applying a feed-forward neural network to the set of filter values associated with the respective image fragment.

68. The microscope system of claim 65, wherein the localized set of coordinates is determined based on applying a feed-forward neural network to the set of image fragments of the detection set, the feed-forward neural network trained based on one or more features of the set of filters.

69. The microscope system of claim 59, wherein the hardware acceleration module is further configured to:extract a set of image fragments associated with the detection set, each image fragment comprising a plurality of pixels proximal to a respective candidate emitter pixel; and perform an iterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates.

70. The microscope system of claim 59, wherein the hardware acceleration module is further configured to: extract a set of image fragments associated with the detection set, each image fragment comprising a plurality of pixels proximal to a respective candidate emitter pixel; and perform a non-iterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates.

71. The microscope system of claim 70, wherein the at least a part of each image fragment comprises a plurality of discretized positions, and wherein the non-iterative optimization of the cost metric comprises non-iteratively optimizing an associated value that is evaluated at every position of the plurality of discretized positions.

72. The microscope system of any of claims 69 to 71, wherein optimizing the cost metric is associated with optimizing a position of a localization kernel.

73. The microscope system of claim 72, wherein the localization kernel comprises a Gaussian kernel.

74. The microscope system of claim 59, wherein, for determining the detection set of candidate emitter pixels, the hardware acceleration module is further configured to: determine a filtering set of candidate emitter pixels corresponding to the respective set of emitters based on applying a first one-dimensional filter along a first dimension of the first image and applying a second one-dimensional filter along a second dimension of the first image; extract a corresponding set of image fragments based on determining the filtering set, each image fragment comprising a plurality of pixels proximal to a respective candidateemitter pixel of the filtering set, the plurality of pixels of each image fragment distributed along the first dimension and the second dimension of the first image; and determine the detection set of candidate emitter pixels corresponding to the respective set of emitters based on applying one or more two-dimensional filters along the first dimension and the second dimension of each image fragment, wherein determining the filtering set of candidate emitter pixels prior to application of the one or more two-dimensional filters and prior to performing the localizing computation facilitates low latency computational processing by the hardware acceleration module, thereby permitting adaptive feedback control of the microscope system for single molecule tracking.

75. The microscope system of claim 59, wherein the localized set of coordinates is determined based on a radial symmetry of each emitter of the set of emitters.

76. The microscope system of claim 74, wherein applying the first onedimensional filter along the first dimension of the first image is simultaneously performed with applying the second one-dimensional filter along the second dimension of the first image.

77. The microscope system of claim 74, wherein a plurality of the one or more two-dimensional filters are simultaneously applied along each image fragment during the computational processing by the hardware acceleration module.

78. The microscope system of claim 59, wherein, prior to determining the detection set of candidate emitter pixels, the hardware acceleration module is further configured to pre-condition the first image.

79. The microscope system of claim 78, wherein the pre-conditioning comprises removing, by the hardware acceleration module, at least a part of fixed pattern noise associated with the first image.

80. The microscope system of claim 78, wherein the pre-conditioning comprises normalizing, by the hardware acceleration module, a respective pixel value associated with each pixel of the first image.

81. The microscope system of claim 78, wherein the pre-conditioning comprises performing, by the hardware acceleration module, a de-noising operation on the first image.

82. The microscope system of claim 78, wherein the pre-conditioning comprises performing, by the hardware acceleration module, a background subtraction operation on the first image.

83. The microscope system of claim 59, wherein, prior to determining the localized set of coordinates, the hardware acceleration module is further configured to filter the detection set of candidate emitter pixels based on one or more filtering criteria.

84. The microscope system of claim 83, wherein the filtering criteria comprises a threshold value of a log likelihood ratio.

85. The microscope system of claim 83, wherein the filtering criteria comprises a threshold value of a signal-to-noise ratio.

86. The microscope system of claim 83, wherein the filtering criteria comprise one or more peak searching criteria.

87. The microscope system of claim 60, further comprising a memory buffer to facilitate consistency of a mean processing time interval of the hardware acceleration module.

88. The microscope system of claim 59, wherein the adaptive feedback operation performed by the control module comprises an adaptive focusing operation.

89. The microscope system of claim 88, wherein the adaptive focusing operation comprises:determining a focusing score generated by an autofocus module of the imaging module; and operating, by the control module and based on optimizing the focusing score, a focusing mechanism of the microscope module to obtain or maintain focus on at least a portion of the set of emitters located at the sample plane, for single molecule tracking.

90. The microscope system of claim 89, wherein operating the focusing mechanism is associated with changing a distance between an objective and a stage of the microscope module, the stage associated with receiving a sample comprising emitters.

91. The microscope system of claim 59, wherein the adaptive feedback operation performed by the control module comprises an adaptive imaging operation.

92. The microscope system of claim 91, wherein the adaptive imaging operation further comprises providing one or more signals to a laser engine module as an image exposure feedback operation, the one or more signals configured to modify an operational set point of a laser output to facilitate live-cell fluorescence imaging by the microscope system.

93. The microscope system of claim 92, wherein the adaptive imaging operation comprises providing one or more respective signals to each of a first laser configured for illuminating the sample plane and a second laser configured for controlling an emission of at least a portion of the set of emitters.

94. The microscope system of claim 91, wherein the adaptive imaging operation further comprises providing one or more signals to the imaging module as a field of view feedback operation, the one or more signals configured to modify a field of view associated with acquiring the sequence of images.

95. The microscope system of claim 94, wherein the field of view is modified based on operating a scanning optical component or a translating optical component of the microscope system.

96. The microscope system of claim 94, wherein the field of view is modified based on operating a galvo mirror of the microscope system.

97. The microscope system of claim 94, wherein the field of view is modified based on altering one or more sensor parameters of the imaging module.

98. The microscope system of claim 91, wherein the adaptive imaging operation further comprises providing one or more signals to the imaging module as a frame rate feedback operation, the one or more signals configured to modify a frame rate or an exposure time of acquiring the sequence of images.

99. The microscope system of claim 98, wherein the adaptive imaging operation further comprises, prior to or concurrent with modifying the frame rate, providing a signal to a laser engine module to modify an operational set point of a laser output.

100. The microscope system of claim 59, wherein the adaptive feedback operation performed by the control module comprises an adaptive screening operation.

101. A microscope system comprising: a microscope module comprising: a light source optically connected for illuminating a sample plane; and an imaging module comprising a detector device and configured to acquire a sequence of images, each image respectively comprising a digital representation of a spatially distributed set of emitters located at the sample plane during a corresponding sampling time interval; a computing module configured to: receive a first image of the sequence of images from the imaging module; determine a detection set of candidate emitter pixels corresponding to the respective set of emitters; extract a set of image fragments associated with the detection set, each image fragment comprising a plurality of pixels proximal to a respective candidate emitter pixel;determine a set of filter values for each image fragment associated with the detection set, wherein each set of filter values is associated with a corresponding set of filters and comprises a reconstructible representation of the corresponding image fragment; and transmit one or more of the sets of filter values associated with the first image to one or more of a storage module or a control module.

102. The microscope system of claim 101, wherein one or more of the sets of filters associated with the detection set comprise orthogonal filters.

103. The microscope system of claim 101, wherein the computing module is further configured to determine a localized set of coordinates based on determining each set of filter values, the localized set of coordinates corresponding to respective locations of the set of emitters, each of the localized set of coordinates having a sub-pixel resolution.

104. The microscope system of claim 103, wherein the computing module is further configured to provide the localized set of coordinates to a control module of the microscope system to facilitate performing one or more adaptive feedback operations.

105. The microscope system of claim 101, wherein the computing module is further configured to: determine a signal-to-noise ratio for each emitter of the set of emitters; and provide one or more of the determined signal-to-noise ratios to one or more of the storage module or the control module.

106. The microscope system of claim 103, wherein the computing module is configured to determine the localized set of coordinates based on applying a feed-forward neural network to the set of filter values associated with the respective image fragment.

107. The microscope system of claim 103, wherein the computing module is configured to determine the localized set of coordinates based on applying a feed-forward neural network to the set of image fragments of the detection set, the feed-forward neural network trained based on one or more features of the set of filters.

108. The microscope system of claim 103, wherein the computing module is further configured to perform an iterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates.

109. The microscope system of claim 103, wherein the computing module is further configured to perform a non-iterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates.

110. The microscope system of claim 109, wherein the at least a part of each image fragment comprises a plurality of discretized positions, and wherein the non-iterative optimization of the cost metric comprises non-iteratively optimizing an associated value that is evaluated at every position of the plurality of discretized positions.

111. The microscope system of any of claims 108 to 110, wherein optimizing the cost metric is associated with optimizing a position of a localization kernel.

112. The microscope system of claim 111, wherein the localization kernel comprises a Gaussian kernel.

113. The microscope system of claim 101, wherein the computing module is configured to receive a second image of the sequence of images, wherein a time interval for computationally processing the first image prior to receiving the second image is less than or equal to a sampling time interval of the imaging module of the microscope system, thereby permitting adaptive control for single molecule tracking by the microscope system.

114. The microscope system of claim 101, wherein the computing module comprises a plurality of computing processors or cores.

115. The microscope system of claim 114, wherein the plurality of computing processors or cores are heterogeneous.

116. The microscope system of claim 101, wherein the computing module comprises a hardware accelerated processing module.

117. The microscope system of claim 116, wherein the hardware accelerated processing module comprises a Field Programmable Gate Array (FPGA).

118. A computing module for a microscope system, the computing module comprising a hardware acceleration module configured to: receive a first image of a sequence of images from an imaging module, each image of the sequence of images respectively comprising a digital representation of a spatially distributed set of emitters located at a sample plane of the microscope system during a corresponding sampling time interval; determine a detection set of candidate emitter pixels corresponding to the respective set of emitters based on performing one or more local filtering operations on the first image; determine a localized set of coordinates corresponding to the respective set of emitters based on performing a localizing computation for each candidate emitter pixel of the detection set, each of the localized set of coordinates having a sub-pixel resolution; and receive a second image of the sequence of images from the imaging module, wherein a processing time interval between determining the respective localized sets of coordinates corresponding to the first image and the second image is less than or equal to the sampling time interval of the imaging module, thereby permitting adaptive control for single molecule tracking by the microscope system.

119. The computing module of claim 118, comprising one or more computing processors or cores.

120. The computing module of claim 119, comprising a Field Programmable Gate Array (FPGA).

121. The computing module of claim 119, comprising a Graphics Processing Unit(GPU).

122. The computing module of claim 119, comprising a Central Processing Unit(CPU).

123. The computing module of claim 118, wherein the hardware acceleration module is further configured to: determine a signal-to-noise ratio for each emitter of the set of emitters; and provide one or more of the determined signal-to-noise ratios to a control module of the microscope system.

124. The computing module of claim 118, wherein, prior to determining the localized set of coordinates, the hardware acceleration module is further configured to: extract a set of image fragments associated with the detection set, each image fragment comprising a plurality of pixels proximal to a respective candidate emitter pixel; and determine a set of filter values for each image fragment associated with the detection set, wherein each set of filter values is associated with a corresponding set of filters and comprises a reconstructible representation of the corresponding image fragment.

125. The computing module of claim 124, wherein one or more of the sets of filters associated with the detection set comprise orthogonal filters.

126. The computing module of claim 124, wherein the localized set of coordinates is determined based on applying a feed-forward neural network to the set of filter values associated with the respective image fragment.

127. The computing module of claim 124, wherein the localized set of coordinates is determined based on applying a feed-forward neural network to the set of image fragments of the detection set, the feed-forward neural network trained based on one or more features of the set of filters.

128. The computing module of claim 118, wherein the hardware acceleration module is further configured to:extract a set of image fragments associated with the detection set, each image fragment comprising a plurality of pixels proximal to a respective candidate emitter pixel; and perform an iterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates.

129. The computing module of claim 118, wherein the hardware acceleration module is further configured to: extract a set of image fragments associated with the detection set, each image fragment comprising a plurality of pixels proximal to a respective candidate emitter pixel; and perform a non-iterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates.

130. The computing module of claim 129, wherein the at least a part of each image fragment comprises a plurality of discretized positions, and wherein the non-iterative optimization of the cost metric comprises non-iteratively optimizing an associated value that is evaluated at_every position of the plurality of discretized positions.

131. The computing module of any of claims 128 to 130, wherein optimizing the cost metric is associated with optimizing a position of a localization kernel.

132. The computing module of claim 131, wherein the localization kernel comprises a Gaussian kernel.

133. The computing module of claim 118, wherein, for determining the detection set of candidate emitter pixels, the hardware acceleration module is further configured to: determine a filtering set of candidate emitter pixels corresponding to the respective set of emitters based on applying a first one-dimensional filter along a first dimension of the first image and applying a second one-dimensional filter along a second dimension of the first image; extract a corresponding set of image fragments based on determining the filtering set, each image fragment comprising a plurality of pixels proximal to a respective candidateemitter pixel of the filtering set, the plurality of pixels of each image fragment distributed along the first dimension and the second dimension of the first image; and determine the detection set of candidate emitter pixels corresponding to the respective set of emitters based on applying one or more two-dimensional filters along the first dimension and the second dimension of each image fragment, wherein determining the filtering set of candidate emitter pixels prior to application of the one or more two-dimensional filters and prior to performing the localizing computation facilitates low latency computational processing by the hardware acceleration module, thereby permitting adaptive feedback control of the microscope system for single molecule tracking.

134. The computing module of claim 118, wherein the localized set of coordinates is determined based on a radial symmetry of each emitter of the set of emitters.

135. The computing module of claim 133, wherein applying the first onedimensional filter along the first dimension of the first image is simultaneously performed with applying the second one-dimensional filter along the second dimension of the first image.

136. The computing module of claim 133, wherein a plurality of the one or more two-dimensional filters are simultaneously applied along each image fragment during the computational processing by the hardware acceleration module.

137. The computing module of claim 118, wherein, prior to determining the detection set of candidate emitter pixels, the hardware acceleration module is further configured to pre-condition the first image.

138. The computing module of claim 137, wherein the pre-conditioning comprises removing, by the hardware acceleration module, at least a part of fixed pattern noise associated with the first image.

139. The computing module of claim 137, wherein the pre-conditioning comprises normalizing, by the hardware acceleration module, a respective pixel value associated with each pixel of the first image.

140. The computing module of claim 137, wherein the pre-conditioning comprises performing, by the hardware acceleration module, a de-noising operation on the first image.

141. The computing module of claim 137, wherein the pre-conditioning comprises performing, by the hardware acceleration module, a background subtraction operation on the first image.

142. The computing module of claim 118, wherein, prior to determining the localized set of coordinates, the hardware acceleration module is further configured to filter the detection set of candidate emitter pixels based on one or more filtering criteria.

143. The computing module of claim 142, wherein the filtering criteria comprises a threshold value of a log likelihood ratio.

144. The computing module of claim 142, wherein the filtering criteria comprises a threshold value of a signal-to-noise ratio.

145. The computing module of claim 142, wherein the filtering criteria comprise one or more peak searching criteria.

146. The computing module of claim 60, further comprising a memory buffer to facilitate consistency of a mean processing time interval of the hardware acceleration module.

147. A computing module for a microscope system, the computing module configured to: receive a first image of a sequence of images from an imaging module, each image of the sequence of images respectively comprising a digital representation ofa spatially distributed set of emitters located at a sample plane of the microscope system during a corresponding sampling time interval; determine a detection set of candidate emitter pixels corresponding to the respective set of emitters; extract a set of image fragments associated with the detection set, each image fragment comprising a plurality of pixels proximal to a respective candidate emitter pixel; determine a set of filter values for each image fragment associated with the detection set, wherein each set of filter values is associated with a corresponding set of filters and comprises a reconstructible representation of the corresponding image fragment; and transmit one or more of the sets of filter values associated with the first image to one or more of a storage module or a control module.

148. The computing module of claim 147, wherein one or more of the sets of filters associated with the detection set comprise orthogonal filters.

149. The computing module of claim 147, wherein the computing module is further configured to determine a localized set of coordinates based on determining each set of filter values, the localized set of coordinates corresponding to respective locations of the set of emitters, each of the localized set of coordinates having a sub-pixel resolution.

150. The computing module of claim 149, wherein the computing module is further configured to provide the localized set of coordinates to a control module of the microscope system to facilitate performing one or more adaptive feedback operations.

151. The computing module of claim 147, wherein the computing module is further configured to: determine a signal-to-noise ratio for each emitter of the set of emitters; and provide one or more of the determined signal-to-noise ratios to one or more of the storage module or the control module.

152. The computing module of claim 149, wherein the computing module is configured to determine the localized set of coordinates based on applying a feed-forward neural network to the set of filter values associated with the respective image fragment.

153. The computing module of claim 149, wherein the computing module is configured to determine the localized set of coordinates based on applying a feed-forward neural network to the set of image fragments of the detection set, the feed-forward neural network trained based on one or more features of the set of filters.

154. The computing module of claim 149, wherein the computing module is further configured to perform an iterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates.

155. The computing module of claim 149, wherein the computing module is further configured to perform a non-iterative optimization of a cost metric over at least a part of each image fragment to determine the localized set of coordinates.

156. The computing module of claim 155, wherein the at least a part of each image fragment comprises a plurality of discretized positions, and wherein the non-iterative optimization of the cost metric comprises non-iteratively optimizing an associated value that is evaluated at_every position of the plurality of discretized positions.

157. The computing module of any of claims 154 to 156, wherein optimizing the cost metric is associated with optimizing a position of a localization kernel.

158. The computing module of claim 157, wherein the localization kernel comprises a Gaussian kernel.

159. The computing module of claim 147, wherein the computing module is configured to receive a second image of the sequence of images, wherein a time interval for computationally processing the first image prior to receiving the second image is less than or equal to a sampling time interval of the imaging module of the microscope system, thereby permitting adaptive control for single molecule tracking by the microscope system.

160. The computing module of claim 147, wherein the computing module comprises a plurality of computing processors or cores.

161. The computing module of claim 160, wherein the plurality of computing processors or cores are heterogeneous.

162. The computing module of claim 147, wherein the computing module comprises a hardware accelerated processing module.

163. The computing module of claim 162, wherein the hardware accelerated processing module comprises a Field Programmable Gate Array (FPGA).

Images