Increased computational efficiency for structured illumination microscopy

Abstract

The disclosed technology relates to structured illumination microscopy (SIM). In particular, the disclosed technology relates to imaging and processing multiple image tiles across a large image plane in real time, dividing them into subtiles, efficiently processing the subtiles, and generating an improved resolution image from the subtiles. The improved resolution images can be combined into an improved image and used in subsequent analysis steps. The disclosed technology includes logic that reduces the computing resources required to generate an improved resolution image from structured illumination of a target. A method is described for generating an improved resolution image from an image of a target imaged under structured illumination. The method applies one or more transforms to non-redundant data and then recovers redundant data from the non-redundant data after the transforms.

Description

[Technical field]

[0001] (Priority Application) This application claims the benefit of and incorporates by reference U.S. Patent Application No. 62 / 924,130, filed October 21, 2019, entitled "Systems and Methods for Stagically Illumination Microscopy," Attorney Docket No. ILLM 1012-1, and U.S. Provisional Patent Application No. 62 / 924,138, filed October 21, 2019, entitled "Increased Calculation Efficiency for Structured Illumination Microscopy." This application also claims the benefit of and incorporates by reference U.S. Patent Application No. 62 / 924,130, filed October 21, 2020, entitled "Systems and Methods for Stagically Illumination Microscopy," Attorney Docket No. ILLM 1022-1. em No. 17 / 075,692, entitled "Increased Calculation Efficiency for Structured Illumination Microscopy," filed on October 21, 2020 (Attorney Docket No. ILLM1012-2), and U.S. Non-provisional Patent Application No. 17 / 075,694, entitled "Increased Calculation Efficiency for Structured Illumination Microscopy," filed on October 21, 2020 (Attorney Docket No. ILLM1022-2), both of which claim the benefit of the above-referenced provisional applications.

[0002] (Built-in) The following are incorporated by reference for all purposes as if fully set forth herein: U.S. Provisional Patent Application No. 62 / 692,303, entitled "Devices for Luminescent Imaging," filed June 29, 2018 (unpublished), and U.S. Nonprovisional Patent Application No. "Device for Luminescent Imaging," filed June 29, 2019 (Attorney Docket No. IP-1683-US).

[0003] FIELD OF THEINVENTION The disclosed technology relates to structured illumination microscopy (SIM). In particular, the disclosed technology relates to imaging and processing multiple image tiles across a large image plane in real time, dividing them into subtiles, efficiently processing the subtiles, and generating improved resolution images from the subtiles, which can be combined into an improved image and used in subsequent analysis steps.

[0004] The disclosed technology relates to structured illumination microscopy. In particular, the disclosed technology relates to reducing the computation required to process a large number of image tiles across a large image plane in real time and generating improved resolution images from the image tiles / subtiles. During several intermediate transformations in the SIM processing chain, roughly half of the multiplexing and splitting that would otherwise be required is eliminated in the described method. Hermite It can be replaced by a search operation using a particular exploitation of the symmetry. [Background technology]

[0005] The subject matter discussed in this section should not be assumed to be prior art merely as a result of its reference in this section. Similarly, it should not be assumed that the problems referenced in this section, or related to the subject matter provided as background, have been previously recognized in the prior art. The subject matter in this section merely represents different approaches, which as such may also correspond to embodiments of the claimed technology.

[0006] More than a decade ago, the pioneers of structured illumination microscopy were awarded the Nobel Prize in Physics. Improving image resolution beyond the Abbe diffraction limit was a key development.

[0007] Both 2D and 3D SIM have been applied to image biological samples, such as parts of individual cells. Much effort has been expended and many alternative technical variants have resulted from the effort to study the interior of cells.

[0008] Resolving millions of sources spread across an image plane presents a problem quite different from that seen inside cells. For example, one new approach that has emerged combines multiple images with specular illumination to produce an improved resolution image after extensive computation. Real-time processing of large image planes with modest resources requires a fundamentally different approach from such recent work.

[0009] Therefore, an opportunity arises to introduce new methods and systems adapted to process large image planes with reduced computing resource requirements. Summary of the Invention [Means for solving the problem]

[0010] An embodiment of the present disclosure provides a method for generating an improved resolution image from an image of a target captured under structured illumination, the method comprising: transforming at least three images of the target imaged by the sensor in the spatial domain from at least three phase displacements of the structured illumination along an angle into a Fourier domain to generate at least three frequency domain matrices including at least non-redundant and redundant conjugate components that are conjugates at complementary matrix positions of the non-redundant components; applying an unmixing matrix to at least the non-redundant components using the estimated reconstruction parameters to generate at least three phase separation matrices in the Fourier domain, the unshifted and shifted matrices including non-redundant non-shifted and non-redundant shifted spatial frequency components derived from the non-redundant components of the frequency domain matrix; performing one or more intermediate transformations on at least non-redundant components of the phase separation matrix, performing a sub-pixel shift of at least non-redundant shifted spatial frequency components of the shift matrix by transforming from the Fourier domain to the spatial domain; applying a translation vector to the data in the spatial domain; and transforming the translated data from the spatial domain data into one or more realigned shift matrices in the Fourier domain, where applying the translation vector effectively realigns duplicate values ​​in the shift matrix in the Fourier domain with a more accurate non-shifted matrix compared to full matrix positions of a discrete Fourier transform; recovering the redundant conjugate components from the non-redundant components after the intermediate transformation by copying transformed values ​​from the non-redundant components to complementary matrix locations of the redundant conjugate components and changing the sign of the imaginary parts of the copied values ​​to generate transformed redundant conjugate components; aligning and summing the overlapping values ​​of the unshifted and shifted matrices to generate an extended frequency coverage matrix; and inverting the extended frequency coverage matrix from the Fourier domain to generate an improved resolution image in the spatial domain. [Brief description of the drawings]

[0011] Color drawings in the priority application corresponding to the black and white drawings in this application may be available from the USPTO in PAIR via the supplemental content tab. In the drawings, like reference characters generally refer to like parts throughout the different views. Also, the drawings are not necessarily to scale, emphasis instead being placed on illustrating the principles of the disclosed technology. In the following description, various embodiments of the disclosed technology are described with reference to the following drawings:

[0012] [Figure 1A] An example of Moire fringe formation by using a grating with one dimensional (1D) modulation is shown. [Figure 1B] A graphical representation of the illumination intensity produced by a two dimensional (2D) structured illumination pattern is presented. [Figure 1C]1 shows exemplary geometric patterns of nanowell arrays. [Diagram 2] 1 shows a structured illumination microscopy imaging system capable of imaging a sample using spatially structured excitation light. [Diagram 3] FIG. 1 is an optical system diagram showing an exemplary optical configuration for a two-arm SIM imaging system. [Figure 4A] FIG. 1 is an exemplary schematic diagram illustrating the optical configuration of a dual optical grating slide SIM photography system. [Figure 4B] FIG. 1 is an exemplary schematic diagram illustrating the optical configuration of a dual optical grating slide SIM photography system. [Figure 5A] 1 illustrates undesirable variations in spacing parameters that can occur in a SIM imaging system. [Figure 5B] 1 illustrates undesirable changes in phase parameters that can occur in a SIM imaging system. [Figure 5C] 1 illustrates undesirable changes in the angle of the orientation parameters that can occur in a SIM imaging system. [Figure 6] 1 shows a simplified illumination fringe pattern that may be projected onto the plane of a sample by the vertical and horizontal gratings of a SIM imaging system. [Figure 7] 1 illustrates a simplified illumination fringe pattern that may be projected onto a plane by a first and second grating of a dual optical grating slide SIM imaging system. [Figure 8A] 1 is a simplified diagram of parallel lines curved due to distortion of a magnifying lens. [Figure 8B] Indicates the spacing between nominally parallel lines. [Figure 8C] Indicates the spacing between nominally parallel lines. [Figure 8D] 1 shows an example of subtiles or subfields of a full field of view (FOV) image. [Figure 9] Generally, it shows a color-coded surface that fits well to the observed data points shown in red. [Figure 10A] 1 shows an inverted bowl shape of the measured spacing versus the spacing in the center-nearby subfields. [Figure 10B] 1 shows an inverted bowl shape of the measured spacing versus the spacing in the center-nearby subfields. [Figure 11A] The measured interval distortion is compared to a quadratic surface fit and a cubic surface fit. [Figure 11B] The measured interval distortion is compared to a quadratic surface fit and a cubic surface fit. [Figure 11C] The measured interval distortion is compared to a quadratic surface fit and a cubic surface fit. [Figure 12A] Measured data points are shown that have not been smoothed by curve fitting. [Figure 12B] 1 shows the actual data compared to the 2D fitted surface. [Figure 13A] 13 illustrates the improvement of curve fitting by cropping along the sensor boundaries. [Figure 13B] 13 illustrates the improvement of curve fitting by cropping along the sensor boundaries. [Figure 13C] 13 illustrates the improvement of curve fitting by cropping along the sensor boundaries. [Figure 13D] 13 illustrates the improvement of curve fitting by cropping along the sensor boundaries. [Figure 13E] 13 illustrates the improvement of curve fitting by cropping along the sensor boundaries. [Figure 14A] The graph shows the reduction in mean squared error (MSE) for different reduction factors. [Figure 14B] We show an improvement in the two-dimensional fit to the angular distortion model by incrementally increasing the demagnification factor applied to the full FOV image data. [Figure 14C] We show an improvement in the two-dimensional fit to the angular distortion model by incrementally increasing the demagnification factor applied to the full FOV image data. [Figure 14D] We show an improvement in the two-dimensional fit to the angular distortion model by incrementally increasing the demagnification factor applied to the full FOV image data. [Figure 14E]We show an improvement in the two-dimensional fit to the angular distortion model by incrementally increasing the demagnification factor applied to the full FOV image data. [Figure 14F] We show an improvement in the two-dimensional fit to the angular distortion model by incrementally increasing the demagnification factor applied to the full FOV image data. [Figure 14G] We show an improvement in the two-dimensional fit to the angular distortion model by incrementally increasing the demagnification factor applied to the full FOV image data. [Figure 14H] 13 is a graphical plot showing the marginal improvement in a two-dimensional surface fit when the reduction factor value is six. [Figure 15A] A two-dimensional fit of the response surface obtained by fitting the fringe spacing distortion (A) and the fringe angle distortion (B) over the full FOV image data is shown. [Figure 15B] A two-dimensional fit of the response surface obtained by fitting the fringe spacing distortion (A) and the fringe angle distortion (B) over the full FOV image data is shown. [Figure 16A] 4 shows the translation of the subtile phase offset from the subtile coordinate system to the full FOV image coordinate system. [Figure 16B] Indicates the location of a point within a subtile with respect to the subtile coordinate system and the full FOV image coordinate system. [Figure 17] 11 is an example phase bias lookup table for a subtile in a full FOV image. [Figure 18] 1 is a high-level overview of the process steps in the non-redundant subtile SIM image reconstruction algorithm. [Figure 19] Demonstrate the symmetry of Fourier space for matrices with even and odd numbers of rows and columns. [Figure 20] 1 shows the non-redundant and redundant halves of the full matrix in the frequency domain representing three acquired images taken at one illumination peak angle. [Figure 21] 13 illustrates the non-redundant half-reshape reordering of three matrices in the frequency domain representing three images acquired at one illumination peak angle. [Figure 22] 22 illustrates the band separation process by multiplication of the inverse band separation matrix with the reshaped matrix from FIG. 21. [Figure 23A] Here is an example of a (1,1) shift operation applied to a matrix: [Figure 23B] 13 illustrates averaging of the first and middle columns in non-redundant SIM image reconstruction. [Figure 23C] 13 illustrates averaging of the first and middle columns in non-redundant SIM image reconstruction. [Figure 24] FIG. 1 is a simplified block diagram of a computer system that can be used to implement the disclosed techniques. [Figure 25A] We present the error rate and percentage improvement of the cluster passing filter (PF) metric with updates in the SIM image reconstruction algorithm. [Figure 25B] We present the error rate and percentage improvement of the cluster passing filter (PF) metric with updates in the SIM image reconstruction algorithm. [Figure 26A] FIG. 13 is a graphical illustration of error rate improvement over multiple cycles in an ordered run. [Figure 26B] FIG. 13 is a graphical illustration of error rate improvement over multiple cycles in an ordered run. [Figure 26C] FIG. 13 is a graphical illustration of error rate improvement over multiple cycles in an ordered run. [Figure 26D] We present graphical plots of the improvement in error rate due to the inclusion of phase learning and stitch fixing techniques in the baseline SIM algorithm. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The following discussion is presented to enable those skilled in the art to make and use the disclosed technology and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the disclosed technology. Thus, the disclosed technology is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

[0014] (Background technology) Structured illumination microscopy in two and three dimensions improves the resolution of imaging systems and helps researchers to investigate the internal structure of living cells and the molecular arrangement of biological materials. See, for example, Sydor, Andrew & Czymmek, Kirk & Puchner, Elias & Mennella, Vito. (2015). Super-Resolution Microscopy: From Single Molecules to Supramolecular Assemblies. Trends in Cell Biology. 25.10.1016 / j.tcb.2015.10.004; also Lal et al. 2015, A., Shan, C., & Xi, P. (2015). Structured illumination microscopy image reconstruction algorithm. IEEE Journal of Selected Topics in Quantum Electronics, 22(4), 50-63. For readers wanting a refresher on SIM processing, Lal et al. (2015) is an excellent combination of examples and mathematical place, except for deriving the equations used. This disclosure extends SIM technology to imaging of flow cells, using several techniques that can be used individually or in combination.

[0015] SIM holds the ability to resolve densely packed samples from flow cells with fluorescent signals from millions of sample points, thereby reducing reagents required for processing and increasing imaging throughput. The trick is to use SIM techniques to resolve densely packed fluorescent samples that are close to each other, even at the Abbe diffraction limit for resolving adjacent sources. Samples can be in regularly spaced nanowells or in randomly distributed clusters. Most of the following discussion is directed to patterned nanowells, but SIM techniques also apply to randomly distributed clusters.

[0016] Technical problems related to lens distortion and required computational resources emerge from resolving closely packed sources, and this disclosure addresses these technical problems.

[0017] Structured illumination can produce images with illumination sources resolved several times more than with normal illumination. Information is not easily created. Multiple images with varying angles and phase shifts of structured illumination are used to convert dense, or otherwise unresolvable, high spatial frequency features into lower frequency signals that can be sensed by the optical system without violating the Abbe diffraction limit. This limit is physically imposed on imaging by the nature of light and optics, and is expressed as a function of the illumination wavelength and numerical aperture (NA) of the final objective lens. By applying SIM reconstruction, information from multiple images is transformed from the spatial domain to the Fourier domain, combined, processed, and then reconstructed into an improved image.

[0018] In SIM, a grating is used or an interference pattern is created between an illumination source and a sample to generate an illumination pattern, such as a pattern whose intensity varies according to a sine or cosine function. In the context of SIM, "grating" may be used to refer to the projected structured illumination pattern in addition to the surface that generates the structured illumination pattern. Alternatively, the structured illumination pattern may be generated as an interference pattern between portions of separate coherent beams.

[0019] For example, the projection of structured illumination onto the sample plane in FIG. 1 mixes the illumination pattern with fluorescent (or reflected) sources in the sample to induce new signals, sometimes called moiré fringes or aliasing. The new signals shift high spatial frequency information to lower spatial frequencies that can be imaged without violating the Abbe diffraction limit. After taking images of a sample illuminated with a 1D intensity modulation pattern as shown in FIG. 1A or a 2D intensity modulation pattern as in FIG. 1B, a system of linear equations is solved and used to extract from the multiple images of the moiré fringes or aliasing a portion of the new signal that contains the information shifted from higher spatial frequencies to lower spatial frequencies. To solve the system of linear equations, three or more images are taken with a structured illumination pattern that is shifted or displaced in stages. Images of various phases (three to seven phases) per angle (with one to five angles) are often taken for analysis and then separated by bands for shifting and recombination in the Fourier domain. Increasing the number of images can improve the quality of the reconstructed image by increasing the signal-to-noise ratio. However, this may also increase the computation time. The Fourier representations of the band-separated images are shifted and summed to generate the reconstruction sum. Finally, an inverse Fast Fourier Transform (FFT) reconstructs a new high-resolution image from the reconstruction sum.

[0020] Applying multiple images and shifting approaches improves image information only in the particular direction in which the 1D structured illumination pattern is shifted, so the structured illumination or grating pattern is rotated and the shifting procedure is repeated. Rotations such as 45, 60, 90, or 120 degrees of the 1D pattern may be applied to generate six or nine images in a set of three steps. Environmental factors such as damage to molecules on the surface of the flow cell caused by blue and green lasers, tilting of tiles on the surface of the flow cell, etc., can cause distortions in the phase bias estimates for image subtiles or subwindows of a full field of view (FOV) image. They can also cause tile-to-tile differences across the substrate. To compensate for these environmental factors, more frequent estimation of SIM image reconstruction parameters can be performed. For example, the phase bias of a subtile can be re-estimated for each tile and each period of reconstruction to minimize these errors. Because the angle and spacing parameters do not change as frequently as the phase bias, increasing their estimation frequency may introduce additional calculations that are not necessary. However, if necessary, the angle and spacing parameters can be calculated more frequently.

[0021] Application of SIM to flow cell image analysis

[0022] Imaging a flow cell with millions of fluorescent samples is more analogous to scanning space with a telescope than to studying the microstructure of a living cell. Scanning a flow cell with economical optics is more analogous to taking images with an Instamatic camera than to the use of adaptive optics at Mount Palmer Observatory. Scanning a flow cell or the sky involves multiple images covering a tile of the target, as opposed to imaging a living cell in the sweet spot of the lens. The number of images required to cover any target depends on the field of view and the degree of overlap of each image tile. Economical lenses have distortion at the edges of the field of view that complicates the production of improved images.

[0023] The calculations used in SIM reconstruction are sensitive to lens distortion, pattern alignment in a particular image set, and thermal effects on pattern alignment over a shooting cycle that proceeds over many hours. Increasing the field of view by using a larger portion of the lens instead of the central sweet spot increases the sensitivity of SIM image reconstruction to distortions caused by aberrations in the lens. These anomalies, such as coma, distort the structured illumination pattern, causing parallel lines of illumination peaks to appear curved, changing the distance between the intensity peaks of the pattern and the apparent pattern angle, as generally shown in FIG. 8A.

[0024] A practical instrument would require characterization that maps distortions and offsets across subtiles of a tile. Due to the duration of flow cell imaging over a period, some instruments would benefit from updated characterization at the beginning of a run or even during a run. Thermal distortion over many hours of flow cell imaging with hundreds of periods may make it useful to occasionally or periodically recharacterize the angle, spacing, and phase (shift) of the structured illumination.

[0025] The angle (rotation) and spacing (scale) of the projected structured illumination pattern can be determined by fringe peak estimation when the instrument is characterized. The estimated phase or displacement of the repeating pattern along the direction of stepping can be expressed as a spatial shift from 0 to 360 degrees or expressed in radians. Distortions across the lens complicate parameter estimation and lead to computationally expensive proposals for reconstruction from multiple images. See, for example, Ayuk et al. (2013) Structured illumination fluorescence microscopy with distorted excitations using a filtered blind-SIM algorithm. Optics letters. 38.4723-6.10.1364 / OL.38.004723, and Mudry et al. (2012) Structured illumination microscopy using unknown speckle patterns. Nature Photonics. 6.312-315. The disclosed technique simplifies iterative parameter estimation and reduces the computations required in an iterative estimation cycle.

[0026] Real-time parameter estimation for SIM reconstruction is computationally challenging. The computational power required for SIM reconstruction increases with a cubic relationship with the number of pixels in the image field or subfield. For example, for an image with a width of M pixels and a height of N pixels, the Fourier transform is * M * N(log(M * N) computational complexity. Thus, the order of magnitude of resources for SIM image reconstruction increases with the number of pixels in the image, becoming quadratic O(N 2 ) and cubic O(N 3) between 1024×1024 and 512×512. Thus, a doubling of image dimensions, such as from 512×512 to 1024×1024, can result in up to an 8-fold increase in computational costs. It is particularly challenging to reconstruct an improved image from six or nine images of 20 megapixels on a CPU at a rate of one completed reconstruction every 0.3 seconds while the scan is in progress. Real-time processing is desirable to reduce storage requirements and track the quality of flow cell processing over many hours as the scan and sequencing progresses. The disclosed technique uses symmetries and near symmetries in the matrix of Fourier domain coefficients to reduce some core calculations by approximately a factor of two.

[0027] Addressing lens distortion

[0028] The disclosed technique addresses lens distortions that are too severe for optical systems within economical manufacturing tolerances by subdividing an imaged image tile into subtiles and treating the near-center subtiles differently from the other subtiles. Images captured by an optical sensor may be referred to as tiles. A capture cycle for a flow cell results in many image tiles being captured with some overlap. Each image tile is divided into independently evaluated subtiles. The subtiles may be reconstructed independently of each other, even in parallel. Reconstructions from the improved subtiles may be stitched together to generate a reconstruction tile with improved spatial resolution.

[0029] The disclosed interpolation technique for relating the reconstruction parameters for the center-nearby subtiles to the other subtiles approximates a nonlinear function, such as a quadratic curve, with a piecewise approximation. The disclosed technique subdivides the image tile into subtiles such that the peak lines are approximately evenly spaced within the subtile, thereby achieving better image quality from the reconstructed subtiles across the field of view of the lens.

[0030] This subtiling approach to reduce fringe distortion introduces a new problem: the reconstruction parameters must be estimated for each subtile. Parameter estimation is the most expensive setup in SIM reconstruction, and subtiling increases the parameter estimation run time by at least an order of magnitude; for example, an image partitioned into 5x5 subtiles produces a 25x slower algorithm. A tile partitioned into 8x11 subtiles requires 88 sets of reconstruction parameters instead of just one set.

[0031] The disclosed technique learns a function that maps distortions and thereby reduces recalculations in a repeat cycle. The learned function remains valid across cycles and image sets because it maps optical properties of the optics that change when the optics is altered, e.g., by realignment. During an estimation cycle, during instrument characterization, expensive parameter estimation calculations focus on a single near-center subtile. Parameters for each subtile map measurements in the cycle from the near-center reference subtile and to other subtiles.

[0032] Three parameters are mapped to the subtiles: illumination peak angle, illumination peak spacing, and phase shift. Illumination peak angle is also referred to as grating angle. Illumination peak spacing is also referred to as grating spacing. The disclosed technique maps the angle and spacing using a quadratic surface distortion model. Phase shift, or simply phase, is the shift of the structured illumination pattern or grating that is projected onto the sample plane.

[0033] Reusable spacing and angular distortion models are calculated a priori using a subtiling approach to characterize the full FOV. The window or subtile approach divides the tile into overlapping windows or subtiles of the image and performs SIM parameter estimation on each subtile. The subtile parameter estimation results are then fed into a least squares regression, which generates a quadratic surface of distortion. The stored formula coefficients can then be used to extrapolate the parameters of the subtile or anywhere in the full field of view image. In an alternative implementation, the coefficients can be stored and used, or they can be converted to subtiles, and a lookup table relates the near-center subtiles to other subtiles. The near-center subtiles are used as a reference because the center of the lens has less distortion compared to the rim.

[0034] For phase relationships between subtiles, the lookup table performs better than the curve fit, as shown below.

[0035] Over the course of runs that extend for hours, periodic recharacterization can be applied to prevent thermal instability and resulting drift. After the extrapolation factors have been redetermined, the process resumes extrapolating the parameter estimates for the near-center subtiles to other subtiles by extrapolation and without expensive parameter estimates for the other subtiles.

[0036] Exploiting symmetries to reduce computational cost

[0037] When photographing a single cell, SIM reconstruction from multiple images is often performed on specialized hardware such as GPU, FPGA, or CGRA. With abundant computing resources, the reconstruction algorithm works with a fully redundant, center-shifted Fourier transform. Instead, using a reasonably priced CPU, SIM reconstruction is disclosed using non-redundant coefficients of the Fourier transformed image. An embodiment is described based on the symmetry of the data in the corner-shifted Fourier transform space. By using a specific symmetry, the amount of data for calculation in the core set of calculations can be reduced by half and the number of calculations required by half, while maintaining approximately the same accuracy, although the program complexity increases. Also, the calculations required to shift the Fourier representations of the images being combined are reduced.

[0038] Adapting to 2D lighting patterns

[0039] Standard algorithms for 1D modulated illumination require modifications when used with 2D modulated illumination patterns. This includes estimation of illumination peak spacing and illumination peak angle, which requires 2D band separation instead of 1D. It also includes Wicker phase estimation, which works from two points (instead of one) to estimate the phase in two dimensions. The 1D interference pattern can be generated by a one-dimensional diffraction grating as shown in FIG. 1A, or as a result of the interference pattern of two beams.

[0040] FIG. 1B shows an intensity distribution that can be generated by a 2D diffraction grating or by the interference of four light beams. Two light beams generate an intensity pattern along the y axis (horizontal bright and dark lines) and are therefore called y-pairs of incident beams. Two more light beams generate an intensity pattern along the x axis (vertical bright and dark lines) and are therefore called x-pairs of incident beams. The interference of the y-pairs with the x-pairs of light beams generates a 2D illumination pattern. FIG. 1B shows the intensity distribution of such a 2D illumination pattern.

[0041] FIG. 1C shows an arrangement of nanowells on the surface of a flow cell positioned at the corners of a rectangle. When using 1D structured illumination, the peak angle of illumination is selected so that images are taken along a straight line connecting the diagonally opposite corners of the rectangle. For example, two sets of three images (six images total) can be taken at angles of +45 degrees and -45 degrees. As the distance along the diagonal becomes longer than the distance between any two sides of the rectangle, a higher resolution image is achieved. The nanowells can be arranged in other geometric arrangements, such as a hexagon. Three or more images can then be taken along each of the three diagonals of the hexagon, resulting in, for example, nine or fifteen images.

[0042] The new specific techniques disclosed can be used individually or in combination to improve scanning performance while detecting the fluorescence of millions of samples distributed throughout the flow cell over multiple cycles. In the following sections, we describe imaging equipment that can be improved using the disclosed techniques and disclose new image improvement techniques that can be used individually or in combination.

[0043] term

[0044] As used herein to refer to structured illumination parameters, the term "frequency" refers to the inverse of the spacing between stripes or lines of a structured illumination pattern (e.g., a stripe or grid pattern) since frequency and period are inversely related. For example, a pattern with greater spacing between stripes will have a lower frequency than a pattern with smaller spacing between stripes.

[0045] As used herein to refer to structured illumination parameters, the term "phase" refers to the phase of the structured illumination pattern that illuminates the sample. For example, the phase can be changed by translating the structured illumination pattern relative to the illuminated sample.

[0046] As used herein to refer to structured illumination parameters, the term "orientation" shall refer to the relative orientation between a structured illumination pattern (e.g., a stripe pattern or a grating pattern) and a sample illuminated by this pattern. For example, the orientation can be changed by rotating the structured illumination pattern relative to the illuminated sample.

[0047] As used herein to refer to structured illumination parameters, the term "predict" or "predicting" shall mean calculating the value of a parameter without directly measuring the parameter or estimating the parameter from an imaging image corresponding to the parameter. For example, the phase of a structured illumination pattern can be predicted at time t1 by interpolation between directly measured or estimated phase values (e.g., from an imaged phase image) at times t2 and t3, where t2 < t1 < t3. As another example, the frequency of a structured illumination pattern can be predicted at time t1 by extrapolating from directly measured or estimated frequency values at times t2 and t3 (e.g., from an imaged phase image), where t2 < t1 < t3.

[0048] As used herein to refer to light diffracted by a diffraction grating, the term "order" or "order number" shall mean an integer number of wavelengths representing the optical path length difference from adjacent slits or structures of the diffraction grating for constructive interference. The interaction of an incident light beam on a repeating series of grating structures or other beam splitting structures can redirect or diffract portions of the light beam into predictable angular directions from the original beam. The term "zeroth order" or "maximum zeroth order" shall refer to the central bright stripe without diffraction emitted by the diffraction grating. The term "first order" shall refer to two bright stripes diffracted on both sides of the zeroth order stripe where the optical path length difference is ±1 wavelength. Higher orders are diffracted at larger angles from the original beam. The characteristics of the grating can be manipulated to control how much beam intensity is directed into the various orders. For example, a phase grating can be fabricated to maximize the transmission of the ±1st orders and minimize the transmission of the zeroth order beam.

[0049] As used herein to refer to a sample, the term "feature" is intended to mean a point or area in a pattern that can be distinguished from other points or areas according to their relative position. An individual feature may contain one or more molecules of a particular type. For example, a feature may contain a single target nucleic acid molecule having a particular sequence, or a feature may contain several nucleic acid molecules having the same sequence (and / or its complementary sequence).

[0050] As used herein, the term "xy-plane" is intended to mean the two-dimensional area defined by linear axes x and y in a Cartesian coordinate system. When used in relation to a detector and an object observed by the detector, the area may be further specified as being orthogonal to the beam axis, or observation direction between the detector and the object being detected.

[0051] As used herein, the term "z-coordinate" is intended to mean information that specifies the location of a point, line, or area along an axis orthogonal to the xy plane. In certain embodiments, the z-axis is orthogonal to the area of ​​the object observed by the detector. For example, the direction of the focus of an optical system may be specified along the z-axis.

[0052] As used herein, the term "optically coupled" refers to one element that is adapted to impart light directly or indirectly to another element.

[0053] SIM Hardware

[0054] This section of the construction related to the present disclosure was previously made by the applicant in U.S. Provisional Patent Application No. 62 / 692,303 (unpublished), filed June 29, 2018. Structured illumination microscopy (SIM) is considered a technique that can image a sample using spatially structured (i.e., patterned) light to increase the lateral resolution of the microscope by more than a factor of two. FIG. 1A shows Moiré fringes (or Moiré pattern) formation by using a grating with 1D modulation. A surface containing a sample is illuminated by a structured pattern of light intensity, typically a sinusoid, that results in Moiré fringe formation. FIG. 1A shows two sinusoidal patterns when their frequency vectors in frequency or reciprocal space are (a) parallel and (b) non-parallel. FIG. 1A is a typical diagram of shifting high frequency spatial information to lower frequencies that can be optically detected. The new signal is called Moiré fringes or aliasing. In some cases, during imaging of a sample, three images of the sample's fringe pattern are acquired at various pattern phases (e.g., 0°, 120°, and 240°) so that each location on the sample is exposed to a range of illumination intensities, and this procedure is repeated by rotating the pattern orientation around the optical axis to two (e.g., 45°, 135°) or three (e.g., 0°, 60°, and 120°) distinct angles. The captured images (e.g., six or nine images) may be assembled into a single image with an extended spatial frequency bandwidth, which can be retransformed back into real space to produce an image with higher resolution than that captured by a conventional microscope.

[0055] In some implementations of the SIM system, the linearly polarized beam is directed by an optical beam splitter that splits the beam into two or more separate orders that can be combined and projected onto the imaged sample as an interference fringe pattern with sinusoidal intensity variation. A diffraction grating is an example of a beam splitter that can generate beams with a high degree of coherence and stable propagation angles. When two such beams are combined, the interference between such beams can create a uniform, regularly repeating fringe pattern whose spacing is determined by factors including the angle between the interfering beams.

[0056] 1B presents an example of 2D structured illumination. 2D structured illumination may be formed by two orthogonal 1D diffraction gratings superimposed on each other. As with the 1D structured illumination pattern, the 2D illumination pattern may be generated either by the use of a 2D diffraction grating or by interference between four light beams that generate a regularly repeating fringe pattern.

[0057] During imaging and / or subsequent assembly or reconstruction of images into a single image with an extended spatial frequency bandwidth, the following structured illumination parameters may need to be considered: the orientation or angle of the fringe pattern, also referred to as the illumination peak angle, relative to the illuminated sample, the spacing between adjacent fringes, also referred to as the illumination peak spacing (i.e., the frequency of the fringe pattern), and the phase displacement of the structured illumination pattern. In an ideal imaging system that is not subject to factors such as mechanical instability and thermal deformation, each of these parameters will not drift or otherwise change over time, and the exact SIM frequency, phase, and orientation parameters associated with a given image sample are known. However, due to factors such as mechanical instability of the excitation beam path and / or thermal expansion / contraction of the imaged sample, these parameters may drift or otherwise change over time.

[0058] Thus, SIM imaging systems may need to estimate structured illumination parameters to account for their variance over time. Because many SIM imaging systems do not perform real-time SIM image processing (e.g., they process captured images offline), such SIM systems may spend a significant amount of computational time processing SIM images to estimate structured illumination parameters for the images.

[0059] 2-4B show three such exemplary SIM imaging systems. Although these systems are described primarily in the context of SIM imaging systems that generate 1D illumination patterns, it should be noted that the techniques disclosed herein may also be implemented in SIM imaging systems that generate higher dimensional illumination patterns (e.g., two-dimensional grid patterns).

[0060] 2 illustrates a structured illumination microscopy (SIM) imaging system 100 in which prediction of structured illumination parameters according to some embodiments described herein can be implemented. For example, system 100 can be a structured illumination fluorescence microscopy system that utilizes spatially structured excitation light to image biological samples.

[0061] 2, light emitter 150 is configured to output a light beam that is collimated by collimation lens 151. The collimated light is structured (patterned) by light structuring optical assembly 155 and directed by dichroic mirror 160 through objective lens 142 to a sample in sample container 110 positioned on motion stage 170. In the case of a fluorescent sample, the sample fluoresces in response to the structured excitation light, and the resulting light is collected by objective lens 142 and directed to an image sensor of camera system 140 for detecting the fluorescence.

[0062] The light structuring optical assembly 155 includes one or more optical gratings or other beam splitting elements (e.g., beam splitter cubes or plates) to generate a pattern of light (e.g., fringes, typically sinusoidal) that is projected onto the sample in the sample container 110. The grating may be a one-dimensional transmission or reflection grating or a two-dimensional transmission or reflection grating. The grating may be a sinusoidal amplitude grating or a sinusoidal phase grating.

[0063] In some embodiments, the grating may not utilize a rotation stage to change the orientation of the structured illumination pattern. In other embodiments, the grating may be attached to a rotation stage. In some embodiments, the grating may be fixed (i.e., does not require rotational or linear motion) during operation of the imaging system. For example, in certain embodiments described further below, the grating may include two fixed one-dimensional transmission gratings (e.g., a horizontal grating and a vertical grating) oriented perpendicular to each other.

[0064] 2, the light structuring optical assembly 155 outputs a 1 st order diffracted light beam (e.g., m=±1 orders) while blocking or minimizing all other orders, including the 0 th order. However, in alternative embodiments, additional orders of light can be projected onto the sample.

[0065] During each imaging cycle, imaging system 100 utilizes light structuring optical assembly 155 to acquire multiple images at various phases, and the fringe pattern is displaced laterally (e.g., in the xy plane and perpendicular to the fringes) to the modulation direction. This procedure is repeated one or more times by rotating the pattern orientation about the optical axis (i.e., relative to the xy plane of the sample). The captured images may then be computationally reconstructed to generate a higher resolution image (e.g., an image having approximately twice the lateral spatial resolution of the individual images).

[0066] In system 100, light emitter 150 may be a non-coherent light emitter (e.g., emitting a light beam output by one or more pump diodes) or a coherent light emitter, such as a light emitter of light output by one or more lasers or laser diodes. As shown in the example of system 100, light emitter 150 includes an optical fiber 152 for guiding the output optical beam. However, other configurations of light emitter 150 may be used. In an embodiment utilizing structured illumination in a multi-channel imaging system (e.g., a multi-channel fluorescence microscope utilizing multiple wavelengths of light), optical fiber 152 may be optically coupled to multiple different light sources (not shown), each emitting light of a different wavelength. Although system 100 is illustrated as having a single light emitter 150 in some embodiments, multiple light emitters 150 may be included. For example, multiple light emitters may be included in the case of a structured illumination imaging system utilizing multiple arms, discussed further below.

[0067] In some embodiments, system 100 may include an imaging lens 156, which may include lens elements coupled along the z-axis to adjust the shape and path of the structured beam. For example, the components of the imaging lens may be coupled to account for a range of sample thicknesses (e.g., various cover glass thicknesses) for the samples in container 110.

[0068] In the example of the system 100, the fluid delivery module or device 190 can direct (and pass) the flow of reagents (e.g., fluorescently labeled nucleotides, buffers, enzymes, cleavage reagents, etc.) to the sample container 110 and waste valve 120. The sample container 110 can include one or more substrates on which the sample is provided. For example, in the case of a system that analyzes a large number of different nucleic acid sequences, the sample container 110 can include one or more substrates on which the nucleic acids to be sequenced are bound, attached, or associated. Substrates can include any inert substrate or substrate to which nucleic acids can be attached, such as, for example, glass surfaces, plastic surfaces, latex, dextran, polystyrene surfaces, polypropylene surfaces, polyacrylamide gels, gold surfaces, and silicon wafers. In some applications, the substrates are within channels or other areas at multiple locations formed in a matrix or array across the sample container 110. The system 100 can also include a temperature station actuator 130 and a heater / cooler 135 that can optionally regulate the temperature state of the fluid in the sample container 110.

[0069] In certain embodiments, the sample container 110 may be implemented as a patterned flow cell that includes a transparent cover plate, a substrate, and a liquid contained between the transparent cover plate and the substrate, and the biological sample may be located on the inner surface of the transparent cover plate or the inner surface of the substrate. The flow cell may include a large number (e.g., thousands, millions, or billions) of wells (also referred to as nanowells) or regions within the substrate that are patterned into a defined array (e.g., a hexagonal array, a rectangular array, etc.). Each region may form a cluster (e.g., a monoclonal cluster) of biological samples, such as DNA, RNA, or other genomic material, that may be sequenced using sequencing by synthesis. The flow cell may be further divided into a large number of spaced lanes (e.g., eight lanes), each lane including a hexagonal array of clusters.

[0070] The sample container 110 may be mounted on a sample stage 170 to provide movement and alignment of the sample container 110 relative to the objective lens 142. The sample stage may have one or more actuators that allow it to move in any of three dimensions. For example, actuators may be provided that allow the stage to move in the X-, Y-, and Z-directions relative to the objective lens in terms of a Cartesian coordinate system. This may allow one or more sample locations on the sample container 110 to be positioned in optical alignment with the objective lens 142. Movement of the sample stage 170 relative to the objective lens 142 may be achieved by moving the sample stage itself, the objective lens, some other component of the imaging system, or any combination of the above. Further embodiments may also include moving the entire imaging system over a stationary sample. Alternatively, the sample container 110 may be fixed during imaging.

[0071] In some implementations, a focus (z-axis) component 175 may be included to control the positioning of the optical components relative to the sample container 110 in a focus direction (typically referred to as the z-axis or z-direction). The focus component 175 may include one or more actuators physically coupled to the optical stage or the sample stage, or both, to move the sample container 110 on the sample stage 170 relative to the optical components (e.g., the objective lens 142) to provide proper focusing for the imaging operation. For example, the actuators may be physically coupled to the respective stages, e.g., by a direct or indirect mechanical, magnetic, fluidic, or other connection or contact with the stage. The one or more actuators may be configured to move the stage in the z-direction while keeping the sample stage in the same plane (e.g., while maintaining a level or horizontal orientation perpendicular to the optical axis). The one or more actuators may also be configured to tilt the stage. This is done, for example, so that the sample container 110 can be dynamically flattened to compensate for any tilt in its surface.

[0072] Structured light emanating from the test sample at the location of the sample being imaged may be directed by a dichroic mirror 160 to one or more detectors of the camera system 140. In some implementations, a filter switching assembly 165 having one or more emission filters may be included, which may be used to pass certain emission wavelengths and block (or reflect) other emission wavelengths. For example, one or more emission filters may be used to switch between different channels of the imaging system. In certain implementations, the emission filters may be implemented as dichroic mirrors that direct emission of different wavelengths to different image sensors of the camera system 140.

[0073] The camera system 140 may include one or more image sensors for monitoring and tracking the imaging (e.g., sequencing) of the sample containers 110. The camera system 140 may be implemented, for example, as a charge-coupled device (CCD) image sensor camera, although other image sensor technologies (e.g., active pixel sensors) may be used.

[0074] Output data (e.g., images) from the camera system 140 may be communicated to a real-time SIM imaging component 191, which may be implemented as a software application that can reconstruct the images captured in each imaging cycle to generate images with higher spatial resolution, as described further below. The reconstructed images can take into account predicted changes in structured illumination parameters over time. Additionally, the SIM imaging component 191 can be used to track predicted SIM parameters and / or predict SIM parameters given previous estimated and / or predicted SIM parameters.

[0075] A controller 195 may be provided to control the operation of the structured illumination imaging system 100, including synchronizing various optical components of the system 100. The controller may be implemented to control aspects of the system operation, such as, for example, the configuration of the light structured optical assembly 155 (e.g., selection of a diffraction grating and / or linear translation), the movement of the imaging lens 156, focusing, stage movement, and imaging operations. The controller may also be implemented to control hardware elements of the system 100 to correct for changes in structured illumination parameters over time. For example, the controller may be configured to send control signals to motors or other devices that control the configuration of the light structured optical assembly 155, the motion stage 170, or some other element of the system 100 to correct or compensate for changes in the phase, frequency, and / or orientation of the structured illumination over time. In embodiments, these signals may be sent in accordance with structured illumination parameters predicted using the SIM imaging components 191. In some embodiments, the controller 195 may include a memory for storing predicted and / or estimated structured illumination parameters corresponding to various times and / or sample positions.

[0076] In various embodiments, the controller 195 may be implemented using hardware, algorithms (e.g., machine-executable instructions), or a combination of the above. For example, in some embodiments, the controller may include one or more CPUs, GPUs, or processors with associated memory. As another example, the controller may include hardware or other circuitry for controlling operations, such as a computer processor and a non-transitory computer-readable medium having machine-readable instructions stored thereon. For example, this circuitry may include one or more of the following: a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logic device (CPLD), a programmable logic array (PLA), a programmable array logic (PAL), and other similar processing devices or circuitry. As yet another example, the controller may include a combination of this circuitry with one or more processors.

[0077] 3 is an optical diagram illustrating an example optical configuration of a two-arm SIM imaging system 200 that can implement prediction of structured illumination parameters according to some embodiments described herein. The first arm of the system 200 includes a light emitter 210A for collimating the light output by the light emitter 210A, a first optical collimator 220A, a diffraction grating 230A at a first orientation relative to the optical axis, a rotating mirror 240A, and a second optical collimator 250A. The second arm of the system 200 includes a light emitter 210B, a first optical collimator 220B for collimating the light output by the light emitter 210B, a diffraction grating 230B at a second orientation relative to the optical axis, a rotating mirror 240B, and a second optical collimator 250B. Although a diffraction grating is shown in this example, in other embodiments, other beam splitting elements such as beam splitter cubes or plates can be used to split the light received in each arm of the SIM imaging system 200.

[0078] Each light emitter 210A-210B may be a non-coherent light emitter (e.g., emitting a light beam output by one or more pump diodes) or a coherent light emitter, such as a light emitter output by one or more lasers or laser diodes. In the example of system 200, each light emitter 210A-210B is an optical fiber that outputs a light beam that is collimated by a respective collimator 220A-220B.

[0079] In some implementations, each optical fiber may be optically coupled to a corresponding light source (not shown), such as a laser. During imaging, each optical fiber may be switched on or off using a high-speed shutter (not shown) positioned in the optical path between the fiber and the light source, or by pulsing the fiber's corresponding light source at a predetermined frequency during imaging. In some implementations, each optical fiber may be optically coupled to the same light source. In such implementations, a beam splitter or other suitable optical element may be used to direct light from the light source to each of the optical fibers. In such an example, each optical fiber may be switched on or off using a high-speed shutter (not shown) positioned in the optical path between the fiber and the beam splitter.

[0080] In the exemplary SIM imaging system 200, a first arm includes a fixed vertical grating 230A for projecting a structured illumination pattern or grating pattern in a first orientation onto the sample, and a second arm includes a fixed horizontal grating 230B for projecting a structured illumination pattern or grating pattern (e.g., a horizontal stripe pattern) in a second orientation onto the sample 271. The gratings of the SIM imaging system 200 do not need to be mechanically rotated or translated, which may provide improved system speed, reliability, and repeatability.

[0081] In an alternative embodiment, gratings 230A and 230B may be mounted on respective linear motion stages that may be translated to change the optical path length (and therefore phase) of the light emitted by gratings 230A and 230B. The axes of motion of the linear motion of the stages may be perpendicular or otherwise offset from the orientation of their respective gratings to achieve translation of the grating patterns along the sample 271.

[0082] Gratings 230A-230B may be transmission gratings including a plurality of diffractive elements (e.g., parallel slits or grooves) formed in a glass substrate or other suitable surface. The gratings may be implemented as phase gratings that provide a periodic variation in the refractive index of the grating material. The spacing of the grooves or features may be selected to diffract light at a suitable angle and may be tailored to the minimum resolvable feature size of the sample being imaged for operation of SIM imaging system 200. In other embodiments, the gratings may be reflective gratings.

[0083] In the example SIM imaging system 200, the vertical and horizontal patterns are offset by approximately 90 degrees. In other implementations, other orientations of the grating may be used to produce the approximately 90 degree offset. For example, the grating may be oriented to project an image that is offset ±45 degrees from the x or y plane of the sample 271. The configuration of the example SIM imaging system 200 may be particularly advantageous for a regularly patterned sample 271 having features on a rectangular grid, since improvements in structure resolution may be achieved using only two perpendicular gratings (e.g., a vertical grating and a horizontal grating).

[0084] The gratings 230A-230B are configured in the example system 200 to diffract the input beam into orders (e.g., 0th order, ±1st order, ±2nd order, etc.), of which the ±1st order may be projected onto the sample 271. As shown in this example, the vertical grating 230A diffracts the collimated light beam into a first order diffracted beam (±1st order) and propagates the first order in the plane of the page, and the horizontal grating 230B diffracts the collimated light beam into a first order diffracted beam and propagates the order above and below the plane of the page (i.e., in a plane perpendicular to the page). To improve the efficiency of the system, the 0th order beam and all other higher order (i.e., ±2nd order or higher) diffracted beams output by each phase mask may be blocked (i.e., filtered out from the illumination pattern projected onto the sample 271). For example, beam blocking elements (not shown), such as order filters, may be inserted in the optical path after each diffraction grating to block the zeroth and higher order beams. In some implementations, diffraction gratings 230A-230B may be configured to diffract the beam into only the first order, and the zeroth order (non-diffracted beam) may be blocked by some beam blocking elements.

[0085] Each arm includes an optical phase modulator or phase shifter 240A-240B for phase shifting the diffracted light output by each of the gratings 230. For example, during structured imaging, the optical phase of each diffracted beam can be shifted by some fraction (e.g., ½, ⅓, ¼, etc.) of the pitch (λ) of each fringe of the structured pattern. In the example of FIG. 3, the phase modulators 240A and 240B are implemented as rotating windows that can rotate and modulate the optical path length of each diffracted beam using a galvanometer or other rotary actuator. For example, the window 240A can be rotated about a vertical axis to shift the image projected by the vertical grating 230A on the sample 271 to the left or right, and the window 240B can be rotated about a horizontal axis to shift the image projected by the horizontal grating 230B on the sample 271 up or down.

[0086] In other embodiments, other phase modulators (e.g., linear translation stages, wedges, etc.) that change the optical path length of the diffracted light may be used. Additionally, although optical phase modulators 240A-240B are shown as being located after gratings 230A-230B, in other embodiments they may be located elsewhere in the illumination system.

[0087] In alternative embodiments, a single phase modulator may be operated in two different directions for different fringe patterns, or a single phase modulator may adjust both path lengths using a single motion. For example, a large rotating optical window may be placed after mirror 260 with holes 261. In this case, the large window may be used in place of windows 240A and 240B to adjust the phase of both sets of diffracted beams output by the vertical and horizontal gratings. Instead of being parallel to the optical axis of one of the gratings, the rotation axis for the large rotating window may be offset by 45 degrees (or some other angle offset) from the optical axis of each of the vertical and horizontal gratings to allow phase shift along both directions along one common rotation axis of the large window. In some embodiments, the large rotating window may be replaced by a wedge optic that rotates about the nominal beam axis.

[0088] In the exemplary system 200, a mirror 260 with a hole 261 combines the two arms in a loss-free manner (e.g., without significant loss of optical power other than a small amount of absorption in the reflective coating) into the optical path. The mirror 260 may be positioned such that the diffracted orders from each of the gratings can be spatially resolved and unwanted orders can be blocked. The mirror 260 passes the first order of light output by the first arm through the hole 261. The mirror 260 reflects the first order of light output by the second arm. Thus, the structured illumination pattern can be switched from a vertical direction (e.g., grating 230A) to a horizontal direction (e.g., grating 230B) by turning on or off each light emitter or by opening or closing an optical shutter that directs the light of the light source through the fiber optic cable. In other embodiments, the structured illumination pattern may be switched by using an optical switch to change the arm that illuminates the sample.

[0089] Also shown in the exemplary imaging system 200 are an imaging lens 265, a semi-reflective mirror 280, an objective lens 270, and a camera 290. For example, the imaging lens 265 may be implemented as linked along the z-axis to adjust the structured beam shape and path. The semi-reflective mirror 280 may be a dichroic mirror that reflects the structured illumination light received from each arm into the objective lens 270 for projection onto the sample 271 and passes light emitted by the sample 271 (e.g., fluorescence emitted at a wavelength different from the excitation) to the camera 290.

[0090] Output data (e.g., images) from camera 290 may be communicated to a real-time SIM imaging component (not shown), which may be implemented as a software application that may reconstruct the images captured during each imaging cycle to generate images with higher spatial resolution, as described further below. The reconstructed images may take into account predicted changes in structured illumination parameters over time. Additionally, the SIM imaging component may be used to track predicted SIM parameters and / or make predictions of SIM parameters given previously estimated and / or predicted SIM parameters.

[0091] A controller (not shown) may be provided to control the operation of the structured illumination imaging system 200, including synchronizing the various optical components of the system 200. The controller may be implemented to control aspects of the system operation, such as, for example, the configuration of each optical arm (e.g., switching each optical arm on / off during acquisition of a phase image, actuation of the phase modulators 240A-240B), movement of the imaging lens 265, stage movement of the sample 271 (if any stage is used), and imaging operations. The controller may also be implemented to control hardware elements of the system 200 to correct for changes in structured illumination parameters over time. For example, the controller may be configured to send control signals to devices that control each optical arm or some other element of the system 100 (e.g., the phase modulators 240A-240B) to correct or compensate for changes in the phase, frequency, and / or orientation of the structured illumination over time. As another example, if the gratings 230A-230B are mounted on a linear motion stage (e.g., instead of using phase modulators 240A-240B), the controller may be configured to control the linear motion stage to correct or compensate for phase changes. In embodiments, these signals may be transmitted according to structured illumination parameters predicted using SIM imaging components. In some embodiments, the controller may include a memory for storing predicted and estimated structured illumination parameters corresponding to various times and / or sample positions.

[0092] It should be noted that for simplicity, the optical components of SIM imaging system 200 may be omitted from the above discussion. Additionally, while system 200 is shown in this example as a single channel system, in other embodiments it may be implemented as a multi-channel system (e.g., by using two different cameras and light sources emitting at two different wavelengths).

[0093] It should be noted that while system 200 illustrates a two-arm structured illumination imaging system including two gratings oriented at two different angles, in other embodiments, the techniques described herein may be implemented in systems using three or more arms. For a regularly patterned sample with features on a rectangular grid, resolution improvement may be achieved with only two perpendicular angles (e.g., vertical and horizontal gratings) as described above. Meanwhile, for image resolution improvement in all directions for other samples (e.g., hexagonally patterned samples), three illumination peak angles may be used. For example, a three-arm system may include three light emitters and three fixed diffraction gratings (one per arm), each oriented around the optical axis of the system to project a respective pattern orientation (e.g., 0° pattern, 120° pattern, or 240° pattern) on the sample. In such a system, additional images of additional gratings may be combined into the system in a loss-free manner using additional mirrors with holes. Alternatively, such a system may utilize one or more polarizing beam splitters to combine the images of each of the gratings.

[0094] 4A and 4B are schematic diagrams illustrating an exemplary optical configuration of a dual optical grating slide SIM imaging system 500 that may implement structured illumination parameter prediction according to some embodiments described herein. In the exemplary system 500, any changes (e.g., pattern phase shift or rotation) to the structured illumination pattern or grating pattern projected onto the sample 570 may be made by linearly translating the motion stage 530 along a single axis of motion to select grating 531 or 532, or to phase shift one of the gratings 531-532.

[0095] The system 500 includes a light emitter 510 (e.g., an optical fiber optically coupled to a light source), a first optical collimator 520 (e.g., a collimation lens) for collimating the light output by the light emitter 510, a linear motion stage 530 on which a first diffraction grating 531 (e.g., a horizontal grating) and a second diffraction grating 532 (e.g., a vertical grating) are mounted, an imaging lens 540, a semi-reflective mirror 550 (e.g., a dichroic mirror), an objective lens 560, a sample 570, and a camera 580. For simplicity, the optical components of the SIM imaging system 500 may be omitted from FIG. 4A. Furthermore, while the system 500 is shown in this example as a single channel system, in other embodiments it may be implemented as a multi-channel system (e.g., by using two different cameras and light sources emitting at two different wavelengths).

[0096] As shown in FIG. 4A, the grating 531 (e.g., a horizontal grating) may diffract the collimated light beam into a first order diffracted light beam (above and below the plane of the page). As shown in FIG. 4B, the grating 532 (e.g., a vertical grating) may diffract the beam into a first order (above and below the plane of the page). In this configuration, only a single light emitter 510 (e.g., an optical fiber) and a single optical arm with a single linear motion stage are required to image the sample 570, which may provide system advantages such as reducing the number of moving system parts to improve speed, complexity, and cost. Furthermore, the absence of polarizers in the system 500 may provide the above-mentioned advantage of high optical efficiency. The configuration of the exemplary SIM imaging system 200 may be particularly advantageous in the case of a regularly patterned sample 570 having features on a rectangular grid, since improved structure resolution may be achieved using only two perpendicular gratings (e.g., a vertical grating and a horizontal grating).

[0097] To improve the efficiency of the system, the zeroth order beam and all other higher order (i.e., ±2nd or higher) diffracted beams output by each grating may be blocked (i.e., filtered out from the illumination pattern projected onto the sample 570). For example, a beam blocking element (not shown), such as an order filter, may be inserted in the optical path after the motion stage 530. In some implementations, the diffraction gratings 531-532 may be configured to diffract the beam into only the first order, and the zeroth order (non-diffracted beam) may be blocked by some beam blocking element.

[0098] In the example system 500, the two gratings may be positioned at about ±45° from the axis of motion (or some other angular offset from the axis of motion, such as about 40° / −50°, about 30° / −60°, etc.) such that a phase shift can be realized for each grating 531 and 532 along a single axis of linear motion. In some implementations, the two gratings may be combined into one physical optical element. For example, one side of the physical optical element may have a structured illumination pattern or grating pattern in a first orientation, and an adjacent side of the physical optical element may have a structured illumination pattern or grating pattern in a second orientation that is orthogonal to the first orientation.

[0099] The single-axis linear motion stage 530 may include one or more actuators to enable it to move along an X-axis relative to the sample plane or along a Y-axis relative to the sample plane. In operation, the linear motion stage 530 may provide sufficient travel (e.g., about 12-15 mm) and precision (e.g., repeatability of less than about 0.5 micrometers) to project precise illumination patterns for efficient image reconstruction. In embodiments in which the motion stage 530 is utilized in an automated imaging system such as a fluorescence microscope, it may be configured to provide high speed operation, minimal vibration generation, and long useful life. In embodiments, the linear motion stage 530 may include crossed roller bearings, linear motors, high precision linear encoders, and / or other components. For example, the motion stage 530 may be implemented as a high precision stepper or piezo motion stage that may be translated using a controller.

[0100] Output data (e.g., images) from camera 580 may be communicated to a real-time SIM imaging component (not shown), which may be implemented as a software application that may reconstruct the images captured during each imaging cycle to generate images with higher spatial resolution, as described further below. The reconstructed images may take into account predicted changes in structured illumination parameters over time. Additionally, the real-time SIM imaging component may be used to track predicted SIM parameters and / or make predictions of SIM parameters given previously estimated and / or predicted SIM parameters.

[0101] A controller (not shown) may be provided to control the operation of the structured illumination imaging system 500, including synchronizing various optical components of the system 500. The controller may be implemented to control aspects of the system operation, such as, for example, translation of the linear motion stage 530, movement of the imaging lens 540, stage movement of the sample 570 (if any stage is used), and imaging operations. The controller may also be implemented to control hardware elements of the system 500 to correct for changes in structured illumination parameters over time. For example, the controller may be configured to send control signals to devices (e.g., the linear motion stage 530) to correct or compensate for changes in structured illumination phase, frequency, and / or orientation over time. In embodiments, these signals may be sent according to structured illumination parameters predicted using SIM imaging components. In some embodiments, the controller may include a memory for storing predicted and estimated structured illumination parameters corresponding to different times and / or sample positions.

[0102] Although the example of Figures 4A-4B shows a dual optical grating slide photography system that may implement structured illumination parameter prediction, structured illumination parameter prediction may also be implemented in a SIM photography system that uses a linear motion actuator to which two or more diffraction gratings are attached.

[0103] Projection Parameter Estimation

[0104] Parameter estimation for SIM imaging may be required to correct for undesired changes in structured illumination parameters over time. As an example, Figures 5A-5C show undesired changes in frequency or illumination peak spacing (Figure 5A), phase or phase shift (Figure 5B), and orientation or illumination peak angle (Figure 5C) that can occur over time in a SIM imaging system projecting a one-dimensional structured illumination pattern onto a regularly patterned sample. In particular, Figure 5A shows a sample 50 having a feature 51 illuminated by a one-dimensional structured illumination pattern having stripes 60 before and after a frequency shift. Before any frequency shift, adjacent stripes 60 have a pitch or center-to-center spacing of P, which corresponds to an initial frequency f. Over time, temperature variations in the system may increase or decrease the pitch P. For example, thermal expansion may cause the pitch P to increase by P+ΔP 1 and accordingly increase the frequency f to f-Δf 1 Conversely, thermal contraction reduces the pitch P to P-ΔP 1 and accordingly, the frequency f is increased to f+Δf 2 can be increased to.

[0105] FIG. 5B shows a sample 50 illuminated by a one-dimensional structured illumination pattern having fringes 60 before and after a change in phase. As shown, before the phase drift, a first phase state Φ may correspond to fringes that fully illuminate all second rows of features 51 of the sample 50. Over time, the position of the fringes 60 relative to the sample 50 may shift such that all phase images are offset by ΔΦ. For example, mechanical vibrations of the SIM imaging system (e.g., in the excitation beam path), inaccuracies in the translation stage used by the grating or sample stage, thermal variations, and / or other factors may cause an undesirable drift in the phase. After a drift in phase by ΔΦ, the first phase state changes to Φ+ΔΦ and the fringes are centered on all second rows of features.

[0106] 5C shows a sample 50 illuminated by a one-dimensional structured illumination pattern having stripes 60 before and after a change in orientation. As shown, before the change in orientation, the orientation of the stripes relative to the sample 50 is perfectly perpendicular. Over time, the orientation may change due to factors such as changes in the excitation beam path, movement of the sample, thermal fluctuations, and / or other factors. After the orientation is rotated by an angle Δθ, the stripes are no longer perfectly perpendicular to the sample.

[0107] Parameter estimation during the SIM imaging process to accurately account for changes in the structured illumination parameters described above helps ensure artifact-free and accurate reconstruction of an image from a set of sampled images. However, such processes can be computationally expensive and are frequently performed after image acquisition. For time-critical SIM imaging systems involving real-time processing and reconstruction of images, and therefore real-time estimation of parameters such as frequency, phase, orientation, and modulation order, these computational requirements can result in a loss of data throughput (e.g., less data may be processed per time unit). In such systems, the rate at which a sample is imaged may exceed the rate at which structured illumination parameters can be directly estimated from the sampled images. Thus, there is a need for a method to generate parameter estimates with low complexity and low processing time.

[0108] To this end, embodiments of the techniques disclosed herein are directed to predicting structured illumination parameters for a particular time, space, and / or temperature using estimates of structured illumination parameters obtained from images captured by a structured illumination system. Certain embodiments are directed to predicting structured illumination frequency, phase, orientation, and / or modulation order parameters.

[0109] According to some implementations, the structured illumination parameters may be predicted for a given time, space, and / or temperature by interpolating estimates of the structured illumination parameters from the image captures. For example, a first frequency may be estimated from a first sampled image, a second frequency may be estimated from a second sampled image, and a frequency corresponding to a time between the first and second captured images (e.g., a frequency for an image taken between the first and second images) may be predicted by interpolation using at least the determined rate of change of frequency between the first and second captured images.

[0110] According to some embodiments, the structured illumination parameters may be predicted for a given time, space, and / or temperature by extrapolation using estimates of the structured illumination parameters obtained from two image captures. For example, a first orientation may be estimated from a first sampled image, a second orientation may be estimated from a second sampled image, and an orientation corresponding to a time point after the first and second captured images (e.g., an orientation of a third image taken after the first and second images) may be predicted by extrapolation using at least the determined rate of change of the orientation from the first captured image to the second captured image. As a second example, a first orientation may be estimated from a first sampled image, a second orientation may be estimated from a second sampled image, and an orientation corresponding to a time point after the first and second captured images (e.g., an orientation of a third image taken after the first and second images) may be predicted by retaining values ​​from the second captured image.

[0111] In embodiments, the estimated and predicted structured lighting parameters can be used to narrow the search space for other predicted structured lighting parameters. For example, given an estimate of a structured lighting parameter for a first time point, space, and / or temperature, a value of the structured lighting parameter for a second time point, space, and / or temperature that is in the vicinity of the first time point, space, and / or temperature can be predicted taking into account the predicted or estimated value of the first time point, space, and / or temperature.

[0112] In an embodiment, the estimated and predicted structured illumination parameters may be stored in a memory of the structured illumination system for later use by the system. For example, the predicted and estimated parameters may be stored in a history file, such as a look-up table. The predicted parameters stored in the memory may be determined from the estimated parameters or may be set based on physical characteristics of the structured illumination system. For example, a nominal grid spacing of the structured illumination system may be stored. The stored parameters may then be referenced to perform operations such as: calibrated image reconstruction, providing feedback to hardware components to compensate for changes in structured illumination parameters, and narrowing the search space when predicting additional structured illumination parameters.

[0113] Lighting stripe pattern

[0114] 6 shows simplified illumination fringe patterns that can be projected onto the plane of the sample 271 by the vertical grating 230A and horizontal grating 230B of the SIM imaging system 200 during one imaging cycle to generate high-resolution images using structured light. In this example, three phase images with a vertical illumination orientation can be captured using the vertical grating 230A, and three phase images with a horizontal illumination orientation can be captured using the horizontal grating 230B. For each orientation, the projected fringes can be phase shifted to the appropriate position in steps of 1 / 3λ (e.g., by setting the phase modulator 230A or 230B to three different positions) to capture three phase images of the orientation pattern.

[0115] During capture of each phase image, light emitted by the sample may be imaged by camera 290. For example, fluorescent dyes located at different features of sample 271 may fluoresce and the resulting light may be collected by objective lens 270 and directed to an image sensor of camera 290 to detect the fluorescence. The six captured images may be used to image the entire sample or a larger sample location.

[0116] Once all images have been taken for an acquisition cycle (six images in this example), a high-resolution image can be constructed from the images taken. For example, a high-resolution image can be reconstructed from the six images shown in FIG. 6. Suitable algorithms can be used to combine these various images to synthesize a single image of the sample that has significantly better spatial resolution than any of the individual component images.

[0117] During construction of the high resolution image, undesired shifts or changes in structured illumination parameters (e.g., angle, spacing, and phase) may be algorithmically compensated for using predicted structured illumination parameters (e.g., predicted changes in angle, spacing, or phase) according to the present disclosure. For example, offsets in phase, angle, and / or spacing of the vertical illumination image and / or horizontal illumination image may be compensated for.

[0118] In some implementations, undesired shifts or changes in structured illumination parameters may be compensated for prior to image capture by controlling one or more hardware elements of system 200 to compensate for those changes in the SIM imaging system. For example, phase drift may be compensated for for each optical arm by adjusting a phase shifting element (e.g., a rotating mirror, a linear actuator, etc.) prior to an imaging sequence and / or during imaging of the images of the imaging sequence. In some implementations, a combination of hardware and algorithmic compensation may be implemented.

[0119] FIG. 6 illustrates an embodiment in which a patterned flow cell includes symmetrically patterned nanowells. In some embodiments, the nanowells may be asymmetrically patterned, e.g., the nanowells may be shaped or configured to form elongated structures. Elongated structures refer to shapes in which the dimension along a first axis is greater than the dimension along a second axis. Symmetrically shaped nanowells may be oval, rectangular, etc. In such embodiments, the sample may be resolved along the direction or axis of the longer side, and SIM is used to increase the resolution along the shorter side. If the shorter side of the nanowell is along the x-axis and the longer side is along the y-axis, the pitch Px of the asymmetrically patterned flow cell is narrow or packed, and along the y-axis, the pitch Py is larger. Thus, the resolution is increased along the x-axis, and only three images are taken to adequately resolve the sample contained within the nanowells of the flow cell.

[0120] 7 shows a simplified illumination fringe pattern that may be projected onto the plane of sample 570 by the first and second gratings of dual optical grating slide SIM imaging system 500 during image capture for a structured illumination imaging cycle. For example, SIM imaging system 500 may use first and second gratings 531 and 532 to generate the illumination pattern shown in FIG. 7. As shown in the example of FIG. 7, these two gratings project a perpendicular fringe pattern onto the surface of sample 570 and are positioned approximately ±45° from the axis of motion of linear motion stage 530.

[0121] For example, a first grating (e.g., grating 531) may project a first illumination fringe on sample 570. Any light emitted by the sample may be imaged by camera 580, and a first phase image of a first pattern (e.g., a +45° pattern) may be imaged to generate a first phase image. To image additional phase-shifted images, the pattern projected by the grating may be phase shifted by translating a linear motion stage. These phase-shifting motions are shown as steps 1 and 2 in FIG. 7. The phase-shifting motions may provide a small (e.g., about 3-5 micrometers or less) movement of the grating to slightly shift the fringe pattern projected onto the grating.

[0122] Following imaging of all phase-shifted images for the grating, system 500 can switch gratings by translating linear motion stage 530 to optically couple another grating to the imaging system's light source (e.g., transition to Figures 4A-4B). This movement is shown as step 3 in the example of Figure 7. When changing gratings, the linear motion stage can provide a relatively large translation (e.g., on the order of 12-15 mm).

[0123] A series of phase images can then be taken for the next grating. For example, as shown in Figure 7, a second diffraction grating can project a first order illumination fringe onto the sample, and the projected fringes can be shifted to the appropriate position by translating the linear motion stage 530 to take three phase images of the grating's pattern (e.g., steps 4 and 5 of Figure 7).

[0124] Once all images have been taken for an acquisition cycle (six images in this example), a high resolution image can be constructed from the images taken. For example, a high resolution image can be reconstructed from the six images shown in FIG. 7. Suitable algorithms can be used to combine these various images to synthesize a single image of the sample that has significantly better spatial resolution than any of the individual component images.

[0125] During construction of the high resolution image, undesired shifts or changes in structured illumination parameters (e.g., angle, spacing, and phase) may be algorithmically compensated for using estimated structured illumination parameters (e.g., predicted changes in angle, spacing, or phase) according to the present disclosure. For example, phase, angle, and / or spacing offsets of the vertical illumination image and / or horizontal illumination image may be compensated for. In one embodiment, one or more of the image phase, angle, and / or spacing offsets may be calculated regularly, such as in the first cycle of the sequencing run. The frequency of these estimates may be increased or decreased based on environmental factors that affect the stability of the process.

[0126] In some implementations, undesired shifts or changes in structured illumination parameters may be compensated for prior to image capture by controlling one or more hardware elements of system 500 to compensate for those changes in the SIM imaging system. For example, phase drift may be compensated for by translating linear motion stage 530 prior to an imaging sequence and / or during capture of the images of the imaging sequence. In some implementations, a combination of hardware and algorithmic compensation may be implemented.

[0127] According to the embodiments described herein, the structured illumination parameters may be predicted at a particular time point using estimates of the structured illumination parameters obtained from images captured before and / or after that time point. For example, computational resource limitations may limit the rate at which a SIM imaging system (e.g., system 100, 200, or 500) may directly estimate structured illumination parameters such as phase, frequency, and / or orientation from captured images. In some cases, the SIM imaging system may be able to directly estimate or measure all phase images of the structured illumination parameters, in which case there may be no need to predict the structured illumination parameters. However, in other cases, the SIM imaging system may only directly estimate or measure the structured illumination parameters for some phase images of an imaging cycle once per imaging cycle, or even less frequently (e.g., every 3, 5, 10, 50, or 100 imaging cycles). In such cases, in order to preserve the image sampling rate of the system, it may be advantageous to leverage direct estimates of the structured illumination parameters obtained for a particular time point and / or space to make predictions regarding the structured illumination parameters at other times and / or spaces.

[0128] To mathematically explain one example of this principle, correlation to a reference is one way to estimate structured illumination parameters.

[0129] Correlation output = Σ r c(x)h(xf) (1)

[0130] where h(x) is a reference that may be known or derived from the image data, c(x) is derived from the image data that is correlated with the reference, and f is a value to be estimated (in this example, a frequency). It should be noted that other alternative estimation techniques may be utilized in accordance with the present disclosure.

[0131] In the example of equation (1), one correlation output may be generated for each of multiple hypothetical values ​​of f. The parameter estimates f may be obtained as the value of f that maximizes the magnitude of the correlation. However, in many cases, multiple hypothetical values ​​of f may need to be tried to maximize the correlation output. A large search space may increase computational requirements, which may result in reduced system throughput (i.e., less data processed per unit time).

[0132] Light sensing (especially lens) distortion

[0133] In the following two sections, techniques for estimating parameters of SIM image reconstruction are presented. Some of the disclosed techniques correct for distorted or curved fringe peak lines due to lens imperfections. Pattern lines that are assumed to be parallel tend to start near the center of the image but converge or become non-parallel near the edge of the lens. This affects the illumination peak angle or orientation, illumination peak spacing, and phase offset. In FIG. 8A, we show the division of the image tile into overlapping regions called subtiles or subwindows or subfields. The subtiles are small enough that parameters can be set that result in sufficient reconstruction of the entire subtile. Parameter estimation can be performed in two steps. First, parameter estimation can be performed for a subtile near the center of the image. Then, parameter estimation can be performed for other subtiles and compared to the near-center subtile to determine the distortion and correction for the distortion for the parameters of the near-center subtile.

[0134] To illustrate implementation of these techniques, exemplary pseudocode is presented for parameter estimation of the "Subtile Angle and Spacing Mapping" algorithm (also referred to as Algorithm 2A) and the "Subtile Phase Mapping" algorithm (also referred to as Algorithm 2B). Also presented is exemplary pseudocode for determining the distortion and correction of estimated parameters for non-central subtiles as compared to near-central subtiles in the "Final Distortion Calibration Algorithm" (also referred to as Algorithm 1).

[0135] 8A-8C show the physical aspects of the full field of view (FOV). In one implementation, a rectangular sensor is used that is 5472 pixels by 3694 pixels. Of course, square shaped sensors or sensors of different sizes, for example 5472 x 5472 pixels or 4800 x 4800 pixels, may be used. When a rectangular sensor is used, distortion is greatest closest to the edge of the lens. Because lenses are often circular, a rectangular sensor is not as close to the edge of the lens on its long side as it is on its short side.

[0136] FIG. 8A shows two diagrams illustrating distortion of fringe spacing across the entire field of view (FOV). The diagram on the left 801 is a simplified diagram 801 of parallel lines that are bent due to magnified lens distortion. The illustrated lines are intended to be parallel in the image plane. When viewed through the lens, the lines appear to converge at the left and right ends relative to the center spacing. The diagram on the right 805 is another exaggerated example. In this diagram, the fringe lines are oriented diagonally between the upper left and lower right corners. The fringe spacing has been exaggerated so that it can be seen more easily. The fringe lines converge at the upper left and lower right corners relative to the center. In lenses from a particular manufacturer, the fringe pattern may be non-uniform.

[0137] Figures 8B and 8C show measurements of the spacing in the images between the peaks of nominally parallel fringes in the image plane for green and blue laser illumination. The color scale shows a variation in spacing from 2.08 to 2.22 (sensor pixels). In both figures, the color scale shows that the center-to-center spacing between parallel lines is approximately 2.14. Irregularities under green wavelength illumination are seen in the upper right corner of Figure 8B. Larger irregularities under blue wavelength illumination are seen along the right and left edges in Figure 8C. In these figures, the fringe pattern is a series of parallel lines from the bottom left to the top right of the figure. Thus, the spacing is measured perpendicular to the general direction of the arrows in Figure 8C. These figures motivate the correction of distortions caused by lenses. Because the lenses are manufactured and mounted individually, calibration and correction of the individual systems after assembly is desired.

[0138] FIG. 8B, drawn in grayscale, includes three notional contour ellipses with major axes along the diagonal from the top left to the bottom right. The contour lines approximately indicate regions with different spacing between the parallel lines. In priority provisional applications with color drawings, the color figures provide more readable details of the estimated variations. The stripe pattern is a series of parallel lines at 45° angles from the bottom left to the top right of the figure. The spacing between the parallel lines increases in the central region, as indicated by the lighter gray shading. As one moves away from the central region to the top right or bottom left regions, the spacing distortion between the stripe lines decreases. The major axes of the three contour lines are approximately perpendicular to the direction of the stripe lines. The spacing distortion decreases as one moves away from the region within the central contour line. FIG. 8C, in grayscale, also includes contour lines. The contour lines in FIG. 8C have the same meaning as explained above, indicating regions with different extents of spacing distortion.

[0139] FIG. 8D illustrates subtiles or subfields of the full field of view (FOV) within an image tile. In this figure, the subtiles illustrated are 512 pixels by 512 pixels. These subtiles may subdivide the illustrated field of view or may overlap. Subtiles may be larger or smaller. For example, subtiles of 400 by 400 and 1024 by 1024 pixels are shown to be useful. The figure illustrates 5 by 7 subtiles. The larger sensor may have 8 by 11 subtiles. Other configurations of subtiles such as 3 by 3, 5 by 5, 5 by 7, 9 by 9, 9 by 16, etc. may be used. Larger sensors may be divided into more subtiles. Subtiles may overlap by at least 2 pixels of the optical sensor. Larger and smaller numbers of pixels may be used for overlap between subtiles. For example, for 512 pixel wide subtiles, a maximum overlap of 256 pixels may be used, and for 1024 pixel wide subtiles, a maximum overlap of 256 pixels may be used. Consistent with Figures 8B and 8C, there are several candidate near-center subtiles 803, all of which are within the sweet spot of the lens and include the central subtile in the odd x odd subtile array. As used herein, a near-center subtile either includes the central pixel of the sensor or is adjacent to the subtile that includes the central pixel. In some optical systems that are flat and have small errors, subtiles from tiles that are adjacent to the central subtile can also be used as references without affecting the overall distortion compensation.

[0140] The disclosed technique includes the distortion of the mapping measured over substantially the entire field of view imaged by the sensor. Three parameters on which improved resolution SIM reconstruction from regular structured illumination depends include fringe spacing, fringe angle, and phase displacement of the fringe pattern. These variables are also referred to as the spacing, angle, and phase offset of the structured illumination or grid pattern. The spacing and angular displacement from the central tile value may be fitted over the entire field of view using polynomial surfaces. Both quadratic and cubic surfaces have been investigated. Higher order polynomials may also be used.

[0141] Both the fringe spacing and the fringe angle across the image tile may be fitted by a quadratic surface. Sensitivity analysis shows that the quadratic surface fits approximately as well as a cubic surface. The quadratic surface is fitted to the following equation:

[0142] f(x,y)=c0+(c1 * x)+(c2 * y)+(c3 * x * y)+(c4 * x 2 )+(c5 * y 2 ) (2)

[0143] In subsequent cycles of imaging, the temperature changes and the spacing from the central subtile can be characterized as shrinking or expanding to apply the same deviation correction to the remainder of the image. Depending on the shrinkage and expansion of the near-center subtile, the deviation correction can be scaled and translated accordingly. It has been observed that the relative deviation of the subtile is constant with respect to the central or near-center subtile used as a reference.

[0144] FIG. 9 shows a color-coded surface fitted to the measured data points shown in red. Because the surface is three-dimensional, reference to a color version of the figure is recommended for understanding the contours. The dark shaded areas of FIG. 9 depicted in grayscale are represented in dark colors at the top and uppermost parts of the three-dimensional plot. For example, the dark shaded top right portion of the graphic plot of FIG. 9 is represented in dark red, and the dark shaded area at the bottom is represented in blue in the color version of the figure. The USPTO will store the color version in the Supplemental Content tab of the PAIR Online system upon filing of the priority application and will make it publicly available in due course.

[0145] In practice, it has been observed that while angular distortions are characteristically saddle-shaped, spacing distortions across the field of view can be described as bowl-shaped or inverted bowl-shaped. Figures 10A and 10B show the inverted bowl shape of the measured spacing versus spacing within the near-center subtiles. Figures 12A and 12B show the saddle shape of the measured angle versus the fringe pattern angle in the near-center subtiles.

[0146] In Figures 10A and 10B, the x-axis is along the length of the sensor. The y-axis is along the width of the sensor. The labels on the x-axis and y-axis are subtile indices. The vertical axis spacing is normalized to 1.000 spacing on the z-axis between the regularly spaced fringes seen in the near-center subtile. Figure 10A shows the measured data points that have not been smoothed by curve fitting. This shows an approximately 1.2 percent variation in fringe spacing observed between measurements at the center of the sensor and measurements in the lower right portion. In Figure 10B, the actual data points, plotted in blue, are compared to a quadratically fitted surface. In the following sections, measuring spacing and angular distortion across the full FOV compared to the near-center subtiles is presented. The grayscale versions of Figures 10A and 10B have darker shaded areas at the top and bottom of the graphic plot. The darker shading in the top portion represents red, and the darker shading in the bottom portion represents blue.

[0147] Spacing and Angular Distortion Calibration

[0148] This section further describes techniques for calibrating the full field of view (FOV) on a sensor applied to imaging of structured illumination and fitting functions to nonlinear effects caused by lens distortion. Structured illumination involves projecting regularly spaced and angled patterns onto an image plane. An image tile of the projected structured illumination pattern is imaged. A near-center subtile is selected from the full field of view. The pattern spacing and angle across the subtile, including the variation within the subtile, are measured. Additionally, the spacing and pattern angle across other subtiles across the full field of view are determined. The spacing and pattern angle across the full field of view are described relative to the spacing and pattern angle within the near-center subtile. A polynomial surface is iteratively fitted to the spacing and pattern angle across substantially the full field of view.

[0149] 11A, 11B, and 11C compare the measured spacing distortion with quadratic and cubic surface fits. Color variation is slight in the three figures, as there is less than 2% variation in spacing across the field of view in all three figures. Visual analysis of these figures and calculation of fitting error suggests that a quadratic polynomial provides a sufficiently good fit to the spacing distortion. The spirit and teachings of this disclosure encompass higher order polynomials and alternative fitting schemes that may produce slightly better or worse results than a quadratic polynomial. The grayscale versions of Figures 11A, 11B, and 11C include dark shading in the center portion representing a dark red color, while the outer portions at the top right and bottom left have lighter shades of gray representing a light red color, transitioning to orange and yellow near the borders.

[0150] 12A and 12B share the x-axis and y-axis configuration with FIG. 10A and 10B. The vertical axis in these figures indicates the angle of the fringe orientation observed across the field of view. Nominally, the angle may be intended to be 45°. The range of angles measured is 44.85 to 45.15 degrees, which is a variation of 0.6%. FIG. 12A shows the measured data points that have not been smoothed by curve fitting. In FIG. 12B, the actual data points, plotted in blue, are compared to a quadratically fitted surface. The grayscale version of FIG. 12A and 12B has darker shading at the top and bottom of the graphic plot. The darker shading at the top represents darker red and the darker shading at the bottom represents darker blue.

[0151] 13A-13E and 14B-14H show how cropping along the sensor boundary can improve curve fitting of spacing and fringe angle on the sensor. The x and y axes of the graphs in these figures represent pixel position along the length and width of the sensor. With reference to FIG. 8C, cropping is shown by the boundary near the edge of the sensor. This figure shows that this cropping or shrinkage removes distortion near the edge of the image captured by the sensor. The figure below shows how this shrinkage of the image allows for a good fit of the FOV data to the curved surface model. The shrinkage factor or shrinkage unit can be equivalent to shrinking the image by one sliding window stride. The stride value can be selected to match the size of the subtile or can be empirically selected based on the number of pixels at the edge of the sensor that may cause distortion. For example, if the stride value is selected as 200 pixels, one shrinkage unit can remove the outer 200 pixels of image data of poor data quality on all sides of the image. Stride values ​​smaller than 200 pixels, such as 25, 50, or 100 pixels, or stride values ​​greater than 200 pixels, such as 225, 250, or 300 pixels, may be used.

[0152] FIG. 13A shows a quadratic model surface for a spacing distortion fit to image data for the full field of view (FOV) without cropping in blue dots. No shrinkage factor is applied to this full FOV image data, so the shrinkage factor is zero. As shown in FIG. 13A, the full FOV data does not provide a good fit to this model. As the shrinkage factor is progressively increased from 1 to 4 (FIGS. 13B-13E), the fit between this model and the data points without cropping improves. Cropping removes outlying data points and shrinks the scale along the z-axis, showing less variation between subtiles. Referring back to FIG. 8C, which shows that the outlying data points are along the edge of the sensor, cropping is applied. In FIG. 13C, many more outlying blue data points are more evident than in 13D-E. In FIG. 13E, the MSE drops to 0.000001 and the z-scale extends to be a bit lower than in FIG. 13D. The grayscale versions of Figures 13A-13E, 14B-14H, and 15A-15B have darker shading at the top and bottom of the graphic plots. The darker shading at the top represents darker colors at the top and bottom of the graphic plots. Some of these graphic plots have light and dark circles located on or near the graphic plots. The light and dark colors of the circles do not indicate any difference between the circles. Some circles are drawn in dark colors to make them visible.

[0153] 14B-14H show the improvement of the quadratic surface fit for angular distortion by incrementally increasing the shrinkage factor applied to the full FOV image data. As the shrinkage factor increases, the scale along the z-axis shrinks again, bringing the majority of the data points closer to the fitted surface. Excluding the outlying distorted values ​​on the edge of the sensor improves the surface fit significantly. In FIG. 14B-14E, the distortion is evident from the planar blue data points below the fitted surface. As the shrinkage factor increases from "1" (FIG. 14C) to "3" (FIG. 14E), the scale along the z-axis shrinks by half from [-45,-20] to [-44,-40]. In FIG. 14F, cropping finally occurs, and the data points and fitted surface appear closer to each other. In FIG. 14F, the MSE drops to 0.000033. Here, the quadratic surface model for angular distortion is consistent with the measured data shown as blue dots. In FIG. 14H, an increasing shrinkage factor of "6" causes marginal improvement. From these figures, shrinkage factors of "4" and above appear to be normal. A threshold based on percentage or absolute MSE improvement can be used to select the shrinkage factor or to determine the impact of the shrinkage factor on resolution capability. The selection of the shrinkage factor can be automated using a ratio or threshold approach.

[0154] Characterizing the imaging system by fitting distortion surfaces can be performed before and / or during operation of the instrument by occasionally re-estimating the spacing and pattern angles within the near-center subtiles and using an extrapolation rate to determine the spacing and pattern angles across the entire field of view. The extrapolation rate can be applied without estimating the spacing and pattern angles beyond the near-center subtiles. Subtiles in the entire field of view can overlap. During calibration, spacing and pattern angles are calculated independently for overlapping subtiles. With this approach, two angles of orientation of the structured illumination pattern can be used such that the two angles are substantially aligned with opposite corners of a rectangular or quadrilateral pattern of nanowells.

[0155] The disclosed techniques therefore include calibrating and recalibrating the instrument to characterize the variations between subtiles of a tile. Characteristic distortions of spacing and angle can be imaged to greatly reduce the need for parameter estimation during instrument operation. Characterization / calibration can be improved by cropping pixels at the edge of the sensor that are distorted to not fit the mapping surface, cropping pixels that disrupt the fit.

[0156] Application of calibrated distances and angles

[0157] The disclosed technique can efficiently reconstruct an improved image across substantially the entire field of view. To this end, the entire field of view is divided into at least nine subtiles or subfields for independent reconstruction on a subtile basis. The already estimated spacing and pattern angle parameters can be estimated or accessed by looking at regularly spaced and angled structured illumination patterns for the near-center subtiles. An extrapolation factor is applied to extend the near-center subtile parameters to other subtiles across the field of view. An improved image reconstruction is performed from the multiple images using the near-center subtile estimated parameters and the extrapolated parameters of the other subtiles. The multiple images are viewed at multiple pattern angles and periodically displaced pattern phases for each pattern angle by applying Structured Illumination Microscopy (SIM) analysis to the multiple images. In one embodiment, the above technique can further estimate a regularly displaced pattern of phases for each pattern angle. The above technique can be extended to stitch subtiles together by identifying regular patterns of noise along the stitching boundaries and mitigating the noise.

[0158] Figures 15A and 15B are examples of distortion models fitted from actual data collected from an ordering device. Figure 15A shows a quadratic fitted response surface obtained by fitting fringe spacing distortion across the entire field of view (FOV). Figure 15B shows a quadratic fitted response surface obtained by fitting fringe angle distortion across the FOV.

[0159] Phase shift estimation

[0160] Phase displacement is the third remaining input parameter for SIM image reconstruction. This section adapts existing techniques for estimating phase displacement. Since the disclosed technique performs estimation on subtiles, a technique is further presented to establish a common frame of reference for the estimated phase displacement across subtiles with respect to the entire FOV tile. One implementation of phase estimation corresponds to the technique proposed by Wicker et al. in Section 3 of the paper entitled "Phase Optimisation for Structured Illumination Microscopy" (2013). The formulas by Lal et al. (2015) and Wicker et al. (2013) entitled "Structured Illumination Microscopy Image Reconstruction Algorithm" help to explain Wicker's phase estimation.

[0161] The following equation (3) derived by Lal et al. in 2015 describes three bands of frequency components:

number

number

number

number

[0162]

number

[0163] This equation, which has three components, is obtained from the Fourier transform for sine or cosine illumination. Different illumination functions will change the equation.

[0164] Therefore, accurate knowledge of the illumination sinusoidal intensity pattern phase is important. Since it is not always possible to precisely control these phases in an experimental setup, it is desirable to determine the illumination pattern phase from the acquired image data. Wicker et al. (2013) present a phase estimation technique for SIM data acquired using coherent sinusoidal illumination at a selected frequency. Coherent illumination produces good pattern contrast from fine gratings with very small illumination peak spacing "s", which improves the reconstructed resolution. The peak frequency of the illumination pattern is used to obtain the illumination pattern phase for the nth image. The peak frequency of the illumination pattern is also referred to as the Fourier peak.

[0165] The following equation (4) by Wicker et al. (2013) is

number

number

number

number

[0166]

number

[0167] In addition, "c" in equation (4) is called the contrast of the illumination pattern. In the absence of noise, "c" is the same as the modulation factor "m" in the mixing matrix M in equation (3). φ n In order to determine

number

number

[0168]

number

[0169] Equation (5) expresses the pattern phase φ n However, the captured image

number

number

[0170]

number

number

number

number

[0171] Third, the pattern frequency

number

number

number

number

[0172] The OTF of an optical system can be determined experimentally. For example, Lal et al. (2015) calculate the OTF by acquiring several images of a sample with sparsely distributed 100 nm fluorescent microspheres. The intensity distributions corresponding to over 100 microspheres were then superimposed and averaged to obtain an approximation of the PSF of the system. The Fourier transform of this PSF estimates the OTF of the system. With this background, we can apply the phase estimation technique described in the subtile.

[0173] Translation of subtile phases to a common frame of reference

[0174] It is useful to estimate the phase displacement of the tile relative to the full field of view (FOV), so that measurements of phase within one subtile can be extrapolated to other subtiles across the tile. The illumination peak angle and illumination peak spacing for the full FOV can be estimated from the illumination peak angle and illumination peak spacing of the subtile using the quadratic model presented above. The phase displacement is less regular because it depends on the pixel shape of the subtile, which may produce an irregular step function instead of a smooth function. Each phase estimate has a "frame of reference" fixed to the top left corner of the image used to estimate the phase. As a result, if it is desired to geometrically correct phase differences across the image, the estimated phase from each subtile needs to be compared to a reference phase (from the central estimation subwindow or subtile) in a common frame of reference. To achieve a common frame of reference, the phase estimate of each subtile is translated to the same geometric location. Translating the phase requires accurate estimates of the fringe angle and spacing of each subtile. The fringe angle and spacing can be obtained from a calibrated quadratic surface distortion model as a result of the distortion calibration process.

[0175] Phase estimates are expressed using a common reference system across subtiles of the full FOV image. The subtile coordinate space is mapped to the full FOV coordinate space. In this section, the translation of point coordinates from one space to another is shown. FIG. 16A shows a typical setting of subtile windows and related variables in the full FOV image window, as well as the corrective phase offset equations to translate coordinates between local and global references. FIG. 16A shows a boundary 1603 representing the full field of view including the subtiles indicated by boundary 1605. The crop angle α is defined by a line 1606 connecting the top left corner of the full FOV window 1603 and the top left corner of the subtile window 1605. The length of the line 1606 is measured in pixels and is labeled "a".

[0176] An illumination peak angle β is formed between the top boundary of the sensor and a line 1607 perpendicular to the window 1603 and the fringe peak line 1609 at the top left corner of the subtile window 1605. This fringe peak line is parallel to the actual fringe peak and is easily offset from the peak. The length of line 1607 from the top left corner of 1603 to 1609 is the projected distance "b". The phase difference φ between the top left corner of tile 1603 and subtile 1605 is a function of the spacing s and distance b. In equation (7), the illumination peak spacing is represented in pixels as "s" and the phase is expressed in degrees as follows: φ o =([b+(φ s / 360 * s)]%s) / s * 360 (7) This formula, derived and explained below, can be applied to translate the phase offset from subtile coordinates to full FOV coordinates. Translation between coordinate systems is illustrated in the two examples below.

[0177] In SIM reconstruction, a one-dimensional (1D) grid can be represented by an intensity function such as:

[0178]

number

[0179] where the parameters have the following meaning: · M is the modulation depth. ·s is the illumination phase interval. ·

number

[0180] The bold x and u in the formula represent vectors, where x is a vector in image space representing the location in image space where the intensity is determined, and u is a unit vector. Intensity values ​​for sinusoidal illumination are positive within the scale range from +M (bright) to -M (dark).

[0181] Once the phase displacement differences between subtiles are characterized, only the phase displacements at the near-center subtiles need to be estimated because the phase displacements of other subtiles can be extrapolated from the near-center subtiles.

[0182] The following calculations are relevant for estimating the phase offset φ. Rather than estimating the phase offset using data from the entire image (full FOV), it is possible to use data from only a subtile of the image. The subtile phase offset is related to, but not identical to, the full FOV phase offset. To recover the full FOV phase offset from the subtile phase offset, it is necessary to characterize how the phase offset transforms under spatial translation. The following mathematical model presents this spatial translation from subtile to full FOV.

[0183] Derivation of coordinate transformation

[0184] Consider the following function that generates grid lines with the orientation as shown in FIG. 16A. The two-dimensional (2D) space of the entire FOV is represented by coordinates (x 1 ,x 2 ) in the coordinate space Ox. 1 ,x 2 ) represents the point 1611. 1 ,x 2 ), as shown in equation (9), can be defined as follows: This function can be used to generate illumination pattern lines with an orientation as shown in FIG.

[0185]

number

[0186] The one-dimensional periodicity of the above function f is expressed by the function in Equation (10). With reference to FIG. 16A, this function describes the repetition of the illumination pattern lines as one moves along line 1607 in FIG. 16A, which is perpendicular to the illumination pattern lines. This function can determine the spacing "s" between two lines by determining where a particular intensity value is repeated. Equation (10) describes the periodicity of the intensity of the illumination pattern. In Equation (10), the line perpendicular to the intensity peak is at a 45 degree angle extending down and to the right from the top left corner, so that point (x 1 ,x 2 ) is the intensity at successive points

number

[0187]

number

[0188] The illumination peak spacing "s" defined by the above function is shown in Equation (11). For convenience, the coordinate system can be scaled to match the spacing of the illumination pattern as shown in Equation (11).

[0189]

number

[0190] The illumination peak angle is defined in equation (12). The illumination peak angle is formed between the top boundary of the entire FOV (horizontal line) and a line perpendicular to the illumination pattern lines shown in Figure 16A. Equation (12) shows that this angle is -45 degrees for the structured illumination pattern generated by the 1D periodic function in equation (9).

[0191]

number

[0192] The zero set of pattern lines formed by the periodicity function presented in equation (9) is the combination of lines and is expressed by equation (13).

[0193]

number

[0194] The zero set includes structured illumination pattern lines spaced apart from each other by a distance “s”. The pattern lines are represented by the 1D function f(x 1 ,x 2 ) is zero. The unit vector perpendicular to the lattice is defined by equation (14).

[0195]

number

[0196] Here, the coordinate (y 1 ,y 2 ) the subtile coordinate space O defined by y From the entire FOV coordinate space O x Two examples are presented for transforming the phase offset of a subtile to the vector

number

number

number

[0197] First Example

[0198]

number

[0199] (y 1 ,y 2 )=T(x 1 ,x 2 )=(x 1 -a,x 2 +a) (15)

[0200] Now, the 1D periodicity function of equation (10) in the subtile coordinate space Oy can be expressed as:

number

[0201] It can be seen that the phase offset introduced by the change in coordinates is given by:

number

[0202] Second Example

[0203] Now consider another position of the subtile relative to the top left corner of the full FOV window.

[0204]

number

[0205] (y 1 ,y 2 )=T(x 1 ,x 2 )=(x 1 -2a,x 2 +a) (18)

[0206] Now, the 1D periodic function of equation (10) can be expressed in the subtile coordinate space Oy as follows:

number

[0207] In this case, the phase offset introduced by the change in coordinates is given by:

number

[0208] General case

[0209] Theory 1.

number

number

[0210]

number

[0211] a) The lattice spacing of f is s. b) Either

number

number

[0212]

number

[0213] Equation (21) shows the periodic function for the entire FOV coordinate space Ox, and the phase offset φ 0 Equation (22) shows the periodicity function for the subtile coordinate space, and the phase offset φ 0 has.

[0214] c) any such φ a satisfies the following equation.

[0215]

number

[0216] Proof. (a) The illumination peak spacing is the length of the fundamental period for f. Vector

number

number

[0217] (b) One solution is

number

[0218]

number

number

[0219] Using the above relationship, the phase displacement estimated in the first coordinate space can be transformed to a second coordinate space, such as between a subtile and a full FOV image tile. This is useful because the phase displacement also undergoes distortion between a near-center subtile and the rim of the lens. In the following section, a phase bias lookup table is presented that can capture the relationship between phase displacement values ​​for non-center subtiles with respect to the central subtile.

[0220] Phase bias estimation

[0221] The disclosed technique involves estimating the phase for each subtile and then converting the phase offset to full FOV coordinates, so that the estimated phase displacement values ​​are expressed in a common reference system. We store the difference between the phase displacement values ​​for non-central subtiles and the central subtiles in a phase bias or phase difference lookup table for each angle of the illumination pattern. In some cases, the influence of environmental factors may increase the instability of the system. In such cases, the phase bias lookup table is generated more frequently to compensate for the increased system instability. For example, the phase bias table may be generated in the first cycle of every ordering run. The frequency of generation of the phase bias lookup table can be increased or decreased according to the stability of the system.

[0222] FIG. 17 shows an exemplary phase bias lookup table 1701 with color coding of phase differences from the central near subtiles at 3, 5 with zero values. Lookup table 1701 contains as many cells as there are subtiles in the full FOV image data, in this example 7×9 in subtiles. The location of the cells in table 1701 may correspond to the location of the subtiles in the full FOV image data. The phase lookup table stores the difference in phase between each of the subtiles relative to the central window (3, 5) estimated phase. The exemplary table includes phase bias for one angle of structured illumination. A similar phase bias table is stored for each angle of the structured illumination pattern. The phase values ​​are sensitive to accumulated errors in the estimates of the other parameters (angle and spacing). Note that the phase values ​​wrap from 0 degrees to 360 degrees. The phase of the central subtile is shown as 0 degrees in the exemplary table. The color coding indicates the degree of difference in phase of the subtile corresponding to the central subtile. Estimating the phase bias requires the fringe angle, spacing, phase of the central subtile or subwindow, and the distortion model (assumed unchanged). The following sections present exemplary algorithms for estimating the angle, spacing, and phase parameters.

[0223] The above sections describe techniques for estimating parameters for subtiles in a full FOV image. The following pseudocode further describes estimating angle, spacing, and phase parameters for subtiles. Exemplary pseudocode for estimating angle and spacing parameters is presented below and is referred to as "Subtile Angle and Spacing Mapping" (also referred to as Algorithm 2A). A quadratic or cubic polynomial fit is used to estimate distortion coefficients for the subtile angle and spacing parameters. Exemplary pseudocode for implementing the phase estimation technique is presented below and is referred to as the "Subtile Phase Mapping" algorithm (also referred to as Algorithm 2B). Phase offset estimates may be stored in a phase bias lookup table for each angle or orientation of the illumination pattern. In one embodiment, a separate lookup table may be used for each ordered tile on the flow cell to account for flow cell orientation, surface defects, and other non-idealities that may skew the estimated results. Pseudocode for an exemplary algorithm for final distortion calibration is presented at the end and is called "Final distortion calibration with quadratic fit and lookup table" (also called Algorithm 1 and Algorithm 2C).

[0224] Algorithm 1: Final distortion calibration by quadratic fit

[0225] In the first pseudocode, cropping can be applied to remove distortion near the edges of the image, allowing a good fit of the full FOV data to a quadratic surface model for angle and spacing estimation during calibration / characterization.

[0226] Input: e.g., acquisition of images for 2 angles and 3 phases per angle Output: Distortion model coefficients algorithm: 1. Build the following image / filter (sized to match the subregion): a.OTF 2. Divide the image into equal sized but overlapping sub-regions, e.g. 512x512, using the sliding window method 3. Estimate parameters at the center of the image, for example using a 512x512 window (see Subtile Angle and Spacing Mapping Algorithm and Subtile Phase Mapping Algorithm below) a. Estimate the illumination peak angle and interval for each angle b. Estimate the modulation for each angle c. Estimate the phase for each image 4. For each sub-area, a. Perform parameter estimation in step 3 b. Store the subtile center x,y coordinates, fringe angle (per angle), spacing (per angle) and phase0 (per angle) of the Wicker phase estimates in an array (2d array) 5. In the case of stripe spacing and angle, a. Subtract the table values ​​from step 4 by the center window estimates from step 3. b. Perform optimal shrinkage estimation of the quadratic surface i. The quadratic surface is condensed by least squares regression (using the same input arrays X and Y corresponding to the x, y coordinates of the subregion centers) into a functional form f(x,y)=a+b for each array (Z is called, for example, the fringe spacing from angle 0) in step 4. * x 2 +c * y 2 +d * xy+e * x+f * Fit to y. ii. The coefficient of determination from step i, R 2 Rate the following. iii. Erosion the X, Y, Z arrays (remove outer edge elements) and repeat step (i) until N erosions have been performed, where N = 3 to 5 (an adjustable parameter). iv. As the array shrinks towards the center of the data, the fit quality is expected to improve, and coefficients are selected at the elbow point (to minimize the amount of shrinkage and also to reduce R 2) and the model fit is sufficiently good (the current standard is R 2 (defined as less than 5% relative improvement in value). Other acceptable criteria that may be used herein include the following: Improvement of the mean squared error of the fit (the mean squared error of the next iteration is smaller than the mean squared error of the current iteration by some predefined threshold) Improvement of the maximum fitting error of the quadratic surface (if successive improvement detects a plateau, each successive iteration should reduce this error and increase this fitting) A normalized measure of error such as MAE (mean absolute error) instead of MSE (mean squared error), using improvement thresholding similar to the previous two criteria. 6. The optimal coefficients from step 5 are the output of the final distortion calibration algorithm.

[0227] Algorithm 2A: Subtile angle and spacing mapping

[0228] Pseudo code for implementation of illumination peak angle and spacing estimation for a subtile, used during steps 3-4 and operations above, is as follows:

[0229] Input: Collect images at the same angle so that the phase of each image is uniformly spread from 0 to 360 degrees output: 1. Location of the peak in Fourier space (or spacing and grid angle) 2. Modulation of Fourier peaks 3. Phase offset of the first image algorithm: 1. Fast Fourier transform of each image 2. Perform band separation 3.2 Acquire component images: a.0 (wide field) and 1 (high resolution area) b. Mask out all non-overlapping Fourier components of each image (also known as the common region) c. Mask low frequency components around DC and around the estimated fringe peaks d. Apply inverse OTF 4. Repeat the grid around the starting point to create a much finer grid. a. For each point on a 10x10 grid, uniformly sample some range i. Apply a phase shift to one of the two image phases ii. Multiplexing one image with the conjugate of the other iii. Sum the results iv. Normalization to the sum of wide-field mask Fourier images v. Saving the shift provides the maximum of the absolute value of the normalized complex correlation 1. The absolute value of this number is the modulation 2. The angle of this complex number is the phase offset

[0230] Algorithm 2B: Subtile Topological Mapping

[0231] Pseudocode for an exemplary implementation of subtile phase shift estimation. The disclosed technique can use the latest best estimates of spacing and angle parameter values ​​for this mapping. The system can also use a distortion model and central subwindow or subtile parameters to obtain estimated spacing and angle parameter values.

[0232] input: 1. Acquire images for a single angle 2. Illumination peak spacing and angle estimated from the above algorithm Output: Phase offset for each image in the acquisition algorithm: 1. Perform a Fast Fourier Transform (FFT) of each image 2. Make a copy of the FFT image 3. Apply OTF to each image 4. Shifting a single image 5. Multiplex both images using the conjugate and sum absolute value of each complex number 6. Phase is the angle of the resulting complex number

[0233] Algorithm 2C: Phase bias lookup learning

[0234] For phase bias, the following steps are performed for all tiles on the flow cell during imaging of the first cycle of the sequencing run.

[0235] Input: e.g., collection of images for 2 angles and 3 phases per angle from each tile of a sequenced flow cell, with the calibrated distortion model from Algorithm 1 Output: Phase bias lookup table algorithm: For each ordering tile (not confused by SIM reconstruction subtiles) i. Apply step 3 of Algorithm 1 to estimate the tile's central window SIM parameters (spacing, angle, phase for each angle-channel pair) ii. Divide the image into subtiles and for each subtile: a. Apply the distortion model via Algorithm 2A to obtain the subtile fringe spacing and subtile angle b. Perform Wicker phase estimation on the first phase image of each channel-angle pair using the given fringe parameters from step a on the local subtile. c. Calculate the difference between the phase obtained in step iib and the central window phase 0 obtained from step i and store the phase bias in the corresponding 2d array for phase bias iii. Save the completed phase bias lookup table in memory with the tile ID as the entry to be used later. Repeat step 1 for all other ordered tiles encountered throughout the row cells. NOTE: For performance reasons, the phase bias lookup for each tile in the first cycle of the ordering run is estimated, but if there are stability concerns, the learning frequency can be adjusted to learn every N cycles.

[0236] In these pseudocodes, techniques were presented to estimate and map the illumination peak angle, illumination peak spacing, and phase shift parameters. The estimated parameters are used as inputs to Structured Illumination Microscopy (SIM) image reconstruction. In the following sections, we present the proposed SIM image reconstruction algorithm that can efficiently reconstruct high-resolution images of subtiles and then combine those subtile images to reconstruct a high-resolution image of the full FOV.

[0237] SIM image reconstruction

[0238] The disclosed SIM image reconstruction technique uses the subtile angle, spacing, and phase estimates to reconstruct a high-resolution image. The disclosed technique can construct a full-FOV high-resolution image in two steps. First, the subtile parameters are used to reconstruct a high-resolution SIM image for the subtile. Second, the high-resolution images for the subtiles can be stitched together to construct a full-FOV image. Alternatively, the high-resolution subtiles may be processed without stitching. This section begins with an exemplary pseudocode for the SIM image reconstruction of a subtile. This is referred to as the "subtile reconstruction" algorithm. The disclosed technique uses non-redundant data in the frequency domain to reconstruct a high-resolution image in Fourier space. Hermite We exploit symmetry, which reduces the computation required for some core operations by roughly half. This section also presents an example pseudocode for combining the SIM images for the subtiles to reconstruct a high-resolution image of the full image. This is referred to as a "full-field reconstruction" algorithm. Note that the above parameter estimation technique can be used with traditional redundant or the disclosed non-redundant SIM image reconstruction technique.

[0239] Algorithm 3: Subtile reconstruction

[0240] Pseudocode for one embodiment for reconstructing high resolution structured illumination microscopy image subtiles is as follows:

[0241] input: a. Some lattice profiles (I angle,phase ) collection of images of the sample. For each angle, at least three phase images are required, evenly spaced from 0 degrees to 360 degrees. b.Parameter: i. Illumination peak spacing and angle for a set of images corresponding to an angle ii. Phase for each image Output: One high-resolution reconstructed image for the subtile algorithm: 1. Pre-process image (e.g. high pass filter) 2. Create a large Fourier image buffer (its size depends on the final upsample factor required) and initialize it to 0. 3.About each angle a. For a series of images, each with a unique phase for the grating for a given angle i. Perform a Fast Fourier Transform on each image b. Perform band separation as images are collected (n>=3 input images → 3 output or component images) i. Two of the three output images contain higher resolution information. ii. The top left image has all the standard information (original resolution image) iii. The phase for each image and modulation needs to be known for the entire set c. Apply the MTF to each of the component images d. Shift each component image from the band separation to their appropriate location in the larger Fourier image buffer and sum the components. 4. Construct a Wiener filter from the input MTF by shifting and summing the 2D MTF image into a large Fourier image a. This is done for each of the input images for each angle b. This does not have to be done for each reconstruction, but can be done once and provided as an input parameter. 5. Applying Wiener and apodization filters in Fourier space to the large Fourier representation of the reconstructed image 6. Perform an inverse fast Fourier transform on a large Fourier image buffer to obtain a high-resolution image

[0242] Since the disclosed technique cannot be explained by referring to the prior art, further explanation is provided for this non-redundant subtile reconstruction pseudocode than for other pseudocodes. The disclosed technique exploits symmetry in Fourier space to increase efficiency and reduce computing load in SIM image reconstruction. Figure 18 presents a high-level overview of our non-redundant subtile construction technique. Three images 1811, 1813, 1815 are acquired for each subtile per angle, preferably with uniformly spaced phases. Separate Fast Fourier Transforms (FFTs) are applied to convert image data from spatial domain (real space) to frequency domain (complex space). FFT symmetry of data representing the three images in frequency domain is exploited. By operating on the non-redundant portion of the image data in frequency domain, the efficiency of reconstructing high-resolution SIM images of the subtiles is improved. Details of the processing steps of the non-redundant subtile reconstruction algorithm are presented below.

[0243] input

[0244] 1-a. Image collection

[0245] An image of the sample divided into subtiles is provided as input to the subtile reconstruction algorithm. The image is divided into subtiles or subwindows or subregions of a predefined size, for example 512x512 pixels. Other sizes of subtiles may also be used.

[0246] The collection of image tiles extends the prior art because many tiles are required to image and capture a plane. In one embodiment, the image tiles are irradiated with a sinusoidal illumination intensity pattern I. θ,ΦThe intensity is acquired with a diffraction grating that produces a 2D spatial position vector (r). The intensity varies according to the location (x, y) represented by the vector r, which is a two-dimensional spatial position vector. For comparison, a basic structured illumination microscopy reconstruction algorithm labeled as "Algorithm 1 SIM-RA" by Lal et al. (2015) is presented. We used the mathematical formulation from Lal et al. (2015) to explain the steps of our SIM image reconstruction algorithm. Note that the reconstruction algorithm presented by Lal et al. (2015) is applied to the full image, whereas our subtile reconstruction technique reconstructs high-resolution SIM image subtiles.

[0247] I θ,Φ A mathematical formulation for (r) is presented in Equation (1) of Lal et al. (2015). For each orientation of the sinusoidal illumination pattern θ, three SIM images of the specimen corresponding to three different illumination phases are acquired. In one embodiment, two sets of images are acquired at two angles of orientation of the sinusoidal illumination pattern, i.e., θ 1 = 45°, and θ 2 = 135°. The orientation angle θ is also referred to as the illumination peak angle β. In such embodiments, the nanowells may be positioned at the corners of a square or, more generally, a rectangle or quadrilateral. The orientation angle is selected such that the image is taken along a line connecting the opposing corners of the rectangle. This increases the apparent distance between the nanowells.

[0248] In another embodiment, three sets of images are taken at three orientation angles, e.g., θ 1 = 0°, θ 2 = 60°, θ 3 = 120° or θ 1 = 30°, θ 2 = 90°, θ 3 = 150°. In this embodiment, the nanowells may be located at opposite corners of a hexagon. Images are acquired along lines connecting opposite ends of the hexagonal geometry. In general, the set of images may be selected to match the number of diagonal lines connecting opposite ends of the geometry in which the nanowells are located.

[0249] For each orientation θ, at least three images spaced from 0° to 360° are captured with three different illumination phases φ 1 =0°, φ 2 = 120°, and φ 3 = 240°. A uniform phase interval is preferred. More images, such as 5, 6 or 7, can be acquired by using smaller phase steps.

[0250] 1-b(i) Structured illumination pattern angle and spacing parameters

[0251] Periodicity of Grating or Sinusoidal Intensity Pattern I θ,Φ The illumination peak spacing "s" that defines the illumination of (r) is also given as an input. The illumination peak spacing "s" and illumination peak angle θ for the subtiles are estimated using the "Subtile Angle and Spacing Mapping" algorithm (also referred to as Algorithm 2A) presented above. A "Final Distortion Calibration" with a "Quadratic Fit and Lookup Table" algorithm (also referred to as Algorithm 1) is applied to compensate for errors in estimating the angle and spacing parameters of non-central subtiles due to photo-sensing distortion. Exemplary pseudocode for the "Subtile Angle and Spacing Mapping" and "Final Distortion Calibration" are presented above, along with the quadratic fit and lookup table algorithms.

[0252] 1-b(ii) Phase shift parameters

[0253] Phase displacement (also called grating phase) "φ" (for structured illumination intensity pattern) I θ,Φ(r) is also given as an input. Phase values ​​for the subtiles are estimated using the "Subtile Phase Mapping" algorithm (also referred to as Algorithm 2B). A "Final Distortion Calibration" is applied along with the "Secondary Fit and Lookup Table" algorithm (also referred to as Algorithm 1) to compensate for errors in the estimation of phase parameters for non-central subtiles due to photo-sensing distortion. The difference between the phase of the non-central subtiles and the central subtile per angle is stored in a lookup table. Exemplary pseudocode for the "Subtile Phase Mapping" algorithm is presented above. With these inputs described, processing is performed.

[0254] Processing step 1: Pre-process image (high pass filter)

[0255] The process for subtile reconstruction may begin with image pre-processing. Pseudocode generally corresponding to this specification is described as follows. The reconstruction of the improved image is performed in Fourier space. However, before applying the Fourier transform to obtain the image data from the spatial domain to the frequency domain, a pre-processing step may be applied to the image tiles, as described below. The following description is based on three images with different phase values ​​along one angle. As mentioned above, more than two orientations may be used.

[0256] The acquired images may be preprocessed to remove noise. There are many sources of noise. For example, exposure to strong excitation light may produce photobleaching that reduces the intensity of emission from the fluorophores. An illustration of noise in acquired SIM images is shown in Figure 6 of Lal et al. (2015). Figure 6, the top left image labeled (a), shows a raw SIM image with background fluorescence blurring. This results in large errors in the frequency content of the raw SIM image in a small neighborhood around the zero or DC frequency. A high pass filter, such as a Gaussian high pass filter or a Butterworth high pass filter, can be used to preprocess the raw SIM image. The second image in Figure 6, labeled as (b), shows the raw SIM image after background removal. The Fourier spectra of both images (a) and (b) are the same as shown. For a detailed discussion of high pass filters, the reader is referred to Chapter 4 of Gonzales and Woods 2008, "Digital Image Processing", 3rd edition. In Chapter 4, Gonzales and Woods present the illustration in Figure 4.52, which includes perspective plots, pictorial representations, and cross sections of an ideal Butterworth high-pass filter and a Gaussian high-pass filter.

[0257] Processing Step 2: Create a large Fourier image buffer

[0258] A buffer in memory may be reserved and initialized to receive the improved Fourier images. The size of the memory buffer depends on the degree of improvement, the upsample factor. The SIM image reconstruction technique may theoretically generate an improved resolution along each angle or orientation of three images with up to twice the resolution of the acquired image along that orientation. For two or three sets of three images, a buffer may be reserved to store the high resolution image with an upsample factor of two.

[0259] Processing step 3: Loop steps 3-a to 3-d for each angle θ

[0260] Processing steps 3-a to 3-d are performed in a loop. In one iteration, the processing steps are performed for three images of a subtile acquired for each angle. Details of the processing steps in the loop are now described.

[0261] Step 3-a (Fast Fourier Transform):

[0262] A discrete fast Fourier transform (FFT) is applied to the set of images acquired at the same angle or orientation "θ". As above, along each orientation, at least three images with different phases are acquired. Each image is transformed from the spatial domain to the frequency domain. For N×N images, the high-order complexity of computing a 2D discrete Fourier transform (DFT) is O(N 4 ), which can be reduced to O(N 3 ) for an N×N image, where N is a power of 2. Further speedup can be achieved when N is a power of 2. The Fast Fourier Transform (FFT) algorithm has a speed of O(N 2 log 2 Complexities of high order (N) can be achieved. An example of utilizing common expressions in rows and columns is by using "butterfly computation". For details, see Heckbert 1995, "Fourier Transforms and the Fast Fourier Transform (FFT) Algorithm", available at http: / / www.cs.cmu.edu / afs / andrew / scs / cs / 15-463 / 2001 / pub / www / notes / fourier / fourier.pdf. Thus, choosing a subtile size such as 512x512 pixels trades off frequency resolution for increased computation time. Subtiles with sides of 128, 256, 512, 1024, 2048, 4096, or ranging from 128 to 4096 pixels may be used.

[0263] Image data in the spatial domain is represented as real numbers. When performing a Fourier transform, the resulting data in the frequency domain is represented as complex numbers. A complex number z can be expressed as z=x+iy, where x and y are its real and imaginary parts. The conjugate of a complex number z is z * =x-iy.

[0264] Two exemplary corner-shifted Fourier domain matrices are shown in FIG. 19, which shows matrices with even (1910) and odd (1920) values ​​of N, illustrating the symmetry of Fourier space. These images in the frequency domain are represented by N×N matrices. The top row and left column of both matrices 1910 and 1920 are referred to as the DC row and DC column, corresponding to the zero or DC frequency component. In contrast to the values ​​of these corner-shifted matrices, plots in the frequency domain typically show a zero or DC frequency at the center of the circle.

[0265] The matrix in Figure 19 is a Fourier or frequency domain Hermite The DC rows and DC columns in matrices 1910 and 1920 have left-right and top-bottom conjugate symmetry. Matrix 1910, which has an even number of rows and columns, has a middle column that also has top-bottom conjugate symmetry. Apart from the top and left columns (1910, 1920), in the even matrices, the remainder of each matrix has Or block has 180 degree rotational conjugate symmetry. These parts of the two 1910 and 1920 are shown in bold. These are sometimes called the first block and the second block. .

[0266] Dividing the full matrix in the frequency domain into two halves after setting aside the rows and columns produces a right half that is the conjugate of the left half after a 180 degree rotation. Figure 19 shows half matrices (1911, 1912) and (1921, 1922) for even and odd dimension matrices (1910, 1920). Values ​​for the right half can be searched from the left half data. Given this search, the left half matrix is ​​the so-called non-redundant data, and the right half matrix is ​​the redundant data. The number of columns in the left half (slightly more than half) is ((N / 2)+1), and the right half columns is (N-(N / 2)-1). The disclosed technique keeps track of symmetry to reduce the computing load.

[0267] When reconstructing a SIM image, we operate on the non-redundant left half of the full image matrix in Fourier space. This increases efficiency since the number of calculations required is greatly reduced. In the SIM reconstruction steps below, further details will be presented on how symmetries in Fourier space are exploited.

[0268] Step 3-b (band separation):

[0269] Since the source locations are randomly distributed with respect to the illumination fringe peaks, shifted and non-shifted frequency components are present in each captured image. This step is performed by using the band separation matrix M -1is used to separate the shifted and non-shifted frequency components in the frequency domain representation of the acquired image. The band separation matrix is ​​also called the unmixing matrix (Wicker et.al. 2013) or reconstruction matrix (Lal et al. 2015). Gustafsson et.al. 2000 in their paper entitled "Doubling the lateral resolution of wide-field fluorescence microscopy using structured illumination" illustrates the band separation process in Figure 5 of the paper. This figure shows the separated frequency components from the observed image. The band separation can be understood by the following equation (31), taken from Lal et al. (2015). This equation is based on three components:

number

[0270]

number

[0271] The first component found in equation (31) is the so-called DC component.

number

number

[0272] The band separation step utilizes the disclosed non-redundant SIM image reconstruction technique as shown in Figures 20-22. The full matrix of each subtile's image in Fourier space is made to have 512 rows and 512 columns as shown in Figure 20. As explained above, the matrix positions in the three frequency domain matrices corresponding to the three images acquired at three phases for one angle are separated into a non-redundant half and a redundant half. Each non-redundant half matrix has 512 rows and 257 columns dominated by non-redundant coefficients including two columns and one column of DC components and with complementary conjugate coefficients to the redundant half matrix. Each non-redundant left half matrix is ​​reshaped into a vector and the three vectors of the three half matrices are combined into a matrix with 3 rows and 131,584 columns as shown in Figure 21. The band separation matrix M is then -1 The inverse of (x,y) is multiplied by the combined matrix to obtain a resulting component separation matrix of the same size, but also with 3 rows and 131,584 columns. The component separation matrix corresponds to the left side of equation (31). The three rows of the resulting matrix are then reshaped to obtain the non-redundant left half of the matrix corresponding to the three frequency components.

number

[0273] 18, after process step 3-b, the full matrix for the so-called DC frequency component can be represented as a square matrix combining a non-redundant half matrix 1831 (left half) and a redundant half matrix 1832 (right half), or it can be a rectangular matrix. Because the DC frequency matrix does not undergo the same shifts as the high frequency matrices, the DC half can be assembled sooner or later without computational penalty. In the case of the two high frequency components, symmetries are treated differently to reduce the computational load.

[0274] Following process step 3-b, the full matrix of high frequency components is organized such that portions from two bands are combined, such as non-redundant half matrix (1) 1843 from the first high frequency image and redundant half matrix (2') 1844 from the second high frequency image. Equivalently, non-redundant half matrix (2) 1845 from the second high frequency component and redundant half matrix (1') 1846 from the first high frequency component can be combined.

[0275] All three matrices are shown in FIG. 18 for the DC and high frequency components, and the operations of steps 3b, 3c, and 3d are processed using symmetry, allowing a band of frequency components to be processed by using half the data.

[0276] A similar process can be applied to five images with 1D structured illumination with phase steps along the angle, separating into frequency components, or separating more images along an angle. Functions other than sine or cosine varying illumination can be used. The disclosed technique can also be applied to sets of images generated with 2D structured illumination.

[0277] Step 3-c (Applying the Modulation Transfer Function):

[0278] Two alternative applications for applying the Modulation Transfer Function (MTF) and / or Optical Transfer Function (OTF) are described: a two-step application of an independent MTF followed by Wiener filtering to reduce noise, and a one-step application of an MTF built on a Wiener filter, both of which can take advantage of the disclosed non-redundant computation techniques.

[0279] In this two-step embodiment, the magnitude or absolute value of the optical transfer function (OTF), (MTF), is calculated by dividing the three components obtained after step 3-b.

number

[0280] The modulation transfer function of a lens is a measure of its ability to transfer contrast from an object to an image at a particular resolution. In other words, MTF is a way of incorporating resolution and contrast into a single specification. Resolution is the ability of a photography system to distinguish details of an object. It can be expressed in terms of line pairs per millimeter (lp / mm), where a line pair is an arrangement of one black line and one white line. This measurement of line pairs per millimeter is also known as frequency. The inverse of frequency results in the millimeter spacing between two resolved lines. A series of equally spaced bar targets, alternating white and black bars (also known as the 1951 USAF target or Ronchi ruling), is often used to test system performance. As the line spacing decreases (i.e., as the frequency increases for the target), the lens has an increasingly difficult time efficiently transferring this decrease in contrast, and as a result, the MTF decreases.

[0281] For further details on the modulation transfer function, the reader is referred to the article entitled "Introduction-to Modulation-Transfer-Function" available at https: / / www.edmundoptics.com / resources / application-notes / optics / introduction-to-modulation-transfer-function / . Figure 5 in the above referenced article shows that as the spatial frequency of the lines increases, the contrast of the image decreases. For the image to appear as specified, blacks must be truly black and whites must be truly white, with a minimum amount of grayscale in between. Figure 6 in this article plots the MTF of an aberration-free image with a rectangular lens. The MTF decreases as the spatial resolution increases. It is important to note that no real system is completely free of aberrations.

[0282] This two-step application of MTF and Wiener filtering can replace the single-step approach presented in step 15 of SIM-RA algorithm 1 in Lal et al. (2015). Again, the first part of the algorithm, step 3-c, performs Wiener filtering in two steps. Alternatively, this step can be performed after step 3-d without affecting the quality of the SIM reconstructed image.

[0283] This two-step approach (i.e., applying the MTF separately from the Wiener filtering) is recommended in the literature for SIM image reconstruction. For example, MGL Gustafsson recommends this two-step approach in a paper entitled "Surpassing the lateral resolution limit by factor of two using structured illumination microscopy," published in the Journal of Microscopy, Vol. 198, Pt. 2, May 2000, pp-82-87. For SIM image reconstruction, the paper suggests multiplying each of the three frequency components by the OTF of the microscope. Following this, the components are added if they overlap. Finally, the sum of the components is divided by the sum of the square of the OTF and a small constant. The paper further states that this process is equivalent to one-step Wiener filtering.

[0284] Wiener filtering of the three components in Eqs. 32, 33, and 34 as shown below (taken from Lal et al. 2015):

number

[0285] The three frequency components obtained on the left side of Equations 32, 33, and 34

number

number

[0286] Noisy estimates

number

number

number

[0287] Wiener filtering or Gustafsson's two-step method allows for a precise approximation of the uncorrupted component, referred to as the ungraded estimate on the left side of Equations 32, 33, and 34 of Lal et al. (2015), as shown above.

[0288] Another important observation from Lal et al. (2015) in eqs. 32, 33, and 34 is that the molecules in eqs. 32, 33, and 34 are OTFs.

number

number

number

number

number

number

[0289] In the non-redundant SIM subtile reconstruction algorithm, after the band separation step (step 3-b), we multiply the non-redundant matrices 1831, 1833, and 1835. This is an element-wise scalar multiplication, which does not change the dimensions of the resulting non-redundant matrix. After this multiplication, we obtain the three non-redundant left half matrices 1831, 1833, and 1835 shown in FIG. 18. After applying OTF, the symmetry of all three matrices is maintained, as described in the band separation step.

[0290] Averaging the first and middle columns

[0291] Here, a computational function is presented that can improve the quality of the reconstructed image using a non-redundant SIM image reconstruction algorithm. In Fig. 23B, the Fourier space matrices of two high frequency images are shown. In the non-redundant algorithm, the left halves of the two matrices labeled as 1843 (left half block 1) and 1845 (left half block 2) are used. These left halves are sometimes called the first block of complex coefficients. These two matrices are shown in the top row of FIG. 23B along with the separated right half matrices 1846 (right block 1) and 1844 (right half block 2'). These right halves are sometimes called the second block of complex coefficients. The labeling of the left and right halves of the two matrices corresponds to that in Figure 18 (fourth row from the top). The non-redundant algorithm reduces the left half of the two high-frequency matrices during image reconstruction. Or the first Due to Fourier space symmetry, the right half is used. Or the secondThe half matrices can be reconstructed using lookup of the corresponding left half matrices. The second row of matrices in Fig. 23B shows stitching of the left half matrix of one high frequency image to the right half matrix of the second high frequency image during image reconstruction. These stitched matrices are also shown in the third and fourth rows of Fig. 18.

[0292] The left half matrices of the two high frequency images in this embodiment have two more columns than their corresponding right half matrices. These two columns are the first and middle columns shown in Fig. 23B with a diagonal line pattern. The first and middle columns of 1843 and 1845 represent DC components that match the perfect instrument and measurement. The two high frequency matrices may have different values ​​in their first and middle columns because the instrument used to capture the images (at different phases), with imperfections and noise, may also be introduced in this process. The reconstruction may be improved by forcing the first and middle columns of 1843 and 1845 to have the same values.

[0293] Averaging the first and middle columns of the non-redundant components of the matrices of the two high frequency components is one way to make these columns match 1843 and 1845 in order to improve the quality of the reconstructed image by introducing symmetry into the values ​​of these columns. FIG. 23C shows the averaging of the first and middle columns of the left half matrices of the two high frequency images in Fourier space. The average values ​​of the first and middle columns of the two matrices make the resulting matrix satisfy Fourier space symmetry. The first and last columns after averaging are shown in FIG. 23C with a cross-hatch pattern.

[0294] Other techniques may also be used to match these column values ​​to 1843 and 1845. For example, the values ​​of the first and middle columns of either the first or second high frequency image may be used and these values ​​may be used in both matrices 1843 and 1845. Or a weighted combination of the values ​​of the first and middle columns of the two high frequency images may be used. For example, the values ​​are combined by assigning a weight of 2 / 3 to the values ​​of one high frequency image and a weight of 1 / 3 to the values ​​of the other high frequency image. It will be appreciated that different relative weights may be assigned to combine the values ​​of corresponding columns.

[0295] Step 3-d (sub-pixel shift):

[0296] In this step, the high frequency components

number

[0297]

number

[0298] In prior art SIM image reconstruction algorithms, such as the SIM-RA algorithm 1 presented in Lal et al. (2015), two Fourier transform operations are performed for each high frequency image to shift the high frequency components to their correct positions, as shown in Equations 35 and 36. The shifting properties of the Fourier transform allow the signal to be shifted. The shifting can be done by taking the information from the frequency domain to the spatial domain and then applying a multiplication (also called applying a transformation vector). A Fourier transform is then applied to this shifted signal in the spatial domain to bring it back to the frequency domain. More information on the shifting properties of the Fourier transform can be found at http: / / www.thefouriertransform.com / transform / properties.php. The shifting of the high frequency components is actually performed in real space by performing an inverse Fourier transform on the untilted image, so as not to constrain the transformation vector for all pixels in the discrete Fourier transform. Thus, sub-pixel shifting can be achieved in real space. The shifted high frequency components are then transformed to the frequency domain by applying a Fourier transform.

[0299] By using symmetries in Fourier space, the efficiency of SIM image reconstruction is improved and the required computational resources are reduced. The so-called DC frequency component does not need to be sub-pixel shifted to be at the reference position as shown in Figure 18. The high frequency component matrix is ​​shifted relative to the DC frequency component matrix.

[0300] In the disclosed non-redundant algorithm, after combining the non-redundant halves of the high frequency components' complementary phase shift matrices shown in FIG. 18, only one matrix of the high frequency components is shifted. The combined halves of the high frequency components are shown as left half 1833 and right half 1834, or labeled 1|2' matrix shown in the middle column of the third row of FIG. 18. This matrix is ​​shifted from the Fourier domain to the spatial domain and back again, instead of the two matrices of the prior art SIM image reconstruction algorithm. The shifted matrix 1|2' is shown in the middle column of the fourth row in FIG. 18, with two arrows pointing diagonally upwards towards the top left corner. This matrix is ​​composed of left half matrix 1843 and right half matrix 1844. The second shifted high frequency matrix, composed of left half matrices 1845 and 1846, or labeled 2|1', is obtained using Fourier space symmetry and the conjugation of the 1|2' matrix. This is shifted the same amount as in 1|2', but in the opposite direction as indicated by the two diagonal arrows in the lower right corner.

[0301] Process steps 3a-3d are repeated in a loop over the illumination peak angles. In this example, two angles or orientations are used, so the above steps are performed for a second set of three images acquired at a second orientation of the structured illumination pattern.

[0302] Processing step 4: Reconstruct the sum and build the Wiener filter

[0303] The Wiener filter may be constructed once and then repeatedly applied to filter the image even as the parameters change. The supporting material for the paper by Karras et.al. 2019, entitled "Successful optimization of reconstruction parameters in structured illumination microscopy-A practical guide," published in Optics Communications, shows the adjustment of the Wiener filter parameter (w), which is the constant part of the denominator in Equations 32, 33, and 34. Figure S1 in this supporting material for Karras et.al. 2019 illustrates the effect of different values ​​of the Wiener filter parameter (w) on the searched image spectrum. Low values ​​of the Wiener filter parameter may result in noise in the searched spectrum. Higher values ​​may result in loss of image data. The parameter values ​​of the Wiener filter may be empirically selected as discussed in the supporting materials of Lal et al. (2015) and Karras et al. (2019). Other types of Fourier space low-pass filters may alternatively be used to reduce noise. For example, a Gaussian low-pass filter, a Butterworth low-pass filter, or a Hamming low-pass filter. Details of these filters are available at http: / / fourier.eng.hmc.edu / e101 / lectures / Fourier_Analysis / node10.html.

[0304] Only the part of the matrix containing high frequency components overlaps with the so-called DC frequency matrix. The extended matrix, in which coefficients from multiple Fourier domain frequency matrices are combined, is combined wider and higher from the individual matrices before combining by an upsample factor, which reflects an improvement in image resolution. The upsample factor can be 1.0 to 2.0 times the width and height of the individual matrices. If the upsample factor is 1.0, there is no improvement in image resolution. There are no high frequency components and images acquired at all phases along the orientation overlap with each other. If the upsample factor is 2.0, the shifted matrices of high frequency components are separated from each other such that their edges (or boundaries) at the center of the unshifted matrix touch. This means that the width and height of the high resolution reconstructed image increases by twice the individual image matrices. This combined matrix is ​​shown as large matrix 1850 in FIG. 18. Matrix 1850 is shown twice as wider and higher than the individual matrices labeled 1|2' and 2|1' to show that the reconstructed image is improved. When the upsampling factor for image resolution improvement is greater than 1.0 and less than 2.0, the shifted matrices 1|2' and 2|1' are located close to each other at the center of the large matrix 1850 and overlap each other.

[0305] In Figure 18, a matrix 1850 is shown constructed from the matrices of three images at a first angle. The non-redundant half of the DC components for the first angle is shown as 1841 in Figure 18. Sub-pixel shift coefficients are then placed for one high frequency image on the top left portion of matrix 1850 and for the other high frequency image on the bottom right portion. The redundant conjugate components required to complete 1850 can be searched for in the complementary non-redundant components, keeping track of the matrix location of the complementary components (coefficients) and applying a changed sign to any imaginary parts of the copy coefficients.

[0306] Adding a second orthogonal angle and three more images (not shown) would fill the square. For the second angle in the lower left part of matrix 1850 and for the second high frequency component in the upper right, the entire matrix is ​​laid out for one high frequency component.

[0307] Combining nine images is shown in Lal et al. (2015). The coefficients at overlapping matrix positions are combined in matrix 1850. This is a six-image variation, for two angles instead of three.

[0308] Visualization of the composite matrix is ​​clearest when the coefficients are shown as center-shifted, with the median of the frequency range at the center of the matrix, although the calculations may in fact be performed using a corner-shifted matrix, with the lowest frequency components at, for example, the top-left or bottom-right corner. The reconstruction of the improved resolution image matrix 1850 is shown in FIG. 18 by placing the low frequency components at the center and the high frequency components away from the center. This placement of frequency components is referred to as a center-shifted spectrum. Application of a Fast Fourier Transform (FFT) to image data in the spatial domain results in a corner-shifted spectrum in which the amplitudes of the low frequency components end at the corners of the two-dimensional spectrum in Fourier space (or the frequency domain). The high frequency components are located near the center of the two-dimensional spectrum. For ease of understanding, it is useful to visualize the frequency components in the frequency domain as a center-shifted spectrum, with the low frequency components located near the center and the high frequency components located away from the center. The symmetry of the quadrants in the corner-shifted spectrum allows for swapping the quadrants on the diagonal to move the low frequency components closer to the center. Therefore, the corner-shifted spectrum can be transformed into the center-shifted spectrum in a simple manner, and vice versa. The conjugation symmetry and redundant regions are preserved in both the corner-shifted and center-shifted spectra. The center-shifted spectrum in FIG. 18 is presented for the ease of understanding of the reader.

[0309] A Wiener filter can be constructed for the combined image using either the one-step (Lal et al. 2015) technique or the two-step (Gustafsson 2000) technique presented above for the band-separated images in step 3-c. For example, in one embodiment, a Wiener filter can be constructed using the MTF (Modulation Transfer Function), which is the magnitude or absolute value of the OTF (Optical Transfer Function). In this embodiment, a reconstruction process is performed to computationally simulate the overlapping frequency information in the Wiener filter. We apply the Wiener filter weighting for the band-separated images to the OTF of the combined image to construct a Wiener filter for the combined image. The Wiener filter is applied to the combined image in the next step.

[0310] Processing step 5: Application of Wiener and apodization filters

[0311] In this step, the Wiener filter or an alternative filter constructed in the previous step is applied to the combined image in Fourier space. In step 3-c, the Wiener filtering is separated into two steps by applying independent MTFs. See equations 32, 33, and 34 in step 3-c. As a two-step filtering process, this step includes applying the remaining part of the Wiener filter (presented in equations 32, 33, and 34) to the combined image in Fourier space. In an alternative embodiment, the Wiener filter may be applied to the Fourier space representation of the three frequency components per angle before combining them to form the combined image. If applied early, the Wiener filter may be applied to only the non-redundant half of the matrix for the band-separated frequency components.

[0312] The application of the Wiener filter improves the signal but may also increase noise in the image. A properly designed apodization filter can avoid hard edges introduced in the image due to high frequency components. The apodization filter is a low pass filter that removes frequency components outside a predefined radius in Fourier space. The apodization filter reduces the error introduced by the Wiener filter. Other low pass filters such as a real space Hamming low pass filter or a Gaussian low pass filter can also be applied to reduce the error introduced by the Wiener filter. Note that the apodization filter is optional and is not used by some SIM reconstruction algorithms such as the SIM-RA algorithm 1 of Lal et al. (2015). The apodization filter can alternatively be applied early in the pipeline to only the non-redundant half of the matrix for the three frequency components.

[0313] Processing step 6: Perform the inverse FFT

[0314] In this step, an inverse Fourier transform is applied to the matrix 1850 for the entire image in the frequency domain. Application of an inverse fast Fourier transform (IFFT) converts the matrix elements in the Fourier domain to an image in the spatial domain (real space). The resulting matrix 1860 is the high-resolution SIM reconstructed image. If a subtile is used, it is a portion of a larger tile.

[0315] SIM image reconstruction of the entire FOV image

[0316] Subtiles of high-resolution SIM images can be combined to reconstruct a high-resolution image of the full FOV. Pseudocode of one embodiment of the disclosed technique is provided below to reconstruct a high-resolution full-field-of-view image from multiple subtile images. The algorithm provided below is referred to as the "full-field reconstruction" algorithm (also referred to as Algorithm 4). It extends Algorithm 3 for subtile reconstruction. It takes as input the distortion coefficients of the acquired images and the estimated parameters. The estimation of the distortion coefficients with respect to the angle and spacing parameters (Input 2), as well as the phase bias look-up table (Input 3) are provided above. The algorithm divides the full FOV image into subtiles of equal sized regions. The SIM image is reconstructed for the subtiles as provided in Algorithm 3 above. Finally, the high-resolution SIM images of the subtiles are combined to reconstruct a high-resolution SIM image of the full FOV data (Algorithm step 4b).

[0317] Algorithm 4: Full-field reconstruction

[0318] input: 1. Acquire images at, for example, two angles and three phases per angle 2. Angle / Spacing Distortion Coefficients (see Final Distortion Calibration in Quadratic Fit and Lookup Table Algorithms) 3. Distortion Lookup Table for Phase (see Final Distortion Calibration in Quadratic Fit and Lookup Table Algorithm) 4. Wiener filter and apodization parameters Output: Reconstructed output algorithm 1. Build the following image / filter (sized to match the subregion): a.OTF b. Wiener C. Apodization 2. Divide the image into equal sized sub-regions, e.g. 512x512 a. If the image is not evenly divided by the subregion size, then at the right edge or bottom, shift the subregion left (or up) so that it falls on the image. b. Set the book to keep border sub-regions to ensure that only non-overlapping portions are copied to the right or bottom edge sub-regions 3. Estimate the parameters in the center of the image, for example using a 512x512 window (see Subtile Angle and Spacing Mapping Algorithm and Subtile Phase Mapping Algorithm) a. Estimate the illumination peak angle and interval for each angle b. Estimate the modulation for each angle c. Estimate the phase for each image 4. For each sub-area, a. Reconstruct the high-resolution sub-regions using a subtile reconstruction algorithm i. Using the lattice and angles found in step 3 and the given distortion coefficients, estimate the local lattice and angles. ii. Using the phase found in step 3 and the given distortion lookup table, estimate the appropriate phase iii. Consider previously constructed OTF, Wiener and apodization of sub-regions b. Copy the output from (a) to the larger image i. For border subregions, copy only the missing data

[0319] In the context of the previous discussion, this pseudocode should be self-explanatory.

[0320] Potential improvements to SIM image reconstruction algorithms.

[0321] The inventors present potential improvements to the SIM image reconstruction algorithm (SIR) disclosed in U.S. Provisional Patent Application No. 62 / 924,130, entitled "SYSTEMS AND METHODS FOR STRUCTURED ILLUMINATION MICROSCOPY" (Attorney Docket No. ILLM1012-1), and U.S. Provisional Patent Application No. 62 / 924,138, entitled "INCREASED CALCULATION EFFICIENCY FOR STRUCTURED ILLUMINATION MICROSCOPY" (Attorney Docket No. ILLM1022-1). Both provisional patent applications were filed on October 21, 2019.

[0322] The first potential improvement is related to the phase bias look-up table. The baseline SIR algorithm uses a static phase bias look-up table. The assumption in the baseline model is that the phase bias and distortion can be learned once and stored in the look-up table. The values ​​in the look-up table are assumed to be stable and applicable for multiple ordering runs. The baseline SIR algorithm assumes that the phase bias look-up table (LUT) remains stable in the spatial domain. This means that the same look-up table can be applied across tiles of a flow cell. Our second assumption in the baseline algorithm is that the LUT remains stable across multiple ordering runs. Experiments have shown that these assumptions may not be true for all flow cells and ordering runs. Tiles in a flow cell may have different tilts or other distortions. The values ​​in the look-up table may be affected by various environmental factors that change across multiple ordering runs.

[0323] Thus, the "All-In" SIM algorithm involves learning the phase bias lookup table for every sequencing run per tile when a new flow cell is loaded into the sequencer and a sequencing run is started. This can increase the processing time required for the first cycle of a sequencing run. However, as shown in the following paragraphs, substantial improvements in error rate and percentage of clusters passing the filter (or %PF) metric are observed.

[0324] A second potential improvement relates to geometric distortion, which causes parallel lines to become bowed or otherwise distorted. Geometric distortion can cause distortions in the imaging or light path and the flow cell. In some cases, all or some tiles in the flow cell may have a slope and may not be flat. This can cause geometric distortion. Characterizing all tiles in the flow cell is preferred because there may be variation in the flow cell from tile to tile. In early versions of the correction, it was assumed that the geometric distortion could be learned once and used across multiple ordering runs. Experiments have shown that this assumption may not be true for all ordering runs. Thus, the "All-In" algorithm includes learning the geometric distortion for all tiles in the flow cell for the first cycle of the ordering run.

[0325] A third potential improvement relates to stitching subtile images to form a full field of view (FOV) image. In the baseline SIM algorithm, when an inverse fast Fourier transform (IFFT) is applied to the subtiles in the frequency domain, the edges of the subtiles are apparent in the spatial domain. Thus, stitched subtiles in the full field of view exhibit stitch lines at the edges of the subtiles. The stitch line problem causes dark artifacts that appear along the boundaries of the subtiles. All SIM algorithms solve the stitch line problem by using overlap between subtiles. For example, subtiles may have an overlap of 5 to 15 pixels. In one implementation, subtiles have an overlap of 8 pixels along their boundaries. When reconstructing a full field of view (FOV) image, an "All-In" algorithm can discard the outer 8 pixels of the subtiles and use the interior region to reconstruct the full FOV image in the spatial domain. In this implementation, the size of the subtiles may be larger than 512 x 512 pixels by adding border pixels (e.g., 8 pixels) on each side. In an alternative embodiment, the boundary pixels of the subtiles may be averaged instead of discarding the boundary pixels.

[0326] A fourth potential improvement is related to optimizing the apodization and Weiner filters. The filter parameters depend on the upsampling. The baseline SIM algorithm uses 1.2x upsampling, which can shave off partial pixels. This can cause image shifting and alignment issues when the subtile images are stitched together in the spatial domain. The shaving off pixels is due to an upsampling factor (i.e., 1.2) that produces non-integer pixel numbers. The All-InSIM algorithm uses 1.5x upsampling to overcome the partial pixel shaving issue by using an upsampling factor that produces integer pixel numbers. Thus, cleaner subtile images for stitching together are provided. In other implementations, other upsampling factor values ​​can be used, such as 1.25x upsampling, which can also produce integer pixel numbers.

[0327] In some embodiments, the disclosed techniques can use dose ramping to compensate for damage to nucleotides caused by exposure cycles in a sequencing run. In dose ramping, the laser power of the blue and green channel lasers is increased.

[0328] Performance comparison results

[0329] The performance of the baseline SIM algorithm was compared with the All-InSIM algorithm. The two algorithms are compared using the quality of the ordering results as indicated by error rate and cluster passing filter (%PF). The error rate metric measures the percentage of base calls made incorrectly when compared to a known reference genome in the ordering process. The cluster passing filter or %PF metric is an indication of the signal quality from each cluster of molecules on the flow cell. Lower error rate and higher %PF values ​​are desirable to indicate good ordering output with high quality base calls. Among the 100+ tiles on the flow cell, the best tile result was selected for each ordering run to compare the algorithms. The tiles in the flow cell may have variability in terms of flatness, tilt, surface polishing, and other environmental factors. The reason for selecting the best tile for the comparison among the 100+ tiles is to reduce the variability caused by these other external factors through the statistical population.

[0330] FIG. 25A presents the error rates of the baseline and All-In algorithms for three ordering runs, labeled as A, B, and C. For each ordering run, the error rates of the best tile are presented for the two algorithms. The top bar shows the error rate for the best tile in the All-In algorithm, and the bottom bar shows the error rate for the best tile in the baseline algorithm for each ordering run. The table at the bottom of the graph tabulates the quantitative improvement in the error rate of the best tile when the All-In algorithm is used. For example, the best tile error rate for ordering run A is 1.85% when the baseline algorithm is used. The best tile error rate for ordering run A is 1.18% when the All-In algorithm is used, indicating a higher quality ordering process with a lower error rate. Similar improvements in lowering the error rate are observed for ordering runs B and C when using the All-In algorithm.

[0331] FIG. 25B presents the results of the cluster pass filter (or %PF) for the best tile in the three ordering runs for those two algorithms. The bottom table presents the %PF values ​​for the best tile in each ordering run for the two algorithms. The All-In algorithm results in higher %PF values ​​for all ordering runs. Specifically, the %PF value for the best tile in ordering run A is 82.7, indicating that 82.7% of the clusters on the flow cell tile are considered good quality clusters that generate base call information. The %PF value for the best tile in the ordering run is only 70.1. A higher %PF value increases the ordering throughput because more clusters pass the filter. A patterned flow cell may have a theoretical maximum limit for the number of clusters equal to the number of nanowells in which molecular clusters can be formed. The results in FIG. 25A and FIG. 25B show that the All-In algorithm results in a lower error rate and higher throughput (by the higher %PF value) compared to the baseline algorithm.

[0332] FIG. 26A, FIG. 26B, and FIG. 26C present the best tile error rate for a period in ordering runs A, B, and C, respectively. Separate time series of error rates in the ordering runs are presented for the baseline and the All-In algorithm. The results in FIG. 26A-FIG. 26C show that the All-In algorithm reduces the error rate in the ordering runs. The error rate improvement using the All-In algorithm for the best tile can vary from period to period. In FIG. 26A, for the initial 25 periods, the error rates of the baseline and All-In algorithms are almost the same, but as the ordering run progresses, the error rate of the All-In algorithm decreases compared to that of the baseline algorithm. FIG. 26B shows that the error rate of the All-In algorithm is slightly better in the initial periods. In later periods of the ordering run, the All-In algorithm performs better than the baseline algorithm. FIG. 26C shows the best tile error rate for a period in ordering run C for the two algorithms. The performance of the All-In algorithm is significantly better than the baseline algorithm for most of the periods. The graph also shows the specific cycles where the All-In algorithm has a higher error rate. These results can help engineers or researchers to focus on the quality issues that cause the increased error rate for a specific cycle.

[0333] FIG. 26D shows the improvement in error rate achieved by certain updates to the All-In algorithm compared to the baseline algorithm. For example, the left plot results in an improvement in the error rate of ordering run A by including phase learning and stitch fixation. Phase learning refers to an estimation of phase bias that can capture the relationship between phase quantile values ​​for non-central subtiles relative to central subtiles. A separate phase bias lookup table is generated and stored for each tile of the flow cell. The error rate of the baseline algorithm for ordering run A is 1.85, as shown in the first column of the table in FIG. 24A. The improvement in error rate is 0.65 or 32% when phase learning is included in the baseline algorithm, and the improvement in error rate in the baseline algorithm is 0.19 or 10% when stitch fixation is included in the baseline model. The graphs can also be interpreted as phase learning reducing the error rate by 0.65 and stitch fixation reducing the error rate by 0.19, respectively, in the baseline algorithm. The remaining error rate after the inclusion of the two techniques is 1.18.

[0334] The right graph plot in FIG. 26D presents the improvement in error rate provided by the phase learning and stitch fixation techniques for ordering B. The inclusion of phase learning in the baseline algorithm reduces the error rate by 0.227 or 33%. The inclusion of stitch fixation in the baseline algorithm reduces the error rate by 0.039 or 6%. The remaining error rate after the inclusion of both techniques is 0.414. Here we present a discussion of the impact of the disclosed SIM parameter estimation and image reconstruction techniques when they are applied to process images from an ordering run.

[0335] Impact of the Disclosed Technology

[0336] A simplified example is presented to show how the disclosed technique can reduce the computational resource requirements for structured illumination image improvement over an entire flow cell over multiple cycles and hours. Machine capabilities vary, so approximate numbers are chosen to simplify the exemplary calculations. Keep in mind that the disclosed techniques can be applied independently, alone, or in combination. In some embodiments, parameters can be estimated more frequently to compensate for system instability due to environmental factors. For example, as described above, the "All-In" SIM algorithm estimates the phase bias lookup table on the first cycle of every ordering run. This can increase the duration for the first cycle by up to 30 percent. However, significant performance improvements in error rate and %PF metrics can be achieved.

[0337] The sequencing run involves the analysis of 150 cycles at two illumination frequencies at a rate of 6 minutes per cycle. A basic image tile imaging at 150 cycles takes 15 hours. Assume two angles + / - 45 degrees are used for structured illumination with 3 images per angle in each cycle. The number of images per tile over 150 cycles is 1800 images. Assume there are 70 subtiles in a 7x10 grid. The processing requirement is 126,000 subtiles in 900 minutes.

[0338] In a first embodiment, we assume that the system is calibrated before the run and that lookup tables generated for spacing, angle, and phase difference, in addition to the quadratic fit, can be used throughout the run. In this embodiment, the near-center subtile corresponds to an estimated 1800 fringe peaks, which is the number of times there is a change in flow cell position or illumination laser source or projected structured illumination phase.

[0339] In this first example, the system avoids 69 out of 70 fringe peak estimates. For 124,200 image subtiles, the other subtiles use pre-calculated relationships with the central subtiles to determine the reconstruction parameters. Estimating the reconstruction parameters is computationally more expensive than the actual reconstruction. Such computational requirements are reduced by more than half. Further reductions can be achieved using the reliance on outlined symmetries.

[0340] The application of symmetry allows the conjugate pair of values ​​to be searched for instead of being calculated by FFT and other methods. Instead of the governing 180 degree rotation and sign change arithmetic symmetry, special care needs to be taken in the subtile section with reflection symmetry. When using an even number of rows and columns in the corner-shifted Fourier transform space, the top row and the left and center+1 columns are treated separately because they are reflected rather than rotationally symmetric. See FIG. 19. When using an odd number of rows and columns in the corner-shifted Fourier transform space, the top row and the left column are treated separately because they are reflected rather than rotationally symmetric. See FIG. 19 and FIG. 23A. Other symmetries can be applied to the center-shifted Fourier transform space.

[0341] The center column of Figure 18 shows how half of the computations in the frequency domain can be avoided at various steps by focusing on non-redundant data. This reduces computations by 40 percent and image coefficient storage by 50 percent during these steps until the entire image is reconstructed at 1850, 1860. When values ​​are needed at 1850, 1860 for a conjugate pair, the conjugate pair does not need to be included in the computations during steps 3b-3d, so that a useful search can be applied.

[0342] Thus, one disclosed technique can save 50 percent of the required computational resources, while the other can save 40 percent. Cumulatively, the computational cost of SIM reconstruction can be reduced by more than two-thirds.

[0343] In a second embodiment, it is assumed that the system is calibrated at the start of the run and again at the midpoint to update the secondary fit and lookup tables. This calibration is assumed to take 8 minutes since it involves fringe peak estimation for all 70 subtiles, followed by fitting and lookup table generation. This estimate extends the run time to 15 hours to 15 hours and 16 minutes. If, instead of applying the disclosed technique, the estimation is performed once per image trio for each subtile and exposure frequency, for a total of 300 parameter estimates, the run time will be longer than triple.

[0344] In the second embodiment, the effect of applying symmetry would not change.

[0345] The combined effect of the disclosed techniques is expected to enable real-time processing of image tiles to be performed using a CPU without adding a GPU, FPGA, or CGRA to the system. Alternatively, simpler computing resources can be used even if a GPU, FPGA, or CGRA implementation is selected. This can facilitate preserving the form factor of existing machines while reducing the complexity of systems and increasing their real-time processing capabilities.

[0346] Specific Implementations

[0347] A subtile of the full field of view imaged by the sensor

[0348] The disclosed subtiling techniques can be described in terms of calibration and / or generation using a calibrated scanner.

[0349] In one calibration embodiment of the disclosed technology, a method is described for the calibration or characterization of a scanner that detects the fluorescence of millions of samples distributed across a flow cell, or more generally, an imaging plane, in images collected over multiple cycles. Structured illumination is used to improve the resolution between fluorescent samples that are positioned closer together than the Abbe diffraction limit for optical resolution. The method of calibrating a scanner that substantially uses the entire field of view projected by a lens onto an optical sensor in the scanner includes imaging an image tile under structured illumination at multiple angles and phase shifts of the structured illumination. Variations of the method may use 2D structured illumination and different phase shift step patterns. The method includes measuring the spacing between and angles of intensity peaks in at least nine subtiles of the image tile, calculating the optical distortion across the image tile, including the subtile near the center, and fitting a spacing surface and an angle surface to the measured spacing and angle, respectively. Of course, other configurations for dividing the image tile may be used, such as 3x3, 5x5, 5x7, 9x9, 8x11, 9x16 subtiles, etc. Images captured by larger sensors are divided into more subtiles, not limited to the enumerated subtile array. The fitted spacing and fitted angle surfaces represent the correction of distortion across the subtiles of the image tile. The resulting fit coefficients are stored, including fitting results relating the spacing of intensity peaks and the angles of intensity peaks across at least nine subtiles to the central subtile. The calibration further includes measuring the phase displacement of the structured illumination at multiple angles and phase displacements within the subfield, and storing a look-up table representing the difference between the phase displacement in the central subtile and the phase displacement in other subtiles of the image tile. This calibration prepares the scanner for generation to analyze the fluorescence of millions of samples distributed across a flow cell or imaging surface in images collected over multiple cycles.

[0350] In a production embodiment, the method produces improved images of the fluorescence of millions of samples distributed across a flow cell or imaging surface from images collected over multiple cycles. This includes processing imaged image tiles covering positions across the entire flow cell, where the image tiles are imaged over multiple cycles and at multiple angles and phase displacements of the structured illumination. The processing targets subtiles of each image tile. It includes accessing stored estimated fitting results or generating estimated fitting results that represent the spacing and angle distortion relationships between the near-center subtiles and other subtiles. The fitting results may be coefficients of a polynomial surface fit or values ​​in a lookup table. The method also includes accessing stored lookup tables or generating lookup tables that represent the phase displacement differences between the near-center subtiles and other subtiles. The fitting results and lookup tables are combined with measurements of the near-center tiles for each imaged image tile, the spacing between intensity peaks, the angles of the intensity peaks, and the phase displacements of the structured illumination. Reconstruction parameters are determined for each imaged image tile, subtile by subtile, and then applied to the SIM reconstruction. This method produces improved resolution images for subtiles at locations across the flow cell or imaging surface over multiple periods with better than Abbe diffraction limit resolution performance.

[0351] For example, for an image tile divided into 9 subtiles, the method includes deriving estimated reconstruction parameters for at least 8 additional subtiles of the image tile in addition to the near-center subtile. The estimated spacing and angular reconstruction parameters of the additional subtiles are derived by combining the accessed or estimated parameters of the near-center subtile with stored fitting results relating spacing parameters and angular parameters across the at least 8 additional subtiles to the near-center subtile. The estimated phase displacement reconstruction parameters for the additional subtiles are derived by combining the accessed or estimated parameters of the near-center subtile with a stored look-up table for phase displacement measurements across the at least 8 additional subtiles relative to the near-center subtile. As noted above, other configurations for dividing the image tile may be used without affecting the above parameter estimation process.

[0352] The improved resolution images can be used to sequence the sample over multiple periods or otherwise study the fluorescence on an image plane imaged over dozens, hundreds, or thousands of image tiles.

[0353] In a combined calibration and generation embodiment, the above operations are combined so that the scanner is calibrated and then used for generation. The disclosed technique includes a method implementation that improves the performance of a scanner to detect the fluorescence of millions of samples distributed across a flow cell in images collected over multiple cycles. Structured illumination is used to improve the resolution between fluorescent samples that are located together closer than the Abbe diffraction limit for optical resolution. Applying this technique, calibrating a scanner using a substantially full field of view projected by a lens onto an optical sensor in the scanner includes imaging an image tile under structured illumination at multiple angles and phase displacements of the structured illumination. It includes measuring the spacing between and angles of intensity peaks in at least nine subtiles of the image tile, calculating optical distortions across the image tile, including the central subtile, and fitting a spacing surface and an angle surface to the measured spacing and angle, respectively. Of course, the image tile can be divided into more subtiles as described above. The fitted spacing and angle surfaces represent the correction of the distortions across the subtiles of the image tile. The resulting coefficients of the fit are stored. The calibration further includes measuring phase displacement of the structured illumination at multiple angles and phase displacements within the subfield and storing a look-up table representing the difference between the phase displacement in the center-nearby subtile and the phase displacement in other subtiles of the image tile.

[0354] The method processes using imaged image tiles covering positions across the entire flow cell, the image tiles being imaged at multiple angles and phase displacements of the structured illumination over multiple periods. Doing so includes combining the stored fitting results and the stored lookup table with measurements for each imaged image tile of the spacing between intensity peaks, the angles of the intensity peaks, and the phase displacement of the structured illumination at the central tile to determine reconstruction parameters for each imaged image tile on a subtile-by-subtile basis. This further includes generating improved resolution images for the subtiles at positions across the flow cell with resolution performance that is better than the Abbe diffraction limit over the multiple periods and using the improved resolution images to sequence the samples over the multiple periods.

[0355] Each of the embodiments of the combined calibration and generation implementations presented above has a separate utility. For example, a first entity may deliver and calibrate a scanner for use by a second entity. The first entity may use a different calibration approach to determine and represent the spacing and angle distortion relationships, as well as the phase displacement differences between the center-nearby subtiles and other subtiles, and still use the described generation techniques. The described calibration techniques may be used, in which case the second entity may use the calibrated scanner in a different manner. Thus, the disclosed technology includes calibration and / or generation implementations.

[0356] Each of this and other embodiments of the disclosed technology may optionally include one or more of the following features and / or features described in connection with the additional disclosed methods. For purposes of brevity, combinations of features disclosed in this application are not individually recited and are not repeated for each base set of features. The reader will understand how the features identified in this section can be readily combined with the basic features identified as an implementation.

[0357] When the imaging plane is a flow cell, the sample may be distributed in millions of nanowells across the flow cell. At least some adjacent pairs of nanowells may be positioned closer together than the Abbe diffraction limit for optical resolution. Structured illumination allows for such close positioning. Alternatively, the sample may be randomly distributed across the flow cell.

[0358] The subtiles each contain at least 512x512 pixels of the optical sensor. Larger or smaller numbers may be used, including 256, 400, 1024, 2048 and 4096, or a range of 256 to 4096 pixels. Performance can be improved if the subtile dimension is a power of 2, with 512 and 1024 pixels being a practical range found to work well.

[0359] The subtiles may overlap by at least 2 pixels of the optical sensor, although a larger or smaller number may be used, for example for a 512 pixel wide window a maximum overlap of 256 pixels may be used, for a 1024 pixel wide window a maximum overlap of 512 may be used.

[0360] The results of fitting the interval and angle surfaces may be stored as coefficients of a quadratic, cubic, or other higher order surface, or in a look-up table calculated from a polynomial fit of the interval and angle surfaces.

[0361] In some implementations, re-measuring the spacing between intensity peaks and the spacing between angles of intensity peaks in a subtile, re-fitting the spacing and angle surfaces, and storing the results of the re-fit for use in processing later imaged image tiles may be performed one or more times during the ordering run. This may be done at intermediate periods, such as the 50th, or at least 100 subsequent periods. In some implementations, the re-measuring may be performed when computing resources are available, such as at the end of the ordering run, or during the time between two readings when chemical steps are being performed for the next reading.

[0362] The nanowells may be arranged in a regular repeating pattern. For a rectangular pattern, two structured illumination angles may be used substantially along two diagonals connecting opposite corners of the rectangles in the pattern, so that the intensity peaks of the structured illumination are oriented substantially perpendicular to the two diagonals. For a repeating hexagonal pattern of nanowells with three diagonals connecting opposite corners of the hexagons in the pattern, three structured illumination angles may be used with the intensity peaks oriented substantially perpendicular to the three diagonals. Alternatively, the samples may be randomly distributed on the imaging plane without nanowells. Or the samples may be regularly arranged on the imaging plane by some means other than nanowells.

[0363] The method may further include aggregating the enhanced resolution images for the subtiles into an enhanced resolution image for the tile and using the enhanced resolution image tile for further analysis, which may include ordering the samples, one position per period.

[0364] As an improvement to the surface fit, the method may further include determining and applying a cropping margin to take into account pixels around the edges of the sensor from consideration before fitting the spacing and angle surfaces to the measured spacing and angles. This margin may be uniform around the edges of the sensor or may vary with the edges, such as being wider at the ends of a rectangular sensor along long edges.

[0365] The method may further include measuring the spacing between the intensity peaks and the angles of the intensity peaks over at least 8x11 subtiles of the image tile. Other configurations for dividing the image tile into subtiles may be used, as described above.

[0366] The disclosed subtiling technique may be used to generate improved resolution images from images of a target imaged under structured illumination. In one image reconstruction implementation of the disclosed technique, a method is described for dividing an image of a target imaged by an optical sensor into nine or more subtiles and operating independently on each of the nine or more subtiles. As described above, more subtiles may be used to sub-divide the imaged image. The method includes transforming at least three images of the target in each subtile into the Fourier domain to generate at least three frequency domain matrices for each subtile. The images are imaged by the sensor in the spatial domain while applying at least three phase shifts of the structured illumination along an angle. An unmixing matrix is ​​calculated using the estimated reconstruction parameters. The inverse matrix is ​​applied to the frequency domain matrix such that at least three phase separation matrices in the Fourier domain are generated, which are a non-shifted matrix and a shifted matrix. A sub-pixel shift on at least one shift matrix is ​​performed. The sub-pixel shift includes transforming the shift matrix from the Fourier domain to the spatial domain, applying a translation vector to the data in the spatial domain, and transforming the transformed data from the spatial domain data. The sub-pixel shift generates two or more realigned shift matrices in the Fourier domain. The method includes aligning and summing overlapping values ​​of the shift matrix and the realigned shift matrix to generate an extended frequency coverage matrix. The extended frequency coverage matrix is ​​inverse transformed from the Fourier domain to generate an improved resolution subtile in the spatial domain. Optionally, an improved resolution image of the target may be generated by merging the improved resolution subtile into nine or more subtiles.

[0367] Each of this and other embodiments of the disclosed technology may optionally include one or more of the following features and / or features described in connection with the additional disclosed methods. For purposes of brevity, combinations of features disclosed in this application are not individually recited and are not repeated for each base set of features. The reader will understand how the features identified in this section can be readily combined with the basic features identified as an implementation.

[0368] The process steps of the method can be applied iteratively to generate a sequence of improved resolution images, which can be used to sequence the sample imaged by the sensor over multiple cycles.

[0369] The subtiles each contain at least 512x512 pixels of the optical sensor. Larger or smaller numbers may be used, including 256, 1024, 2048 and 4096 or between 256 and 4096 pixels. Performance can be improved if the subtile dimension is a power of 2, with 512 and 1024 pixels being a practical range found to work well.

[0370] The subtiles may overlap by at least 2 pixels of the optical sensor, although larger or smaller numbers can be used, for example in the case of 512 an overlap of 256 pixels would be appropriate, and in the case of 1024 an overlap of 512 would be appropriate.

[0371] The method may further include aggregating the improved resolution image for the subtile into an improved resolution aggregate image, and ordering the sample over multiple periods using the improved resolution aggregate.

[0372] The method can be further applied to photograph tiles at locations throughout the flow cell to image the fluorescence of millions of samples distributed throughout the flow cell. Application of such methods produces improved images of the flow cell.

[0373] When the imaging plane is a flow cell, the samples may be distributed in millions of nanowells across the flow cell. At least some adjacent pairs of nanowells may be positioned closer together than the Abbe diffraction limit for optical resolution. Structured illumination allows for such close positioning. Alternatively, the samples may be randomly distributed across the flow cell. In such an embodiment, the method further includes generating a sequence of enhanced images for the flow cell and recalling the sequence of samples using the sequence of enhanced images.

[0374] The above computer-implemented methods may be executed in a system including computer hardware. The computer-implemented system may execute one or more of the above methods. The computer-implemented system may incorporate any of the method features described immediately above or throughout this application that apply to the method performed by the system. For brevity, alternative combinations of system features are not individually listed. Features applicable to systems, methods, and articles of manufacture are not repeated for each statutory classification set of basic features. The reader will understand how the features identified in this section can be readily combined with basic features in other statutory classifications.

[0375] As an article of manufacture rather than a method, program instructions executable by a processor may be stored on a non-transitory computer readable medium (CRM). When the program instructions are executed, one or more of the computer-implemented methods described above are performed. Alternatively, the program instructions may be stored on a non-transitory CRM and, when combined with appropriate hardware, may become one or more components of a computer-implemented system that performs the disclosed methods.

[0376] Each of the features discussed in this specific embodiment section for the method embodiment applies equally to the CRM and system embodiments, and as noted above, all method features are not repeated here but should be considered again by reference.

[0377] Apply non-redundant SIM image reconstruction

[0378] The disclosed techniques relate to reducing the computing resources required to generate improved resolution images from structured illumination of a target.

[0379] In one embodiment of the disclosed technology, a method is described for generating an enhanced resolution image from an image of a target captured under structured illumination by applying one or more transforms to non-redundant data and then recovering redundant data from the non-redundant data after the transforms.

[0380] The method includes transforming at least three images of the target captured by a sensor in a spatial domain. The sensor captures images from at least three phase shifts of the structured illumination along an angle. The three captured images in the spatial domain are transformed to a Fourier domain to generate at least three frequency domain matrices. The frequency domain matrices include at least non-redundant and redundant conjugate components. The redundant conjugate components are conjugates of the non-redundant components at complementary matrix locations. The method further includes generating an unmixing matrix using the estimated reconstruction parameters and applying the unmixing matrix to at least the non-redundant components. This generates at least three phase separation matrices in the Fourier domain that are non-shifted and shifted matrices. The matrices include non-redundant non-shifted and non-redundant shifted spatial frequency components derived by applying the unmixing matrix to the non-redundant components of the frequency domain matrices.

[0381] The method also includes performing one or more intermediate transformations on at least the non-redundant components of the phase separation matrix. One intermediate operation is a sub-pixel shift. This includes transforming at least the non-redundant shifted spatial frequency components of the shift matrix from the Fourier domain to the spatial domain. Then, applying a transformation vector to the data in the spatial domain, transforming the transformed data from the spatial domain data in the Fourier domain back to one or more realigned shift matrices. By applying the transformation vector, duplicate values ​​in the shift matrix are efficiently realigned in the Fourier domain with a more accurate non-shift matrix compared to the full matrix positions of the discrete Fourier transform. The redundant conjugate components can be recovered from the non-redundant components after the intermediate transformation by copying the transformed values ​​from the non-redundant components to the complementary matrix positions of the redundant conjugate components and changing the sign of the imaginary parts of the copied values ​​to generate the transformed redundant conjugate components. This reduces the resources required during the intermediate transformation. Constructing the improved resolution image further includes aligning and summing overlapping values ​​of the unshifted and shifted matrices to generate an extended frequency coverage matrix, and then inverse transforming the extended frequency coverage matrix from the Fourier domain to generate an improved resolution image in the spatial domain.

[0382] In some implementations, a series of improved resolution images are generated by iteratively applying the previous operation. Optionally, for example in sequencing by synthesis, the disclosed techniques may be applied to sequence the sample imaged by the sensor over multiple cycles using a series of improved resolution images.

[0383] Each of this and other embodiments of the disclosed technology may optionally include one or more of the following features and / or features described in connection with the additional disclosed methods. For purposes of brevity, combinations of features disclosed in this application are not individually recited and are not repeated for each base set of features. The reader will understand how the features identified in this section can be readily combined with the basic features identified as an implementation.

[0384] The disclosed techniques may be applied to elements arranged in a corner shift matrix, where the complementary matrix position of a redundant conjugate element is rotated 180 degrees from the matrix position of the non-redundant element.

[0385] The method may further include applying one-step Wiener filtering or two-step Wiener filtering during the intermediate operation to reduce noise. When two-step Wiener filtering is applied, a separate modulation transfer function may be applied to the phase-separated matrix as an intermediate transformation to compensate for the optical transfer of contrast by the objective lens, which reduces its ability to transfer contrast with increasing spatial frequency. The noise reduction component of Wiener filtering may be applied separately later. The intermediate operation may further include applying apodization filtering.

[0386] In some implementations, the phase separation matrix in the Fourier domain includes an even number of rows and an even number of columns. For these even-sized matrices, column averaging can be applied as an intermediate operation that averages the DC column of the DC component along with the center column, which is one column beyond the center of the even-sized matrix. This column averaging may precede the sub-pixel shifting.

[0387] In some implementations, the phase separation matrix in the Fourier domain is corner shifted such that the DC components then appear in the top or bottom row and left or right columns of the matrix, and the redundant conjugate components appear in blocks rotated 180 degrees from the orientation of the non-redundant components.

[0388] In some embodiments, the phase separation matrix in the Fourier domain includes an odd number of rows and an odd number of columns with no columns between blocks of non-redundant components and blocks of redundant conjugate components, where in an odd number of matrices the redundant conjugate components also appear in blocks rotated 180 degrees from the orientation of the non-redundant components.

[0389] In embodiments where the target comprises regularly spaced nanowells arranged in a rectangular pattern, the method may be applied to a set of at least three images from phase displacement along two angles of structured illumination of the target, the two angles being along diagonals between opposite corners of the rectangle. Alternatively, in embodiments where the target comprises regularly spaced nanowells arranged in a rectangular pattern, the method may be applied to a set of at least three images from phase displacement along three angles of structured illumination of the target, the three angles being along diagonals between opposite corners of the rectangle.

[0390] The method may further include applying Wiener filtering during intermediate operations in two steps, including applying an independent modulation transfer function to the phase separation matrix to compensate for the light transfer of contrast that decreases with increasing spatial frequency by the objective lens that transfers the target to the sensor under structured illumination, which further includes applying a noise reduction component of Wiener filtering or another equivalent filter, when performed in two steps.

[0391] The method may further include applying an apodization filter during the intermediate operations.

[0392] The phase separation matrix in the Fourier domain may include an even number of rows and an even number of columns. For such an even number of matrices, the method may further include applying column averaging during intermediate operations to average DC columns of the DC components as well as intermediate columns between the blocks of non-redundant components and the blocks of redundant conjugate components. The column averaging may precede the sub-pixel shifting.

[0393] If the phase separation matrix in the Fourier domain is shifted to the top left corner, the DC components will appear in the top row and left column of the matrix, and the redundant conjugate components will appear in blocks rotated 180 degrees from their orientation as non-redundant components. Alternatively, the phase separation matrix can be shifted to the bottom right corner.

[0394] Alternatively, the phase separation matrix in the Fourier domain may include an odd number of rows and an odd number of columns, such that such an odd matrix does not include an intermediate column between the block of non-redundant components and the block of redundant conjugate components.

[0395] The method may further be applied to a set of at least three images of the target imaged through a lens by a sensor in the spatial domain from phase shifts along two angles of structured illumination of the target. When the target includes regularly spaced nanowells, the two angles that may be substantially oriented are along diagonals between opposite corners of quadrilaterals between the regularly spaced nanowells. These quadrilaterals may be squares, and the two angles may be substantially orthogonal within 1-3 degrees. Alternatively, the method may be applied to a set of images from phase shifts along three angles of structured illumination of the target. This is useful when the target includes spaced nanowells that are regularly spaced in a hexagonal pattern. In this pattern, the three angles may be along diagonals between opposite corners of hexagons in the pattern.

[0396] The above computer-implemented methods may be executed in a system including computer hardware. The computer-implemented system may execute one or more of the above methods. The computer-implemented system may incorporate any of the method features described immediately above or throughout this application that apply to the method performed by the system. For brevity, alternative combinations of system features are not individually listed. Features applicable to systems, methods, and articles of manufacture are not repeated for each statutory classification set of basic features. The reader will understand how the features identified in this section can be readily combined with basic features in other statutory classifications.

[0397] As an article of manufacture rather than a method, program instructions executable by a processor may be stored on a non-transitory computer readable medium (CRM). When the program instructions are executed, one or more of the computer-implemented methods described above are performed. Alternatively, the program instructions may be stored on a non-transitory CRM and, when combined with appropriate hardware, may become one or more components of a computer-implemented system that performs the disclosed methods.

[0398] Each of the features discussed in this specific embodiment section for the method embodiment applies equally to the CRM and system embodiments, and as noted above, all method features are not repeated here but should be considered again by reference.

[0399] The foregoing description is presented to enable making and using the disclosed technology. Various modifications to the disclosed embodiments will be apparent, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the disclosed technology. Thus, the disclosed technology is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein. The scope of the disclosed technology is defined by the appended claims.

[0400] Computer Systems

[0401] FIG. 24 is a simplified block diagram of a computer system that may be used to implement the disclosed techniques. A computer system typically includes at least one processor that communicates with a number of peripheral devices via a bus subsystem. These peripheral devices may include, for example, storage subsystems such as memory devices and file storage subsystems, user interface input devices, user interface output devices, and a network interface subsystem. The input and output devices enable user interaction with the computer system. The network interface subsystem provides an interface to external networks, including interfaces to corresponding interface devices in other computer systems.

[0402] In one embodiment, a parameter estimator 2401 for estimating three parameters (angle, spacing, and phase offset) is communicatively coupled to the storage subsystem and the user interface input device.

[0403] User interface input devices may include keyboards and pointing devices such as a mouse, trackball, touchpad, or graphics tablet, scanners, touch screens integrated into displays, audio input devices such as voice recognition systems and microphones, as well as other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and ways of inputting information into a computer system.

[0404] The user interface output devices may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), a projection device, or some other mechanism for producing a visible image. The display subsystem may also provide non-visual displays such as an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from a computer system to a user or to another machine or computer system.

[0405] The storage subsystem stores programming and data constructs that provide the functionality of some or all of the modules and methods described herein. These software modules are generally executed by the processor alone or in combination with other processors.

[0406] The memory used in the storage subsystem may include multiple memories, such as a main random access memory (RAM), a read only memory (ROM) with fixed instructions stored therein, for storing instructions and data during program execution. The file storage subsystem may provide persistent storage for program and data files and may include a hard disk drive, associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of a particular embodiment may be stored by the file storage subsystem in the storage subsystem or in other machines accessible by the processor.

[0407] The bus subsystem provides a mechanism for allowing the various components and subsystems of a computer system to communicate with each other as intended. Although the bus subsystem is shown generally as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0408] The computer system itself can be of various types, including a personal computer, a portable computer, a workstation, a computer terminal, a network computer, a television, a mainframe, a server farm, a loosely distributed set of loosely networked computers, or any other data processing system or user device. Due to the ever-changing nature of computers and networks, the depiction of the computer system shown in Figure 24 is intended only as a specific example for purposes of illustrating the disclosed technology. Many other configurations of a computer system can have more or fewer components than the computer system shown in Figure 24.

[0409] The deep learning processor may be a GPU or FPGA and may be hosted by a deep learning cloud platform such as Google Cloud Platform, Xilinx, and Cirrascale. Examples of deep learning processors include Google's Tensor Processing Unit (TPU), rackmount solutions such as the GX4 Rackmount Series, GX8 Rackmount Series, NVIDIA DGX-1, Microsoft's Stratix V FPGA, Graphcore's Intelligent Processor Unit (IPU), Qualcomm's Zeroth Platform with Snapdragon processors, NVIDIA's Volta, NVIDIA's DRIVE PX, NVIDIA's JETSON TX1 / TX2 MODULE, Intel's Nirvana, Movidius VPU, Fujitsu DPI, ARM's DynamicIQ, IBM TrueNorth, and others. [Explanation of symbols]

[0410] 191 Real-time SIM shooting components 195 Controller 531 Lattice 532 Lattice 2401 Parameter Estimator

Claims

1. 1. A method for generating an improved resolution image from an image of a target captured under structured illumination, comprising: transforming at least three images of the target imaged by a sensor in the spatial domain from at least three phase displacements of the structured illumination along an angle into a Fourier domain to generate at least three frequency domain matrices, each matrix of the at least three frequency domain matrices including a first block of complex coefficients and a redundant second block of complex coefficients that is a conjugate, each complex coefficient of the first block of complex coefficients being a complex conjugate of each complex coefficient of the second block of complex coefficients; applying an unmixing matrix to the first block using the estimated reconstruction parameters to generate at least three phase-separating half matrices in the Fourier domain, a non-shifted half matrix and a shifted half matrix; performing one or more intermediate transforms on the phase-separating half-matrix, performing a sub-pixel shift of the shift-half matrix by transforming from the Fourier domain to the spatial domain; multiplying data in the spatial domain by a vector to generate translated data; and transforming the translated data from the spatial domain data back to the Fourier domain to generate a realigned shift-half matrix; calculating complex coefficients of a second block in the Fourier domain to generate a full matrix from the half matrix by copying complex coefficients from the unshifted half matrix and the reordered shifted half matrix as conjugates to respective second block positions after the intermediate transformation, and generating a shifted full matrix and an unshifted full matrix from the half matrix, including changing signs of imaginary parts of the copied complex coefficients; aligning and summing overlapping complex coefficients of the unshifted full matrix and the shifted full matrix to generate an extended frequency coverage matrix; and inverse transforming the extended frequency coverage matrix from the Fourier domain to generate an enhanced resolution image in the spatial domain.

2. 2. The method of claim 1, further comprising sequencing samples imaged by the sensor over multiple cycles using a sequence of improved resolution images produced by iteratively applying the operations of claim 1.

3. A method as described in claim 1 or 2, wherein the positions of the complex coefficients of the second block are rotated 180 degrees from the positions of the complex coefficients of the first block when arranged such that high frequency components are closer to the center of their respective matrices and low frequency components are farther away from the center of their respective matrices.

4. The method according to any one of claims 1 to 3, further comprising applying one or two steps of Wiener filtering during the intermediate transformation so as to reduce noise.

5. The method according to any one of claims 1 to 3, further comprising applying apodization filtering during the intermediate transformation.

6. 6. The method of claim 1, wherein the phase separation matrix in the Fourier domain includes an even number of rows and an even number of columns, and further comprising setting a DC column of a DC complex coefficient and intermediate columns of other complex coefficients equal between the shift matrices.

7. 6. The method of claim 1, wherein at least one of the phase separation matrices in the Fourier domain is corner-shifted such that a DC complex coefficient appears in the top row and left column of the matrix and the complex coefficient conjugates appear in a second block that is rotated 180 degrees from the orientation of the first block.

8. 6. The method of claim 1, further comprising applying the method to a set of at least three images of the target imaged through a lens by a sensor in the spatial domain from phase displacements along two angles of the structured illumination of the target, the target comprising regularly spaced nanowells, the two angles of the structured illumination being along diagonals between opposite corners of a quadrilateral between the regularly spaced nanowells.

9. 8. The method of claim 1, further comprising applying the method to a set of at least three images of the target imaged through a lens by a sensor in the spatial domain from phase displacements along three angles of the structured illumination of the target, the target comprising regularly spaced nanowells, the three angles being along diagonals between opposite corners of a hexagon between the regularly spaced nanowells.

10. 1. A system including one or more processors coupled to a memory, the memory being loaded with computer instructions for generating an enhanced resolution image from an image of a target captured under structured illumination, the computer instructions, when executed on the one or more processors, comprising: transforming at least three images of the target imaged by a sensor in the spatial domain from at least three phase displacements of the structured illumination along an angle into a Fourier domain to generate at least three frequency domain matrices, each matrix of the at least three frequency domain matrices including a first block of complex coefficients and a redundant second block of complex coefficients that is a conjugate, each complex coefficient of the first block of complex coefficients being a complex conjugate of each complex coefficient of the second block of complex coefficients; applying an unmixing matrix to the first block using the estimated reconstruction parameters to generate at least three phase-separating half matrices in the Fourier domain, a non-shifted half matrix and a shifted half matrix; performing one or more intermediate transforms on the phase-separating half-matrix, performing a sub-pixel shift of the shift-half matrix by transforming from the Fourier domain to the spatial domain; multiplying data in the spatial domain by a vector to generate translated data; and transforming the translated data from the spatial domain data back to the Fourier domain to generate a realigned shift-half matrix; calculating complex coefficients of a second block in the Fourier domain to generate a full matrix from the half matrix by copying complex coefficients from the unshifted half matrix and the reordered shifted half matrix as conjugates to respective second block positions after the intermediate transformation, and generating a shifted full matrix and an unshifted full matrix from the half matrix, including changing signs of imaginary parts of the copied complex coefficients; aligning and summing overlapping complex coefficients of the unshifted full matrix and the shifted full matrix to generate an extended frequency coverage matrix; inverting the extended frequency coverage matrix from the Fourier domain to generate an improved resolution image in the spatial domain; A system that implements behavior including

11. 11. The system of claim 10, wherein the computer instructions, when executed on the one or more processors, further implement an operation including sequencing samples imaged by the sensor over multiple cycles using a sequence of improved resolution images generated by repeatedly applying the operations of claim 10.

12. The system described in claim 10 or 11, wherein the positions of the complex coefficients of the second block are rotated 180 degrees from the positions of the complex coefficients of the first block when arranged such that high frequency components are closer to the center of their respective matrices and low frequency components are farther away from the center of their respective matrices.

13. 13. The system of claim 10, wherein the computer instructions, when executed on the one or more processors, further implement operations including applying one or two steps of Wiener filtering to reduce noise during the intermediate transformation.

14. The system of any one of claims 10 to 12, wherein the computer instructions, when executed on the one or more processors, further implement operations including applying apodization filtering during the intermediate transformation.

15. 15. The system of claim 10, wherein the computer instructions, when executed on the one or more processors, further implement an operation comprising applying a phase shift along two angles of the structured illumination of the target to a set of at least three images of the target imaged through a lens by a sensor in the spatial domain, the target comprising regularly spaced nanowells, the two angles of the structured illumination being along diagonals between opposite corners of a quadrilateral between the regularly spaced nanowells.

Images