Systems and methods for structured illumination microscopy

Abstract

The technology disclosed relates to structured illumination microscopy (SIM). In particular, the technology disclosed relates to capturing and processing a large number of image tiles on a large image plane in real time, dividing them into sub-tiles, efficiently processing the sub-tiles, and generating an enhanced resolution image from the sub-tiles. The enhanced resolution image can be combined into an enhanced image and can be used for subsequent analysis steps.

Description

[0001] This application is a divisional application of Chinese patent application No. 202080072888.3, ​​entitled "System and Method for Structured Illumination Microscopy".

[0002] Priority application

[0003] This application claims the benefit of the following U.S. provisional patent applications, which are incorporated herein by reference: U.S. Provisional Patent Application No. 62 / 924,130, filed October 21, 2019, entitled “Systems and Methods for Structured Illumination Microscopy” (Attorney’s File No. ILLM 1012-1), and U.S. Provisional Patent Application No. 62 / 924,138, filed October 21, 2019, entitled “Increased Calculation Efficiency for Structured Illumination Microscopy” (Attorney’s File No. ILLM 1022-1). This application also claims priority to the following U.S. non-provisional patent applications, which are incorporated herein by reference: U.S. non-provisional patent application No. 17 / 075,692, filed October 21, 2020, entitled “Systems and Methods for Structured Illumination Microscopy” (Attorney’s File No. ILLM 1012-2), and U.S. non-provisional patent application No. 17 / 075,694, filed October 21, 2020, entitled “Increased Calculation Efficiency for Structured Illumination Microscopy” (Attorney’s File No. ILLM 1022-2), both of which claim the benefit of the aforementioned provisional applications.

[0004] Literature merged

[0005] The following references are incorporated herein by reference, as if shown in their entirety, for all purposes: U.S. Provisional Application No. 62 / 692,303 (unpublished), filed June 29, 2018, entitled “Device for Luminescent Imaging”, and U.S. Non-Provisional Patent Application No. P-1683-US, filed June 29, 2019, entitled “Device for Luminescent Imaging”. Technical Field

[0006] The technology disclosed in this invention relates to Structured Imaging Microscopy (SIM). Specifically, the technology disclosed in this invention relates to the real-time capture and processing of numerous image blocks on a large image plane, dividing them into sub-blocks, efficiently processing these sub-blocks, and generating enhanced resolution images from these sub-blocks. These enhanced resolution images can be combined into an enhanced image and used in subsequent analysis steps.

[0007] The techniques disclosed in this invention relate to structured imaging microscopy. Specifically, the techniques disclosed in this invention relate to reducing the computation required for real-time processing of large image blocks on a large image plane, and generating enhanced resolution images from image blocks / sub-blocks. During some intermediate transformations in the SIM processing chain, almost half of the multiplications and divisions that would otherwise be required can be replaced by lookup operations that utilize specific exploits of the described symmetries. Background Technology

[0008] The topics discussed in this section should not be considered prior art simply because they are mentioned here. Similarly, problems mentioned in this section or related to the topics provided as background art should not be assumed to have been previously recognized in the prior art. The topics in this section merely represent different methods, which themselves may correspond to specific implementations of the technology protected by the claims.

[0009] More than a decade ago, the pioneers of structured imaging microscopy were awarded the Nobel Prize in Physics. Enhancing image resolution beyond the Abbe diffraction limit was a remarkable achievement.

[0010] Both 2D and 3D SIM have been applied to imaging biological samples, such as portions of a single cell. Much effort has been put into studying the interior of cells, resulting in numerous alternative technological changes.

[0011] Resolving millions of sources distributed across an image plane presents very different problems compared to examining the interior of a cell. For example, one emerging approach combines a large number of images with specular illumination to produce an enhanced resolution image after extensive computation. Processing large image planes in real time with limited resources requires a completely different approach than recent works of this kind.

[0012] Therefore, there is an opportunity to introduce new methods and systems that are suitable for processing large image planes with reduced computational resource requirements. Summary of the Invention

[0013] According to a first aspect of the invention, a method for generating sub-block calibration parameters is provided, the method being used with a scanner to produce an enhanced resolution image of the fluorescence of a sample positioned closer than the Abbe diffraction limit of optical resolution between at least some adjacent pairs of the sample, the method comprising:

[0014] The scanner is calibrated using a substantially full field of view projected onto an optical sensor in the scanner by a lens, including:

[0015] At least one image block is captured under structured illumination with multiple angles and phase shifts of the structured illumination;

[0016] Calculate the optical distortion on the image block, including

[0017] The spacing between intensity peaks in at least nine sub-blocks, including a near-center sub-block, of the image block and the angle of the projected illumination pattern indicated by the intensity peaks are measured.

[0018] The spacing surface and the angle surface are respectively fitted to the measured spacing and angle, wherein both the spacing surface and the angle surface represent corrections for distortion on the sub-blocks of the image block; and

[0019] The phase displacement of the structured illumination is measured using the multiple angles and the phase displacement within the sub-block, and a lookup table is generated to represent the difference between the phase displacement in the near-center sub-block of the image block and the phase displacement in other sub-blocks;

[0020] The sub-block calibration parameters, including the lookup table and the fitting results including relating the spacing of the intensity peaks on the at least eight additional sub-blocks and the angle of the projected illumination pattern to the near-center sub-block, are stored for use by the scanner in production.

[0021] According to a second aspect of the invention, a method is provided for generating an enhanced resolution image of a sample using a calibrated scanner, the sample being positioned closer than the Abbe diffraction limit of optical resolution between at least some adjacent pairs of the sample, the method comprising the following actions:

[0022] For the near-center sub-block of an image block captured by the optical sensor in the scanner, access or generate estimated reconstruction parameters, including

[0023] Spacing parameters, for the spacing between intensity peaks, and pattern angle parameters, for the angle of the projected illumination pattern indicated by the intensity peaks, the spacing parameters being generated from measurements of imaging a regularly spaced and angled structured illumination pattern projected onto the near-center sub-block.

[0024] Phase displacement parameters, which are used for the relative phase relationship between the sub-blocks, are generated at the sub-block reference point from further measurements of multiple phase displacements imaged from an angle along the structured illumination pattern projected onto the near-center sub-block.

[0025] The estimated reconstruction parameters for at least eight additional sub-blocks of the image block and the near-center sub-block are derived by combining the accessed or estimated reconstruction parameters of the near-center sub-block with the following:

[0026] The saved fitting results correlate the spacing parameters and angle parameters of the at least eight additional sub-blocks with the near-center sub-block.

[0027] A saved lookup table, the saved lookup table being used for phase displacement measurements relative to the near-center sub-block on the at least eight additional sub-blocks; and

[0028] The estimated reconstruction parameters are applied to the near-center sub-block and the at least eight additional sub-blocks to generate an enhanced resolution image for the sub-block that is superior to the Abbe diffraction limit.

[0029] According to a third aspect of the present invention, a method is provided for generating an enhanced resolution image from an image of a target captured under structured illumination, the method comprising the following actions:

[0030] The captured image of the target, captured across the full field of view of the optical sensor, is divided into nine or more sub-blocks, and operations are performed independently on each of the nine or more sub-blocks, including:

[0031] Transform at least three images of the target into the Fourier domain in the corresponding sub-block to generate at least three frequency domain matrices for the corresponding sub-block, the at least three images being captured by a sensor in the spatial domain while applying at least three phase shifts of the structured illumination along an angle;

[0032] The estimated reconstruction parameters determined by the sub-block are used to compensate for optical or lens distortion in the sub-block within the full field of view. An inverse mixing matrix is ​​applied to the frequency domain matrix to generate at least three phase-separated matrices in the Fourier domain, the at least three phase-separated matrices being an unoffset matrix and an offset matrix.

[0033] Performing sub-pixel offset on at least one offset matrix includes transforming the offset matrix from the Fourier domain to the spatial domain, applying a transformation vector to data in the spatial domain, transforming the transformed data from the data in the spatial domain, and generating two or more realigned offset matrices in the Fourier domain.

[0034] Align and sum the overlap values ​​of the unoffset matrix and the realigned offset matrix to produce an extended frequency coverage matrix; and

[0035] The extended frequency coverage matrix is ​​inversely transformed from the Fourier domain to generate sub-blocks with enhanced resolution in the spatial domain; and

[0036] The enhanced resolution sub-blocks of the nine or more sub-blocks are merged into an enhanced resolution image. Attached Figure Description

[0037] The color drawings in the priority application correspond to the black and white drawings in this application and are available from the PAIR database of the U.S. Patent and Trademark Office via the Supplemental Contents tab. In the drawings, similar reference numerals generally refer to similar parts in all different views. Furthermore, the drawings are not necessarily drawn to scale but are intended to illustrate the principles of the disclosed technology. Various specific embodiments of the disclosed technology are described in the following description with reference to the following drawings, wherein:

[0038] Figure 1A Moiré fringes are shown formed by using a grating with one-dimensional (1D) modulation.

[0039] Figure 1B An illustration of the lighting intensity generated by a two-dimensional (2D) structured lighting pattern is presented.

[0040] Figure 1C An example geometric pattern of nanopore arrangement is shown.

[0041] Figure 2 A structured illumination microscopy imaging system is shown that can image samples using spatially structured excitation light.

[0042] Figure 3 This is an optical system diagram illustrating an example optical configuration of a dual-arm SIM imaging system.

[0043] Figure 4A and Figure 4B This is a schematic diagram illustrating an example optical configuration of a dual-grating sliding SIM imaging system.

[0044] Figure 5A This illustrates undesirable variations in the spacing parameters that may occur in a SIM imaging system.

[0045] Figure 5B This illustrates unwanted variations in phase parameters that may occur in SIM imaging systems.

[0046] Figure 5C This illustrates undesirable variations in orientation angle parameters that may occur in SIM imaging systems.

[0047] Figure 6 A simplified illumination stripe pattern is shown that can be projected onto the sample plane by vertical and horizontal gratings through a SIM imaging system.

[0048] Figure 7 A simplified illumination stripe pattern is shown that can be projected onto a plane by the first and second gratings of a dual-grating sliding SIM imaging system.

[0049] Figure 8A This is a simplified description of how parallel lines bend due to distortion caused by a magnifying lens.

[0050] Figure 8B and Figure 8C The spacing between the nominal parallel lines is shown.

[0051] Figure 8D An example of a sub-block or subfield of a full field of view (FOV) image is shown.

[0052] Figure 9 The color-coded surface was roughly depicted, and it fits well to the observed data points indicated in red.

[0053] Figure 10A and Figure 10B An inverted bowl shape is shown, representing the measured spacing relative to the spacing in the near-center subfield.

[0054] Figure 11A , Figure 11B and Figure 11C The measurement spacing distortion was compared with that of quadratic and cubic surface fitting.

[0055] Figure 12A The measurement data points that were not smoothed by curve fitting are shown.

[0056] Figure 12B The actual data compared to the quadratic fitted surface are shown.

[0057] Figures 13A to 13E This demonstrates how to improve curve fitting by cropping along the sensor boundary.

[0058] Figure 14A The diagram illustrates the reduction in mean squared error (MSE) under different shrinkage factors.

[0059] Figures 14B to 14G The paper demonstrates how to improve the quadratic fit of the angle distortion model by incrementally increasing the shrinkage factor applied to the full FOV image data.

[0060] Figure 14H This is a graph showing a slight improvement in the quadratic surface fit when the shrinkage factor is six.

[0061] Figure 15A and Figure 15B The quadratic fit of the response surface obtained by fitting the stripe spacing distortion (A) and stripe angle distortion (B) on full FOV image data is shown.

[0062] Figure 16A This illustrates the transformation of the sub-block phase offset from the sub-block coordinate system to the full FOV image coordinate system.

[0063] Figure 16B The positions of points in the sub-block relative to the sub-block coordinate system and the full FOV image coordinate system are shown.

[0064] Figure 17 This is an example phase offset lookup table for sub-blocks in a full FOV image.

[0065] Figure 18 This is a high-level overview of the processing steps in the non-redundant sub-block SIM image reconstruction algorithm.

[0066] Figure 19 The symmetry in Fourier space of a matrix with even and odd rows and columns is shown.

[0067] Figure 20 The representation of the full matrix in the frequency domain is shown, along with the non-redundant and redundant halves of three acquired images obtained at an illumination peak angle.

[0068] Figure 21 The representation of three matrices in the frequency domain is shown as the reshaping of the non-redundant half of three images acquired at an illumination peak angle.

[0069] Figure 22 This demonstrates how to separate the frequency bands by using the inverse matrix from... Figure 21 The band separation process of multiplying the reshaping matrix.

[0070] Figure 23A An example of the (1, 1) offset operation applied to a matrix is ​​shown.

[0071] Figure 23B and Figure 23C The average of the first and middle columns is shown in the non-redundant SIM image reconstruction.

[0072] Figure 24It is a simplified block diagram of a computer system that can be used to implement the technology disclosed in this invention.

[0073] Figure 25A and Figure 25B The results demonstrate the improvement in error rate and the percentage of clusters passing the filter (%PF) as the SIM image reconstruction algorithm is updated.

[0074] Figures 26A to 26C The diagram illustrates the improvement in error rate across multiple cycles in the sequencing run.

[0075] Figure 26D The graphs show how the error rate is improved by incorporating phase learning and stitching fixation techniques into the Baseline SIM algorithm. Detailed Implementation

[0076] The following discussion is presented to enable any person skilled in the art to implement and use the disclosed technology, and is provided in the context of a particular application and its requirements. Various modifications to the specific embodiments disclosed will be apparent to those skilled in the art, and the general principles defined herein can be applied to other specific embodiments and applications without departing from the spirit and scope of the disclosed technology. Therefore, the disclosed technology is not intended to be limited to the specific embodiments shown, but is to be accorded the broadest scope consistent with the principles and features disclosed herein.

[0077] Introduction

[0078] Two-dimensional and three-dimensional structured illumination microscopy helps researchers study the internal structure of living cells and even the molecular arrangement of biological materials because it improves the resolution of image capture systems. See, for example: Sydor, Andrew and Czymmek, Kirk and Puchner, Elias and Mennella, Vito (2015), “Super-Resolution Microscopy: From Single Molecules to Supramolecular Assemblies,” Trends in Cell Biology, Vol. 25, 10.1016 / j.tcb.2015.10.004; and Lal et al. 2015, A., Shan, C. and Xi, P. (2015), “Structured illumination microscopy image reconstruction algorithm,” IEEE Journal of Selected Topics in Quantum Electronics, Vol. 22, No. 4, pp. 50–63. For readers wishing to revisit SIM processing, Lal et al. 2015 offers a perfect combination of illustrations and mathematical exposition, only without deriving the equations used. This disclosure extends SIM technology to image processing with flow pools that can be used individually or in combination.

[0079] SIM has the potential to resolve densely packed samples from a flow cell containing fluorescence signals from millions of sample points, thereby reducing the reagents required for processing and increasing the throughput of image processing. The trick is to use SIM to resolve densely packed fluorescent samples that are even closer together than the Abbe diffraction limit for resolving adjacent light sources. These samples can be regularly spaced nanopores or they can be randomly distributed clusters. Much of the following description focuses on patterned nanopores, but SIM is also applicable to randomly distributed clusters.

[0080] Technical problems related to lens distortion and required computational resources arise when resolving densely filled sources. This disclosure addresses those technical problems.

[0081] Structured illumination can produce images with several times the resolution of illumination sources compared to normal illumination. Information cannot be simply created this way. Instead, multiple images with different angles and phase shifts of structured illumination transform closely spaced, otherwise indistinguishable high spatial frequency features into lower frequency signals that can be sensed by an optical system without violating the Abbe diffraction limit. This limit is physically imposed on imaging by the properties of light and optics and is expressed as a function of the illumination wavelength and the numerical aperture (NA) of the final objective lens. SIM reconstruction is applied, transforming information from multiple images from the spatial domain to the Fourier domain, combining and processing it, and then reconstructing the enhanced image.

[0082] In SIM, gratings or interference patterns are used between the illumination source and the sample to produce illumination patterns, such as patterns where the intensity varies according to a sine or cosine function. In the context of SIM, "grating" is sometimes used to refer to the projected structured illumination pattern, as well as the surface from which the structured illumination pattern is generated. The structured illumination pattern can also optionally be generated as an interference pattern between portions of a split coherent beam.

[0083] Structured illumination is projected onto the sample plane, as shown in Figure 1, and the illumination pattern is mixed with a fluorescent (or reflective) source in the sample to induce new signals, sometimes called Mohs fringes or aliasing. These new signals shift high spatial frequency information to lower spatial frequencies that can be captured without violating the Abbe diffraction limit. Capture is achieved using 1D intensity modulation patterns (such as...) Figure 1A (as shown) or using 2D intensity modulation patterns (such as Figure 1B After obtaining images of the illuminated samples (as shown), a system of linear equations is solved and used to extract a portion of a new signal containing information shifted from higher to lower spatial frequencies from multiple images of moiré fringes or aliasing. To solve the linear equations, three or more images are captured using a structured illumination pattern with progressive shifts or displacements. Typically, images of different phases (3 to 7 phases) at each angle (with one to five angles) are captured for analysis and then band-separated for Fourier domain shifting and reconstruction. Increasing the number of images can improve the quality of the reconstructed image by enhancing the signal-to-noise ratio. However, it also increases computation time. The Fourier representations of the band-separated images are shifted and summed to produce a reconstructed sum. Finally, an inverse fast Fourier transform (FFT) reconstructs a new high-resolution image from the reconstructed sum.

[0084] This multi-image and offset method enhances image information of the 1D structured illumination pattern only along the specific direction of its offset, thus rotating the structured illumination or grating pattern and repeating the offset process. Rotations of the 1D pattern, such as 45, 60, 90, or 120 degrees, can be applied to generate six or nine images in groups of three steps. Environmental factors, such as the damage to molecules on the flow cell surface by blue and green lasers, and the tilting of flow cell surface patches, can cause distortion in the phase deviation estimation of image sub-blocks or sub-windows in the full field of view (FOV) image. They can also cause differences between blocks on the substrate. More frequent estimation of SIM image reconstruction parameters can be performed to compensate for these environmental factors. For example, the phase deviation of sub-blocks can be re-estimated for each block and each reconstruction cycle to minimize these errors. Angle and spacing parameters do not change as frequently as phase deviations, and therefore, increasing their estimation frequency may introduce unnecessary additional computation. However, angle and spacing parameters can be calculated more frequently if needed.

[0085] Application of SIM in flow cell image analysis

[0086] Imaging a flow cell with millions of fluorescence samples is more like scanning space with a telescope than studying the microscopic structure of living cells. Scanning a flow cell with an economical optics system is more like capturing an image with a static camera than using the adaptive optics system at Palomar Observatory. Scanning a flow cell or the sky involves a large number of images covering a target area, rather than imaging living cells in the optimal position of the lens. The number of images required to cover any target depends on the field of view and the degree of overlap for each image area. Economical lenses exhibit distortion at the edges of the field of view, which complicates the generation of enhanced images.

[0087] The computations used in SIM reconstruction are sensitive to lens distortion, pattern alignment within a specific image set, and the thermal effects of pattern alignment over imaging cycles lasting several hours. Using the majority of the lens, rather than the center, to increase the field of view makes SIM image reconstruction susceptible to distortions caused by lens aberrations. These aberrations, such as coma, distort the structured illumination pattern and make parallel lines of illumination peaks appear curved, as... Figure 8A As roughly shown, the distance between the brightness peak of the pattern and the apparent pattern angle was changed.

[0088] Practical instruments require characterization of distortions and biases mapped onto sub-blocks of a block. Due to the duration of flow cell imaging in cycles, some instruments will benefit from characterizations updated at the start of a run or even during the run. Thermal deformations in flow cell imaging over hundreds of cycles and hours make it very useful to periodically or occasionally recharacterize the angles, spacing, and phase (offset) of structured illumination.

[0089] When characterizing the instrument, the angle (rotation) and spacing (scale) of the projected structured illumination pattern can be determined by fringe peak estimation. The estimated phase or displacement of the repeating pattern along the step direction can be expressed as a spatial offset from 0 to 360 degrees, or in radians. Distortion on the lens complicates parameter estimation, leading to proposals for computationally expensive reconstructions from large numbers of images. See, for example, Ayuk et al. (2013), “Structured illumination fluorescence microscopy with distorted excitations using a filtered blind-SIMalgorithm,” Optics Letters, Vol. 38, pp. 4723-4726, 10.1364 / OL.38.004723; and Mudry et al. (2012), “Structured illumination microscopy using unknown speckle patterns,” Nature Photonics, Vol. 6, pp. 312-315. The technique disclosed in this invention simplifies repeating parameter estimation and reduces the computation required for repeating estimation cycles.

[0090] Real-time parameter estimation for SIM reconstruction is computationally challenging. The computational power required for SIM reconstruction increases with the cubic relationship of the number of pixels in the image field or subfield. For example, for an image with M pixels wide and N pixels high, the Fourier transform may have k... M N(log(M The computational complexity of SIM image reconstruction is O(N). Therefore, the resource order of SIM image reconstruction can increase quadratically with the number of pixels in the image. 2 ) and cubic O(N 3 The image dimensions increase by a factor of two. Therefore, doubling the image dimension (e.g., from 512×512 to 1024×1024) can lead to a computational cost increase of up to eight times. Reconstructing enhanced images from six or nine 20-million-pixel images on a CPU at a rate of one complete reconstruction every 0.3 seconds during scanning is particularly challenging. As scanning and sequencing proceed, real-time processing is required to reduce storage needs and to track the quality of flow pool processing over several hours. The technique disclosed in this invention reduces some core computations by approximately half using symmetry and approximate symmetry in the Fourier domain coefficient matrix.

[0091] Solving lens distortion

[0092] The technique disclosed in this invention addresses lens distortion, which is less severe for optical systems within economical manufacturing tolerances, by subdividing captured image blocks into sub-blocks and processing near-center sub-blocks in a manner different from other sub-blocks. An image captured by an optical sensor can be referred to as a block. The imaging cycle of a flow cell captures many image blocks with some overlap. Each image block is divided into independently evaluated sub-blocks. Sub-blocks can be reconstructed independently of each other, or even in parallel. Reconstructions from enhanced sub-blocks can be stitched together to create reconstructed blocks with enhanced spatial resolution.

[0093] The inference technique disclosed in this invention, which correlates the reconstruction parameters of a near-center sub-block with other sub-blocks, approximates a nonlinear function, such as a quadratic curve, through piecewise approximation. The technique subdivides an image block into sub-blocks such that the peak lines are spaced approximately evenly within the sub-blocks, thereby obtaining better image quality from the reconstructed sub-blocks in the lens's field of view.

[0094] This sub-blocking method, which mitigates stripe distortion, creates a new problem: reconstruction parameters must be estimated for each sub-block. Parameter estimation is the most expensive setting in SIM reconstruction, and sub-blocking increases the parameter estimation runtime by at least an order of magnitude; for example, dividing the image into 5×5 sub-blocks creates an algorithm that is 25 times slower. A block divided into 8×11 sub-blocks requires 88 sets of reconstruction parameters instead of just one.

[0095] The technique disclosed in this invention learns a function of mapping distortion, thereby reducing recalculations in repeated loops. The learned function remains valid between loops and image sets because it maps the optical characterizations of the optical system, which change as the optical system changes, for example, through realignment. During estimation loops, costly parameter estimation calculations are concentrated on a single near-center sub-block between instrument characterizations. The parameters of each sub-block map the measurements of the loop from the near center, the reference sub-block, and to other sub-blocks.

[0096] Three parameters are mapped to the sub-blocks: illumination peak angle, illumination peak spacing, and phase shift. The illumination peak angle is also known as the grating angle. The illumination peak spacing is also known as the grating spacing. The technique disclosed in this invention uses a quadratic surface distortion model to map the angle and spacing. Phase shift, or simply phase, is the offset of the structured illumination pattern or grating projected onto the sample plane.

[0097] The reusable spacing and angular distortion models are computed a priori using a sub-blocking method to characterize the full field of view (FOV). The windowing or sub-blocking method divides the image into overlapping windows or sub-blocks and performs SIM parameter estimation on each sub-block. The sub-block parameter estimates are then fed into a least-squares regression, which generates a quadratic surface of distortion. The stored equation coefficients can then be used to extrapolate parameters at any location within the sub-blocks or the full field of view. In an alternative implementation, the coefficients can be stored and used, or they can be converted into sub-blocks, i.e., a lookup table relating the near-center sub-block to other sub-blocks. The near-center sub-block is used as a reference because the center of the lens has less distortion than the edges.

[0098] For phase relationships between sub-blocks, lookup tables are superior to curve fitting, as shown below.

[0099] During operation lasting several hours, periodic recharacterization can be applied to prevent thermal instability and the resulting drift. After the extrapolation factors have been redefined, the process continues by extrapolating the parameter estimates of the near-center sub-block to other sub-blocks without requiring costly parameter estimates for those sub-blocks.

[0100] Using symmetry to reduce computational cost

[0101] When imaging a single pool, SIM reconstruction of a large number of images is typically performed on dedicated hardware such as GPUs, FPGAs, or CGRAs. Using abundant computational resources, reconstruction algorithms operate with fully redundant, center-shifted Fourier transforms. In contrast, using reasonably priced CPUs, we disclose SIM reconstruction using non-redundant coefficients of the Fourier-shifted images. The specific implementation is described based on the symmetry of the data in the angular-shifted Fourier transform space. Using certain symmetries increases the complexity of the procedure but halves the amount of computational data in the core computation set and the required computation, while maintaining nearly the same accuracy. We also reduce the computation required for the Fourier representation of the images with shifted combinations.

[0102] Adapting to 2D lighting patterns

[0103] When used with 2D modulated illumination patterns, the standard algorithm for 1D modulated illumination needs modification. This includes estimation of illumination peak spacing and illumination peak angle, which requires 2D band separation instead of 1D. It also includes Wicker phase estimation, which must work from two points (instead of one) to estimate the phase in 2D. 1D interference patterns can be derived from... Figure 1A The one-dimensional diffraction grating shown is generated, or is the result of the interference pattern of two beams.

[0104] Figure 1BThe diagram illustrates the intensity distribution produced by either a two-dimensional (2D) diffraction grating or by the interference of four beams. Two beams produce intensity patterns (horizontal bright and dark lines) along the y-axis and are therefore referred to as the y-pair of incident beams. The other two beams produce intensity patterns (vertical bright and dark lines) along the x-axis and are referred to as the x-pair of incident beams. The interference of the y-pair and x-pair beams produces a 2D illumination pattern. Figure 1B The intensity distribution of this 2D lighting pattern is shown.

[0105] Figure 1C The diagram illustrates an arrangement of nanopores on the surface of a flow cell located at the corners of a rectangle. When using 1D structured illumination, the illumination peak angle is selected such that images are captured along lines connecting opposite corners of the diagonals of the rectangle. For example, two sets of three images (a total of six images) can be captured at angles of +45 degrees and -45 degrees. Higher resolution images are achieved because the distance along the diagonals is greater than the distance between any two sides of the rectangle. The nanopores can be arranged in other geometric configurations, such as hexagons. Three or more images can then be captured along each of the three diagonals of the hexagon, resulting in, for example, nine or fifteen images.

[0106] The novel and proprietary techniques disclosed in this invention can be used alone or in combination to improve scanning performance while detecting fluorescence in millions of samples distributed on a flow cell over multiple cycles. In the following sections, we introduce terminology, describe imaging instruments that can be improved using the techniques disclosed in this invention, and disclose novel image enhancement techniques that can be used alone or in combination.

[0107] the term

[0108] When used herein to refer to structured lighting parameters, the term "frequency" is intended to refer to the reciprocal of the spacing between the stripes or lines of a structured lighting pattern (e.g., a stripe or grid pattern), because frequency and period are inversely related. For example, a pattern with a larger spacing between stripes will have a lower frequency than a pattern with a smaller spacing between stripes.

[0109] As used herein to refer to structured lighting parameters, the term "phase" is intended to refer to the phase of the structured lighting pattern illuminating the sample. For example, the phase can be changed by transforming the structured lighting pattern relative to the illuminated sample.

[0110] When used herein to refer to structured lighting parameters, the term "orientation" is intended to refer to the relative orientation between the structured lighting pattern (e.g., a stripe or grid pattern) and the sample illuminated by the pattern. For example, the orientation can be changed by rotating the structured lighting pattern relative to the illuminated sample.

[0111] As used herein when referring to structured illumination parameters, the term "predict" (or "predicting") is intended to mean calculating the value of a parameter without directly measuring the parameter or estimating the parameter from a captured image corresponding to the parameter. For example, the phase of a structured illumination pattern can be predicted at time t1 by interpolation between phase values directly measured or estimated (e.g., from a captured phase image) at times t2 and t3, where t2 < t1 < t3. As another example, the frequency of a structured illumination pattern can be predicted at time t1 by extrapolation of frequency values directly measured or estimated (e.g., from a captured phase image) at times t2 and t3, where t2 < t3 < t1.

[0112] As used herein when referring to light diffracted by a diffraction grating, the term "order" or "order number" is intended to mean the number of integer wavelengths that represent the path length difference of light from adjacent slits or structures of a diffraction grating for constructive interference. The interaction of an incident light beam on a repetitive series of grating structures or other beam splitting structures can redirect or diffract portions of the light beam from the original beam into predictable angular directions. The term "zero order" or "zero order maximum" is intended to refer to the central bright fringe emitted by a diffraction grating where there is no diffraction. The term "first order" refers to two bright fringes diffracted to either side of the zero order fringe, where the path length difference is ±1 wavelength. Higher orders are diffracted from the original beam at larger angles. The characteristics of the grating can be manipulated to control how much beam intensity is directed into different orders. For example, a phase grating can be fabricated to maximize the transmission of the ±1 order beams and minimize the transmission of the zero order beam.

[0113] As used herein when referring to a sample, the term "feature" is intended to denote a point or region in a pattern that can be distinguished from other points or regions based on relative position. A single feature can include one or more specific types of molecules. For example, a feature can contain a single target nucleic acid molecule having a specific sequence, or a feature can contain several nucleic acid molecules having the same sequence (and / or its complementary sequence).

[0114] As used herein, the term "xy plane" is intended to denote a 2-dimensional region defined by the straight axes x and y in a Cartesian coordinate system. When used in reference to a detector and an object being observed by the detector, this region can be further specified as being orthogonal to the beam axis, or orthogonal to the direction of observation between the detector and the object being detected.

[0115] As used herein, the term "z coordinate" is intended to denote information specifying the position of a point, line, or region along an axis orthogonal to the xy plane. In a particular embodiment, the z axis is orthogonal to the region of the object being observed by the detector. For example, the direction of focus of an optical system can be specified along the z axis.

[0116] As used herein, the term "optically couple" is intended to mean that one element is adapted to impart light directly or indirectly to another element.

[0117] SIM hardware

[0118] This section is based on the disclosure made by the applicant in U.S. Provisional Application No. 62 / 692,303 (Unpublished), filed June 29, 2018. Structured Illumination Microscopy (SIM) describes a technique by which spatially structured (i.e., patterned) light can be used to image a sample to increase the lateral resolution of a microscope by two times or more. Figure 1A The diagram illustrates moiré fringes (or moiré patterns) formed using a grating with 1D modulation. The surface containing the sample is illuminated by a structured pattern of light intensity (typically sinusoidal) to achieve the formation of the moiré fringes. Figure 1A Two sinusoidal modes are shown when their frequency vectors are (a) parallel and (b) non-parallel in frequency or reciprocal space. Figure 1A This is a typical illustration of shifting high-frequency spatial information to lower, optically detectable frequencies. The new signal is called moiré fringes or aliasing. In some cases, during sample imaging, three fringe pattern images of the sample are acquired at various pattern phases (e.g., 0°, 120°, and 240°), such that each location on the sample is exposed to a range of illumination intensities. This process is repeated by rotating the pattern orientation around the optical axis to two (e.g., 45°, 135°) or three (e.g., 0°, 60°, and 120°) separate angles. The captured images (e.g., six or nine images) can be assembled into a single image with an extended spatial frequency bandwidth, which can then be re-transformed into real space to generate an image with a higher resolution than that captured by conventional microscopy.

[0119] In some specific implementations of SIM systems, a linearly polarized beam is guided through a beam splitter that divides the beam into two or more individual orders. These orders can be combined and projected onto an imaging sample as an interference fringe pattern with sinusoidal intensity variations. A diffraction grating is an example of a beam splitter that can produce beams with high coherence and stable propagation angles. When two such beams are combined, their interference creates a uniform, regularly repeating fringe pattern, where the spacing is determined by factors including the angle between the interfering beams.

[0120] Figure 1B An example of 2D structured lighting is presented. 2D structured lighting can be formed by two orthogonal 1D diffraction gratings superimposed on each other. As with the case of 1D structured lighting patterns, 2D lighting patterns can be generated by using 2D diffraction gratings or by interference between four beams that create a regularly repeating fringe pattern.

[0121] During the capture and / or subsequent assembly or reconstruction of images into a single image with an extended spatial frequency bandwidth, the following structured illumination parameters may need to be considered: the orientation or angle of the fringe pattern, also known as the illumination peak angle relative to the illumination sample; the spacing between adjacent fringes, known as the illumination peak spacing (i.e., the frequency of the fringe pattern); and the phase displacement of the structured illumination pattern. In an ideal imaging system, each of these parameters will not drift or otherwise change over time, unaffected by factors such as mechanical instability and thermal variations, and the precise SIM frequency, phase, and orientation parameters associated with a given image sample will be known. However, these parameters can drift or otherwise change over time due to factors such as mechanical instability of the excitation beam path and / or thermal expansion / contraction of the imaging sample.

[0122] Therefore, SIM imaging systems may need to estimate structured illumination parameters to account for their changes over time. Since many SIM imaging systems do not perform SIM image processing in real time (e.g., they process captured images offline), such SIM systems can spend a considerable amount of computation time processing SIM images to estimate the structured illumination parameters of those images.

[0123] Figures 2 to 4B Three such example SIM imaging systems are shown. It should be noted that although these systems are described primarily in the context of SIM imaging systems that generate 1D illumination patterns, the techniques disclosed herein can be implemented with SIM imaging systems that generate higher-dimensional illumination patterns, such as two-dimensional mesh patterns.

[0124] Figure 2 A structured illumination microscopy (SIM) imaging system 100, which enables the prediction of structured illumination parameters according to some specific implementations described herein, is illustrated. For example, system 100 may be a structured illumination fluorescence microscopy system that utilizes spatially structured excitation light to image biological samples.

[0125] exist Figure 2 In this example, the light emitter 150 is configured to output a beam collimated by the collimating lens 151. The collimated light is structured (patterned) by the optical structured optics 155 and guided by the dichroic mirror 160 through the objective lens 142 onto the sample in the sample container 110 positioned on the stage 170. In the case of a fluorescent sample, the sample fluoresces in response to the structured excitation light, and the resulting light is collected by the objective lens 142 and guided to the image sensor of the camera system 140 to detect the fluorescence.

[0126] The optical structure optics 155 includes one or more optical diffraction gratings or other beam-splitting elements (e.g., beam splitter cubes or plates) to generate a pattern (e.g., stripes, typically sinusoidal) of light projected onto a sample in the sample container 110. The diffraction grating can be a one-dimensional or two-dimensional transmission or reflection grating. The diffraction grating can be a sinusoidal amplitude grating or a sinusoidal phase grating.

[0127] In some embodiments, the diffraction grating may not utilize a rotating stage to change the orientation of the structured illumination pattern. In other embodiments, the diffraction grating may be mounted on a rotating stage. In some embodiments, the diffraction grating may be fixed during operation of the imaging system (i.e., no rotation or linear motion is required). For example, in a particular embodiment further described below, the diffraction grating may comprise two fixed one-dimensional transmission diffraction gratings (e.g., a horizontal diffraction grating and a vertical diffraction grating) oriented perpendicularly to each other.

[0128] like Figure 2 As shown in the example, the optical structure optics 155 outputs a first-order diffracted beam (e.g., m = ±1 order) while blocking or minimizing all other orders, including the zeroth order. However, in an alternative embodiment, additional optical orders can be projected onto the sample.

[0129] During each imaging cycle, the imaging system 100 utilizes the optical structure optics 155 to acquire multiple images of various phases, wherein the fringe pattern is laterally shifted in the modulation direction (e.g., in the xy plane and perpendicular to the fringe), and this process is repeated once or multiple times by rotating the pattern orientation about the optical axis (i.e., relative to the xy plane of the sample). The captured images can then be computationally reconstructed to generate a higher resolution image (e.g., an image with approximately twice the lateral spatial resolution of a single image).

[0130] In system 100, the light emitter 150 can be an incoherent light emitter (e.g., emitting a beam of light output from one or more excitation diodes) or a coherent light emitter (such as an emitter emitting light output from one or more lasers or laser diodes). As shown in the example of system 100, the light emitter 150 includes an optical fiber 152 for guiding the beam to be output. However, other configurations of the light emitter 150 may be used. In specific implementations of structured illumination utilizing a multi-channel imaging system (e.g., a multi-channel fluorescence microscope utilizing light of multiple wavelengths), the optical fiber 152 may be optically coupled to multiple different light sources (not shown), each emitting light of a different wavelength. Although system 100 is shown as having a single light emitter 150, in some specific implementations, multiple light emitters 150 may be included. For example, multiple light emitters may be included in the case of a structured illumination imaging system utilizing multiple arms, as discussed further below.

[0131] In some implementations, system 100 may include a telescope lens 156, which may include lens elements to articulate along the z-axis to adjust the structured beam shape and path. For example, components of the telescope lens may articulate to accommodate a range of sample thicknesses (e.g., different coverslip thicknesses) in the container 110.

[0132] In an example of system 100, a fluid delivery module or device 190 may direct the flow of reagents (e.g., fluorescently labeled nucleotides, buffers, enzymes, lysis reagents, etc.) to (and through) sample container 110 and waste valve 120. Sample container 110 may include one or more substrates on which a sample is provided. For example, in the case of a system for analyzing a large number of different nucleic acid sequences, sample container 110 may include one or more substrates on which the nucleic acid to be sequenced is bound, attached, or associated. The substrate may include any inert substrate or matrix on which the nucleic acid can attach, such as glass surfaces, plastic surfaces, latex, dextran, polystyrene surfaces, polypropylene surfaces, polyacrylamide gels, gold surfaces, and silicon wafers. In some applications, the substrate is in channels or other regions located at multiple locations on sample container 110 formed in a matrix or array. System 100 may also include a temperature station actuator 130 and a heater / cooler 135 that optionally regulate the temperature of the fluid conditions within sample container 110.

[0133] In a particular embodiment, sample container 110 may be implemented as a patterned flow cell, comprising a translucent cover, a substrate, and a liquid contained therebetween, with the biological sample located on the inner surface of the translucent cover or the inner surface of the substrate. The flow cell may include a large number (e.g., thousands, millions, or billions) of pores (also referred to as nanopores) or regions designed on the substrate to define an array (e.g., a hexagonal array, a rectangular array, etc.). Each region may form a cluster (e.g., a monoclonal cluster) of a biological sample (such as DNA, RNA, or another genomic material) that can be sequenced, for example, using sequencing-by-synthesis. The flow cell may be further divided into multiple spaced-apart lanes (e.g., eight lanes), each lane comprising a cluster of hexagonal arrays.

[0134] Sample container 110 may be mounted on sample stage 170 to provide movement and alignment of sample container 110 relative to objective lens 142. The sample stage may have one or more actuators to allow it to move in any of the three dimensions. For example, in a Cartesian coordinate system, actuators may be provided to allow the sample stage to move relative to the objective lens in the X, Y, and Z directions. This allows one or more sample positions on sample container 110 to be positioned optically aligned with objective lens 142. Movement of sample stage 170 relative to objective lens 142 may be achieved by moving the sample stage itself, the objective lens, some other component of the imaging system, or any combination thereof. Further embodiments may include moving the entire imaging system onto a fixed sample. Alternatively, sample container 110 may be fixed during imaging.

[0135] In some implementations, a focusing (z-axis) component 175 may be included to control the positioning of the optical components relative to the sample container 110 in the focusing direction (typically referred to as the z-axis or z-direction). The focusing component 175 may include one or more actuators physically coupled to the optical stage or sample stage, or both, to move the sample container 110 on the sample stage 170 relative to the optical components (e.g., objective lens 142) to provide appropriate focusing for imaging operations. For example, the actuator may be physically coupled to the corresponding sample stage, such as through mechanical, magnetic, fluid, or other attachment or direct or indirect contact with the sample stage. The one or more actuators may be configured to move the sample stage in the z-direction while maintaining the sample stage in the same plane (e.g., maintaining a horizontal or horizontal orientation perpendicular to the optical axis). The one or more actuators may also be configured to tilt the sample stage. For example, this can be done such that the sample container 110 can be dynamically leveled to account for any slope on its surface.

[0136] Structured light emitted from the test sample at the imaged sample location can be guided by a dichroic mirror 160 to one or more detectors of the camera system 140. In some embodiments, a filter switching assembly 165 with one or more emission filters may be included, wherein the one or more emission filters can be used to allow specific emission wavelengths to pass and block (or reflect) other emission wavelengths. For example, the one or more emission filters can be used to switch between different channels of the imaging system. In a particular embodiment, the emission filters may be implemented as dichroic mirrors that guide emitted light of different wavelengths to different image sensors of the camera system 140.

[0137] Camera system 140 may include one or more image sensors to monitor and track imaging (e.g., sequencing) of sample container 110. Camera system 140 may be implemented, for example, as a charge-coupled device (CCD) image sensor camera, but other image sensor technologies (e.g., active pixel sensors) may be used.

[0138] Output data (e.g., images) from camera system 140 can be transmitted to a real-time SIM imaging component 191, which can be implemented as a software application, as further described below. This software application can reconstruct the images captured during each imaging cycle to create an image with high spatial resolution. The reconstructed image can take into account changes in predicted structure illumination parameters over time. Additionally, SIM imaging component 191 can be used to track predicted SIM parameters and / or predict SIM parameters given previously estimated and / or predicted SIM parameters.

[0139] A controller 195 may be provided to control the operation of the structured illumination imaging system 100, including synchronizing various optical components of the system 100. The controller may be implemented to control aspects of system operation, such as the configuration of the optical structured optics 155 (e.g., selection and / or linear conversion of the diffraction grating), movement of the barrel lens 156, focusing, stage movement, and imaging operations. The controller may also be implemented to control the hardware elements of the system 100 to correct for changes in structured illumination parameters over time. For example, the controller may be configured to transmit control signals to motors or other devices that control the configuration of the optical structured optics 155, the stage 170, or some other element of the system 100 to correct or compensate for changes in the phase, frequency, and / or orientation of the structured illumination over time. In some implementations, these signals may be transmitted based on structured illumination parameters predicted using the SIM imaging component 191. In some implementations, the controller 195 may include a memory for storing predicted and / or estimated structured illumination parameters corresponding to different times and / or sample locations.

[0140] In various implementations, controller 195 may be implemented using hardware, algorithms (e.g., machine-executable instructions), or a combination thereof. For example, in some implementations, the controller may include one or more CPUs, GPUs, or processors with associated memory. Alternatively, the controller may include hardware or other circuitry to control operations, such as a computer processor and a non-transitory computer-readable medium storing machine-readable instructions therein. For example, the circuitry may include one or more of the following: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), programmable logic arrays (PLAs), programmable array logic (PALs), and other similar processing devices or circuitry. Alternatively, the controller may include a combination of this circuitry and one or more processors.

[0141] Figure 3This is an optical diagram illustrating an example optical configuration of a dual-arm SIM imaging system 200 with implementable structured illumination parameter prediction according to some specific embodiments described herein. The first arm of system 200 includes a light emitter 210A, a first optical collimator 220A for collimating the light output from the light emitter 210A, a diffraction grating 230A in a first orientation relative to the optical axis, a rotating mirror 240A, and a second optical collimator 250A. The second arm of system 200 includes a light emitter 210B, a first optical collimator 220B for collimating the light output from the light emitter 210B, a diffraction grating 230B in a second orientation relative to the optical axis, a rotating mirror 240B, and a second optical collimator 250B. Although a diffraction grating is shown in this example, in other specific embodiments, other beam-splitting elements such as beam splitter cubes or plates may be used to separate the light received at each arm of the SIM imaging system 200.

[0142] Each optical transmitter 210A to 210B can be an incoherent optical transmitter (e.g., emitting a beam of light output by one or more excitation diodes) or a coherent optical transmitter (such as an transmitter of light output by one or more lasers or laser diodes). In the example of system 200, each optical transmitter 210A to 210B is an optical fiber whose output is a beam collimated by a corresponding collimator 220A to 220B.

[0143] In some embodiments, each fiber may be optically coupled to a corresponding light source (not shown), such as a laser. During imaging, each fiber can be turned on or off using a high-speed shutter (not shown) positioned in the optical path between the fiber and the light source, or by pulses applied to the corresponding light source at a predetermined frequency during imaging. In some embodiments, each fiber may be optically coupled to the same light source. In such embodiments, a beam splitter or other suitable optical element may be used to guide light from the light source into each of the fibers. In such examples, a high-speed shutter (not shown) positioned in the optical path between the fiber and the beam splitter can be used to turn each fiber on or off.

[0144] In the example SIM imaging system 200, the first arm includes a fixed vertical grating 230A to project a structured illumination pattern or grating pattern onto a first orientation (e.g., a vertical stripe pattern) on the sample, and the second arm includes a fixed horizontal grating 230B to project a structured illumination pattern or grating pattern onto a second orientation (e.g., a horizontal stripe pattern) on the sample 271. The gratings in the SIM imaging system 200 do not require mechanical rotation or conversion, which provides improved system speed, reliability, and repeatability.

[0145] In an alternative embodiment, gratings 230A and 230B may be mounted on respective linear motion stages that can be switched to change the optical path length (and thus the phase) of the light emitted by gratings 230A and 230B. The axes of motion of the linear motion of these stages may be perpendicular to or otherwise biased to the orientation of their respective gratings to achieve a change in the pattern of the gratings along sample 271.

[0146] Gratings 230A to 230B may be transmission diffraction gratings, comprising multiple diffraction elements (e.g., parallel slits or grooves) formed on a glass substrate or other suitable surface. The grating may be implemented as a phase grating providing a periodic variation in the refractive index of the grating material. The grooves or feature spacing may be selected to diffract light at a suitable angle and tuned to the minimum resolvable feature size of the imaging sample for operation of the SIM imaging system 200. In other specific embodiments, the grating may be a reflection diffraction grating.

[0147] In the example of SIM imaging system 200, the vertical and horizontal patterns are offset by approximately 90 degrees. In other specific implementations, other orientations of the gratings can be used to create an offset of approximately 90 degrees. For example, the gratings may be oriented such that they project an image offset by ±45 degrees from the x or y plane of sample 271. The configuration of example SIM imaging system 200 may be particularly advantageous in the case of a regularly patterned sample 271 with features on a rectangular grid, since structured resolution enhancement can be achieved using only two vertical gratings (e.g., a vertical grating and a horizontal grating).

[0148] In the example of system 200, gratings 230A to 230B are configured to diffract the input beam into multiple orders (e.g., 0th order, ±1st order, ±2nd order, etc.), where the ±1st order can be projected onto sample 271. As shown in this example, vertical grating 230A diffracts the collimated beam into a first-order diffracted beam (±1st order), unfolding one order on the plane of the page, and horizontal grating 230B diffracts the collimated beam into a first-order diffracted beam, unfolding these orders above and below the plane of the page (i.e., in a plane perpendicular to the page). To improve the efficiency of the system, the zeroth-order beam and all other higher-order beams (i.e., ±2nd order or higher) can be blocked (i.e., filtering out the illumination pattern projected onto sample 271). For example, beam-blocking elements (not shown), such as order filters, can be inserted into the optical path after each diffraction grating blocks the 0th-order beam and higher-order beams. In some specific implementations, diffraction gratings 230A to 230B can be configured to diffract the beam to only the first order, and the 0th order (undiffracted beam) can be blocked by some beam blocking elements.

[0149] Each arm includes optical phase modulators or phase shifters 240A to 240B to phase-shift the diffracted light output by each of the gratings 230. For example, during structured imaging, the optical phase of each diffracted beam can be shifted by a portion of the spacing (λ) of each fringe of the structured pattern (e.g., 1 / 2, 1 / 3, 1 / 4, etc.). Figure 3 In the example, phase modulators 240A and 240B are implemented as rotating windows that can be rotated and adjusted using a galvanometer or other rotary actuator to control the optical path length of each diffracted beam. For example, window 240A can be rotated about a vertical axis to shift the image projected onto sample 271 by vertical grating 230A to the left or right, and window 240B can be rotated about a horizontal axis to shift the image projected onto sample 271 by horizontal grating 230B up or down.

[0150] In other embodiments, other phase modulators that alter the optical path length of the diffracted light (e.g., linear transition stages, wedges, etc.) may be used. Furthermore, although optical phase modulators 240A to 240B are shown positioned after gratings 230A to 230B, in other embodiments they may be placed at other locations within the illumination system.

[0151] In alternative embodiments, the single-phase modulator can operate in two different directions for different fringe patterns, or the single-phase modulator can use a single movement to adjust the two path lengths. For example, a large rotating optical window can be placed after a mirror 260 with aperture 261. In this case, a large window can be used instead of windows 240A and 240B to adjust the phase of two sets of diffracted beams output from vertical and horizontal diffraction gratings. The rotation axis of the large rotating window is not parallel to the optical axis of one of the gratings, but can be offset by 45 degrees (or some other angular offset) with the optical axis of each of the vertical and horizontal gratings to allow phase offset in both directions along a common rotation axis of the large window. In some embodiments, the large rotating window can be replaced by a wedge-shaped optics that rotates about the nominal beam axis.

[0152] In example system 200, mirror 260 with aperture 261 combines the two arms into the optical path in a non-destructive manner (e.g., with no significant loss of optical power except for minor absorption in the reflective coating). Mirror 260 can be positioned such that the diffraction order from each of the gratings is spatially resolved and can block unwanted orders. Mirror 260 allows first-order light output from the first arm to pass through aperture 261. Mirror 260 reflects first-order light output from the second arm. Thus, the structured illumination pattern can be switched from a vertical orientation (e.g., grating 230A) to a horizontal orientation (e.g., grating 230B) by opening or closing each emitter, or by opening and closing the optical shutter that guides the light source through the fiber optic cable. In other embodiments, the structured illumination pattern can be switched using an optical switch to change the arms of the illumination sample.

[0153] The example imaging system 200 also shows a barrel lens 265, a semi-reflective mirror 280, an objective lens 270, and a camera 290. For example, the barrel lens 265 may be implemented to articulate along the z-axis to adjust the shape and path of the structured beam. The semi-reflective mirror 280 may be a dichroic mirror to reflect the structured illumination light received from each arm downwards into the objective lens 270 to project onto the sample 271 and allow light emitted from the sample 271 (e.g., fluorescence, which emits at a different wavelength than the excitation) to pass through to the camera 290.

[0154] Output data (e.g., images) from camera 290 can be transmitted to a real-time SIM imaging component (not shown), which can be implemented as a software application, as further described below. This software application can reconstruct the images captured during each imaging cycle to create an image with high spatial resolution. The reconstructed image can take into account changes in predicted structure illumination parameters over time. Additionally, the real-time SIM imaging component can be used to track predicted SIM parameters and / or predict SIM parameters given previously estimated and / or predicted SIM parameters.

[0155] A controller (not shown) may be provided to control the operation of the structured illumination imaging system 200, including synchronizing the various optical components of the system 200. The controller may be implemented to control aspects of system operation, such as the configuration of each optical arm (e.g., opening / closing each optical arm during phase image capture, actuation of phase modulators 240A to 240B), movement of the barrel lens 265, stage movement of the sample 271 (if any stage is used), and imaging operations. The controller may also be implemented to control the hardware elements of the system 200 to correct for changes in structured illumination parameters over time. For example, the controller may be configured to transmit control signals to devices (e.g., phase modulators 240A to 240B) that control the configuration of each optical arm or some other element of the system 100 to correct or compensate for changes in the phase, frequency, and / or orientation of the structured illumination over time. For example, when gratings 230A to 230B are mounted on a linear motion stage (e.g., instead of phase modulators 240A to 240B), the controller can be configured to control the linear motion stage to correct or compensate for phase changes. In specific implementations, these signals can be transmitted based on structured illumination parameters predicted using the SIM imaging component. In some implementations, the controller may include memory for storing predicted and / or estimated structured illumination parameters corresponding to different times and / or sample locations.

[0156] It should be noted that, for simplicity, the optical components of the SIM imaging system 200 can be omitted from the foregoing discussion. Furthermore, although the system 200 is shown as a single-channel system in this example, in other specific implementations it can be implemented as a multi-channel system (e.g., by using two different cameras and light sources emitting at two different wavelengths).

[0157] Although System 200 illustrates a two-arm structured illumination imaging system comprising two gratings oriented at two different angles, it should be noted that in other specific embodiments, the techniques described herein can be implemented using more than two arms. In the case of regularly patterned samples characterized on a rectangular grid, resolution enhancement can be achieved using only two vertical angles (e.g., a vertical grating and a horizontal grating), as described above. On the other hand, for image resolution enhancement in all directions for other samples (e.g., hexagonal patterned samples), three illumination peak angles can be used. For example, a three-arm system may include three light emitters and three fixed diffraction gratings (one per arm), wherein each diffraction grating is oriented around the optical axis of the system to project a corresponding pattern orientation (e.g., a 0° pattern, a 120° pattern, or a 240° pattern) onto the sample. In such systems, additional images of the additional gratings can be combined into the system in a non-destructive manner using an additional mirror with an aperture. Alternatively, such systems may utilize one or more polarization beam splitters to combine the images of each of the gratings.

[0158] Figure 4A and Figure 4B This is a schematic diagram illustrating an example optical configuration of a dual-optical grating sliding SIM imaging system 500 capable of implementing structured illumination parameter prediction according to some specific embodiments described herein. In the example system 500, all changes (e.g., pattern phase shift or rotation) to the structured illumination pattern or grating pattern projected onto the sample 570 can be made by linearly switching the motion stage 530 along a single axis of motion to select one of gratings 531 or 532 (i.e., selecting the grating orientation) or phase-shifted gratings 531 to 532.

[0159] System 500 includes a light emitter 510 (e.g., an optical fiber optically coupled to a light source), a first optical collimator 520 for collimating the light output from the light emitter 510, a linear motion stage 530 equipped with a first diffraction grating 531 (e.g., a horizontal grating) and a second diffraction grating 532 (e.g., a vertical grating), a barrel lens 540, a semi-reflective mirror 550 (e.g., a dichroic mirror), an objective lens 560, a sample 570, and a camera 580. For simplicity, Figure 4A The optical components of the SIM imaging system 500 may be omitted. Furthermore, although the system 500 is shown as a single-channel system in this example, it may be implemented as a multi-channel system in other specific embodiments (e.g., by using two different cameras and light sources emitting at two different wavelengths).

[0160] like Figure 4A As shown, grating 531 (e.g., a horizontal diffraction grating) can diffract a collimated beam into a first-order diffracted beam (on the plane of the page). Figure 4B As shown, a diffraction grating 532 (e.g., a vertical diffraction grating) diffracts the beam into first order (above and below the plane of the page). In this configuration, only a single optical arm with a single emitter 510 (e.g., an optical fiber) and a single linear motion stage is required to image the sample 570, which provides system advantages such as a reduction in the number of moving system components to improve speed, complexity, and cost. Additionally, in system 500, the absence of a polarizer provides the previously mentioned advantage of high optical efficiency. The configuration of the example SIM imaging system 200 may be particularly advantageous for regularly patterned samples 570 with features on a rectangular grid, as structured resolution enhancement can be achieved using only two vertical gratings (e.g., a vertical grating and a horizontal grating).

[0161] To improve system efficiency, the zeroth-order diffracted beam from each grating output and all other higher-order diffracted beams (i.e., ±2nd order or higher) can be blocked (i.e., filtering out the illumination pattern projected onto sample 570). For example, beam-blocking elements (not shown), such as order filters, can be inserted after stage 530. In some specific embodiments, diffraction gratings 531 to 532 can be configured to diffract the beam to only the first order, and the zeroth-order (undiffracted beam) can be blocked by some beam-blocking elements.

[0162] In the example of system 500, the two gratings can be arranged at approximately ±45° to the axis of motion (or offset at other angles to the axis of motion, such as approximately +40° / -50°, approximately +30° / -60°, etc.), such that for each grating 531 and 532, a phase shift can be achieved along a single axis of linear motion. In some specific implementations, the two gratings can be combined into a single physical optical element. For example, one side of the physical optical element may have a structured illumination pattern or grating pattern in a first orientation, and the adjacent side of the physical optical element may have a structured illumination pattern or grating pattern in a second orientation orthogonal to the first orientation.

[0163] A single linear stage 530 may include one or more actuators to allow it to move relative to the sample plane along the X-axis or relative to the sample plane along the Y-axis. During operation, the linear stage 530 provides sufficient travel (e.g., about 12 mm to 15 mm) and accuracy (e.g., repeatability of less than 0.5 micrometers) to result in accurate illumination patterning for effective image reconstruction. In specific implementations utilizing the stage 530 in automated imaging systems such as fluorescence microscopes, it can be configured to provide high-speed operation, minimal vibration generation, and long service life. In specific implementations, the linear stage 530 may include crossed roller bearings, a linear motor, a high-precision linear encoder, and / or other components. For example, the stage 530 may be implemented as a high-precision stepper or piezoelectric stage that can be converted using a controller.

[0164] Output data (e.g., images) from camera 580 can be transmitted to a real-time SIM imaging component (not shown), which can be implemented as a software application, as further described below. This software application can reconstruct the images captured during each imaging cycle to create an image with high spatial resolution. The reconstructed image can take into account changes in predicted structure illumination parameters over time. Additionally, the real-time SIM imaging component can be used to track predicted SIM parameters and / or predict SIM parameters given previously estimated and / or predicted SIM parameters.

[0165] A controller (not shown) may be provided to control the operation of the structured illumination imaging system 500, including synchronizing the various optical components of the system 500. The controller may be implemented to control aspects of system operation, such as switching of the linear stage 530, movement of the barrel lens 540, stage movement of the sample 570 (if any stage is used), and imaging operations. The controller may also be implemented to control the hardware elements of the system 500 to correct for changes in structured illumination parameters over time. For example, the controller may be configured to transmit control signals to devices (e.g., the linear stage 530) to correct or compensate for changes in the phase, frequency, and / or orientation of the structured illumination over time. In specific implementations, these signals may be transmitted based on structured illumination parameters predicted using the SIM imaging components. In some specific implementations, the controller may include memory for storing predicted and / or estimated structured illumination parameters corresponding to different times and / or sample locations.

[0166] although Figures 4A to 4B The example illustrates a dual-optical-grating sliding imaging system that can implement structured illumination parameter prediction, but structured illumination parameter prediction can be implemented in a SIM imaging system using linear motion actuators with more than two diffraction gratings.

[0167] Projection parameter estimation

[0168] Parameter estimation may be needed for SIM image processing to correct for unwanted variations in structured lighting parameters over time. For example, Figures 5A to 5C This illustrates a SIM imaging system that projects a one-dimensional structured illumination pattern onto a regularly patterned sample, where the frequency or illumination peak spacing ( Figure 5A ), phase or phase shift ( Figure 5B ) and orientation or illumination peak angle ( Figure 5C Undesirable changes that can occur over time. In particular, Figure 5A Sample 50 is shown, in which features 51 are illuminated by a one-dimensional structured illumination pattern with stripes 60 before and after a frequency shift. Before any frequency shift, adjacent stripes 60 have a spacing or center-to-center spacing P corresponding to an initial frequency f. Over time, as the temperature in the system changes, the spacing P may increase or decrease. For example, thermal expansion may cause the spacing P to increase to P+ΔP1, correspondingly decreasing the frequency f to f-Δf1. Conversely, thermal contraction may cause the spacing P to decrease to P-ΔP1, correspondingly increasing the frequency f to f+Δf2.

[0169] Figure 5BThe illustration shows a sample 50 illuminated by a one-dimensional structured illumination pattern with stripes 60 before and after a phase shift. As shown, before the phase shift, a first phase state Φ corresponds to stripes of every other column of features 51 in a fully illuminated sample 50. Over time, the position of stripes 60 relative to the sample 50 can shift, such that all phase images are offset by ΔΦ. For example, mechanical vibrations in a SIM imaging system (e.g., in the excitation beam path), inaccuracies in the converter stage used by the grating or sample stage, thermal variations, and / or other factors can cause undesirable phase shifts. After the phase shift ΔΦ, the first phase state becomes Φ+ΔΦ, and the stripes are no longer centered on features every other column.

[0170] Figure 5C The image shows a sample 50 illuminated by a one-dimensional structured illumination pattern with stripes 60 before and after an orientation change. As shown, before the orientation change, the stripes are perfectly vertical relative to the sample 50. Over time, the orientation may change due to factors such as variations in the excitation beam path, sample movement, thermal changes, and / or other factors. After an orientation rotation angle Δθ, the stripes are no longer perfectly vertical relative to the sample.

[0171] Parameter estimation during SIM imaging to accurately account for variations in structured illumination parameters as described above helps ensure artifact-free and accurate reconstruction of images from a set of sampled images. However, this process can be computationally expensive and is often performed after image acquisition. For time-critical SIM imaging systems involving real-time image processing and reconstruction, and therefore real-time estimation of parameters such as frequency, phase, orientation, and modulation order, these computational requirements can lead to a loss of data throughput (e.g., less data may be processed per unit time). In such systems, the rate at which samples are imaged can exceed the rate at which structured illumination parameters can be estimated directly from sampled images. Therefore, a method for generating parameter estimates with low complexity and low processing time is needed.

[0172] Therefore, specific implementations of the technology disclosed herein are aimed at predicting structured lighting parameters at specific time points, spatial points, and / or temperature points using estimates of structured lighting parameters obtained from images captured by a structured lighting system. Specific implementations are aimed at predicting structured lighting frequency, phase, orientation, and / or modulation order parameters.

[0173] According to some specific implementations, structured lighting parameters can be predicted for a given time point, spatial point, and / or temperature point by extrapolating estimates of structured lighting parameters from image captures. For example, a first frequency can be estimated from a first sampled image, a second frequency can be estimated from a second sampled image, and the frequency corresponding to a time point between the first and second captured images (e.g., the frequency of images taken between the first and second images) can be predicted by extrapolating using at least a determined rate of frequency change between the first and second captured images.

[0174] According to some specific implementations, structured lighting parameters can be predicted for a given time point, spatial point, and / or temperature point by extrapolating estimates of structured lighting parameters obtained from two image captures. For example, a first orientation can be estimated from a first sampled image, a second orientation can be estimated from a second sampled image, and the orientation corresponding to a time point after the first and second capture images (e.g., the orientation of a third image taken after the first and second images) can be predicted by extrapolating using at least a determined rate of orientation change from the first capture image to the second capture image. As a second example, the first orientation can be estimated from the first sampled image, the second orientation can be estimated from the second sampled image, and the orientation corresponding to a time point after the first and second capture images (e.g., the orientation of a third image taken after the first and second images) can be predicted by holding the values ​​from the second capture image.

[0175] In practice, the estimated and predicted structured lighting parameters can be used to narrow down the search space for other predicted structured lighting parameters. For example, given the estimated values ​​of the structured lighting parameters at a first time, space, and / or temperature point, and considering the predicted or estimated values ​​at the first time, space, and / or temperature point, the values ​​of the structured lighting parameters at a second time, space, and / or temperature point near the first time, space, and / or temperature point can be predicted.

[0176] In practice, the estimated and predicted structured lighting parameters can be stored in the memory of the structured lighting system for later use. For example, the predicted and estimated parameters can be stored in a history file (such as a lookup table). The predicted parameters stored in memory can be determined based on the estimated parameters or set based on the physical characterization of the structured lighting system. For example, the nominal grid spacing of the structured lighting system can be stored. The stored parameters can then be referenced to perform operations such as calibrating image reconstruction, providing feedback to hardware components to correct for variations in the structured lighting parameters, and narrowing the search space when predicting additional structured lighting parameters.

[0177] Lighting stripe pattern

[0178] Figure 6A simplified illumination fringe pattern is illustrated that can be projected onto the plane of sample 271 by a vertical grating 230A and a horizontal grating 230B of a SIM imaging system 200 during one imaging cycle to create a high-resolution image using structured light. In this example, three phase images with a vertical illumination orientation can be captured using the vertical grating 230A, and three phase images with a horizontal illumination orientation can be captured using the horizontal grating 230B. For each orientation, the position of the projected fringes can be shifted by a step phase shift of 1 / 3λ (e.g., by setting the phase modulators 230A or 230B to three different positions) to capture three phase images of the orientation pattern.

[0179] During the capture of each phase image, any light emitted by the sample can be captured by camera 290. For example, fluorescent dyes located at different features of sample 271 may fluoresce, and the resulting light can be collected by objective lens 270 and directed to the image sensor of camera 290 to detect the fluorescence. The six captured images can be used to image the entire sample or the location of a larger sample.

[0180] Once all images have been captured for the imaging cycle (six images in this example), a high-resolution image can be constructed from the captured images. For example, it can be obtained from... Figure 6 High-resolution images are reconstructed from the six images shown. A suitable algorithm can be used to combine these various images to synthesize a single image with a sample having significantly better spatial resolution than any single component image.

[0181] During the construction of a high-resolution image, undesired offsets or variations in structured lighting parameters (e.g., angle, spacing, and phase) can be algorithmically compensated using predicted variations in structured lighting parameters (e.g., angle, spacing, or phase) as predicted according to this disclosure. For example, offsets in phase, angle, and / or spacing of vertically illuminated images and / or horizontally illuminated images can be compensated.

[0182] In some implementations, those variations in the SIM imaging system are compensated for by one or more hardware elements of the control system 200, allowing undesirable shifts or variations in structured illumination parameters to be compensated for before image capture. For example, phase drift of each optical arm can be compensated for by adjusting phase shift elements (e.g., rotating mirrors, linear actuators, etc.) before and / or between image captures in the imaging sequence. In some implementations, a combination of hardware and algorithmic compensation can be implemented.

[0183] Figure 6A specific embodiment of a patterned flow cell comprising symmetrically patterned nanopores is illustrated. In some embodiments, the nanopores can be asymmetrically patterned; for example, the nanopores can be shaped or configured to form elongated structures. An elongated structure refers to a shape whose dimension along a first axis is larger than its dimension along a second axis. The asymmetrically shaped nanopores can be elliptical, rectangular, etc. In such embodiments, the sample can be resolved along the longer side or axis, and SIM is used to increase the resolution along the shorter side. If the shorter side of the nanopore is along the x-axis and the longer side is along the y-axis, the spacing Px of the asymmetrically patterned flow cell is narrow or compact, resulting in increased resolution, while along the y-axis, the spacing Py is larger. Therefore, the resolution is increased along the x-axis, and only three images are captured to adequately resolve the sample contained within the nanopores of the flow cell.

[0184] Figure 7 A simplified illumination fringe pattern is shown that can be projected onto the plane of sample 570 by a first and second diffraction grating of a dual-grating sliding SIM imaging system 500 during image capture in a structured illumination imaging cycle. For example, the SIM imaging system 500 can use the first diffraction grating 531 and the second diffraction grating 532 to generate Figure 7 The lighting pattern shown. (Example) Figure 7 As shown in the example, two gratings project a vertical stripe pattern onto the surface of sample 570 and are arranged at approximately ±45° to the motion axis of linear motion stage 530.

[0185] For example, a first grating (e.g., grating 531) can project first-order illumination fringes onto sample 570. Any light emitted by the sample can be captured by camera 580, and a first phase image of a first pattern (e.g., a 45° pattern) can be captured to create a first phase image. To capture additional phase-shifted images, the pattern projected by the grating can be phase-shifted by converting a linear motion stage. These phase-shifting motions are shown as... Figure 7 Steps 1 and 2 in the process. Phase shift motion can provide small (e.g., about 3 to 5 micrometers or less) grating movement to slightly shift the stripe pattern projected on the grating.

[0186] After capturing all phase-shifted images for the diffraction grating, system 500 can optically couple another diffraction grating to the light source of the imaging system (e.g., from the linear motion stage 530 to switch the diffraction grating) by switching the linear motion stage 530. Figures 4A to 4B (The transformation). Figure 7 In the example, this motion is shown as step 3. With changes in the diffraction grating, a linear motion stage can provide a relatively large transition (e.g., approximately 12 mm to 15 mm).

[0187] A series of phase images can then be captured for the next grating. For example, such as Figure 7As shown, the second diffraction grating can project first-order illumination fringes onto the sample, and the projected fringes can be shifted by the conversion linear motion stage 530 to capture three-phase images of the grating pattern (e.g., Figure 7 Steps 4 and 5).

[0188] Once all images have been captured for the imaging cycle (six images in this example), a high-resolution image can be constructed from the captured images. For example, it can be obtained from... Figure 7 High-resolution images are reconstructed from the six images shown. A suitable algorithm can be used to combine these various images to synthesize a single image with a sample having significantly better spatial resolution than any single component image.

[0189] During the construction of high-resolution images, undesired offsets or variations in structured illumination parameters (e.g., angle, spacing, and phase) can be algorithmically compensated using predicted variations in structured illumination parameters (e.g., angle, spacing, or phase) estimated according to this disclosure. For example, biases in the phase, angle, and / or spacing of vertically illuminated images and / or horizontally illuminated images can be compensated. In one specific implementation, one or more of the phase, angle, and / or spacing biases of the images can be calculated periodically (e.g., during the first cycle of a sequencing run). The frequency of these estimates can be increased or decreased based on environmental factors affecting process stability.

[0190] In some implementations, undesirable shifts or variations in the structured illumination parameters can be compensated for by one or more hardware elements of the control system 500 before image capture. For example, phase drift can be compensated by switching the linear motion stage 530 before and / or between image captures in the imaging sequence. In some implementations, a combination of hardware and algorithmic compensation can be implemented.

[0191] According to the specific implementation described herein, the structured illumination parameters at a specific time point can be predicted using estimates of structured illumination parameters obtained from images captured before and / or after that time point. For example, computational resource constraints can limit the rate at which a SIM imaging system (e.g., system 100, 200, or 500) can directly estimate structured illumination parameters (such as phase, frequency, and / or orientation) from captured images. In some cases, the SIM imaging system can directly estimate or measure the structured illumination parameters for each phase image, in which case it may not be necessary to predict the structured illumination parameters. However, in other cases, the SIM imaging system may only be able to directly estimate or measure the structured illumination parameters for a particular phase image in an imaging cycle, once per imaging cycle or even less frequently (e.g., once every 3, 5, 10, 50, or 100 imaging cycles). In such cases, to keep up with the system's image sampling rate, it may be advantageous to use direct estimates of the structured illumination parameters obtained for a specific time and / or spatial point to predict the structured illumination parameters for other time and / or spatial points.

[0192] To illustrate this principle mathematically, correlation with a reference is one way to estimate structured lighting parameters.

[0193] (1)

[0194] Where h(x) is a reference that may be known or derived from the image data, c(x) is derived from the image data associated with the reference, and f is the value to be estimated (in this example, the frequency). It should be noted that other alternative estimation techniques may be used according to this disclosure.

[0195] In the example of equation (1), a correlation output can be generated for each of the multiple f-hypothesis values. The parameter estimate f can be obtained as the value of f that maximizes the magnitude of the correlation. However, in many cases, it may be necessary to try a large number of f-hypothesis values ​​to maximize the correlation output. A large search space can increase computational requirements and may therefore lead to a reduction in system throughput (i.e., less data processed per unit time).

[0196] Optical sensing (especially lens) distortion

[0197] In the following two sections, we present techniques for estimating parameters for SIM image reconstruction. Some of the techniques disclosed in this invention compensate for fringe peak lines that are distorted or bent due to lens defects. Pattern lines that should be parallel begin to be parallel near the image center but tend to converge or become non-parallel near the edges of the lens. This affects the illumination peak angle or orientation, illumination peak spacing, and phase offset. Figure 8AIn the diagram, we illustrate dividing an image patch into overlapping regions called sub-blocks, sub-windows, or subfields. The sub-blocks are small enough that parameters can be set to provide a satisfactory reconstruction for the entire sub-block. Parameter estimation can be performed in two steps. First, parameter estimation can be performed for the near-center sub-block of the image. Then, parameter estimation can be performed for the other sub-blocks and compared with the near-center sub-block to determine distortion and corrections relative to the parameters of the near-center sub-block.

[0198] To illustrate concrete implementations of these techniques, we present example pseudocode for parameter estimation in the "Sub-block Angle and Spacing Mapping" algorithm (also known as Algorithm 2A) and the "Sub-block Phase Mapping" algorithm (also known as Algorithm 2B). We also present example pseudocode to determine the distortion and correction for distortion in the estimated parameters for non-central sub-blocks compared to near-central sub-blocks in the "Final Distortion Calibration Algorithm" (also known as Algorithm 1).

[0199] Figures 8A to 8C The physical aspects of the full field of view (FOV) are illustrated. In one specific implementation, a rectangular sensor with 5472 pixels × 3694 pixels is used. Of course, square sensors or sensors of different sizes can be used, such as 5472 pixels × 5472 pixels or 4800 pixels × 4800 pixels. When using a rectangular sensor, the distortion is greatest near the lens edge. Lenses are typically circular, so a rectangular sensor will not be as close to the lens edge on the long side as it is on the short side.

[0200] Figure 8A Two illustrations are presented illustrating the distortion of fringe spacing across the entire field of view (FOV). Figure 801 on the left is a simplified depiction of curved parallel lines caused by magnified lens distortion. The depicted lines are intended to be parallel in the image plane. Viewed through the lens, the spacing appears to converge at the left and right ends relative to the center. Figure 805 on the right is another exaggerated example. In this figure, the fringe lines are diagonally oriented between the upper left and lower right corners. The fringe spacing is exaggerated to make it easier to see. Relative to the center, the fringe lines converge at the upper left and lower right corners. For lenses from a particular manufacturer, the fringe pattern may be non-uniform.

[0201] Figure 8B and Figure 8C The measurements of the spacing between the nominal parallel fringe peaks in the image plane are depicted for green and blue laser illumination. A color scale indicates the spacing variation (sensor pixels) between 2.08 and 2.22. In both figures, the color scale indicates a center-to-center spacing of approximately 2.14 between the parallel lines. Figure 8B Irregularities can be seen in the upper right corner under green wavelength illumination. Figure 8CIn the middle, more pronounced irregularities are visible under blue wavelength illumination along the right and left edges. In these figures, the stripe pattern is a series of parallel lines running from the upper left to the lower right of the figure. Therefore, the spacing is orthogonal to... Figure 8C The overall direction of the arrow in the middle is measured. These figures facilitated the correction of distortion caused by the lens. Because the lenses are manufactured and installed separately, the individual systems need to be calibrated and corrected after assembly.

[0202] Drawing in grayscale Figure 8B The figure comprises three conceptual elliptical outlines, their principal axes running diagonally from the upper left to the lower right corner. The outlines roughly indicate areas with varying spacing between the parallel lines. In this provisional application, color figures are used, providing clearer detail in estimating the variation. The stripe pattern is a series of parallel lines at 45° angles, running from the lower left to the upper right corner of the figure. The spacing between the parallel lines increases in the central region, as indicated by the light gray shading. The distortion in spacing between the stripes decreases as we move from the central region to the upper right or lower left region. The principal axes of the three outlines are approximately orthogonal to the direction of the stripes. The spacing distortion decreases as we move away from the area of ​​the central outline. Figure 8C The grayscale also includes the outline. Figure 8C The outline in the text has the same meaning as explained above and indicates areas with different spacing distortion ranges.

[0203] Figure 8D The diagram illustrates sub-blocks or subfields of the full field of view (FOV) within an image block. In this figure, the sub-blocks shown are 512 pixels × 512 pixels. These sub-blocks can be subdivided into fields of view, displayed individually, or overlapped. Sub-blocks can be larger or smaller. For example, 400 pixel × 400 pixel and 1024 pixel × 1024 pixel sub-blocks have been shown as feasible. This figure shows a 5 × 7 sub-block. Larger sensors, as described above, can have 8 × 11 sub-blocks. Other configurations of sub-blocks, such as 3 × 3, 5 × 5, 5 × 7, 9 × 9, and 9 × 16, can be used. Larger sensors can be divided into even more sub-blocks. Sub-blocks can overlap with the optical sensor by at least 2 pixels. Larger and smaller numbers of pixels can be used for overlap between sub-blocks. For example, for a sub-block that is 512 pixels wide, up to 256 pixels of overlap can be used, and for a sub-block that is 1024 pixels wide, up to 256 pixels of overlap can be used. Figure 8B and Figure 8CConsistently, several candidate near-center sub-blocks 803 exist, all at the optimal position of the lens, including the center sub-block in an odd-numbered sub-block array. As used herein, a near-center sub-block includes the center pixel of the sensor or a sub-block adjacent to the center pixel. In some flat optical systems with small errors, sub-blocks far from the center sub-block can be used as references without affecting overall distortion compensation.

[0204] The techniques disclosed in this invention include measuring mapping distortion across the entire field of view, essentially captured by a sensor. The enhanced resolution SIM reconstruction from regularly structured illumination relies on three parameters: fringe spacing, fringe angle, and phase shift of the fringe pattern. These variables are also referred to as the spacing, angle, and phase offset of the structured illumination or grating pattern. Polynomial surfaces can be used to fit the spacing and angle deviations from the central sub-block values ​​across the entire field of view. Both quadratic and cubic surfaces have been investigated. Higher-order polynomials can also be used.

[0205] Both the fringe spacing and fringe angle on an image patch can be fitted using a quadratic surface. Sensitivity analysis shows that the fitting of the quadratic surface is very close to that of the cubic surface. The quadratic surface fitting equation is:

[0206] f(x,y) = c0 + (c1 x) + (c2 y) + (c3 x y) + (c4 x 2 ) + (c5 y 2 (2)

[0207] In later loops of the imaging process, we can characterize the shrinkage or expansion of the distance from the center sub-block due to temperature variations, and apply the same bias correction to the rest of the image. Depending on the shrinkage and expansion of the near-center sub-block, the bias correction can be scaled and transformed accordingly. We observe that the relative bias within the sub-block is constant relative to the center or near-center sub-block used as a reference.

[0208] Figure 9 The surface is depicted using color-coded mapping fitted to the measurement data points indicated in red. Since the surface is three-dimensional, it is recommended to refer to the color version of this plot for an understanding of its contours. (Drawn in grayscale.) Figure 9 The dark shading indicates the darker areas at the top and bottom of the 3D drawing. For example, in the colored version of the figure, Figure 9The dark shading in the upper right of the chart represents dark red, and the dark shading at the bottom represents blue. The United States Patent and Trademark Office (USPTO) saves the color version to the supplementary content tab of the PAIR online system when filing a priority application and makes it publicly available as appropriate.

[0209] In practice, it has been observed that spacing distortion in the field of view can be represented as a bowl or inverted bowl shape, while angular distortion is characterized as a saddle shape. Figure 10A and Figure 10B An inverted bowl shape is shown, representing the measured spacing relative to the spacing in the near-center sub-block. Figure 12A and Figure 12B The saddle shape is shown as the measurement angle relative to the stripe pattern angle in the near-center sub-block.

[0210] exist Figure 10A and Figure 10B In the diagram, the x-axis is along the length of the sensor, and the y-axis is along the width of the sensor. The labels on the x and y axes are sub-block indices. The vertical axis spacing is normalized to a 1.000 spacing on the z-axis between regularly spaced stripes observed in approximately central sub-blocks. Figure 10A The measurement data points, not smoothed by curve fitting, are depicted. It shows an approximately 1.2% variation in fringe spacing observed between measurements at the sensor center and the lower right portion. Figure 10B In the diagram, the actual data points, depicted in blue, are compared to the quadratically fitted surface. In the following sections, we present the measured spacing and angular distortions across the full field of view, compared to the near-center sub-block. Figure 10A and Figure 10B The grayscale version of the chart has dark shading at the top and bottom. The dark shading at the top represents red, and the dark shading at the bottom represents blue.

[0211] Calibration of spacing and angle distortion

[0212] In this section, we further describe techniques for calibrating the full field of view (FOV) on a sensor applied to capture structured illumination and for providing fitting functions for nonlinear effects caused by lens distortion. Structured illumination involves projecting regularly spaced and angled patterns onto an image plane. An image block of the projected structured illumination pattern is captured. A near-center sub-block is selected from the full field of view. The pattern spacing and angle on the sub-block are measured, including variations within the sub-block. We further determine the spacing and pattern angle on other sub-blocks in the full field of view. We express the spacing and pattern angle in the full field of view relative to the spacing and pattern angle in the near-center sub-block. We repeatedly fit a polynomial surface to the spacing and pattern angle essentially across the full field of view.

[0213] Figure 11A , Figure 11B and Figure 11C The measured spacing distortion was compared with quadratic and cubic surface fittings. In these three plots, the color variations are subtle because the spacing variation across the field of view is less than 2% in all three plots. Visual analysis of these plots and calculation of the fitting error indicate that the quadratic polynomial provides a sufficiently good fit for the spacing distortion. The spirit and teachings of this disclosure cover higher-order polynomials and alternative fitting schemes that may produce slightly better or worse results than the quadratic polynomial. Figure 11A , 11B The grayscale version of 11C includes a dark shade in the center, representing deep red, and lighter gray shades in the outer parts of the upper right and lower left, representing light red that transitions to orange and yellow near the edges.

[0214] Figure 12A and Figure 12B and Figure 10A and Figure 10B Shared x-axis and y-axis configuration. The vertical axis in these figures indicates the orientation angle of the fringes observed in the field of view. Nominally, this angle may be 45°. The measured angle ranges from 44.85 degrees to 45.15 degrees, which is a variation of 0.6%. Figure 12A The measurement data points that were not smoothed by curve fitting are depicted. Figure 12B In the diagram, the actual data points depicted in blue are compared with the quadratic fitted surface. Figure 12A and Figure 12B The grayscale version of the chart has dark shading at the top and bottom. The dark shading at the top represents dark red, and the dark shading at the bottom represents dark blue.

[0215] Figures 13A to 13E as well as Figures 14B to 14H This illustrates how cropping along the sensor boundaries can improve curve fitting for spacing and stripe angles on the sensor. The x and y axes of the graphs in these figures represent pixel positions along the length and width of the sensor. Reference Figure 8CThe cropping is indicated by the boundary near the edge of the sensor. This figure illustrates how this cropping or shrinkage removes distortion near the edges of the image captured by the sensor. The figure below shows how this image shrinkage achieves a good fit between the FOV data and the surface model. The shrinkage factor or shrinkage unit can be equal to shrinking the image by one sliding window step. The step value can be chosen to match the size of the sub-blocks, or empirically selected based on the number of pixels at the sensor edges that might cause distortion. For example, if the step value is chosen to be 200 pixels, one shrinkage unit removes the outer 200 pixels of the image data, which have poor data quality on all sides of the image. Step values ​​of less than 200 pixels (such as 25, 50, or 100 pixels) or greater than 200 pixels (such as 225, 250, or 300 pixels) can be used.

[0216] Figure 13A The blue dots represent the quadratic model surface showing spatial distortion fitted to the full field of view (FOV) image data, without cropping. No shrinkage factor was applied to this full FOV image data; that is, the shrinkage factor is zero. Figure 13A As shown, the full FOV data cannot fit the model well. This is because the shrinkage factor increases from 1 to 4 ( Figures 13B to 13E Therefore, the fit between the model and the uncropped data points is improved. Because pruning eliminates outliers, the scale shrinks along the z-axis, indicating less variation between sub-blocks. Re-reference Figure 8C The abnormal data points were observed to be located at the edge of the sensor, indicating that cropping had been applied. Figure 13C In, compared to Figures 13D to 13E There are significantly more abnormal blue data points. Figure 13E In the meantime, the MSE has decreased to 0.000001, and the z-scale has been stretched beyond... Figure 13D Slightly lower than average. Figures 13A to 13E , Figures 14B to 14H as well as Figures 15A to 15B The grayscale version of the chart has dark shading at the top and bottom. The dark shading at the top indicates the darker areas at the top and bottom of the chart. Some of these charts have light and dark circles positioned on or near the chart. The light and dark colors of the circles do not indicate any difference between them. Some circles are drawn in dark to make them visible.

[0217] Figures 14B to 14H This paper demonstrates how to improve angular distortion in a quadratic surface fitting by incrementally increasing the shrinkage factor applied to the full FOV image data. As the shrinkage factor increases, the scale along the z-axis shrinks again, bringing the data points closer to the fitted surface. Eliminating anomalous distortion values ​​at the sensor edges significantly improves the surface fitting. Figures 14B to 14EIn the figure, distortion is clearly visible in the blue data points on the plane below the fitted surface. When the shrinkage factor changes from "1" (… Figure 14C Increased to "3" ( Figure 14E When [-45, -20] is reached, the scale along the z-axis shrinks by half to [-44, -40]. Figure 14F In the end, cropping seems to bring the data points closer to the fitted surface. Figure 14F In the mean square, the MSE decreased to 0.000033. The secondary surface model used for angular distortion now fits the measurement data as shown in blue dots. Figure 14H Increasing the shrinkage factor to "6" produces a slight improvement. Based on the graph, a shrinkage factor of "4" or higher seems reasonable. Thresholds based on percentage or absolute MSE improvement can be used to select the shrinkage factor or determine its impact on resolution. The shrinkage factor can be automatically selected using ratio or threshold methods.

[0218] Imaging system characterization by fitting distorted surfaces can be performed before and / or during instrument operation by periodically reestimating the spacing and pattern angles in near-center sub-blocks and using extrapolation factors to determine the spacing and pattern angles across the entire field of view. Extrapolation factors can be applied without estimating spacing and pattern angles beyond the near-center sub-blocks. Sub-blocks across the entire field of view can overlap. During calibration, spacing and pattern angles are calculated independently for overlapping sub-blocks. This approach utilizes two orientation angles of a structured illumination pattern, such that these angles are substantially aligned with the opposite angles of a rectangular or quadrilateral pattern of nanopores.

[0219] Therefore, the techniques disclosed in this invention include calibrating and recalibrating instruments to characterize variations between sub-blocks. Characterization distortions of spacing and angles can be captured by significantly reducing the need for parameter estimation during instrument operation. Characterization / calibration can be enhanced by cropping pixels at sensor edges that are distorted to the point of being unable to fit the mapped surface, and by cropping pixels that are interfering with the fit.

[0220] Application of calibration spacing and angle

[0221] The technique disclosed in this invention can effectively reconstruct enhanced images across essentially the entire field of view. For this purpose, the entire field of view is divided into at least nine sub-blocks or subfields for independent reconstruction based on these sub-blocks. Estimated spacing and pattern angle parameters can be estimated or accessed by examining the regular spacing and angled structured illumination patterns of the near-center sub-blocks. Extrapolation factors are applied to extend the near-center sub-block parameters to other sub-blocks in the field of view. Enhanced image reconstruction is performed from multiple images using the estimated parameters from the near-center sub-blocks and the extrapolated parameters from the other sub-blocks. The multiple images are observed with multiple pattern angles and periodically shifted pattern phases of the corresponding pattern angles by applying Structured Illumination Microscopy (SIM) analysis to them. In one embodiment, the technique can further estimate periodically shifted patterns of the phases of the corresponding pattern angles. By identifying and mitigating regular noise patterns along the stitching boundaries, the technique can be extended to stitching sub-blocks together.

[0222] Figure 15A and Figure 15B This is an example of a distorted model fitted based on real data collected from sequencing instruments. Figure 15A The quadratic fitted response surface is shown by fitting the fringe spacing distortion over the full field of view (FOV). Figure 15B The quadratic fitted response surface is shown by fitting the stripe angle distortion on the FOV.

[0223] Phase displacement estimation

[0224] Phase shift is the third remaining input parameter for SIM image reconstruction. This section employs existing techniques to estimate the phase shift. The techniques disclosed in this invention perform estimation on sub-blocks; therefore, we further present a technique to establish a common reference frame for estimating the phase shift over sub-blocks relative to the full FOV block. A specific implementation of the phase estimation employs the technique proposed by Wicker et al. in Chapter 3 of their 2013 paper, “Phase optimization for structured illumination microscopy.” Equations from Lal et al. (2015), entitled “Structured illumination microscopy image reconstruction algorithm,” and from Wicker et al. (2013) help to explain Wicker's phase estimation.

[0225] Equation (3) below is taken from Lal et al.'s 2015 article, which divides the frequency components into three frequency bands: from the acquired image of The blending matrix uses a sinusoidal illumination intensity pattern corresponding to the pattern angle or orientation θ. I θ,Φ (r) Phase of the captured image , and The estimate. Wicker et al. (2013) defined the phase of the nth image in a given orientation as... If the phase is not known with sufficient precision, the demixing or band separation process will not be fully realized in the frequency domain from the observed image. Spatial frequency components are separated from the sample. In fact, there are three spatial frequency components. It will contain more or less residual information from other components, such as that represented by the noise term.

[0226] (3)

[0227]

[0228] This formula, with three components, comes from the Fourier transform of sine or cosine illumination. Different illumination functions will change the equation.

[0229] Therefore, precise knowledge of the phase of the illumination sinusoidal intensity pattern is important. Since it is not always possible to precisely control these phases in experimental settings, it is necessary to determine the illumination pattern phase from the acquired image data. Wicker et al. (2013) presented a phase estimation technique for SIM data acquired using coherent sinusoidal illumination at selected frequencies. Coherent illumination produces good pattern contrast from fine gratings with very small illumination peak spacing “s”, which improves the resolution of the reconstruction. We retrieve the illumination pattern phase of the nth image using the peak frequency of the illumination pattern. The peak frequency of the illumination pattern is also known as the Fourier peak.

[0230] The following equation (4) from Wicker et al. (2013) presents a generalized form of equation (3), where the frequency in the frequency domain is... The image was obtained from the above. Each image consists of three components, which are called... And superimposed with different phases. Note that these three components are related to those in equation (4). They are the same three components.

[0231]

[0232] (4)

[0233] Please note that in equation (4) " c "This is referred to as the contrast of the lighting pattern. In the absence of noise,"c "and the mixing matrix in equation (3) M The modulation factor in m "Same. In order to determine" The frequency in equation (4)

[0234] Replace with the peak frequency of the lighting pattern Thus, equation (5) is obtained.

[0235] (5)

[0236] Equation (5) shows that the pattern phase

[0237] Approximately equal to at frequency Images obtained from n ( The phase of the pattern. This pattern of phase occurs when three criteria are followed. An approximate estimate will yield good results. First, the contrast of the lighting pattern... c It should be large enough. Secondly, the sample power spectrum should decrease sufficiently rapidly as the frequency increases. When these two criteria are followed, equation (5) is dominated by the last term and can therefore be simplified to:

[0238] (6)

[0239] For any real-value sample, the center frequency (0) will be the actual value. Furthermore, if the point spread function (PSF)... h ( If is a real number and symmetric, then the optical transfer function (OTF) is... ( The OTF will be a real number. The OTF is the convolution of the point spread function (PSF). The point spread function is the spatial domain version of the optical transfer function of an imaging system. The name "point spread function" indicates that all physical optical systems will blur (spread) a point of light to some extent, where the degree of blurring depends on the quality of the optical elements. The resolution of an imaging system is limited by the size of the PSF. For asymmetric PSFs, the phase of the OTF must be considered.

[0240] Third, pattern frequency The OTF at the location should be large enough to overcome noise. If the OTF is too small, noise in the acquired image can significantly alter its appearance. The phase is measured at [location]. This phase estimation method cannot be used for pattern frequencies. External support is provided to enable OTF detection. For this type of frequency, = 0.

[0241] The OTF of an optical system can be determined experimentally. For example, Lal et al. (2015) calculated the OTF by obtaining sample images of several 100 nm fluorescent microspheres with a sparse distribution. The intensity distributions corresponding to more than 100 microspheres were then superimposed and averaged to obtain an approximation of the system's PSF. The Fourier transform of this PSF provides an estimate of the system's OTF. Using this background, we can apply the phase estimation technique to sub-blocks.

[0242] Convert sub-block phases to a common reference frame

[0243] Estimating the phase displacement of a block relative to the full field of view (FOV) is useful, allowing phase measurements in one sub-block to be extrapolated to other sub-blocks on the block. The illumination peak angle and illumination peak spacing for the full FOV can be estimated from the illumination peak angle and illumination peak spacing of the sub-blocks using the quadratic model presented above. The phase displacement is somewhat irregular because it depends on the pixel geometry of the sub-block, which can produce an irregular step function rather than a smooth one. Each phase estimate has a “reference frame” anchored to the top-left corner of the image used for phase estimation. Therefore, when we want to geometrically correct phase differences on the image, we need to compare the phase estimated from each sub-block with a reference phase in a common reference frame (from the central estimation sub-window or sub-block). We transform the phase estimates of each sub-block to the same geometric location to achieve a common reference frame. Transforming the phase requires accurately estimating the fringe angle and spacing of each sub-block. The fringe angle and spacing can be obtained from a quadratic surface distortion model calibrated according to the distortion calibration process.

[0244] We use a common reference frame on sub-blocks of the full FOV image to represent the phase estimation. The sub-block coordinate space is mapped to the full FOV coordinate space. In this section, we illustrate the transformation of point coordinates from one space to another. Figure 16A The general settings and related variables for sub-block windows of a full FOV image window are shown, along with a phase offset correction formula that enables coordinate transformation between local and global references. Figure 16A A boundary 1603 representing the full field of view is shown, which contains the sub-blocks shown by boundary 1605. The clipping angle α is defined by a line 1606 connecting the top-left corner of the full FOV window 1603 and the top-left corner of the sub-block window 1605. The length of line 1606 is measured in pixels and labeled "a".

[0245] The illumination peak angle β is formed between the top boundary of the sensor (window 1603) and line 1607, which is perpendicular to the upper left corner of the fringe peak line 1609 of the sub-block window 1605. This fringe peak line will be parallel to the actual fringe peaks and may be offset from the peaks. The length of line 1607 from the upper left corner of 1603 to 1609 is the projected distance “b”. The phase difference between the upper left corner of block 1603 and sub-block 1605. It is a function of the spacing s and the distance b. In formula (7), the peak lighting spacing is expressed as "s" in pixels and as phase in degrees:

[0246] o = ([b + ( s / 360 s)] % s) / s 360 (7)

[0247] The formula derived and explained below can be used to transform the phase offset from sub-block coordinates to full FOV coordinates. The transformation between coordinate systems is shown in the following two examples.

[0248] In SIM reconstruction, one-dimensional (1D) gratings can be described by intensity functions, such as:

[0249] (8)

[0250] The parameters have the following mean:

[0251] Modulation depth

[0252] It is the illumination phase spacing

[0253] It is a unit vector

[0254] β is the peak illumination angle

[0255] In the equation, the bold x and u denote vectors: x is a vector in image space, representing the location of a specific intensity in image space; u is a unit vector. The intensity value of sinusoidal illumination is positive, ranging from +M (bright) to -M (dark).

[0256] Once the phase shift difference between sub-blocks has been characterized, it is necessary to estimate the phase shift at the near-center sub-block, since the phase shift of another sub-block can be derived from the near-center sub-block.

[0257] The following calculations are related to phase offset. The estimation is related to the phase offset. Instead of using data from the entire image (full FOV) to estimate the phase offset, data from only sub-blocks of the image can be used. The phase offset of a sub-block is related to, but not identical to, the phase offset of the full FOV. To recover the phase offset of the full FOV from the phase offset of the sub-blocks, it is necessary to characterize how the phase offset transforms under spatial transformation. The following mathematical model presents this spatial transformation from sub-blocks to the full FOV.

[0258] Derivation of coordinate transformation

[0259] Consider the following function, which in the case of Figure 16A The orientation shown generates grating lines. The full FOV's two-dimensional (2D) space consists of a coordinate space O with coordinates (x1, x2). x Indicates, such as Figure 16B As shown. We use relative to O x Location ( x 1, x 2) Represents point 1611. Function f ( x 1, x 2) It can be defined as shown in equation (9). This function can be used in situations such as Figure 16A The orientation shown generates the lighting pattern lines.

[0260] (9)

[0261] The above function f The one-dimensional periodicity is represented by the function in equation (10). (Reference) Figure 16A This function represents what happens when we move along a line perpendicular to the lighting pattern line. Figure 16A The line 1607 in the equation repeats the lighting pattern lines as it moves. This function determines the spacing “s” between two lines by determining the location where a specific intensity value is repeated. Equation (10) shows the periodicity of the lighting pattern intensity. In equation (10), the lines orthogonal to the intensity peaks are at a 45-degree angle, extending downwards and to the right from the upper left corner, so the point ( x 1, x 2) The intensity at continuous points downwards and to the right along line 1607. The repetition occurred.

[0262] (10)

[0263] The lighting peak spacing “s” defined by the above function is shown in equation (11). For convenience, the coordinate system can be scaled to match the spacing of the lighting pattern, as shown in equation (11).

[0264] (11)

[0265] The peak illumination angle is defined by equation (12). Figure 16A As shown, the illumination peak angle is formed between the top boundary (horizontal line) of the full FOV and the line perpendicular to the illumination pattern line. Equation (12) shows that this angle is -45 degrees for a structured illumination pattern generated by the 1D periodic function in Equation (9).

[0266] (12)

[0267] The zero set of the pattern lines formed by the periodic functions presented in equation (9) is the union of the lines and is represented by equation (13).

[0268] (13)

[0269]

[0270] The zero set comprises structured lighting pattern lines spaced apart from each other by a distance “s”. These pattern lines are generated by 1D functions in equation (9). f ( x 1 , x 2 Points with a value of zero are formed. The unit vector orthogonal to the grating is defined by equation (14).

[0271] (14)

[0272] We now present the phase offset of the sub-block from the sub-block coordinate space O defined by coordinates (y1, y2). y Transform to full FOV coordinate space O x Two examples. Figure 16B Point 1611 is shown, which can be represented in sub-block coordinates by a vector originating from the upper left corner of sub-block window 1605. The same point 1611 can be represented in full FOV coordinates by a vector originating from the upper left window 1603 of the full FOV image. Indicates. For example... Figure 16B As shown, vector Connect the top-left corner of the full FOV image window 1603 to the sub-block window 1605. In the following sections, we will present an example of how the position of point 1611 can be transformed from one coordinate system to another.

[0273] First Example

[0274] set up ,express Figure 16BThe top left corner of sub-block 1605. This means that the top left corner of sub-block 1605 is shifted by a distance "a" along the horizontal axis x1 and a distance "-a" along the vertical axis x2 from the top left corner of the full FOV window 1603. We now place the full FOV coordinate space O x The point in ( x 1 , x 2 Convert to sub-block coordinate space O y The point in ( y 1 , y 2 Consider the given coordinate transformations:

[0275] (15)

[0276] We can now express the 1D periodic function of equation (10) in the subblock coordinate space Oy as:

[0277] (16)

[0278] We can see that the phase offset introduced by the coordinate transformation is given as:

[0279] (17)

[0280] Second example

[0281] Now let's consider another position of the sub-block relative to the top-left corner of the full FOV window. Let a = (2a, -a) ∈ R^2, in which case the coordinate changes are given as:

[0282] (18)

[0283] The 1D periodic function of equation (10) can now be represented in the subblock coordinate space Oy as:

[0284] (19)

[0285] In this case, the phase offset introduced by the coordinate transformation is given as:

[0286] (20)

[0287] However, <u,a> represents the dot product of vectors u and a. This dot product is defined as the sum of the corresponding components of the two vectors.

[0288] General situation

[0289] Theorem 1.

[0290] Let s > 0,

[0291] Let u ∈ R, and let u ∈ R 2 It is a unit vector. Consider the function.

[0292] (twenty one)

[0293] a) The grating spacing of f is s.

[0294] b) For any a∈R 2 ,exist ∈R, such that

[0295] (twenty two)

[0296] Equation (21) shows a periodic function of the full FOV coordinate space Ox, and has a phase offset.

[0297] Equation (22) shows a periodic function in the sub-block coordinate space and has a phase offset.

[0298] c) Any of this kind

[0299] Satisfy the equation

[0300] (twenty three)

[0301] The test (a) shows that the peak spacing of the illumination is the length of the fundamental period of f. The vector su is a period of length s, because...

[0302] (twenty four)

[0303] (25)

[0304] (26)

[0305] Any smaller period means that the period of sin is less than 2π.

[0306] (b) One solution is

[0307] (27)

[0308] because

[0309] (28)

[0310] (29)

[0311] c) If For all t∈R, then a≡b mod 2π (30)

[0312] The relationships derived above can be used to convert phase displacements estimated in a first coordinate space to a second coordinate space, such as between sub-blocks and full FOV image blocks. This is useful because phase displacements are also affected by distortion between near-center sub-blocks and lens edges. In the following sections, we present a phase offset lookup table that captures the relationships between phase displacement values ​​of non-center sub-blocks relative to the center sub-block.

[0313] Phase deviation estimation

[0314] The technique disclosed in this invention involves estimating the phase of each sub-block and then converting the phase offset to full FOV coordinates, such that the estimated phase shift values ​​are represented in a common reference frame. For each angle of the illumination pattern, the difference between the phase shift values ​​of the non-central and near-central sub-blocks is stored in a phase offset or phase difference lookup table. In some cases, environmental factors can increase system instability. In such cases, the phase offset lookup table is generated more frequently to compensate for the increased system instability. For example, the phase offset table can be generated in the first cycle of each sequencing run. The generation frequency of the phase offset lookup table can be increased or decreased depending on the system stability.

[0315] Figure 17 Example phase deviation lookup table 1701 is presented, where the phase difference from the near-center sub-block at positions 3, 5 is color-coded, with the sub-block having a zero value. Lookup table 1701 contains the same number of pools as the number of sub-blocks in the full FOV image data, which is 7×9 sub-blocks in this example. The positions of the pools in Table 1701 correspond to the positions of the sub-blocks in the full FOV image data. The phase lookup table stores the phase difference between the estimated phases of each sub-block relative to the center window (3, 5). The example table contains the phase deviation for one angle of structured lighting. A similar phase deviation table will be stored for each angle of the structured lighting pattern. Phase values ​​are sensitive to cumulative errors in the estimates of other parameters (angle and spacing). Note that the phase values ​​wrap around from 0 degrees to 360 degrees. The phase of the center sub-block is shown as 0 degrees in the example table. The color coding indicates the degree of phase difference among the sub-blocks corresponding to the center sub-block. Estimating the phase deviation requires the fringe angle, spacing, and phase of the central sub-block or sub-window, as well as the distortion model (which we assume remains unchanged). In the following sections, we present example algorithms for estimating the angle, spacing, and phase parameters.

[0316] In the preceding sections, we described techniques for estimating parameters of sub-blocks in a full FOV image. In the following pseudocode, we further illustrate the angle, spacing, and phase parameters for the sub-blocks. Example pseudocode for estimating the angle and spacing parameters is presented below and is referred to as “Sub-block Angle and Spacing Mapping” (also known as Algorithm 2A). We use quadratic or cubic polynomial fitting to estimate the distortion coefficients of the angle and spacing parameters of the sub-blocks. Example pseudocode implementing the phase estimation technique is presented below and is referred to as the “Sub-block Phase Mapping” algorithm (also known as Algorithm 2B). For each angle or orientation of the illumination pattern, the estimate of the phase bias can be stored in a phase bias lookup table. In one specific implementation, a separate lookup table can be used for each sequencing block on the flow cell to account for minor differences in flow cell orientation, surface defects, and other non-ideals that may skew the estimation results. Finally, pseudocode for an example algorithm for final distortion calibration is presented, and this pseudocode is referred to as “Final Distortion Calibration Using Quadratic Fitting and Lookup Tables” (also known as Algorithm 1 and Algorithm 2C).

[0317] Algorithm 1: Using the final distortion calibration of quadratic fitting

[0318] In the first pseudocode, we can apply cropping to remove distortion near the image edges. This allows for a good fit of the full FOV data to a secondary surface model during calibration / characterization for angle and spacing estimation.

[0319] Input: A set of images, such as two angles and three phases per angle.

[0320] Output: Distortion model coefficients

[0321] algorithm:

[0322] 1. Construct the following image / filter (size designed to match the sub-regions).

[0323] a. OTF

[0324] 2. Divide the image into equal-sized but overlapping sub-regions, for example, using a sliding window method to divide it into 512×512 sub-regions.

[0325] 3. Use, for example, a 512×512 window to estimate the parameters of the image center (see the sub-block angle and spacing mapping algorithm and the sub-block phase mapping algorithm below).

[0326] a. Estimate the peak illumination angle and spacing at each angle.

[0327] b. Estimate the modulation at each angle

[0328] c. Estimate the phase of each image

[0329] 4. For each sub-region

[0330] a. Perform parameter estimation in step 3.

[0331] b. Save the x, y coordinates of the sub-block center, stripe angles (for each angle), spacing (for each angle), and phase 0 (for each angle) of the Wicker phase estimate to an array (2D array).

[0332] 5. Regarding stripe spacing and angle

[0333] a. Subtract the table value from step 4 from the center window estimate in step 3.

[0334] b. Perform optimal shrinkage estimation for the secondary surface.

[0335] i. For each array in step 4 (e.g., denoted as Z, the fringe spacing starting from angle 0), fit the quadratic surface to the functional form f(x,y)=a+b via least squares regression. x 2 +c y 2 +d xy+e x+f y (using the same input arrays X and Y, which correspond to the X,Y coordinates of the sub-region center).

[0336] ii. Evaluate the R of step i 2 Coefficient of determination.

[0337] iii. Shrink the X, Y, Z array (remove outer edge elements) and repeat step (i) until N shrinkages have been performed (N=3 to 5 and is a tunable parameter).

[0338] iv. As the array shrinks towards the center of the data, the expected improvement in fit quality is achieved. We select the coefficients at the elbow (minimizing the shrinkage while maximizing R0). 2 ), where the model fits well enough (the current standard is defined as R). 2 The relative improvement in the value is less than 5%.

[0339] Other acceptance criteria that may be used here include:

[0340] Improvement of the mean squared error of the fit (the mean squared error in the next iteration is smaller than the mean squared error in the current iteration by a predefined threshold).

[0341] Improvement of the maximum fitting error for the quadratic surface (if we detect that the continuous improvement is in a stationary state, then the error should be reduced and the fit increased with each successive iteration).

[0342] The error (such as MAE (mean absolute error) instead of MSE (mean squared error)) is regularized using an improvement threshold similar to the first two criteria.

[0343] 6. The optimal coefficient in step 5 is the output of the final distortion calibration algorithm.

[0344] Algorithm 2A: Sub-block Angle and Spacing Mapping

[0345] The pseudocode used in steps 3 to 4 above and during the operation for estimating the illumination peak angle and spacing of the sub-blocks is shown below:

[0346] Input: A set of images at the same angle, such that the phase of each image is uniformly distributed from 0 degrees to 360 degrees.

[0347] Output:

[0348] 1. The position of the peak in Fourier space (or the spacing and angle of the grating).

[0349] 2. Modulation of Fourier peaks

[0350] 3. Phase offset of the first image

[0351] algorithm:

[0352] 1. Fast Fourier Transform for each image

[0353] 2. Perform frequency band separation

[0354] 3. Use two component images:

[0355] a. 0 (wide field) and +1 (high resolution portion)

[0356] b. Mask all non-overlapping Fourier components (also known as common regions) of each image.

[0357] c. Shield low-frequency components near DC and near the estimated fringe peak.

[0358] d. Apply inverted OTF

[0359] 4. Repeat a finer grid around the starting point.

[0360] a. For each point in a 10×10 grid, uniformly sample a certain range.

[0361] i. Applying phase shift to one of the two image phases

[0362] ii. Multiply one image by the conjugate of another image.

[0363] iii. Summing the results

[0364] iv. Normalize relative to the sum of the wide-field masked Fourier images

[0365] v. Saving the offset provides the maximum value of the absolute value of the normalized complex correlation.

[0366] 1. The absolute value of this number is the modulation.

[0367] 2. This complex angle is the phase offset.

[0368] Algorithm 2B: Sub-block Phase Mapping

[0369] The following is pseudocode for an example implementation of phase offset estimation for sub-blocks. The technique disclosed in this invention can use the nearest-best estimate of the spacing and angle parameter values ​​of this mapping. The system can also obtain estimated spacing and angle parameter values ​​using a distortion model and center sub-window or sub-block parameters.

[0370] enter:

[0371] 1. A collection of images from a single angle

[0372] 2. Illumination peak spacing and angle estimated from the previously described algorithm.

[0373] Output: Phase offset of each image in the set.

[0374] algorithm:

[0375] 1. Perform a Fast Fourier Transform (FFT) on each image.

[0376] 2. Create a copy of the FFT image

[0377] 3. Apply OTF to each image

[0378] 4. Offset an image

[0379] 5. Multiply the two graphs using conjugates and sum the absolute values ​​of each complex number.

[0380] 6. The phase is the angle obtained as a complex number.

[0381] Algorithm 2C: Phase Deviation Finding Learning

[0382] To address phase bias, the following steps are performed on each block in the flow cell during the first cycle of imaging in the sequencing run.

[0383] Input: A set of images, for example, 2 angles and 3 phases for each block from the sequencing flow cell, and a calibration distortion model from Algorithm 1.

[0384] Output: Phase deviation lookup table

[0385] algorithm:

[0386] For each sequencing block (do not obfuscate sub-blocks reconstructed by SIM),

[0387] i. Apply step 3 of Algorithm 1 to estimate the SIM parameters of the block's center window (spacing, angle, and phase for each angle-channel pair).

[0388] ii. Divide the image into sub-blocks. For each sub-block:

[0389] a. Obtain the sub-block stripe spacing and angle of the sub-block by applying a distortion model via Algorithm 2A.

[0390] b. Using the fringe parameters given in step a., perform Wicker phase estimation on the first phase image for each channel-angle pair in the local sub-block.

[0391] c. Calculate the difference between the phase obtained in step iib and the central window phase 0 obtained in step i, and save the phase deviation to the corresponding 2D array for phase deviation calculation.

[0392] iii. Store the completed phase offset lookup table in memory, with the block ID as an entry for later use.

[0393] Repeat step 1 for each other sequencing block encountered throughout the flow cell.

[0394] Please note: For performance reasons, we estimate the phase bias lookup for each block in the first loop of the sequencing run. However, if stability issues exist, the learning frequency can be adjusted to learn once every N loops.

[0395] In these pseudocodes, we present techniques for estimating and mapping illumination peak angle, illumination peak spacing, and phase shift parameters. The estimated parameters are used as inputs for Structured Illumination Microscopy (SIM) image reconstruction. In the following sections, we present a proposed SIM image reconstruction algorithm that efficiently reconstructs high-resolution images of sub-blocks and then combines these sub-block images to reconstruct a high-resolution image of the full field of view (FOV).

[0396] SIM image reconstruction

[0397] The SIM image reconstruction technique disclosed in this invention uses angle, spacing, and phase estimation of sub-blocks to reconstruct high-resolution images. The technique can construct a high-resolution FOV image in two steps. First, a high-resolution SIM image is reconstructed for each sub-block using sub-block parameters. Second, the high-resolution images of the sub-blocks can be stitched together to construct a full FOV image. Alternatively, high-resolution sub-blocks can be processed without stitching. This section begins with example pseudocode for SIM image reconstruction of sub-blocks. This is referred to as the "sub-block reconstruction" algorithm. The technique disclosed in this invention utilizes symmetry in Fourier space by using non-redundant data in the frequency domain. This reduces the computational cost required for some core operations by nearly half. This section also presents example pseudocode for combining SIM images of sub-blocks to reconstruct a high-resolution image of the full image. This is referred to as the "full field-of-view reconstruction" algorithm. Note that the parameter estimation techniques disclosed above can be used with conventional redundant or non-redundant SIM image reconstruction techniques disclosed in this invention.

[0398] Algorithm 3: Sub-block Reconstruction

[0399] A pseudocode implementation for reconstructing sub-blocks of high-resolution structured imaging microscopy images.

[0400] enter:

[0401] a. Has a certain grating profile ( 角度、相位 A set of sample images. For each angle, we need at least 3 phase images, evenly spaced from 0 degrees to 360 degrees.

[0402] b. Parameters:

[0403] i. Illumination peak spacing and angle of a set of images corresponding to the angle

[0404] ii. Phase for each image

[0405] Output: A high-resolution reconstructed image for the sub-block

[0406] algorithm:

[0407] 1. Preprocess the image (e.g., use a high-pass filter).

[0408] 2. Create a large Fourier image buffer (its size will depend on the desired final upsampling factor) and initialize it to 0.

[0409] 3. For each angle

[0410] a. For a set of images, each image has a unique phase for a grating at a given angle.

[0411] i. Perform a Fast Fourier Transform on each image

[0412] b. Perform frequency band separation on the image set (n≥3 input images → 3 output or component images)

[0413] i. Two of the three output images contain higher resolution information.

[0414] ii. The remaining images contain all standard information (original resolution images).

[0415] iii. It is necessary to know the phase of each image and the modulation of the entire set.

[0416] c. Apply MTF to each component image

[0417] d. Separate each component image from the frequency band and offset it to the appropriate position in the larger Fourier image buffer, and sum the components.

[0418] 4. Construct a Wiener filter from the input MTF by shifting and summing the 2D MTF image to obtain a large Fourier image.

[0419] a. We perform this operation on each of the input images from each angle.

[0420] b. This operation does not need to be performed for every rebuild; it can be performed once and provided as an input parameter.

[0421] 5. Applying Wiener filters and apodization filters in Fourier space to the large Fourier representation of the reconstructed image.

[0422] 6. Perform an inverse fast Fourier transform on a large Fourier image buffer to obtain a high-resolution image.

[0423] Since the technology disclosed in this invention cannot be explained by referring to existing technologies, it provides a more detailed description of this non-redundant sub-block reconstruction pseudocode compared to other pseudocodes. The technology disclosed in this invention utilizes symmetry in Fourier space to increase efficiency and reduce computational load in SIM image reconstruction. Figure 18 This paper presents a high-level view of our non-redundant sub-block construction technique. Three images (1811, 1813, 1815) are acquired for each sub-block at each angle, preferably with uniformly spaced phases. We apply a Discrete Fast Fourier Transform (FFT) to transform the image data from the spatial domain (real space) to the frequency domain (complex space). We leverage the FFT symmetry of the data representing the three images in the frequency domain. We improve the efficiency of reconstructing high-resolution SIM images of sub-blocks by operating on the non-redundant portions of the image data in the frequency domain. The processing steps of the non-redundant sub-block reconstruction algorithm are detailed below.

[0424] enter

[0425] 1-a. Image collection

[0426] The image, divided into sub-blocks, is given as input to the sub-block reconstruction algorithm. The image is divided into sub-blocks, sub-windows, or sub-regions of a predefined size, such as 512 pixels × 512 pixels. Other sub-block sizes can be used.

[0427] The set of image blocks extends existing techniques because so many blocks are needed to capture and image the plane. In one specific implementation, a sinusoidal illumination intensity pattern is generated. I θ,Φ Image patches are acquired using a diffraction grating (r). The intensity varies with position (x, y) and is represented by the vector r, which is a two-dimensional spatial position quantity. For comparison, Lal et al. (2015) presented a basic structured illumination microscopy reconstruction algorithm, labeled "Algorithm 1 SIM-RA". To explain the steps of our SIM image reconstruction algorithm, we have used the mathematical formulas from Lal et al. (2015). Note that the reconstruction algorithm presented by Lal et al. (2015) applies to the entire image, while our sub-block reconstruction technique reconstructs sub-blocks of high-resolution SIM images.

[0428] I θ,Φ The mathematical formula for (r) is presented in equation (1) of Lal et al. 2015. For each orientation θ of the sinusoidal illumination pattern, three SIM images of the specimen are acquired corresponding to three different illumination phases. In one embodiment, two image sets are acquired using two orientation angles of the sinusoidal illumination pattern, namely θ1 = 45° and θ2 = 135°. The orientation angle θ is also referred to as the illumination peak angle β. In such embodiments, the nanopores may be located at the corners of a square or, more generally, at the corners of a rectangle or quadrilateral. The orientation angles are chosen such that images are taken along lines connecting opposite corners of the rectangle. This increases the apparent distance between the nanopores.

[0429] In another embodiment, three image sets can be captured at three orientation angles, such as θ1=0°, θ2=60°, θ3=120° or θ1=30°, θ2=90°, θ3=150°. In this embodiment, the nanopores can be arranged at opposite corners of a hexagon. Images are captured along lines connecting opposite ends of the hexagonal geometry. Typically, the image sets can be selected to match the number of diagonals connecting opposite ends of the geometry with the nanopores.

[0430] For each orientation θ, there are three different illumination phases. =0°、

[0431] =120° and Acquire at least three images with a phase angle of 240° and a spacing of 0° to 360°. Uniform phase spacing is preferred. More images, such as five, six, or seven, can be acquired by using smaller phase steps.

[0432] 1-b (i) Structured lighting pattern angle and spacing parameters

[0433] Define the periodic or sinusoidal intensity pattern of the grating I θ,Φ The illumination peak spacing "s" of the illumination (r) is also given as input. The "sub-block angle and spacing mapping" algorithm (also known as Algorithm 2A) presented above is used to estimate the illumination peak spacing and illumination peak angle θ of the sub-block. The "final distortion calibration using quadratic fitting and lookup table" algorithm (also known as Algorithm 1) is applied to compensate for the estimation errors of the angle and spacing parameters of the non-central sub-block due to optical sensing distortion. Sample pseudocode for "sub-block angle and spacing mapping" and "final distortion calibration using quadratic fitting and lookup table" is presented above.

[0434] 1-b (ii) Phase displacement parameters

[0435] Structured lighting intensity pattern I θ,Φ The phase shift (also known as grating phase) of (r) The input is also given. The "sub-block phase mapping" algorithm (also known as Algorithm 2B) is used to estimate the phase value of the sub-block. The "final distortion calibration using quadratic fitting and lookup table" algorithm (also known as Algorithm 1) is applied to compensate for the phase parameter estimation error of the non-central sub-block due to optical sensing distortion. The difference between the phase of the non-central sub-block and the central sub-block at each angle is stored in a lookup table. The above presents example pseudocode for the "sub-block phase mapping" algorithm. By describing these inputs, we move on to processing.

[0436] Processing Step 1: Image Preprocessing (High-Pass Filter)

[0437] Processing for sub-block reconstruction can begin with image preprocessing. The pseudocode, typically corresponding to this specification, is explained later. Image reconstruction is performed in Fourier space. However, before applying a Fourier transform to convert the image data from the spatial domain to the frequency domain, we can apply the preprocessing steps described below to the image blocks. The following description is based on three images with different phase values ​​along an angle. As mentioned above, two or more orientations will be used.

[0438] The acquired images can be preprocessed to remove noise. Noise has many sources. For example, exposure to strong excitation light can produce photobleaching, which reduces the emission intensity from the fluorophore. (Lal et al., 2015) Figure 6 The noise description in the acquired SIM image is shown. Figure 6 The top-left image (labeled (a)) shows the raw SIM image with background fluorescence blurring. This introduces significant errors into the frequency content of the raw SIM image in a small neighborhood around zero or DC frequencies. High-pass filters, such as Gaussian high-pass or Butterworth high-pass filters, can be used to preprocess the raw SIM image. Figure 6 The second image (labeled (b)) shows the original SIM image after background subtraction. The Fourier spectra of both images (a) and (b) are the same as those shown. For a detailed discussion of high-pass filters, please refer to Chapter 4, “Digital Image Processing,” Gonzales and Woods, 2008, 3rd edition. In Chapter 4, Gonzales and Woods present an illustration in Figure 4.52 that includes perspective views, image representations, and cross-sections of ideal, Butterworth, and Gaussian high-pass filters.

[0439] Processing Step 2: Create a large Fourier image buffer

[0440] A buffer in memory can be reserved and initialized to receive the enhanced Fourier images. The size of the memory buffer depends on the upsampling factor and the degree of enhancement. SIM image reconstruction technology can theoretically produce enhanced resolution along each angle or orientation of the three images, up to twice the resolution of the image acquired along that orientation. For two or three sets of three images, a buffer can be reserved to store high-resolution images with an upsampling factor of 2.

[0441] Processing step 3: For each angle θ, repeat steps 3-a to 3-d.

[0442] Processing steps 3-a through 3-d are executed cyclically. In one iteration, processing steps are performed on three images of the sub-block acquired at each angle. We will now describe the details of the processing steps in the loop.

[0443] Step 3-a (Fast Fourier Transform):

[0444] The Discrete Fast Fourier Transform (FFT) is applied to a set of images acquired at the same angle or orientation "θ". As described above, we acquire at least three images with different phases along each orientation. Each image is transformed from the spatial domain to the frequency domain. For N×N images, the complexity of computing the 2D Discrete Fourier Transform (DFT) is O(n log n). O (N 4 By utilizing common subexpressions from row to row and column to column, this complexity can be reduced to [value missing]. O (N 3If N is a power of 2, further acceleration can be achieved. When N is a power of 2, for an N×N image, the Fast Fourier Transform (FFT) algorithm can achieve a complexity of [missing information - likely a time complexity]. O (N 2 (log2N). An example of utilizing common expressions in rows and columns is through the use of "butterfly computation". For details, see Heckbert 1995, "Fourier Transforms and the Fast Fourier Transform (FFT) Algorithm", at http: / / www.cs.cmu.edu / afs / andrew / scs / cs / 15-463 / 2001 / pub / www / notes / fourier / fourier.pdf. Therefore, choosing a sub-block size such as 512 pixels × 512 pixels involves a trade-off between increased frequency resolution and computation time. Sub-blocks with side lengths of 128 pixels, 256 pixels, 512 pixels, 1024 pixels, 2048 pixels, 4096 pixels, or in the range of 128 pixels to 4096 pixels can be used.

[0445] Image data in the spatial domain is represented as real numbers. When we perform a Fourier transform, the resulting data in the frequency domain is represented as complex numbers. z This can be represented as z= x + iy ,in x and y These are the real part and the imaginary part. The conjugate of the complex number z is denoted as zi. = x - iy .

[0446] Figure 19 Two example angularly offset Fourier domain matrices are shown, depicting matrices with even (1910) and odd (1920) values ​​of N to illustrate symmetry in Fourier space. These images in the frequency domain are represented by N×N matrices. The top row and left column of the two matrices 1910 and 1920 are referred to as the DC row and DC column corresponding to the zero or DC frequency components. Compared to the angularly offset matrices with these values, the graphs in the frequency domain typically show the zero or DC frequencies at the circles.

[0447] Figure 19The matrices in the matrix exhibit symmetry in the Fourier or frequency domain. The DC rows and DC columns in matrices 1910 and 1920 have conjugate symmetry both horizontally and vertically. Matrix 1910, with an even number of rows and columns, has a middle column that also has vertical conjugate symmetry. Leaving aside the top row and left column (1910, 1920) and the middle column (1910) in the even-numbered matrices, the remainder of each matrix has 180-degree rotational conjugate symmetry. These portions of both 1910 and 1920 are shown in bold.

[0448] Divide the complete matrix in the frequency domain into two halves. After leaving rows and columns, a right half is produced. After rotating 180 degrees, it is the conjugate of the left half. Figure 19 The half-matrices (1911, 1912) and (1921, 1922) of the even- and odd-dimensional matrix (1910, 1920) are shown. Values ​​in the right half can be found from the data in the left half. Considering this lookup, the left half is so-called non-redundant data, and the right half is redundant data. The number of columns in the left half (slightly more than half) is ((N / 2)+1), and the number of columns in the right half is (N-(N / 2)-1). The technique disclosed in this invention maintains tracking symmetry to reduce computational load.

[0449] When reconstructing the SIM image, we operate on the non-redundant left half of the full image matrix in Fourier space. This increases efficiency as the required computation is significantly reduced. In the following SIM reconstruction steps, we will present further details on how to utilize the symmetry in Fourier space.

[0450] Step 3-b (Band Separation):

[0451] In each captured image, offset and unoffset frequency components are presented because the source location is randomly distributed relative to the fringe peaks of the illumination. This step uses a band separation matrix M. -1 The offset and unoffset frequency components in the frequency domain representation of the acquired image are separated. The band separation matrix is ​​also called the inverse mixing matrix (Wicker et al., 2013) or the reconstruction matrix (Lal et al., 2015). Gustafsson et al. in 2000 illustrated the band separation process in Figure 5 of their paper entitled “Doubling the lateral resolution of wide-field fluorescence microscopy using structured illumination.” This figure shows the separated frequency components from the observed image. Band separation can be understood from the following equation (31), taken from Lal et al., 2015. This equation shows that the three components Separate from the three observed images on the right side of the equation. Obtain the three images using the angle θ of the illumination pattern.

[0452] (31)

[0453] The first component extracted from equation (31) is the so-called DC component. This is the component with the unshifted spatial frequency. When separated using precise phase and modulation parameters, this component does not include high-frequency content beyond the Abbe diffraction limit. The second and third components... , It contains high frequency content in the bandpass region that is shifted to the Abbe diffraction limit.

[0454] like Figures 20 to 22 As shown, the frequency band separation step utilizes the non-redundant SIM image reconstruction technique disclosed in this invention. It is assumed that the full matrix of each sub-block image in Fourier space has 512 rows and 512 columns, as follows... Figure 20 As shown above, we divide the matrix positions in the three frequency domain matrices corresponding to the three images acquired at one angle across three phases into non-redundant and redundant halves. Each non-redundant half-matrix has 512 rows and 257 columns, comprising two columns and one row of DC components, and is dominated by non-redundant coefficients with complementary conjugate coefficients in the redundant half-matrix. We reshape each non-redundant left half-matrix into a vector and combine the three vectors of the three half-matrices into a matrix with 3 rows and 131,584 columns, as shown. Figure 21 As shown. Then, we will use the frequency band separation matrix M -1 The inverse matrix is ​​multiplied by the combination matrix to obtain a component separation matrix of the same size, also with 3 rows and 131,584 columns. The component separation matrix corresponds to the left side of equation (31). We then reshape the three rows of the resulting matrix to obtain the components corresponding to the three frequencies. , , The non-redundant left half of the matrix. Figure 22 (and Figure 18 The diagram shows three non-redundant matrices 1831, 1833, and 1835. In our case, the separated components are in the form of non-redundant left-half matrices.

[0455] Now for reference Figure 18After processing step 3-b, the full matrix of the so-called DC frequency component can be represented as a square matrix that combines a non-redundant half matrix 1831 (left half) and a redundant half matrix 1832 (right half), or it can be a rectangular matrix. The DC frequency matrix does not undergo the same offset as the high-frequency matrix, so the DC half can be assembled earlier or later without computational loss. The symmetry is handled differently for the two high-frequency components to reduce computational load.

[0456] After processing step 3-b, the full matrix of the high-frequency components is organized to combine portions from the two frequency bands, such as the non-redundant half matrix (1) 1843 from the first high-frequency image and the redundant half matrix (2') 1844 from the second high-frequency image. Equivalently, the non-redundant half matrix (2) 1845 from the second high-frequency component and the redundant half matrix (1') 1846 from the first high-frequency component can be combined.

[0457] Although Figure 18 The diagram shows three full matrices for DC and high-frequency components, but symmetry can be used to process the operations in steps 3b, 3c, and 3d, thereby processing the frequency band of the frequency components using half the data.

[0458] A similar process can be applied to five images of 1D structured lighting with a phase step along a certain angle, divided into frequency components, or to a large number of images along a certain angle. Functions other than sinusoidal or cosine-varying lighting can be used. The technique disclosed in this invention can also be applied to image sets created using 2D structured lighting.

[0459] Step 3-c (Applying the modulation transfer function):

[0460] Two alternative schemes for applying modulation transfer function (MTF) and / or optical transfer function (OTF) are described: a two-step application of the independent MTF followed by a Wiener filter to reduce noise, and a one-step application of the MTF integrated into the Wiener filter. Both alternative schemes can utilize the non-redundant computational techniques disclosed in this invention.

[0461] In these two specific implementation steps, we will compare the (MTF) (which is the magnitude or absolute value of the optical transfer function (OTF)) with the three components obtained after step 3-b. , , Multiply each of the terms in the table. Note that steps 3-b and 3-c are... Figure 18 The MTF is merged because its application does not affect the symmetry of the matrix.

[0462] A lens's modulation transfer function (MTF) is a measure of its ability to transfer contrast from an object to an image at a given resolution. In other words, MTF is the way resolution and contrast are combined into a single specification. Resolution is the ability of an imaging system to distinguish details of an object. It can be expressed in line pairs per millimeter (lp / mm), where a line pair is a sequence of one black and one white line. This measure of line pairs per millimeter is also called frequency. The reciprocal of the frequency produces the spacing between the two resolving lines, measured in millimeters. Bar targets with a series of equally spaced, alternating white and black bars (also known as the 1951 USAF target or Ronchi rule) are commonly used to test system performance. As the line spacing decreases (i.e., the frequency on the target increases), the lens becomes increasingly less effective at transferring this reduction in contrast; as a result, the MTF decreases.

[0463] For more detailed information on the modulation transfer function, please refer to the article entitled "Introduction to Modulation Transfer Function," available at https: / / www.edmundoptics.com / resources / application-notes / optics / introduction-to-modulation-transfer-function / . Figure 5 in the aforementioned reference article illustrates the decrease in image contrast as the spatial frequency of the lines increases. For an image to appear sharp, blacks must be true blacks and whites must be true whites, with minimal grayscale in between. The article... Figure 6 The MTF of an aberration-free image using a rectangular lens was plotted. The MTF decreases with increasing spatial resolution. It is important to note that real-world systems are not entirely free of aberrations.

[0464] This two-step application of MTF and Wiener filtering can be replaced by a single-step method, as presented in step 15 of the SIM-RA algorithm 1 in Lal et al. (2015). Similarly, step 3-c is the first part of the algorithm, performing Wiener filtering in two steps. Alternatively, this step can be performed after step 3-d without affecting the quality of the SIM-reconstructed image.

[0465] A two-step method (i.e., applying the MTF and Wiener filtering separately) is recommended in the literature on SIM image reconstruction. For example, MGL Gustafsson recommended this two-step method in his paper entitled "Surpassing the lateral resolution limit by factor of two using structured illumination microscopy," published in the Journal of Microscopy, Vol. 198, No. 2, May 2000, pp. 82-87. For SIM image reconstruction, the paper suggests multiplying each of the three frequency components by the microscope's OTF. After this, the components are added to their overlap. Finally, the sum of the components is divided by the sum of squares of the OTF and a small constant. The paper further shows that this process is equivalent to a one-step Wiener filtering.

[0466] The Wiener filtering for the three components of equations 32, 33, and 34 is shown below (excerpted from Lal et al., 2015):

[0467] (32)

[0468] (33)

[0469] (34)

[0470] Lal et al. (2015) obtained three frequency components from the left-hand side of equations 32, 33, and 34. u(k) (k) u(kp θ ) (k) u(k+p θ ) (k) is called the noise estimation by applying Wiener filtering. (k) (k) (kp θ ) (k) (k+p θ ) (k) Perform so-called ungraded estimation. The term ungraded appears to be borrowed from the ungraded mesh in finite element analysis. Lal et al.'s SIM-RA algorithm 1 (2015) operates under the assumption that the original SIM image is corrupted by white noise. The expressions in square brackets in equations 32, 33, and 34 are called Wiener filters, and m It is the modulation factor.

[0471] Gonzales and Woods, in their book "Digital Image Processing," on page 351, in chapter 5.7, discuss noise estimation. , (kp θ ) (k) (k+p θ ) (k) is called the degraded image. Gonzales and Woods used OTF... (k) is called the degradation function. In a noise-free ideal system, a simple "inverse filter" can be applied to retrieve the ungraded image. However, the acquired image contains most of the real-world system noise. Therefore, the inverse filter cannot be used to recover the ungraded components. (k) (kp θ )and (k+p θ The Wiener filter recovers the graded image by minimizing the mean square error between the degraded and graded images. Therefore, the Wiener filter is also known as the "minimum mean square error" filter. Further details about the Wiener filter are presented in Chapter 5.8 on page 352 of Gonzales and Woods' "Digital Image Processing".

[0472] Wiener filtering or Gustafsson's two-step method allows us to recover approximate values ​​of the ungraded components, which are referred to as ungraded estimates on the left-hand side of equations 32, 33, and 34 of Lal et al. (2015), as shown above.

[0473] Another important observation by Lal et al. in equations 32, 33, and 34 (2015) is that the molecule of equations 32, 33, and 34 is OTF. The conjugate of (k). If the point spread function (PSF) of the optical system... H If (r) is a real number and symmetric, then it means that OTF (k) is a real number. For real numbers, OTF is equal to its conjugate, i.e. (k)= (k). Furthermore, if OTF is positive, then MTF (the absolute value of OTF) equals OTF, i.e. Based on our previous observations, this further means This observation indicates that MTF can be used as a numerator in equations 32, 33, and 34 (when OTF is a real number and positive), which means that step 3-c of the process is equivalent to combining the numerators of equations 32, 33, and 34 with the three degenerate components of the three equations 32, 33, and 34, respectively. , (kp θ ) (k) (k+p θ ) (k) multiply.

[0474] In the non-redundant SIM sub-block reconstruction algorithm, we multiply the non-redundant matrices 1831, 1833, and 1835 obtained after the band separation step (step 3-b). This is an element-wise scalar multiplication and does not change the dimension of the resulting non-redundant matrices. After this multiplication, we obtain three non-redundant left-hand matrices 1831, 1833, and 1835. Figure 18 As shown in the diagram, after applying the OTF, as explained in the band separation step, the symmetry in the three full matrices remains unchanged.

[0475] Average of the first column and the middle column

[0476] The computational features we present can be used to improve the quality of reconstructed images using non-redundant SIM image reconstruction algorithms. Figure 23B In the diagram, we present the Fourier space matrices of two high-frequency images. In the non-redundant algorithm, we use the left halves of two matrices labeled 1843 (left half 1) and 1845 (left half 2). These two matrices are shown together with the separated right half matrices 1846 (right half 1') and 1844 (right half 2'). Figure 23B In the top row. The labels of the left and right halves of the two matrices and Figure 18 Consistent (fourth row from the top). The non-redundant algorithm uses the left halves of two high-frequency matrices during image reconstruction. Due to Fourier space symmetry, the right half can be reconstructed using a lookup of the corresponding left half matrix. Figure 23B The second row of the matrix shows how the left half of a high-frequency image is stitched to the right half of a second high-frequency image during image reconstruction. These stitched matrices are also shown in... Figure 18 In lines 3 and 4.

[0477] In this specific implementation, the left half of the matrix of the two high-frequency images has two more columns than their corresponding right half. These two columns are the first column and the middle column, which are located in... Figure 23B The images are shown in shaded patterns. The first and middle columns of 1843 and 1845 represent the DC components, which will be matched using perfect instrumentation and measurements. Due to defects in the instruments used to capture the images (at different phases) and the introduction of noise in the process, the first and middle columns of the two high-frequency matrices may have different values. Reconstruction can be improved by forcing the first and middle columns of 1843 and 1845 to have the same values.

[0478] Averaging the first and middle columns of the non-redundant components of the matrices of the two high-frequency components is one way to match these columns in 1843 and 1845, thereby improving the quality of the reconstructed image by introducing symmetry in the values ​​of these columns. Figure 23C This shows the average of the first and middle columns of the left half matrix of two high-frequency images in Fourier space. The average of the first and middle columns of the two matrices ensures that the resulting matrix satisfies Fourier space symmetry. The averaged first and last columns are... Figure 23C It is shown in the middle with a crosshair pattern.

[0479] Other techniques can also be used to match the values ​​of these columns in matrices 1843 and 1845. For example, we can use values ​​from the first and middle columns of the first or second high-frequency image and use those values ​​in matrices 1843 and 1845. Alternatively, a weighted combination of the values ​​from the first and middle columns of the two high-frequency images can be used. For example, these values ​​can be combined by assigning two-thirds of the weight to the values ​​of one high-frequency image and one-third of the weight to the values ​​of the other high-frequency image. It should be understood that different relative weights can be assigned to combine the values ​​of the corresponding columns.

[0480] Step 3-d (sub-pixel offset):

[0481] In this step, high-frequency components Each of the components is offset to overcome the quantization effects of the Discrete Fast Fourier Transform (OTF). Since the pixel positions in the Fourier space matrix correspond to discrete frequency ranges, we call this sub-pixel offset. To explain this, Karras et al., in their 2019 paper "Successful optimization of reconstruction parameters in structured illumination microscopy – A practical guide" published in Optics Communications, illustrate that when using structured illumination patterns, higher frequency components are offset into the passband of the OTF (see Figure 1, Karras et al., 2019). During the sub-pixel offset, the true positions of the high-frequency components are respectively at frequencies p in the frequency domain. θ and p θ Centered on. Equations 35 and 36, taken from Lal et al. 2015, illustrate this offset.

[0482] (35)

[0483] (36)

[0484] In existing SIM image reconstruction algorithms, such as the SIM-RA algorithm presented by Lal et al. (2015), two Fourier transform operations are performed on each high-frequency image to shift the high-frequency content to its correct location, as shown in equations 35 and 36. The shifting properties of the Fourier transform allow us to shift the signal. This shift can be accomplished by taking the information from the frequency domain to the spatial domain and then applying a multiplication (also known as applying the transform vector). The Fourier transform is then applied to the shifted signal in the spatial domain to bring it back to the frequency domain. More information on the shifting properties of the Fourier transform can be found at http: / / www.thefouriertransform.com / transform / properties.php. The shifting of high-frequency content is actually performed in real space by performing an inverse Fourier transform on the ungraded image so that the transform vector is not limited to the entire pixel in the discrete Fourier transform. Therefore, we can achieve sub-pixel shifting in real space. The shifted high-frequency components are then transformed back to the frequency domain by applying a Fourier transform.

[0485] We improve the efficiency of SIM image reconstruction and reduce the required computational resources by utilizing symmetry in Fourier space. The so-called DC frequency component does not require sub-pixel offset as in the reference location, such as... Figure 18 As shown. The high-frequency component matrix is ​​offset relative to the DC frequency component matrix.

[0486] In the non-redundant algorithm disclosed in this invention, after combining the non-redundant halves of the complementary phase offset matrix of the high-frequency components, we only offset one high-frequency component matrix, such as... Figure 18 As shown. The combined half of the high-frequency components is shown as the left half 1833 and the right half 1834, or in Figure 18 The middle column of the third row is labeled as matrix 1|2'. This matrix is ​​offset from the Fourier domain to the spatial domain and then back again, instead of the two matrices used in existing SIM image reconstruction algorithms. The offset matrix 1|2' is shown in... Figure 18 In the middle column of the fourth row, two arrows point diagonally upwards to the top left corner. This matrix consists of a left half matrix 1843 and a right half matrix 1844. Using the Fourier space symmetry and conjugate of the 1|2' matrix, we obtain a second offset high-frequency matrix consisting of the left half matrices 1845 and 1846, or labeled 2|1'. We offset it by the same amount as offset 1|2', but in the opposite direction, indicated by two diagonally pointing downwards to the right corner.

[0487] Steps 3a through 3d are repeated in a loop of illumination peak angles. In this example, we use two angles or orientations, so we perform the above steps for a second set of three images obtained with a second orientation of the structured illumination pattern.

[0488] Processing step 4: Reconstructing the sum and constructing the Wiener filter

[0489] The Wiener filter can be constructed once and then repeatedly applied to the filter image even if the parameters change. Supplemental material to the 2019 paper by Karras et al., entitled “Successful optimization of reconstruction parameters in structured illumination microscopy – A practical guide,” published in Optics Communications, illustrates the Wiener filter parameters (…). w The tuning of the Wiener filter parameter is the constant part of the denominator in equations 32, 33, and 34. Figure S1 in the supplementary material by Karras et al. (2019) shows the Wiener filter parameter ( w The effects of different values ​​of the Wiener filter parameters on the retrieved image spectrum are investigated. Low values ​​of the Wiener filter parameters may introduce noise into the retrieved spectrum. Values ​​that are too high may result in image data loss. The Wiener filter parameter values ​​can be selected empirically, as described in the supplementary materials of Lal et al. (2015) and Karras et al. (2019). Other types of Fourier space low-pass filters can be alternatively used to reduce noise. For example, Gaussian low-pass filters, Butterworth low-pass filters, or Hamming low-pass filters. Detailed information on these filters can be found at http: / / fourier.eng.hmc.edu / e101 / lectures / Fourier_Analysis / node10.html.

[0490] The portion of the matrix containing only the high-frequency components overlaps with the so-called DC frequency matrix. The expanded matrix, a combination of coefficients from these multiple Fourier domain frequency matrices, is wider and higher than the individual matrices before combination by an upsampling factor, reflecting an enhancement in image resolution. The upsampling factor can be between 1.0 and 2.0 in the width and height of the individual matrices. When the upsampling factor is 1.0, there is no enhancement in image resolution. There are no high-frequency components, and images captured in all phases along a certain orientation overlap each other. When the upsampling factor is 2.0, the offset matrices of the high-frequency components are separated from each other, such that their edges (or boundaries) touch the center of the unoffset matrix. This means that the width and height of the high-resolution reconstructed image are increased by twice that of the individual image matrices. This combined matrix is ​​shown as follows. Figure 18The large matrix 1850 is shown in the diagram. Matrix 1850 is depicted as being twice as wide and high as the individual matrices labeled 1|2' and 2|1' to illustrate the reconstructed image enhancement. When the upsampling factor used for image resolution enhancement is greater than 1.0 and less than 2.0, the offset matrices 1|2' and 2|1' are positioned closer together and overlap each other at the center of the large matrix 1850.

[0491] exist Figure 18 In the diagram, we show the construction of matrix 1850 within the matrix of the three images at the first angle. The non-redundant half of the DC component for the first angle is shown as... Figure 18 In matrix 1841, we then place the subpixel shift coefficients of one high-frequency image in the upper left part of matrix 1850 and the subpixel shift coefficients of another high-frequency image in the lower right part. The redundant conjugate components required to complete 1850 can be found in the complementary non-redundant components, thus maintaining the matrix position that tracks the complementary components (coefficients) and applying the changed sign to any imaginary part of the replicated coefficients.

[0492] Adding a second orthogonal angle and three additional images (not shown) will fill the square. We place the full matrix of one high-frequency component of the second angle in the lower left part of matrix 1850, and the full matrix of the second high-frequency component in the upper right part.

[0493] Lal et al. (2015) demonstrated combining nine images. The coefficients at the overlapping matrix positions are combined in matrix 1850. This represents a variation of six images, for two angles instead of three.

[0494] The visualization of the combination matrix is ​​clearest when the coefficients are depicted as center-offset, with the midpoint of the frequency range at the center of the matrix. However, calculations can actually be performed using an angular offset matrix, for example, where the lowest frequency component is at the top left or bottom right corner. Figure 18 The image matrix 1850, with its enhanced resolution, is shown by placing low-frequency components at the center and high-frequency components further away. This arrangement of frequency content is called a center-shifted spectrum. Applying a Fast Fourier Transform (FFT) to the image data in the spatial domain results in an angularly shifted spectrum, where the amplitudes of the low-frequency components terminate at the corners of the two-dimensional spectrum in Fourier space (or the frequency domain). The high-frequency components are positioned closer to the center of the two-dimensional spectrum. For ease of understanding, the frequency content in the frequency domain can be viewed as a center-shifted spectrum, where low-frequency components are positioned closer to the center and high-frequency components are positioned further away. The symmetry of the quadrants in the angularly shifted spectrum allows for diagonal swapping of quadrants to bring the low-frequency components closer to the center. Therefore, an angularly shifted spectrum can be directly converted to a center-shifted spectrum, and vice versa. Combination symmetry and redundant regions are maintained in both the angularly shifted and center-shifted spectra. For the reader's convenience, a diagram is presented. Figure 18The center-shifted spectrum in.

[0495] For the band-separated image in step 3-c, a Wiener filter can be constructed for the combined image using the one-step (Lal et al. 2015) or two-step (Gustafsson 2000) technique presented above. For example, in one implementation, the Wiener filter can be constructed using the amplitude or absolute value of the MTF (modulation transfer function) of the OTF (optical transfer function). In this implementation, we perform a reconstruction process to computationally simulate the overlapping frequency information in the Wiener filter. We apply the Wiener filter weights from the band-separated image to the OTF of the combined image to construct the combined image for the Wiener filter. The Wiener filter is then applied to the combined image in the next step.

[0496] Processing step 5: Apply Wiener filter and apodization filter

[0497] In this step, we apply the Wiener filter or an alternative filter established in the previous steps to the combined image in Fourier space. In step 3-c, we divide the Wiener filtering into two steps by applying independent MTFs. See equations 32, 33, and 34 in step 3-c. As a two-step filtering process, this step involves applying the remainder of the Wiener filter (presented in equations 32, 33, and 34) to the combined image in Fourier space. In an alternative implementation, we can apply the Wiener filter to the Fourier space representations of the three frequency components at each angle and then combine them to form the combined image. When applied early, we can apply the Wiener filter only to the non-redundant half of the matrix that separates the frequency components by frequency bands.

[0498] The application of the Wiener filter enhances the signal, but it can also increase noise in the image. A properly designed apodization filter can avoid hard edges introduced into the image due to high-frequency content. An apodization filter is a low-pass filter that removes frequency content outside a predefined radius in Fourier space. The apodization filter reduces the error introduced by the Wiener filter. Other low-pass filters, such as real-space Hamming low-pass filters or Gaussian low-pass filters, can also be applied to reduce the error introduced by the Wiener filter. Note that the apodization filter is optional and is not used by some SIM reconstruction algorithms (such as the SIM-RA algorithm of Lal et al., 2015). Alternatively, the apodization filter can be applied only to the non-redundant half of the matrix of the first three frequency components in the pipeline.

[0499] Processing step 6: Perform inverse FFT

[0500] In this step, we apply the inverse Fourier transform to matrix 1850 of the full image in the frequency domain. The inverse fast Fourier transform (IFFT) is then applied to transform the matrix components in the Fourier domain into the image in the spatial domain (in real space). The resulting matrix 1860 is the high-resolution SIM-reconstructed image. When using sub-blocks, it is a portion of a larger block.

[0501] SIM Image Reconstruction for Full FOV Images

[0502] Sub-blocks of a high-resolution SIM image can be combined to reconstruct a high-resolution image of the full field of view (FOV). The following presents pseudocode for a specific embodiment of the technology disclosed in this invention to reconstruct a high-resolution full-field-of-view image from multiple sub-block images. The algorithm presented below is referred to as the "Full Field of View Reconstruction" algorithm (also known as Algorithm 4). It extends Algorithm 3 for sub-block reconstruction. It takes the acquired image and distortion coefficients with applied estimated parameters as input. The above presents estimates of the distortion coefficients for angle and spacing parameters (Input 2) and a phase deviation lookup table (Input 3). The algorithm divides the full FOV image into sub-blocks of equal size. As presented in Algorithm 3 above, SIM images are reconstructed for the sub-blocks. Finally, the high-resolution SIM images of the sub-blocks are combined to reconstruct a high-resolution SIM image of the full FOV data (Algorithm step 4b).

[0503] Algorithm 4: Full Field of View Reconstruction

[0504] enter:

[0505] 1. A set of images, such as two angles and three phases per angle.

[0506] 2. Angle / spacing distortion coefficients (see Final Distortion Calibration using Quadratic Fitting and Look-Up Table Algorithms)

[0507] 3. Distortion lookup table for phase (see Final distortion calibration using quadratic fitting and lookup table algorithms)

[0508] 4. Wiener filter and apodization parameters

[0509] Output: Rebuild output

[0510] algorithm

[0511] 1. Construct the following image / filter (size designed to match the sub-regions).

[0512] a. OTF

[0513] b. Wiener Method

[0514] c. Atomic method

[0515] 2. Divide the image into equal-sized sub-regions, for example, 512×512.

[0516] a. If the image is not divisible by the size of the sub-region, then offset the sub-region to the left (or up) at the right edge or bottom so that it falls on the image.

[0517] b. Set up bookkeeping for boundary sub-regions to ensure that only non-overlapping portions are copied to the right or bottom edge sub-regions.

[0518] 3. Use, for example, a 512×512 window to estimate the parameters of the image center (see Sub-block Angle and Spacing Mapping Algorithm and Sub-block Phase Mapping Algorithm).

[0519] a. Estimate the peak illumination angle and spacing at each angle.

[0520] b. Estimate the modulation at each angle

[0521] c. Estimate the phase of each image

[0522] 4. For each sub-region

[0523] a. Reconstruct higher resolution sub-regions using sub-block reconstruction algorithms.

[0524] i. Use the grating and angle found in step 3, along with the given distortion coefficients, to estimate the local grating and angle.

[0525] ii. Use the phase found in step 3 and the given distortion lookup table to estimate the correct phase.

[0526] iii. Based on the previously constructed OTF, Wiener method, and apodization method for sub-regions

[0527] b. Copy the output from (a) to the large image.

[0528] i. If it is a boundary sub-region, only copy the missing data.

[0529] In the context described above, this pseudocode should be self-interpretive.

[0530] Potential improvements to SIM image reconstruction algorithms

[0531] We present potential improvements to the SIM image reconstruction (SIR) algorithm in the following patents: U.S. Provisional Patent Application No. 62 / 924,130, entitled “SYSTEMS AND METHODS FOR STRUCTURED ILLUMINATION MICROSCOPY” (Attorney’s File No. ILLM 1012-1); and U.S. Provisional Patent Application No. 62 / 924,138, entitled “INCREASED CALCULATION EFFICIENCY FOR STRUCTURED ILLUMINATION MICROSCOPY” (Attorney’s File No. ILLM 1022-1). Both provisional applications were filed on October 21, 2019.

[0532] The first potential improvement relates to the phase bias lookup table. The Baseline SIR algorithm uses a static bias lookup table. The baseline model assumes that phase bias and distortion can be learned once and stored in the lookup table. It also assumes that the values ​​in the lookup table are stable and applicable across multiple sequencing runs. The Baseline SIR algorithm assumes that the phase bias lookup table (LUT) remains stable in the spatial domain. This means that the same lookup table can be applied to blocks in the flow cell. Our second assumption in the Baseline algorithm is that the LUT remains stable across multiple sequencing runs. Experiments show that these assumptions may not be true for all flow cells and sequencing runs. Blocks in the flow cell can have different tilts or other distortions. The values ​​in the lookup table can be affected by various environmental factors that change over multiple sequencing runs.

[0533] Therefore, the "All In" SIM algorithm involves learning a phase bias lookup table for each sequencing run for each block, after loading a new flow cell into the sequencer and starting the sequencing run. This increases the processing time required for the first cycle of the sequencing run. However, significant improvements in error rate and the percentage of clusters measured by filtering (or %PF) are observed, as shown in the following paragraphs.

[0534] The second potential improvement involves geometric distortion, which causes parallel lines to bend or otherwise become distorted. Geometric distortion can lead to distortion in imaging or optical paths, as well as in the flow cell. In some cases, all or some blocks in the flow cell may be tilted and may not be flat. This can cause geometric distortion. Characterizing all blocks in the flow cell is preferred because variations may exist between blocks within the flow cell. Early versions of the correction assumed that geometric distortion could be learned once and used in multiple sequencing runs. Experiments showed that this assumption may not be true for all sequencing runs. Therefore, the "All In" algorithm includes learning the geometric distortion of each block in the flow cell in the first loop of the sequencing run.

[0535] A third potential improvement involves stitching sub-block images to form a full field-of-view (FOV) image. In the Baseline SIM algorithm, when the Inverse Fast Fourier Transform (IFFT) is applied to sub-blocks in the frequency domain, the edges of the sub-blocks are readily apparent in the spatial domain. Therefore, stitched sub-blocks in the full field of view show stitching lines at the sub-block edges. The stitching line problem causes dark artifacts to appear along the boundaries of the sub-blocks. The All-In SIM algorithm addresses the stitching line problem by using overlap between sub-blocks. For example, sub-blocks can have an overlap of 5 to 15 pixels. In one specific implementation, sub-blocks have an overlap of 8 pixels along the boundary. When reconstructing the full field-of-view (FOV) image, the All-In algorithm discards the outer 8 pixels of the sub-block and reconstructs the full field-of-view image in the spatial domain using the inner regions. In this specific implementation, by adding border pixels (e.g., 8 pixels) on each side, the size of the sub-block can be larger than 512 pixels × 512 pixels. In another specific implementation, the boundary pixels of a sub-block can be averaged instead of discarded.

[0536] The fourth potential improvement involves optimizing the apodization and Wiener filters. The filter parameters depend on the upsampling. The Baseline SIM algorithm uses 1.2x upsampling, which culls some pixels. This can lead to image shifting and alignment problems when sub-block images are stitched together in the spatial domain. Pixel culling is due to the upsampling factor (i.e., 1.2) that produces non-integer pixels. The All-In SIM algorithm uses 1.5x upsampling, overcoming the partial pixel culling problem by using an upsampling factor that produces integer pixels. Therefore, a sharper sub-block image is provided for stitching together. In other specific implementations, other upsampling factor values, such as 1.25x upsampling, can be used, which also produces integer pixels.

[0537] In some specific implementations, the techniques disclosed in this invention can use dose ramping to compensate for nucleotide damage caused by exposure cycles during sequencing runs. During dose ramping, the laser power of the blue and green channel lasers is increased.

[0538] Performance comparison results

[0539] We compared the performance of the Baseline SIM algorithm with that of the All-In SIM algorithm. The two algorithms were compared using sequencing result quality indicated by error rate and the percentage of clusters through the filter (%PF). The error rate metric measures the percentage of incorrectly detected bases when compared to a known reference genome during sequencing. The cluster through-filter or %PF metric is an indicator of the signal quality from each molecular cluster on the flow cell. Lower error rates and higher %PF values ​​are expected, as they indicate good sequencing output with high-quality base detection. We selected the best blocks from over one hundred blocks on the flow cell for each sequencing run to compare the algorithms. Blocks in the flow cell can be variable in terms of flatness, tilt, surface polishing, and other environmental factors. The reason for selecting the best blocks from over one hundred blocks for comparison is to reduce the variability caused by these other external factors through statistical summarization.

[0540] Figure 25A The error rates for the Baseline and All-In algorithms are presented for three sequencing runs (labeled A, B, and C). For each sequencing run, the error rate of the best block is presented for both algorithms. The top column indicates the error rate of the best block in the All-In algorithm, and the bottom column indicates the error rate of the best block in the corresponding sequencing run in the Baseline algorithm. The table at the bottom of the chart lists the quantitative improvement in the best block error rate when using the All-In algorithm. For example, the best block error rate for sequencing run A is 1.85% when using the Baseline algorithm. When using the All-In algorithm, the best block error rate for sequencing run A is 1.18%, indicating a higher quality sequencing process with a lower error rate. Similar improvements in error rate reduction were observed for sequencing runs B and C when using the All-In algorithm.

[0541] Figure 25B The results of cluster filtering (or %PF) for the optimal block in three sequencing runs for both algorithms are presented. The table below shows the %PF values ​​for the optimal block for each algorithm in their respective sequencing runs. The All-In algorithm results in higher %PF values ​​for all sequencing runs. Specifically, the %PF value for the optimal block in sequencing run A is 82.7, indicating that 82.7% of the clusters on the flow cell block are considered good clusters for generating base detection information. The %PF value for the optimal block in the other sequencing run is only 70.1. Higher %PF values ​​result in more clusters passing through the filter, thus increasing sequencing throughput. Patterned flow cells can theoretically limit the number of clusters, which is equal to the number of nanopores in which molecular clusters can be formed. Figure 25A and Figure 25B The results show that, compared to the Baseline algorithm, the All-In algorithm results in a lower error rate and higher throughput (due to a higher %PF value).

[0542] Figure 26A , Figure 26B and Figure 26C The optimal block error rates for sequencing runs A, B, and C are presented separately. Separate time series of error rates during sequencing runs are presented for the Baseline and All-In algorithms. Figures 26A to 26C The results show that the All-In algorithm reduces the error rate during sequencing runs. The improvement in error rate for optimal blocks using the All-In algorithm may vary depending on the cycle. Figure 26A In the initial 25 cycles, the error rates of the Baseline and All-In algorithms were almost the same. However, as the sequencing run progressed, the error rate of the All-In algorithm decreased compared to that of the Baseline algorithm. Figure 26B The results show that the All-In algorithm has a slightly better error rate in the initial loop. In subsequent loops of the sequencing run, the All-In algorithm outperforms the Baseline algorithm. Figure 26C The optimal block error rates for sequencing runs C using both algorithms are shown. For most cycles, the All-In algorithm significantly outperforms the Baseline algorithm. These figures also illustrate specific cycles where the All-In algorithm exhibits a higher error rate. These results can help technicians or researchers focus on quality issues leading to increased error rates in specific cycles.

[0543] Figure 26D The improvement in error rate achieved through specific updates to the All-In algorithm compared to the Baseline algorithm is presented. For example, the left figure shows the improvement in the error rate of sequencing run A by including phase learning and splice repair. Phase learning refers to the estimation of phase bias, which captures the relationship between the phase shift values ​​of non-centrifugal sub-blocks relative to centrifugal sub-blocks. A separate phase bias lookup table is generated and stored for each block of the flow cell. The error rate of the Baseline algorithm used for sequencing run A is 1.85, as shown in the figure. Figure 25A The first column of the table shows that when phase learning is included in the Baseline algorithm, the error rate is improved by 0.65% or 32%, and when stitching fixation is included in the Baseline model, the error rate of the Baseline algorithm is improved by 0.19% or 10%. The figure can also be interpreted as phase learning reducing the error rate by 0.65% and stitching fixation reducing the error rate by 0.19% respectively in the Baseline algorithm. The residual error rate after including both techniques is 1.18%.

[0544] Figure 26DThe chart on the right illustrates the improvement in error rate resulting from the phase learning and splicing fixation techniques used in sequencing run B. Including phase learning in the baseline algorithm reduces the error rate by 0.227% or 33%. Including splicing fixation in the baseline algorithm reduces the error rate by 0.039% or 6%. The residual error rate after incorporating both techniques is 0.414%. We now present a discussion of the impact of the disclosed techniques when the SIM parameter estimation and image reconstruction techniques disclosed in this invention are applied to processing images from sequencing runs.

[0545] Impact on the technology disclosed in this invention

[0546] Simplified examples are presented to illustrate how the techniques disclosed in this invention can reduce the computational resource requirements for structured illumination image enhancement on flow cells over multiple cycles and several hours. Machine capabilities vary, so approximately integer values ​​were chosen to make the example computations easy to follow. Remember that the techniques disclosed in this invention can be applied independently, alone, or in combination. In some specific implementations, parameters can be estimated more frequently to compensate for system instabilities due to environmental factors. For example, as described above, the “All-in” SIM algorithm estimates the phase bias lookup table in the first cycle of each sequencing run. This can increase the duration of the first cycle by up to 30%. However, significant performance improvements in error rate and %PF metrics can be achieved.

[0547] Assume the sequencing run involves 150 analysis cycles, with two illumination frequencies at a rate of 6 minutes per cycle. Basic image block capture within the 150 cycles takes 15 hours. Assume two angles (+ / - 45 degrees) are used for structured illumination, with three images per angle per cycle. The number of images per block within the 150 cycles is 1800 images. Assume 70 sub-blocks exist within a 7×10 grid. The processing requirement is 126,000 sub-blocks within 900 minutes.

[0548] In this first embodiment, it is assumed that the system has been calibrated before operation and that a quadratic fit plus a lookup table generated for the spacing, angle, and phase difference can be used throughout the operation. In this embodiment, the near-center sub-block will be subjected to 1800 fringe peak estimates. This is the number of times the flow cell position or the phase of the illumination laser source or projected structured illumination changes.

[0549] In this first example, the system will avoid estimating 69 out of 70 fringe peaks. For the 124,200 image sub-blocks, the remaining sub-blocks will use pre-computed relationships with near-center sub-blocks to determine reconstruction parameters. Estimating reconstruction parameters is computationally more expensive than the actual reconstruction. The computational requirements are reduced by more than half. This can be further reduced by using the outlined dependence on symmetry.

[0550] The application of symmetry allows for the finding of conjugate pairs, rather than computation via FFT and other methods. Special attention needs to be paid to sub-blocks with reflection symmetry, rather than the dominant 180-degree rotation and sign-change symmetry. In cases where the number of rows and columns in the angularly offset Fourier transform space is even, the top row and the left and middle +1 columns are treated differently due to reflection symmetry rather than rotational symmetry. See also Figure 19 In the case where the number of rows and columns in the angularly offset Fourier transform space is odd, the top row and left column are treated differently due to reflection symmetry rather than rotational symmetry. See also Figure 19 and Figure 23A Other symmetries can be applied to the center-offset Fourier transform space.

[0551] Figure 18 The central diagram illustrates how half of the frequency domain computation is avoided in each step by focusing on non-redundant data. This is assumed to reduce computation by 40% and image coefficient storage by 50% in these steps until the entire image is reconstructed at 1850 and 1860. During steps 3b through 3d, conjugate pairs do not need to be included in the computation because value lookup can be applied when values ​​for the conjugate pairs are needed at 1850 and 1860.

[0552] Therefore, one technique disclosed in this invention can save 50% of the required computing resources, and another can save 40%. Cumulatively, the computing overhead of SIM reconstruction can be reduced by more than two-thirds.

[0553] In the second specific implementation, it is assumed that the system is calibrated at the start of operation and recalibrated at the midpoint to update the quadratic fit and lookup table. This calibration is assumed to take 8 minutes because it involves estimating the fringe peaks for all 70 sub-blocks, followed by the fit and lookup table generation. This estimation extends the runtime from 15 hours to 15 hours and 16 minutes. If the technique disclosed in this invention is not applied, and instead estimation is performed for each sub-block and exposure frequency, in groups of three for each image, totaling 300 parameter estimates, the runtime would be more than three times longer.

[0554] In the second specific implementation, the effect of applying symmetry will not change.

[0555] The combined effect of the technologies disclosed in this invention is expected to be enabling real-time processing of image blocks using a CPU without adding a GPU, FPGA, or CGRA to the system. Alternatively, even if a GPU, FPGA, or CGRA implementation is chosen, simpler computing resources can be used. This reduces system complexity and facilitates maintaining the form factor of existing machines while increasing their real-time processing capabilities.

[0556] Specific implementation

[0557] Sub-blocks of the full field of view captured by the sensor

[0558] The subblocking technology disclosed in this invention can be described from a calibration and / or production perspective using a calibrated scanner.

[0559] In a specific calibration embodiment of the technology disclosed in this invention, a method for calibrating or characterizing a scanner that detects fluorescence in millions of samples distributed on a flow cell or more generally an imaging plane in images collected over multiple cycles. Structured illumination is used to improve the resolution between the fluorescent samples, the distances between them being positioned closer than the Abbe diffraction limit of optical resolution. This calibration scanner, using a substantially full field-of-view method projected onto an optical sensor in the scanner by a lens, involves capturing image blocks under structured illumination with multiple angles and phase shifts of the structured illumination. Variations of this method can be achieved using 2D structured illumination and different phase shift step modes. The method involves calculating optical distortion in the image block, including measuring the spacing and angle between intensity peaks in at least nine sub-blocks (including a near-center sub-block) of the image block, and fitting a spacing surface and an angle surface to the measured spacing and angle, respectively. Of course, other configurations for dividing the image block, such as 3×3, 5×5, 5×7, 9×9, 8×11, and 9×16 sub-blocks, can be used. Images captured by a larger sensor are divided into more sub-blocks, not limited to the listed array of sub-blocks. Fitted spacing and angle surfaces represent corrections for distortion in the sub-blocks of the image block. The resulting fitting coefficients are saved, including fitting results that correlate the spacing and angles of intensity peaks on at least nine sub-blocks with the near-center sub-block. Calibration further includes measuring the phase shift of the structured illumination in terms of multiple angles and phase shifts within the subfield, and saving a lookup table representing the difference between the phase shift in the near-center sub-block of the image block and the phase shifts in other sub-blocks. This calibration prepares the scanner for production use in analyzing the fluorescence of millions of samples distributed across a flow cell or imaging plane in images collected over multiple cycles.

[0560] In a production implementation, this method produces enhanced images of fluorescence from millions of samples distributed across a flow cell or imaging plane from images collected over multiple cycles. This involves processing captured image blocks covering locations on the flow cell within multiple cycles, these blocks being captured with multiple angular and phase shifts of structured illumination. The processing is applied to sub-blocks within each image block. It includes accessing or generating estimated fits representing the spacing and angular distortion relationships between near-center sub-blocks and other sub-blocks. These fits can be coefficients of a polynomial surface fit or values ​​from a lookup table. The method also includes accessing or generating a lookup table representing the phase shift differences between near-center sub-blocks and other sub-blocks. For the near-center block of each captured image block, the fits and lookup tables are combined with measurements of the spacing between intensity peaks, the angle of the intensity peaks, and the phase shift of the structured illumination. Reconstruction parameters for each captured image block are determined on a sub-block basis and then applied to SIM reconstruction. This method produces enhanced resolution images with superior Abbe diffraction limit for sub-blocks at locations on the flow cell or imaging plane within multiple cycles.

[0561] For example, for an image block divided into nine sub-blocks, the method includes deriving estimated reconstruction parameters for at least eight additional sub-blocks and a near-center sub-block of the image block. Estimated spacing and angle reconstruction parameters for the additional sub-blocks are derived by combining accessed or estimated parameters of the near-center sub-blocks with saved fitting results that correlate the spacing and angle parameters on the at least eight additional sub-blocks with the near-center sub-block. Estimated phase displacement reconstruction parameters for the additional sub-blocks are derived by combining accessed or estimated parameters of the near-center sub-blocks with a saved lookup table of phase displacement measurements relative to the near-center sub-block on the at least eight additional sub-blocks. As described above, other configurations for dividing the image block can be used without affecting the parameter estimation process described above.

[0562] Enhanced resolution images can be used to sequence samples in multiple cycles, or otherwise to study fluorescence on an image plane captured by dozens, hundreds, or even thousands of image blocks.

[0563] In the combined calibration and production implementation, the above actions are combined, thus calibrating the scanner and then using it for production. The technology disclosed in this invention includes a method embodiment for improving the performance of a scanner that detects fluorescence in millions of samples distributed on a flow cell in images collected over multiple cycles. Structured illumination is used to improve the resolution between fluorescent samples, the distances between them being positioned closer than the Abbe diffraction limit of optical resolution. Applying this technique, the scanner is calibrated using a substantially full field of view projected onto an optical sensor in the scanner by a lens, including capturing image blocks under structured illumination with multiple angles and phase shifts of the structured illumination. It involves calculating optical distortion in the image block, including measuring the spacing and angle between intensity peaks in at least nine sub-blocks (including a near-center sub-block) of the image block, and fitting a spacing surface and an angle surface to the measured spacing and angle, respectively. Of course, as mentioned above, the image block can be divided into more sub-blocks. The fitted spacing and angle surfaces represent corrections for distortion in the sub-blocks of the image block. The resulting fitting coefficients are saved. The calibration further includes measuring the phase displacement of the structured illumination with multiple angles and phase displacements within the subfield, and maintaining a lookup table that represents the difference between the phase displacement in the near-center sub-block of an image block and the phase displacement in other sub-blocks.

[0564] This method processes captured image blocks covering locations on the flow cell within multiple cycles, capturing image blocks with multiple angles and phase shifts of structured illumination. For each captured image block in a near-center block, this involves combining saved fitting results and saved lookup tables with measurements of the spacing between intensity peaks, the angle of the intensity peaks, and the phase shift of the structured illumination to determine reconstruction parameters for each captured image block on a per-sub-block basis. It further includes generating enhanced-resolution images with better-than-Abbe diffraction limits for sub-blocks at locations on the flow cell within these multiple cycles, and using the enhanced-resolution images to sequence samples from multiple cycles.

[0565] Each of the aforementioned combined calibration and production implementations has its own utility. For example, a first entity can deliver and calibrate a scanner for use by a second entity. The first entity can use different calibration methods to determine and represent the spacing and angular distortion relationships and phase displacement differences between the near-center sub-block and other sub-blocks, and can still use the described production techniques. The described calibration techniques can be used, and then the second entity can use the calibrated scanner in different ways. Therefore, the techniques disclosed in this invention include calibration and / or production implementations.

[0566] The method and other specific embodiments of the technology disclosed in this invention may each optionally include one or more of the following features and / or features described in combination with the additional methods disclosed in this invention. For the sake of brevity, the combinations of features disclosed in this application are not listed separately and are not repeated with each basic feature group. The reader will understand how easily the features identified in this section can be combined with the basic feature sets identified in specific embodiments.

[0567] When the imaging plane is a flow cell, the sample can be distributed across millions of nanopores within the flow cell. At least some adjacent pairs of nanopores can be positioned closer than the Abbe diffraction limit of optical resolution. Structured illumination allows for such close positioning. Alternatively, the sample can be randomly distributed across the flow cell.

[0568] Each sub-block comprises at least 512 pixels × 512 pixels from the optical sensor. Larger or smaller numbers can be used, including 256 pixels, 400 pixels, 1024 pixels, 2048 pixels, and 4096 pixels, or within the range of 256 to 4096 pixels. Performance improves when the dimensions of the sub-blocks are powers of 2; we found that 512 pixels and 1024 pixels were the best-performing practical ranges.

[0569] Sub-blocks can overlap with the optical sensor by at least 2 pixels. Larger or smaller numbers can be used. For example, for a wide window of 512 pixels, up to 256 pixels of overlap can be used, and for a wide window of 1024 pixels, up to 512 pixels of overlap can be used.

[0570] The fitting results for the spacing and angle surfaces can be stored as coefficients of second-, third-, or higher-order surfaces, or in a lookup table calculated based on the polynomial fitting of the spacing and angle surfaces.

[0571] In some implementations, one or more remeasurements of the spacing and angle between intensity peaks in sub-blocks can be performed during a sequencing run, the spacing and angle surfaces can be refitted, and the refit results can be saved for processing subsequently captured image blocks. For example, this can be done in an intermediate cycle of at least 100 cycles, such as the 50th cycle or later. In some implementations, remeasurements can be performed when computational resources are available, such as at the end of a sequencing run or during the time between two reads while a chemical step is in progress for the next read.

[0572] Nanopores can be arranged in a repeating pattern. For rectangular patterns, two structured illumination angles can be used, substantially along the two diagonals connecting opposite corners of the rectangles in the pattern, such that the intensity peaks of the structured illumination are substantially orthogonal to these two diagonals. For repeating hexagonal patterns of nanopores, where three diagonals connect opposite corners of the hexagons in the pattern, three structured illumination angles can be used, where the intensity peaks are substantially orthogonal to the three diagonals. Alternatively, samples can be randomly distributed on an imaging plane without nanopores. Alternatively, samples can be regularly arranged on the imaging plane using methods other than nanopores.

[0573] The method may further include aggregating the enhanced resolution images used for sub-blocks into enhanced resolution images used for blocks, and performing further analysis using the enhanced resolution image blocks. Further analysis may include sequencing the samples, one location per cycle.

[0574] As an improvement to the surface fitting, the method may further include determining a clipping margin and applying it to remove pixels around the sensor edges from the calculation without consideration before fitting the spacing and angle surfaces to the measured spacing and angle. This clipping margin may be uniform around the sensor edges, or it may vary with the edges, such as being wider at the ends of a rectangular sensor than along the longer edges.

[0575] The method may further include measuring the spacing between intensity peaks and the angle of the intensity peaks on at least 8 × 11 sub-blocks of the image block. As described above, other configurations for dividing the image block into sub-blocks may be used.

[0576] The sub-blocking technique disclosed in this invention can be used to generate enhanced resolution images from target images captured under structured illumination. In a specific embodiment of the image reconstruction technique disclosed in this invention, a method is described for dividing a captured image of a target captured by an optical sensor into nine or more sub-blocks and operating independently on each of these nine or more sub-blocks. As described above, more sub-blocks can be used to further subdivide the captured image. The method includes transforming at least three images of the target in the respective sub-blocks into the Fourier domain to generate at least three frequency domain matrices for the respective sub-blocks. These images are captured by a sensor in the spatial domain while at least three phase shifts of structured illumination are applied along an angle. An inverse mixing matrix is ​​calculated using estimated reconstruction parameters. The inverse matrix is ​​applied to the frequency domain matrices to generate at least three phase-separated matrices in the Fourier domain, these matrices being un-shifted and shifted matrices. Sub-pixel shifting is performed on at least one shift matrix. Sub-pixel shifting includes transforming the shift matrix from the Fourier domain to the spatial domain, applying a transformation vector to the data in the spatial domain, and transforming the data from the spatial domain data. Subpixel offsets generate two or more realigned offset matrices in the Fourier domain. The method involves aligning and summing the overlap values ​​of the unoffset and realigned offset matrices to produce an extended frequency coverage matrix. The extended frequency coverage matrix undergoes an inverse Fourier transform to generate enhanced-resolution sub-blocks in the spatial domain. Optionally, an enhanced-resolution image of the target can be generated by merging nine or more enhanced-resolution sub-blocks.

[0577] The method and other specific embodiments of the technology disclosed in this invention may each optionally include one or more of the following features and / or features described in combination with the additional methods disclosed in this invention. For the sake of brevity, the combinations of features disclosed in this application are not listed separately and are not repeated with each basic feature group. The reader will understand how easily the features identified in this section can be combined with the basic feature sets identified in specific embodiments.

[0578] The processing steps of the reproducible method produce sequences of images with enhanced resolution. These sequences can then be used to sequence samples imaged by the sensor over multiple cycles.

[0579] Each sub-block comprises at least 512 pixels × 512 pixels from the optical sensor. Larger or smaller numbers can be used, including 256 pixels, 1024 pixels, 2048 pixels, and 4096 pixels, or 256 to 4096 pixels. Performance improves when the sub-block dimensions are powers of 2; we found that 512 pixels and 1024 pixels were the best-performing practical ranges.

[0580] The sub-block can overlap with the optical sensor by at least 2 pixels. Larger or smaller numbers can be used; for example, for 512, an overlap of 256 pixels would be appropriate, and for 1024, an overlap of 512 pixels would be appropriate.

[0581] The method may further include aggregating the enhanced resolution images of the sub-blocks into an enhanced resolution aggregated image, and using the enhanced resolution aggregated image to sequence the sample in multiple cycles.

[0582] This method can be further applied to image blocks located on the flow cell that capture the fluorescence of millions of samples distributed across the flow cell. Such applications of the method produce enhanced images of the flow cell.

[0583] When the imaging plane is a flow cell, the sample can be distributed across millions of nanopores on the flow cell. At least some adjacent pairs of nanopores can be positioned closer than the Abbe diffraction limit of optical resolution. Structured illumination allows for such close positioning. Alternatively, the sample can be randomly distributed on the flow cell. In such a specific implementation, the method further includes generating a sequence of enhanced images of the flow cell and using the sequence of enhanced images to recall a sequence of samples.

[0584] The computer-implemented methods described above can be practiced in systems including computer hardware. The computer-implemented system can practice one or more of the methods described above. The computer-implemented system can incorporate any features immediately preceding or throughout the methods described herein, which are applicable to the methods implemented by that system. For the sake of brevity, no alternative combinations of system features are listed separately. The features applicable to the system, method, and article of manufacture do not repeat the basic features for each group of statutory classifications. The reader will understand how easily the features identified in this section can be combined with basic features in other statutory classifications.

[0585] As an article of manufacture rather than a method, a non-transitory computer-readable medium (CRM) may be loaded with processor-executable program instructions. When the program instructions are executed, one or more of the computer-implemented methods described above are carried out. Alternatively, the program instructions may be loaded on a non-transitory CRM and, when combined with suitable hardware, become part of one or more computer-implemented systems practicing the methods disclosed in this invention.

[0586] Each of the features discussed in this particular implementation section of the method also applies to the CRM and system implementations. As shown above, for the sake of brevity, all method features are not repeated here and should be considered as repeated by reference.

[0587] Non-redundant SIM image reconstruction applications

[0588] The technology disclosed in this invention relates to reducing the computational resources required to generate enhanced resolution images from structured illumination of a target.

[0589] In one specific embodiment of the technology disclosed in this invention, a method for generating an enhanced resolution image from an image of a target captured under structured illumination is described. The method applies one or more transformations to non-redundant data, and then recovers redundant data from the non-redundant data after the transformations.

[0590] The method includes transforming at least three images of a target captured by a sensor in the spatial domain. The sensor captures images from at least three phase shifts of structured illumination along an angle. The three captured images in the spatial domain are transformed to the Fourier domain to produce at least three frequency domain matrices. These frequency domain matrices include at least non-redundant components and redundant conjugate components. The redundant conjugate components are the conjugates of the non-redundant components at complementary matrix locations. The method further includes generating an inverse mixing matrix using estimated reconstruction parameters and applying the inverse mixing matrix to the at least non-redundant components. This produces at least three phase-separated matrices in the Fourier domain, which are un-shifted and shifted matrices. These matrices include non-redundant un-shifted components and non-redundant shifted spatial frequency components derived by applying the inverse mixing matrix to the non-redundant components of the frequency domain matrices.

[0591] The method also includes performing one or more intermediate transformations on the non-redundant components of at least the phase-separated matrices. One intermediate operation is sub-pixel offsetting. This involves transforming at least the non-redundant offset spatial frequency components of the offset matrix from the Fourier domain to the spatial domain. The transformation vector is then applied to the data in the spatial domain, transforming the transformed data back from the spatial domain data to one or more realigned offset matrices in the Fourier domain. Applying the transformation vector effectively realigns the overlap values ​​of the offset matrices in the Fourier domain with the unoffset matrix more accurately compared to the entire matrix position of the discrete Fourier transform. After the intermediate transformation, the redundant conjugate components are recovered from the non-redundant components by copying the transform values ​​from the non-redundant components to the complementary matrix positions of the redundant conjugate components and changing the sign of the imaginary part of the copied values ​​to produce the transformed redundant conjugate components. This reduces the resources required during the intermediate transformation. Constructing the enhanced resolution image further includes aligning and summing the overlap values ​​of the unoffset and offset matrices to produce an extended frequency coverage matrix, which is then inversely transformed from the Fourier domain to produce the enhanced resolution image in the spatial domain.

[0592] In some specific implementations, a series of enhanced-resolution images are generated by repeatedly applying the aforementioned actions. Optionally, for example, in sequencing-by-synthesis, the techniques disclosed in this invention can be applied to use these series of enhanced-resolution images to sequence sequenced samples imaged by sensors in multiple cycles.

[0593] The method and other specific embodiments of the technology disclosed in this invention may each optionally include one or more of the following features and / or features described in combination with the additional methods disclosed in this invention. For the sake of brevity, the combinations of features disclosed in this application are not listed separately and are not repeated with each basic feature group. The reader will understand how easily the features identified in this section can be combined with the basic feature sets identified in specific embodiments.

[0594] The technology disclosed in this invention can be applied to components arranged in an angular offset matrix, wherein the complementary matrix position of the redundant conjugate component is rotated 180 degrees from the matrix position of the non-redundant component.

[0595] The method may further include applying one-step or two-step Wiener filtering during intermediate operations to reduce noise. When applying two-step Wiener filtering, independent modulation transfer functions can be applied to phase-separated matrices as intermediate transforms to compensate for the optical transfer of contrast through objectives that have the ability to reduce transferred contrast with increasing spatial frequency. Noise removal components of the Wiener filtering can then be applied separately. Intermediate operations may further include applying apodization filtering.

[0596] In some implementations, the phase separation matrix in the Fourier domain comprises an even number of rows and even number of columns. For these even-dimensional matrices, column averaging can be used as an intermediate operation to average the DC columns of the DC components, where the intermediate column extends beyond the center column of the even-dimensional matrix. This column averaging can be performed before sub-pixel offsets.

[0597] In some specific implementations, the phase separation matrix in the Fourier domain is angularly offset. Then, the DC component appears in the top or bottom row and left or right column of the matrix, and the redundant conjugate component appears in a block rotated 180 degrees from the orientation of the non-redundant component.

[0598] In some specific implementations, the phase separation matrix in the Fourier domain comprises odd rows and odd columns, without columns between non-redundant component blocks and redundant conjugate component blocks. In the odd-numbered matrix, redundant conjugate components also appear in blocks rotated 180 degrees from the orientation of the non-redundant components.

[0599] In an embodiment where the target comprises regularly spaced nanopores arranged in a rectangular pattern, the method can be applied to a set of at least three images of phase shifts from two angles of structured illumination along the target, wherein the two angles are along the diagonal between opposite angles of the rectangle. Alternatively, in an embodiment where the target comprises regularly spaced nanopores arranged in a hexagonal pattern, the method can be applied to a set of at least three images of phase shifts from three angles of structured illumination along the target, wherein the three angles are along the diagonal between opposite angles of the hexagon.

[0600] The method may further include applying Wiener filtering during intermediate operations in two steps. When performed in two steps, it includes compensating for the optical transfer of contrast, which decreases with increasing spatial frequency, by applying independent modulation transfer functions to a phase-separated matrix to transfer the target under structured illumination through the objective lens to the sensor. It further includes a noise removal component that applies Wiener filtering or another equivalent filter.

[0601] This method may further include applying an apodization filter during intermediate operations.

[0602] The phase separation matrix in the Fourier domain can include an even number of rows and an even number of columns. For such an even-numbered matrix, the method may further include applying a column average during intermediate operations to average the DC columns of the DC components, where the intermediate columns lie between the non-redundant component blocks and the redundant conjugate component blocks. The column average may be performed before the sub-pixel offset.

[0603] When the phase separation matrix in the Fourier domain is offset from the top left corner, the DC component can appear in the top row and left column of the matrix, and the redundant conjugate component appears as a non-redundant component in a block rotated 180 degrees from its orientation. Alternatively, the phase separation matrix can be offset from the bottom right corner.

[0604] The phase separation matrix in the Fourier domain may optionally include an odd number of rows and columns. Such odd matrices do not include intermediate columns between blocks of non-redundant components and blocks of redundant conjugate components.

[0605] This method can be further applied to a set of at least three images of a target captured by a sensor in the spatial domain from phase displacements of two angles from structured illumination along the target. When the target comprises regularly spaced nanopores, the two angles can be oriented substantially along the diagonal between opposite angles of quadrilaterals within the regularly spaced nanopores. These quadrilaterals can be square, and the two angles can be substantially orthogonal, within one to three degrees. Alternatively, the method can be applied to a set of images from phase displacements of three angles from structured illumination along the target. This is useful when the target comprises spaced nanopores opened in a regularly spaced hexagonal pattern. Using this pattern, the three angles can be oriented along the diagonal between opposite angles of the hexagons within the pattern.

[0606] The computer-implemented methods described above can be practiced in systems including computer hardware. The computer-implemented system can practice one or more of the methods described above. The computer-implemented system can incorporate any features immediately preceding or throughout the methods described herein, which are applicable to the methods implemented by that system. For the sake of brevity, no alternative combinations of system features are listed separately. The features applicable to the system, method, and article of manufacture do not repeat the basic features for each group of statutory classifications. The reader will understand how easily the features identified in this section can be combined with basic features in other statutory classifications.

[0607] As an article of manufacture rather than a method, a non-transitory computer-readable medium (CRM) may be loaded with processor-executable program instructions. When the program instructions are executed, one or more of the computer-implemented methods described above are carried out. Alternatively, the program instructions may be loaded on a non-transitory CRM and, when combined with suitable hardware, become part of one or more computer-implemented systems practicing the methods disclosed in this invention.

[0608] Each of the features discussed in this particular implementation section of the method also applies to the CRM and system implementations. As shown above, for the sake of brevity, all method features are not repeated here and should be considered as repeated by reference.

[0609] The foregoing description is presented to enable the manufacture and use of the disclosed technology. Various modifications to the specific embodiments disclosed will be readily apparent, and the general principles defined herein may be applied to other specific embodiments and applications without departing from the spirit and scope of the disclosed technology. Therefore, the disclosed technology is not intended to be limited to the specific embodiments shown, but is to be accorded the broadest scope consistent with the principles and features disclosed herein. The scope of the disclosed technology is defined by the appended claims.

[0610] Computer System

[0611] Figure 24 This is a simplified block diagram of a computer system that can be used to implement the techniques disclosed in this invention. The computer system typically includes at least one processor that communicates with a plurality of peripheral devices via a bus subsystem. These peripheral devices may include a storage subsystem, which includes, for example, memory devices and file storage subsystems, user interface input devices, user interface output devices, and a network interface subsystem. The input and output devices allow users to interact with the computer system. The network interface subsystem provides an interface to external networks, including interfaces to corresponding interface devices in other computer systems.

[0612] In one implementation, the parameter estimator 2401, which estimates three parameters (angle, spacing, and phase offset), can be communicatively linked to the storage subsystem and the user interface input device.

[0613] User interface input devices may include keyboards; pointing devices such as mice, trackballs, touchpads, or graphics tablets; scanners; touchscreens integrated into displays; audio input devices such as speech recognition systems and microphones; and other types of input devices. Generally, the term "input device" is used to encompass all possible types of devices and methods for inputting information into a computer system.

[0614] User interface output devices may include display subsystems, printers, fax machines, or non-visual displays such as audio output devices. Display subsystems may include cathode ray tubes (CRTs), flat panel devices such as liquid crystal displays (LCDs), projection devices, or other mechanisms for producing visible images. Display subsystems may also provide non-visual displays such as audio output devices. Generally, the term "output device" is intended to encompass all possible types of devices and methods for outputting information from a computer system to a user or to another machine or computer system.

[0615] The storage subsystem stores the programming and data construction of some or all of the functions and methods described herein. These software modules are typically executed by a processor alone or in combination with other processors.

[0616] The memory used in the storage subsystem may include multiple memories, including main random access memory (RAM) for storing instructions and data during program execution and read-only memory (ROM) for storing fixed instructions. The file storage subsystem may provide persistent storage for program files and data files and may include hard disk drives, floppy disk drives and associated removable media, CD-ROM drives, optical disk drives, or magnetic tape drives. Modules implementing certain specific functionalities may be stored by the file storage subsystem within the storage subsystem or on other machines accessible to the processor.

[0617] A bus subsystem provides a mechanism for enabling various components and subsystems of a computer system to communicate with each other as intended. Although a bus subsystem is schematically shown as a single bus, alternative implementations of a bus subsystem may use multiple buses.

[0618] Computer systems themselves can be of different types, including personal computers, portable computers, workstations, computer terminals, network computers, televisions, mainframes, server clusters, a loosely networked group of widely distributed computers, or any other data processing system or user equipment. Due to the constantly evolving nature of computers and networks, [the following text appears to be incomplete and requires further context: "for..."] Figure 24 The description of the computer system depicted is intended only as a specific example to illustrate the disclosed techniques. Many other configurations of the computer system are possible, which have more advanced features than those described above. Figure 24The computer system depicted in the text has more or fewer components.

[0619] Deep learning processors can be GPUs or FPGAs and can be hosted by deep learning cloud platforms such as Google Cloud Platform, Xilinx, and Cirrascale. Examples of deep learning processors include Google's Tensor Processing Unit (TPU), rack-mount solutions (such as the GX4 rack-mount series and GX8 rack-mount series), NVIDIA DGX-1, Microsoft's Stratix V FPGA, Graphcore's Intelligent Processing Unit (IPU), Qualcomm's Zeroth Platform with Snapdragon processors, NVIDIA's Volta, NVIDIA's DRIVE PX, NVIDIA's JETSON TX1 / TX2 MODULE, Intel's Nirvana, Movidius VPU, Fujitsu DPI, ARM's DynamicIQ, IBM TrueNorth, and others.

Claims

1. A method for generating sub-block calibration parameters, the method being used with a scanner to produce an enhanced resolution image of the fluorescence of a sample, the sample being positioned closer than the Abbe diffraction limit of optical resolution between at least some adjacent pairs of the sample, the method comprising: The scanner is calibrated using a substantially full field of view projected onto an optical sensor in the scanner by a lens, including: At least one image block is captured under structured illumination with multiple angles and phase shifts of the structured illumination; Calculate the optical distortion on the image block, including The spacing between intensity peaks in at least nine sub-blocks, including a near-center sub-block, of the image block and the angle of the projected illumination pattern indicated by the intensity peaks are measured. The spacing surface and the angle surface are respectively fitted to the measured spacing and angle, wherein both the spacing surface and the angle surface represent corrections for distortion on the sub-blocks of the image block; and The phase displacement of the structured illumination is measured using the multiple angles and the phase displacement within the sub-block, and a lookup table is generated to represent the difference between the phase displacement in the near-center sub-block of the image block and the phase displacement in other sub-blocks; The sub-block calibration parameters, including the lookup table and the fitting results including relating the spacing of the intensity peaks on the at least eight additional sub-blocks and the angle of the projected illumination pattern to the near-center sub-block, are stored for use by the scanner in production.

2. The method of claim 1, wherein the samples comprise millions of samples distributed across a flow cell in images collected over multiple cycles.

3. The method according to any one of claims 1 to 2, wherein each of the sub-blocks comprises at least 512 pixels × 512 pixels of the optical sensor.

4. The method according to any one of claims 1 to 2, wherein the sub-block overlaps with the optical sensor by at least 2 pixels.

5. The method according to any one of claims 1 to 2, the method further comprising saving the fitting result of the spacing surface and the angle surface as coefficients of the quadratic surface.

6. The method according to any one of claims 1 to 2, the method further comprising saving the fitting result of the spacing surface and the angle surface as coefficients of the cubic surface.

7. The method according to any one of claims 1 to 2, the method further comprising storing the fitting result in a lookup table calculated based on a quadratic fitting of the spacing surface and the angle surface.

8. The method according to any one of claims 1 to 2, the method further comprising storing the fitting result in a lookup table calculated based on a cubic fit of the spacing surface and the angle surface.

9. The method according to any one of claims 1 to 2, the method further comprising applying a clipping margin to eliminate the use of pixels around the sensor edge when fitting the spacing surface and the angle surface.

10. The method according to any one of claims 1 to 2, the method further comprising measuring the spacing between intensity peaks on at least 8 × 11 sub-blocks of the image block and the angle of the intensity peaks.

11. A method for generating an enhanced resolution image of a sample using a calibrated scanner, the sample being positioned closer than the Abbe diffraction limit of optical resolution between at least some adjacent pairs of the sample, the method comprising the following actions: For the near-center sub-block of an image block captured by the optical sensor in the scanner, access or generate estimated reconstruction parameters, including Spacing parameters, for the spacing between intensity peaks, and pattern angle parameters, for the angle of the projected illumination pattern indicated by the intensity peaks, the spacing parameters being generated from measurements of imaging a regularly spaced and angled structured illumination pattern projected onto the near-center sub-block. Phase displacement parameters, which are used for the relative phase relationship between the sub-blocks, are generated at a sub-block reference point from further measurements of multiple phase displacements imaged from an angle along the structured illumination pattern projected onto the near-center sub-block. The estimated reconstruction parameters for at least eight additional sub-blocks of the image block and the near-center sub-block are derived by combining the accessed or estimated reconstruction parameters of the near-center sub-block with the following: The saved fitting results correlate the spacing parameters and angle parameters of the at least eight additional sub-blocks with the near-center sub-block. A saved lookup table, the saved lookup table being used for phase displacement measurements relative to the near-center sub-block on the at least eight additional sub-blocks; and The estimated reconstruction parameters are applied to the near-center sub-block and the at least eight additional sub-blocks to generate an enhanced resolution image for the sub-block that is superior to the Abbe diffraction limit.

12. The method of claim 11, further comprising repeatedly applying the actions of the method of claim 11 to generate a sequence of enhanced resolution images for the sub-block, and using the sequence of the enhanced resolution images to sequence samples imaged by the sensor in multiple cycles.

13. The method according to any one of claims 11 to 12, wherein each of the sub-blocks comprises at least 512 pixels × 512 pixels of the optical sensor.

14. The method according to any one of claims 11 to 12, wherein the sub-block overlaps the optical sensor by at least 2 pixels.

15. The method according to any one of claims 11 to 12, the method further comprising aggregating the enhanced resolution images of the sub-blocks into enhanced resolution images of the blocks, and sequencing the samples using the enhanced resolution images of the blocks in a plurality of cycles.

16. The method of claim 12, further comprising, at an intermediate loop between the plurality of loops, remeasuring the spacing between intensity peaks in the sub-blocks and the angle of the intensity peaks, refitting the spacing surface and the angle surface, and saving the result of the refit for processing subsequently captured image blocks.

17. The method of claim 16, wherein the intermediate cycle is the 50th cycle or a later cycle in a set of at least 100 cycles.

18. The method according to any one of claims 11 to 12, further applied to an image block at a location on the flow cell, the image block capturing fluorescence of millions of samples distributed on the flow cell, including generating an enhanced image of the flow cell.

19. The method of claim 18, wherein the sample is distributed in millions of nanopores on the flow cell, and at least some adjacent pairs of nanopores are positioned closer than the Abbe diffraction limit of optical resolution.

20. The method of claim 18, the method further comprising generating a sequence of the enhanced images of the flow cell, and using the sequence of the enhanced images to invoke a sequence of the samples.

21. The method of claim 19, wherein the nanopores are arranged in a repeating rectangular pattern, wherein two diagonals connect opposite angles of the rectangles in the pattern, the method further comprising using two structured illumination angles at which the intensity peaks are substantially orthogonal to the two diagonal orientations.

22. The method of claim 19, wherein the nanopores are arranged in a repeating hexagonal pattern, wherein three diagonals connect opposite angles of the hexagons in the pattern, the method further comprising using three structured illumination angles at which the intensity peaks are substantially orthogonal to the three diagonal orientations.

23. A method for generating an enhanced resolution image from an image of a target captured under structured illumination, the method comprising the following actions: The captured image of the target, captured across the full field of view of the optical sensor, is divided into nine or more sub-blocks, and operations are performed independently on each of the nine or more sub-blocks, including: Transform at least three images of the target into the Fourier domain in the corresponding sub-block to generate at least three frequency domain matrices for the corresponding sub-block, the at least three images being captured by a sensor in the spatial domain while applying at least three phase shifts of the structured illumination along an angle; The estimated reconstruction parameters determined by the sub-block are used to compensate for optical or lens distortion in the sub-block within the full field of view. An inverse mixing matrix is ​​applied to the frequency domain matrix to generate at least three phase-separated matrices in the Fourier domain, the at least three phase-separated matrices being an unoffset matrix and an offset matrix. Performing sub-pixel offset on at least one offset matrix includes transforming the offset matrix from the Fourier domain to the spatial domain, applying a transformation vector to data in the spatial domain, transforming the transformed data from the data in the spatial domain, and generating two or more realigned offset matrices in the Fourier domain. Align and sum the overlap values ​​of the unoffset matrix and the realigned offset matrix to produce an extended frequency coverage matrix; as well as The extended frequency coverage matrix is ​​inversely transformed from the Fourier domain to generate sub-blocks with enhanced resolution in the spatial domain; as well as The enhanced resolution sub-blocks of the nine or more sub-blocks are merged into an enhanced resolution image.

24. The method of claim 23, further comprising repeatedly applying the actions of the method of claim 23 to generate a sequence of images with enhanced resolution, and using the sequence of images with enhanced resolution to sequence samples imaged by the sensor in multiple cycles.

25. The method according to any one of claims 23 to 24, wherein each of the sub-blocks comprises at least 512 pixels × 512 pixels captured by the optical sensor.

26. The method according to any one of claims 23 to 24, wherein the sub-block overlaps with the optical sensor by at least 2 pixels.

27. The method of claim 24, further comprising aggregating the enhanced resolution images of the sub-blocks into an enhanced resolution aggregated image, and sequencing the sample using the enhanced resolution aggregated image in multiple cycles.

28. The method of any one of claims 23 to 24, further applied to an image block at a location on the flow cell, the image block capturing fluorescence of millions of samples distributed on the flow cell, including generating an enhanced image of the flow cell.

29. The method of claim 28, wherein the sample is distributed in millions of nanopores on the flow cell, and at least some adjacent pairs of nanopores are positioned closer than the Abbe diffraction limit of optical resolution.

30. The method of claim 28, the method further comprising generating a sequence of the enhanced images of the flow cell, and using the sequence of the enhanced images to invoke a sequence of the samples.

Images