Method for a high-performance electron microscope

Abstract

A method is provided for correcting one or more image aberrations in an electron microscope image (including a cryo-EM image). The method includes obtaining a plurality of electron microscope (EM) images of an internal reference grid sample having one or more known characteristics, the plurality of electron microscope images being obtained for a plurality of optical conditions and for a plurality of coordinated beam-image shifts. Among other features, the method may in particular include determining an aberration correction function that uses kernel canonical correlation analysis (KCCA) to predict the aberration of each point in the imaged region.

Description

[0001] Recognition of Government Support

[0002] This invention was made with government support under Grant Numbers R01GM117080 and R01GM118619 awarded by the National Institutes of Health. The government has certain rights in this invention.

[0003] Cross - Reference to Related Applications

[0004] This application claims the benefit of priority of U.S. Provisional Application Serial No. 62 / 885,154, filed on August 9, 2019, the entire content of which is incorporated herein by reference for all purposes. Technical Field

[0005] The inventive concept is directed to a method for correcting image aberrations in an electron microscope. Background Art

[0006] An electron microscope is a unique device because it can image biological objects at the nanoscale in detail. After electron microscope imaging, the three - dimensional structure of the object being analyzed can be determined down to the atomic level without the need for highly organized molecular material (e.g., crystals or fibers) required by other structural biology methods that provide high - resolution structures of biological materials. However, the success and quality of three - dimensional structure determination are affected by the imaging characteristics of the microscope, which are technically described by a combination of image distortion, defocus, and image aberrations, including effects deliberately introduced from a phase plate. Although some effects, such as defocus or phase - plate variations, are deliberately introduced, other aberrations are preferentially minimized, although this is not necessary if these aberrations are precisely calibrated and corrected during image analysis and structure determination.

[0007] Minimization of aberrations is achieved by applying a complex process called microscope alignment, which has not been fully automated. Inherent in electron microscope alignment is that the imaging characteristics of minimizing these aberrations and distortions are achieved only in a small area, so observing different parts of a sample requires repeated mechanical translation, which severely hinders efficient data collection in Cryo - EM single - particle reconstruction or analysis (SPA or SPR). Currently, the use of SPA methods is expanding rapidly and the total investment in SPA - related hardware is around $1 billion. Therefore, additional techniques and methods for correcting or minimizing image aberrations in an electron microscope are desired. Summary of the Invention

[0008] The following brief description is provided to indicate the nature of the subject matter disclosed herein. While certain aspects of the inventive concept are described below, the overview is not intended to limit the scope of the inventive concept.

[0009] The presently disclosed methods, systems, and apparatuses provide complex distortion and aberration patterns to be characterized, calibrated, and inverted during computational data analysis after data acquisition in electron microscopy (EM), including cryo-EM. In particular, the presently disclosed methods provide (1) more detailed (e.g., higher resolution) results; (2) more efficient data acquisition such that an instrument can achieve multiple-fold improvements in data acquisition throughput; (3) elimination of frustration for users inexperienced in microscope alignment or elimination of the need for high-level expertise; and (4) enablement of more informative data collection modes involving phase plates and beam tilts.

[0010] According to one aspect of the present disclosure, methods, systems, and apparatuses are provided for correcting EM image aberrations, particularly non-axial aberrations, by: experimentally obtaining calibration (mapping) of local changes in axial aberrations and performing kernel-normalized analysis of these local changes to recover a global description (predictor) of non-axial aberrations. The global predictor can then be applied to obtain the pattern of aberrations at any position away from the optical axis and off-axis height to navigate the image space at distances away from the optical axis without the aberrations affecting data quality. While it is often desirable to calibrate the available beam-image shifts over the entire xy range, it is not necessary to calibrate the entire available range of z translations.

[0011] According to at least one aspect of the present disclosure, a method for correcting one or more image aberrations in an electron microscope image is provided. The method can include: (a) obtaining a plurality of electron microscope (EM) images of an internal reference grid sample having one or more known characteristics, the plurality of electron microscope images being obtained for a plurality of optical conditions and for a plurality of coordinated beam-image shifts, wherein the plurality of optical conditions are selected from a plurality of defocuses, a plurality of z heights, a plurality of beam tilts, and any combination thereof; (b) correcting sample drift of the plurality of EM images by aligning and motion correcting the plurality of EM images to produce an EM micrograph; (c) calculating a Fourier transform (FT) of the micrograph to produce an FT image; (d) deconvolving the FT image using a predetermined deconvolution coefficient selected from a series of deconvolution coefficient values to produce a deconvolved FT image; (e) applying a high-pass filter to the deconvolved FT image to remove all frequencies below to produce a filtered deconvolved FT image;

[0012] (f) calculating an inverse FT of the filtered deconvolved FT image to produce an aberration-corrected EM micrograph; (g) determining an intensity distribution for the aberration-corrected EM micrograph; (h) calculating moments of the intensity distribution; and (i) repeating (c)-(h) using a predetermined deconvolution coefficient selected from a series of deconvolution coefficient values different from the previous iteration until an optimal deconvolution coefficient is determined based on maximization of the moments in (i).

[0013] The method may further include (j) using the optimal deconvolution coefficients obtained in (i) and kernel canonical correlation analysis (KCCA) associated with the multiple optical conditions and multiple coordinated beam-image shifts used in (a) to determine an aberration correction function for predicting axial aberration for each point in the imaging region. The method may further include: (k) obtaining one or more EM images of a calibration check grid sample having one or more known characteristics different from at least one of the one or more known characteristics of an internal reference grid sample; (l) applying the aberration correction function to the one or more EM images obtained in (k) to produce aberration-corrected EM images; and (m) determining the applicability of the aberration correction function based on a comparison of one or more features in the aberration-corrected EM images corresponding to the one or more known characteristics of the calibration check grid sample.

[0014] According to another aspect of the present disclosure, a method for correcting geometric distortion in an electron microscope image is provided. The method may include: (a) obtaining a plurality of electron microscope (EM) images of an internal reference grid sample having one or more known characteristics, the plurality of electron microscope images being obtained for a plurality of optical conditions and for a plurality of coordinated beam-image shifts, wherein the plurality of optical conditions are selected from a plurality of defocus, a plurality of z-heights, a plurality of beam tilts, and any combination thereof; wherein the internal reference grid sample includes an amorphous material distributed on a crystalline support having known unit cell dimensions, the amorphous material having a thickness of five or fewer atomic layers; (b) correcting sample drift of the plurality of EM images by alignment and motion correction to produce an EM micrograph; (c) calculating a Fourier transform (FT) of the micrograph to produce an FT image; (d) identifying diffraction peaks of a crystalline lattice corresponding to the internal reference grid sample on the FT image; (e) performing double spatial filtering on the FT image by masking the identified diffraction peaks, thereby retaining only the diffraction peaks and their corresponding intensities while discarding all information between the diffraction peaks to produce a filtered FT image; (f) calculating an inverse FT of the filtered FT image to produce a filtered EM micrograph; (g) selecting a portion of the filtered EM micrograph, calculating an FT image corresponding to the portion to produce an FT sub-image, identifying a subgroup of diffraction peaks on the FT sub-image, indexing the diffraction maxima of the subgroup of diffraction peaks, and determining unit cell parameters of the subgroup of diffraction peaks; (h) using a distortion matrix to determine whether the one or more unit cell parameters determined in (g) are consistent with the known unit cell dimensions of the crystalline support of the internal reference grid sample; and (i) calculating a metric tensor based on the distortion matrix.

[0015] The method may further include (j) using kernel canonical correlation analysis (KCCA) of the metric tensor obtained in (i) and the plurality of optical conditions used in (a) to determine an aberration correction function that predicts geometric distortion for each point in the imaging region. The method may further include (k) obtaining one or more EM images of a calibration check grid sample having one or more known characteristics that are different from at least one of the one or more known characteristics of the internal reference grid sample; (l) applying the aberration correction function to the one or more EM images obtained in (i) to produce aberration-corrected EM images; and (m) determining the applicability of the aberration correction function based on a comparison of one or more features in the aberration-corrected EM images corresponding to the one or more known characteristics of the calibration check grid sample. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present description will be more fully understood with reference to the following drawings and data graphs, which are presented as various embodiments of the inventive concept and should not be construed as a complete recitation of the scope of the inventive concept, wherein:

[0017] Figure 1 Depicts protein data bank (PDB) cumulative statistics according to an embodiment of the inventive concept, showing the growth of X-rays (hatched) and cryo-EM depositions (double-hatched);

[0018] Figure 2 Depicts data showing the accuracy of beam tilt refinement of 257 micrographs according to an embodiment of the inventive concept;

[0019] Figure 3 Depicts according to an embodiment of the inventive concept Region of resolution reconstruction, where the uncorrected coma aberration caused by beam tilt is severe enough to limit the resolution to

[0020] Figure 4 Depicts multi-step robust navigation between a preview image and a higher magnification for data collection according to an embodiment of the inventive concept;

[0021] Figure 5 Depicts reanalysis of electron microscopy public image archive (EMPIAR) depositions 10185 and 10186 with coma, trefoil, and anisotropic magnification correction according to an embodiment of the inventive concept;

[0022] Figure 6is a schematic diagram showing elements of kernel canonical correlation analysis (KCCA) between independent parameters (i.e., observation conditions and their non-linear functions) according to an embodiment of the inventive concept and observed phase shifts and geometric distortion aberrations; and

[0023] Figure 7 is a flowchart depicting a method for correcting one or more image aberrations in an electron microscope image according to an embodiment of the inventive concept. DETAILED DESCRIPTION

[0024] The inventive concept can be understood by reference to the following detailed description in conjunction with the drawings described herein. Note that, for clarity of illustration, some elements in the respective drawings may not be drawn to scale.

[0025] The present disclosure provides a method for correcting one or more image aberrations in an electron microscope image. The method can include obtaining a plurality of electron microscope (EM) images of an internal reference grid sample having one or more known characteristics under a plurality of optical conditions and at a plurality of coordinated beam image shifts. The plurality of optical conditions can include, for example, defocus, a plurality of z-heights, a plurality of beam tilts, and any combination thereof. One or more known characteristics of the internal reference grid sample can be, for example, the atomicity of the layer and / or the unit cell dimension of the grid support. As used herein, the term "atomicity" refers to a material in which the molecules are individual and separated from other molecules, e.g., they are not deposited in clusters or aggregates.

[0026] The internal reference grid sample can include a solid support and an amorphous material distributed on the solid support. In at least some cases, the solid support can be a crystalline support having a known unit cell dimension. In such cases, the crystalline support can be made of, for example, graphene, graphene oxide, silicon, or silicon nitride. However, in cases where a crystalline support is used, any solid support that generates a diffraction pattern with defined unit cell parameters can be used, especially those materials that are capable of forming a thin support structure operable to receive an amorphous coating having characteristics suitable for use as an internal reference grid sample according to the presently disclosed method.

[0027] The amorphous material distributed on the solid support can have a thickness of five or fewer atomic layers. Preferably, the thickness of the amorphous material is one atomic layer, or one to three atomic layers. In at least some cases, the amorphous material includes a metal. In such cases, the metal can be any metal that is heavy enough to be sputtered onto the surface of the solid support, and the layer thickness can be controlled by current and vacuum. In at least some cases, the amorphous material can be, for example, gold, platinum, iridium, palladium, and any combination thereof. These metals are suitable due to their weight and relative inertness (e.g., do not oxidize).

[0028] Multiple coordinated beam-image shifts can be collected using a beam-image shift method. The beam-image shift method involves keeping the position of the sample unchanged and only shifting the positions of the beam and the image in order to illuminate different parts of the sample (e.g., a grid) (beam shift) and image them (image shift). The shifted positions of the beam and the image introduce different orders of aberration. Two sets of lenses are used in an EM instrument to perform the beam-image shift method. The condenser lens moves the beam, while the objective lens directs the image onto the detector. Both sets of lenses are electronically controlled. In at least one embodiment, the beam-image shifts are coordinated or synchronized, where a coil is used to shift the beam and the shift of the beam is accompanied by a coordinated shift of the image using a coil. In other cases, a desynchronized beam-image shift can be used, where the beam or the image can be moved without moving its "partner". The desynchronized beam-image shift is performed to determine the degree of beam parallelism, as the degree of beam parallelism affects aberration. In at least some cases, an electron microscope (EM) image is an exposure to an electron beam that takes several to dozens of seconds and is acquired without changing the image position, but contains many shorter exposures (frames). In at least some cases, multiple EM images can be obtained at multiple magnifications. In some cases, multiple EM images can be obtained without changing the position of the grid sample. In at least some cases, a fractional factorial design can be used to select the points for data collection and the parameters for changing the microscope optical settings.

[0029] The presently disclosed methods for correcting one or more image aberrations in an electron microscope image can also include correcting sample drift of multiple EM images by alignment and motion correction to produce an EM micrograph. As used herein, the term "micrograph" can refer to the entire image or a portion of the image, such as a sub-image of the image. The Fourier transform (FT) of the micrograph can then be calculated to produce an FT image. The FT image contains the information of the micrograph, but in reciprocal space. The FT image is affected by the same degree of unknown type and unknown magnitude of aberration as the real image, because the FT does not change the information content. However, operating in Fourier space (reciprocal space) has mathematical advantages, because the convolution theorem states that convolution in real space becomes multiplication in reciprocal (Fourier) space.

[0030] Then, the FT image is deconvolved using a predetermined deconvolution coefficient selected from a series of deconvolution coefficient values to produce a deconvolved FT image. The Fourier representation of a micrograph involves complex numbers, which have an amplitude and a phase. Thus, for a micrograph affected by aberrations, the amplitude and phase will be the product of the FT of the aberration and the FT of the image. The deconvolution of these phase and amplitude products involves phase factor multiplication (phase addition) and amplitude multiplication. The phase factor multiplication component is the complex conjugate, derived from the phase-transformed representation of the aberration. The amplitude variation component is derived from two considerations: (1) the inverse of the amplitude change optimized for the resultant aberration and noise, which gives the Wiener filter contribution. The predetermined deconvolution coefficient is a "guess" regarding the aberration value selected from a series of possible deconvolution coefficient values representing the possible range of aberrations. Thus, the predetermined deconvolution coefficient is an initial deconvolution coefficient value that describes the magnitude of the aberration as an arbitrary phase and amplitude product during the initialization process. As described below, the deconvolution coefficient will be optimized in subsequent calculation cycles.

[0031] Then, a high-pass filter is applied to the deconvolved FT image to remove all frequencies below a certain value, so as to produce a filtered deconvolved FT image. For example, the high-pass filter can remove all frequencies below or below . The high-pass filter removes image information with very low resolution, because this information does not represent the atomic information used in the optimization process.

[0032] The method may also include calculating the inverse FT of the filtered deconvolved FT image to produce an aberration-corrected EM micrograph. Thus, the aberration-corrected EM micrograph is an image corresponding to the real-space image of the starting micrograph but with aberration correction applied. Next, the intensity distribution is determined for the aberration-corrected EM micrograph. For example, a histogram of all intensity values in the aberration-corrected image (millions of pixels) can be generated. The histogram will have a non-Gaussian shape, with its tail towards more positive values. The non-Gaussian distribution is caused by the grid characteristics (e.g., thin layers of metal atoms). If the image is fully corrected, then these atoms should be "brighter", and there may also be empty spaces between them. These two features both make the intensity histogram non-Gaussian. The non-Gaussianity is due to the "non-uniformity" of certain features of the distribution, e.g., the intensity signal features caused by sparse sputtering of metal on the grid support. Aberrations distort that signal, making the image of the grid sample more "uniform" in terms of intensity distribution.

[0033] The method also includes calculating moments of the intensity distribution. The moments quantitatively describe the shape of the distribution. A specific moment is selected from a series of options, such as skewness, kurtosis, negative entropy, or any other function applicable to independent component analysis (ICA). Any function that can be used by ICA can be suitable for the currently disclosed techniques and methods. In at least some cases, calculating the moments includes quantifying the shape of the intensity distribution based on optimizing negative entropy.

[0034] The method also includes repeating the FT calculation, deconvolving the FT image, applying a high-pass filter, calculating the inverse FT, determining the intensity distribution, and calculating the moments using a predetermined deconvolution coefficient different from the previous iteration until an optimal deconvolution coefficient is determined based on moment analysis. A numerical optimization method of changing the value of the deconvolution coefficient is employed until the deconvolution coefficient is optimized or maximized.

[0035] The method may also include using the optimal deconvolution coefficient and kernel canonical correlation analysis (KCCA) related to multiple optical conditions and multiple coordinated beam-image shifts to determine an aberration correction function that predicts aberrations for each point in the imaging region, as Figure 6 shown. Figure 6 is a schematic diagram of the KCCA process. In particular, Figure 6 depicts the KCCA elements between independent parameters (i.e., observation conditions and their non-linear functions) and the observed phase shift and geometric distortion aberrations. The result of the analysis is a function that predicts aberrations for any observation condition. Additionally, it can be defined whether the conditions for a specific observation are obtained by interpolation or extrapolation of the input data.

[0036] KCCA extends and refines the aberration description such that a complete map of the imaging region is obtained. The parameters of each micrograph (e.g., multiple optical conditions and coordinated beam-image shifts) are independent variables and they can be used to create one of the matrices used in canonical correlation, as Figure 6 shown. Various non-linear combinations of these parameters (e.g., x 2 , y 2 , xy) can also be used in the matrix. In other words, the characteristics of KCCA can include not only independent parameter values but also their non-linear functions. As Figure 6 shown, a second matrix is formed as a result of the deconvolution coefficient (aberration coefficient) optimization process. In at least some cases, KCCA can be run hierarchically to correct time-varying components of one or more aberrations.

[0037] Data can be collected using a beam-image shift method for a specific region to evaluate a current calibration determined by the method, where the specific region uses a grid different from the grid used for calibration. Thus, the grids can have significantly different characteristics (such as z-height). Accordingly, the method can also include obtaining one or more EM images of a calibration check grid sample having one or more known characteristics different from at least one of the one or more known characteristics of an internal reference grid sample. An aberration correction function can then be applied to the obtained one or more EM images to produce aberration-corrected EM images. The method can include determining the applicability of the aberration correction function based on a comparison of one or more features in the aberration-corrected EM images corresponding to one or more known characteristics of the calibration check grid sample. If the aberration correction function is within the range that only requires interpolation, e.g., within the calibration map generated by the method up to this point, then no further calibration is required. However, if the aberration correction function is outside the range that only requires interpolation and extrapolation is required, then the above method steps may need to be repeated because extrapolation has a greater error than interpolation.

[0038] In at least some cases, the method can include further iterative refinement of the aberration. For example, an initial calibration will determine the time-invariant component of the calibration, but the instrument hardware changes (drifts). In such cases, the initial calibration map can be used, and multiple EM images can be obtained for a smaller image region by using a denser network of measurement points or different combinations of data collection parameters, and then the method can be repeated with these data points, but starting from the time-invariant aberration values. In other cases, the input from the initial calibration can be used to obtain experiment-specific aberration correction, and then these results can be used for KCCA.

[0039] Image aberrations that can be corrected using the currently disclosed method can be any monochromatic aberration that affects the phase and amplitude of the Fourier transform of the image. For example, image aberrations that can be corrected using the method can include a constant phase offset, image displacement, defocus, second-order astigmatism, axial coma, third-order astigmatism (trefoil), spherical aberration, star aberration, fourth-order astigmatism, fifth-order axial coma, trefoil aberration, fifth-order astigmatism, sixth-order spherical aberration, sixth-order star aberration, flowerlet aberration, and sixth-order astigmatism. Image aberrations can also be geometric distortion and curvature of the image field.

[0040] According to another aspect of the present disclosure, a method for correcting geometric distortion in an electron microscope (EM) image is provided. Geometric distortion is also referred to as angular distortion and elliptical distortion. The method involves obtaining calibration of geometric distortion in the image based on an atomic reference, such as a graphene lattice. The method may include obtaining a plurality of EM images of an internal reference grid sample having one or more known characteristics, wherein a plurality of electron microscope images are obtained for a plurality of optical conditions and for a plurality of coordinated beam-image shifts. The plurality of optical conditions may be, for example, a plurality of defocuses, a plurality of z-heights, a plurality of beam tilts, and any combination thereof. The internal reference grid sample may include a crystalline solid support having known unit cell dimensions, wherein an amorphous material is distributed on the crystalline support. The amorphous material may have a thickness of five or fewer atomic layers. Preferably, the thickness of the amorphous material is one atomic layer, or one to three atomic layers.

[0041] In at least some cases, the amorphous material may be a metal. In such cases, the metal may be any metal heavy enough to be sputtered onto the surface of the solid support, and the layer thickness may be controlled by current and vacuum. In at least some cases, the amorphous material may be, for example, gold, platinum, iridium, palladium, and any combination thereof. These metals are suitable due to their weight and relative inertness (e.g., do not oxidize). The crystalline support may be made of, for example, graphene, graphene oxide, silicon, or silicon nitride. However, any solid support that generates a diffraction pattern with defined unit cell parameters may be used, especially those materials that can form a thin support structure operable to receive an amorphous coating having characteristics suitable for use as an internal reference grid sample according to the presently disclosed method.

[0042] The method may further include correcting sample drift of the plurality of EM images by aligning and motion-correcting the plurality of EM images to produce an EM micrograph. Then, a Fourier transform (FT) of the micrograph is calculated to generate an FT image. Diffraction peaks on the FT image corresponding to the crystalline lattice of the internal reference grid sample are identified, and double spatial filtering is performed on the FT image by masking the identified diffraction peaks, thereby retaining only the diffraction peaks and their corresponding intensities while discarding all information between the diffraction peaks, thereby producing a filtered FT image. Then, an inverse FT of the filtered FT image may be calculated to produce a filtered EM micrograph.

[0043] Perform the above steps because in real space, crystalline materials such as graphene can consist of overlapping crystal lattices (more than one crystal), or the crystal can be smaller than the imaging area. Therefore, after FT, all non-crystalline materials will blur the data analysis. Thus, the method finds the diffraction peaks in reciprocal space (Fourier space) and masks them, i.e., only the peaks and their intensities are retained while all information between the peaks is discarded. Then the inverse (reverse) FT is calculated to transform the filtered information from Fourier space to real space. This operation often allows for better segmentation of the image in real space. After better segmentation, the FT of each segment is calculated again, i.e., moved back to Fourier space, in order to obtain a "clearer" diffraction pattern.

[0044] Next, the unit cell parameters of the "clearer" diffraction pattern are determined using standard indexing procedures. Specifically, the method includes selecting a portion of the filtered EM micrograph and calculating the FT image corresponding to that portion to produce an FT sub-image. Then, a subgroup of the diffraction peaks on the FT sub-image is identified, and the diffraction maxima of the subgroup of diffraction peaks are indexed so that the unit cell parameters of the subgroup of diffraction peaks can be determined. Then the distortion matrix (M) is determined and used to determine whether one or more unit cell parameters are consistent with the known unit cell dimensions of the crystalline support of the internal reference grid sample. The distortion (M) matrix contains a description of the rotation between the reference lattice and the observed lattice, so it is not guaranteed that the rotations between the images are the same. Then by taking the distortion matrix M and multiplying it by its conjugate matrix M * , the metric tensor (MM * ) is calculated based on the distortion matrix in order to eliminate the rotation that is part of the distortion matrix M. The metric tensor (MM * ) no longer contains rotation and can be used to perform KCCA.

[0045] The method can also include using the metric tensor along with kernel canonical correlation analysis (KCCA) of multiple optical conditions and multiple coordinated beam-image shifts to determine an aberration correction function that predicts geometric distortion for each point in the imaging area, as Figure 6 shown. The method can also include obtaining one or more EM images of a calibration check grid sample that has one or more known characteristics different from at least one of the one or more known characteristics of the internal reference grid sample, and applying the aberration correction function to the one or more EM images to produce aberration-corrected EM images. Finally, the method can include determining the applicability of the aberration correction function based on a comparison of one or more features in the aberration-corrected EM image corresponding to one or more known characteristics of the calibration check grid sample.

[0046] Figure 7Depicts an exemplary method 700 for correcting one or more image aberrations in an electron microscope image according to an embodiment of the inventive concept. At 705, an internal reference grid sample having known atomicity and / or unit cell parameters and a high-contrast atomic pattern is obtained. At 710, a plurality of electron microscope (EM) images of the internal reference grid sample having one or more known characteristics are obtained for a plurality of optical conditions and a plurality of coordinated beam-image shifts. At 715, distortion calibration is determined for each image or sub-image according to the method described herein. At 745, aberration calibration is determined for each image or sub-image according to the method described herein. Kernel Canonical Correlation Analysis (KCCA) is performed at 720 using the distortion calibration and at 750 using the aberration calibration. The results of the KCCA analysis are interpreted at 725, and if there is no convergence at 730, the process is repeated starting from step 710. If there is convergence, the process can be repeated at a lower resolution 735 starting from step 710. Alternatively, once convergence is achieved at 730, the results of the KCCA analysis can be used to correct one or more data images and / or to align the objective aperture, energy filter, and beam-image stage. The KCCA at 720 and 750 can be performed together or separately, i.e., KCCA can be performed on an input that contains information from both paths.

[0047] The currently disclosed method has advantages over existing power-spectrum-based methods for correcting aberrations. In particular, the present disclosure provides a novel aberration correction that enables long-range (tens of microns) electron repositioning to a point for data acquisition (i.e., the beam-image shift method), including the following innovations: (1) using the beam-image shift method at an unprecedented spatial scale, made possible by computationally inverting the complex image distortion associated with such a shift; and (2) using Kernel Canonical Correlation Analysis (KCCA) to identify the parameters required to invert the image distortion, measuring the distortion at multiple locations on the grid, and then performing a global fit with KCCA to generate data for the correction of image distortion and facilitate the complete alignment of the microscope, thereby eliminating cumbersome alignment procedures.

[0048] The currently disclosed method provides an exact inversion of the complex image amplitude, phase, and geometric distortion resulting from ultrafast data collection methods that rely on imaging samples far from the optical axis of the microscope. Multiple studies have been performed to confirm that such exact distortion maps can be generated and validated for a range of expected experimental conditions. We reanalyzed data previously collected by the beam-image shift method and stored in the EMPIAR database by calibrating the image and phase distortion and then correcting it. As a result, we improved the resolution of proteasome reconstruction from to For data collected on a 200 kV Talos Arctica, this change represents an improvement from mediocre (by today's standards) to relatively impressive results. For in-house data collected by the team, high-resolution reconstructions were performed on samples with high (~20 μm) optical axis displacements and thus very high coma distortion. Results from instruments of the same type also have resolution. Without this computational correction, the resolution of the results would be close to

[0049] In one aspect, the presently disclosed method provides precise and automatic calibration of EM optics required for high-throughput, beam-image shift methods. In particular, the presently disclosed method can be used in cryogenic electron microscopy (cryo-EM), which is increasingly used for high-resolution biomolecular structure determination, as Figure 1 shown in Figure 1 depicted in

[0050] Under bright-field (phase contrast) imaging conditions used in cryo-EM single-particle reconstruction, image distortion does not result in loss of information but rather in its rearrangement, meaning that the effects of these distortions can be computationally reversed with an appropriate protocol. Reversal of some distortions is already an inherent feature of standard cryo-EM software. For example, cryo-EM data are typically acquired at a large amount of defocus and are affected by spherical aberration and also often to some extent by astigmatism. These distortions, described by the contrast transfer function (CTF), are reversed during structure reconstruction. We extended this method to higher-order aberration correction functions for reference-based refinement during particle reconstruction and independent refinement for each micrograph. We found that this aberration refinement for phase shift, defocus, astigmatism, coma, trefoil, and anisotropic magnification is very accurate, as Figure 2 shown in Figure 2 because the results are very reproducible between individual movies when such reproducibility is expected.

[0051] The aberration refinement is also very precise. For data with a large coma caused by a 20-μm shift from the optical axis, we achieved a very high resolution under conditions that are considered quite challenging for high-resolution work. As Figure 3 shown, i.e., using Talos Arctica, no phase plate, no energy filter, 200 kV, and a particle size of 144 kDa. We confirmed that this protocol is also applicable to the data deposited in EMPIAR 10186(7), for proteasome data collected using the beam-image shift method within a smaller shift range using the same type of instrument. This deposit reported a resolution, while our phase correction protocol yielded a resolution.

[0052] Calibration (global parameterization) maps of the distortions present under all experimental conditions encountered can be generated, including initial sample alignment and preview as well as normal high-resolution data collection. The distortion maps fully cover the spatial and resolution ranges encountered in these steps, including the off-center height distances necessary for collecting data with a tilted sample. These calibrations also automatically provide the exact position and orientation of the optical axis, beam tilt, and focal plane. In an embodiment, we use apoferritin as our high-quality reference molecule, but other rigid and stable macromolecules can be substituted. We create a unified distortion map by fitting a single global function, where the optical terms are a description of the off-axis aberrations. However, for each small imaging region, these off-axis aberrations can be converted into a local axial aberration description, making it easy to interface this global description with existing programs. KCCA is a very suitable method because it can robustly refine the linear and non-linear relationships between two sets of parameters, one set describing the explanatory variables, in this case the positions in real space and angle (reciprocal) space, and the other set describing the data characteristics, in this case the magnification and phase distortion, parameterized as the product of real space and angle space Zernike polynomials. After we create this distortion calibration, for actual data collection, we identify the factors that can drift during data collection or between different data collection runs; this drift mainly affects translation (x, y, and focus positions) and astigmatism. In the currently collected data, these data are often adjusted, even for each micrograph, although astigmatism drift tends to be slower.

[0053] There are different ways to perform correction of image distortion. During reconstruction, all corrections can be incorporated into one operation, as in our own 3D reconstruction calculations, but they can also be split into an image manipulation part and the rest as reconstruction steps. Since such splitting is the norm when using other programs, we adopt calculations that correct distortion at the image level so that the results are compatible with existing software. Convenient features of coma and trefoil distortion are that they can be corrected at the image level without any loss of accuracy, and this extends to performing them consecutively multiple times, such as an initial correction followed by additional corrections during reference-based refinement. For magnification distortion, such a rule does not hold, although the procedure is typically done at the image level because if the data collected is oversampled, then the information loss is low. Even when applied to large spatial scales and a large number (up to hundreds) of holes, the beam-image shift data collection procedure produces accurate results and can perform robust automatic calibration of electron optical parameters that can vary between experiments in the same instrument, so it is typically performed for each individual experiment.

[0054] According to another aspect of the present disclosure, there is provided a method for previewing and characterizing the quality pattern in a sample, having an exact mapping between magnifications, including the magnifications for data collection. The method innovatively includes splitting the positioning uncertainty into two parts during the preview process: (1) indexing the holes and (2) precise detection of the hole edges. The splitting of uncertainty when using the beam-image shift method in combination at the data collection magnification enables us to precisely correlate the positioning at the data collection magnification with the low magnification preview image and eliminates the need to use an intermediate magnification in the current procedure. Since the hysteresis and drift associated with the objective lens are eliminated, this has a surprisingly large impact on the data quality, generated in each hole pair step using the current protocol that uses an intermediate magnification for this purpose.

[0055] The method includes optical distortion mapping and a client-server architecture for experimental control optimized for structural biology experimenters. In terms of cryo-EM, we have collected dozens of datasets and reconstructed seven different structures, four of which have a resolution better than Using a cryo-EM instrument that requires full electron beam calibration before data collection. To map aberrations and distortions over a large area, we perform KCCA to fit the contrast transfer function (CTF) to the geometrically distorted images on the grid collected by the beam-image shift method. The quality of such a spatial map is defined by the degree to which the distortion changes from the position prediction. In our earlier results, we found that the main canonical correlation coefficient for such relationships was 0.994.

[0056] Throughput inefficiencies in cryo-EM also stem from the long lag times between collecting micrographs and from the time spent aligning microscopes, assessing samples, and setting up data collection. The result of this approach is a reduction in the lag time between images collected in a beam-image shift group and an acceleration of instrument alignment. In this approach, we address other bottlenecks caused by creating preview images, assessing them, and setting up data collection. Current protocols cannot scale to collecting ten thousand or more micrographs per day, and we have taken a number of steps to address this issue.

[0057] Many inefficiencies in cryo-EM data acquisition are due to poor alignment of preview images in their montage. These inefficiencies are caused by the requirement for excessive overlap and by image distortion that inaccurately points to positions at different magnifications, especially when it comes to the magnification desired for data collection. Electron optics have an inherent pincushion distortion, which can be parameterized by non-linear radial displacement relative to the optical axis and which is established in our protocol by collecting the same image at different stage positions. The refinement protocol adjusts the distortion to optimize the correlation between images with increasingly large shifts. Correcting pincushion and related higher-order distortions makes the images in the montage match precisely, thus reducing the requirement for overlap and therefore cutting the time required to collect them by a factor of two. These pincushion distortions are stable and should therefore be part of the instrument calibration process rather than the experimental calibration process.

[0058] Additional gains are obtained from the reduction of positional errors in the preview montage. Currently, this error results in poor prediction of data collection positions and creates the requirement for additional hole coverage images at intermediate magnifications to precisely locate holes. We replace it with a more robust and automated protocol that correlates square positions at different resolutions. To precisely align the grid within the square, we use the beam-image shift group so that we can obtain precise positioning information without applying intermediate magnifications. The penalty for the current protocol of acquiring intermediate-resolution images between data collection micrographs is very severe and underestimated. It involves large changes in the current in the objective lens, which creates a change in magnetic force, resulting in an inelastic drift response within the lens, thus disturbing the position of the image. This inelastic response takes a long time to stabilize and also produces hysteresis when repeated in cycles, so that the imaging position and image distortion change heterogeneously with such cycles. Therefore, eliminating such cycles is very beneficial for the stability of the instrument and its calibration, reducing image drift and recalibration frequency, and lower drift improves the quality of dose-slicing results.

[0059] As Figure 4As shown, we eliminated the need for an intermediate magnification by a multi-step, rapid alignment data collection strategy at higher magnifications. First, due to the limited field of view, we created very rapid clips to find holes and implemented a rapid grid search to robustly index them. The novel aspect of our navigation is splitting the positional uncertainty into (1) precisely locating the hole and (2) defining which hole it is (represented by the index in our protocol). We collect two separate sets of images to address these problems separately. This addresses the problem that current methods often produce positional errors by aligning mis-indexed holes. Recovering from such frequent errors requires additional steps that are cumbersome and even more time-consuming. Our protocol compares the ice thickness in two rows of vertical holes at different magnifications, thus robustly resolving the indexing ambiguity problem and requiring minimal time.

[0060] Data collection is performed in sessions, where each session represents data acquisition on one or more grid squares. Our protocol uses a constant magnification throughout the session, so we eliminate the contribution of objective lag to drift. In our earlier studies, we found that drift was smaller in the beam-image shift method. Our analysis of the proteasome data deposited at EMPIAR, which was generated from a single sample but collected with two different protocols, confirmed this.

[0061] Figure 4 Depicts multi-step robust navigation between preview images and higher magnifications for data collection according to an embodiment of the inventive concept. The steps of establishing precise geometric relationships between very different resolutions involve first finding holes in a regular grid (a), then defining defocus and drift by collecting exposures (blue rectangles in b) from the carbon portion (b) or the edges of the holes for a gold-on-gold grid. Based on (a) and (b), a smaller sub-grid is placed above the hole to precisely define its boundaries (c), and then the indexing ambiguity problem is resolved by collecting very short exposures (green rectangles in d) with a brief stabilization time to establish the pattern of ice thickness on the grid (d) of nearby holes so that the edges of the visible rows and columns of holes define a clear index relative to the preview image. The complete scheme includes obvious remedies such as starting the initial search at different points and considering failure modes at different levels of the instrument hierarchy, such as the beam not reaching the detector. Note that the sub-panels have different scales, but the rectangles defining the exposures and the holes have the same dimensions.

[0062] We analyzed a conventionally collected dataset and a dataset collected with a limited range, beam-image shift involving two-by-two hole squares. Without distortion correction, the conventionally collected dataset has a higher resolution relative to the dataset obtained with the beam-image shift method, asFigure 5 as shown. After correcting for distortion, this situation is reversed, which results in a resolution of the conventionally acquired dataset of while the resolution of the dataset collected by applying the beam-image shift method at a finite distance is We also implemented a novel protocol for monitoring long-term sample drift throughout the session. In a specific beam-image shift frame group, we estimate the drift by recollecting a short exposure at the first position. We then integrate the drift from multiple groups of the same data collection session. We found that a data acquisition session using only one magnification has relatively high stability, but our protocol also checks for excessive accumulation of drift, which triggers additional recalibration.

[0063] Our goal is to make position navigation across different magnifications precise enough that neither manual intervention nor associated time and expertise are required. For this reason, we identified some steps that seem complex but do not require a large amount of time to execute. Additionally, we have built-in inspection and calibration protocols to robustly maintain position relationships over time.

[0064] CTF mapping by global distortion characterization and automatic EM calibration. There are two methods for establishing the defocus or height (z) of a sample point: determining the contrast transfer function (CTF) of defocus, and the beam tilt method as a faster method for defining z. CTF requires high-contrast materials, more exposure, and more complex analysis; however, it is more informative because it can be used to determine multiple phase distortions. The CTF method can also be combined with beam tilt for additional verification of results. We developed a simple but fast and robust (with respect to large astigmatism) CTF estimator for integration with the rest of the software.

[0065] Calibration begins by verifying that sample drift is stable so that it can be ignored or corrected. Then, we apply our CTF estimator to data collected in regions sparsely distributed (~10 to 20) on carbon and for significantly different defocus values (electron-accessible simulations varying the z coordinate). The results from the CTF and from post-defocus translational shifts are the input to KCCA, which will provide a distortion predictor for any x, y, z triple in the image. This predictor first identifies positions on the image that are invariant with respect to defocus; this is the position of the optical axis. The main factor causing CTF variation is the interaction between beam tilt and spherical aberration. We parameterize the beam tilt to have a helical distortion dependence on the position relative to the optical axis and the beam direction. The fitted value of the beam tilt at the optical axis position defines the direction of the optical axis. This provides the main component of our automated calibration procedure and defines the coordinate system with respect to the optical axis. This procedure can determine the aberrations affecting the CTF and, through interaction with spherical aberration, can also determine the beam tilt and thus coma. However, it cannot determine aberrations such as trefoil and other aberrations with higher-order terms. Outside the optical axis, we have found significant trefoil in one of our data sets and in the proteasome EMPIAR depositions 10185 and 10186. We determine the dependence of this distortion on the distance from the optical axis by refinement based on a reference high-quality sample such as ferritin. Since they originate from the shape effects of the objective lens, they are quite stable even over long periods of time. Thus, the patterns of these aberrations should be considered part of the instrument calibration rather than specific to an experiment. While imaging carbon provides the fastest means of establishing a map of the distortion, analysis of the CTF determined from data collection can also be used to find the global distortion map and its potential drift. Additional control can be obtained by collecting more data with beam tilt to distinguish defocus changes due to non-verticality and non-planarity of the sample from imaging lens aberrations.

[0066] The results of this method include a time-efficient procedure for collecting preview images and precise navigation between magnifications. This allows for rapid and exact collection at the desired position and defocus. The procedure allows a data collection session to completely avoid the destabilizing effects of changing microscope magnification, thus greatly reducing the time required for image stabilization. The precision of data collection benefits the faster convergence of downstream procedures and leads to more efficient feedback calculations for monitoring data collection and readjusting it according to experimental needs.

[0067] Features of the presently disclosed method include: analyzing higher order axial aberrations rather than just the five “standard” Seidel aberrations; analyzing off-axis (i.e., reflection angle and image position dependent) aberrations; the analysis of off-axis aberrations is performed by: (1) globally refining the function parameters of an aberration that describes the entire imaging field covering many potential micrographs; (2) propagating the refined parameters for the entire imaging field to each separate micrograph and converting the description of off-axis aberrations into a description using local axial aberrations; (3) allowing a similar conversion to be separately applied to the images of individual particles, which facilitates the use of non-parallel illumination without degrading the quality of the final reconstruction.

[0068] Other features of the presently disclosed method include: refining complex off-axis aberrations (which includes anisotropic magnification distortion) using kernel specification related methods; optimizing aberrations by optimizing the skewness or correlation statistics of the signal from a high contrast source image to guide instrument calibration; performing calibration using a direct electron detector without switching to a separate camera during calibration since a focused beam is not required (image processing is used to align the condenser without a focused beam); identifying unstable components of aberrations during a data collection series and improving them more frequently (per movie, per set of movies, to correct alignment); mapping off-axis and axial aberrations and image rotation at a large number of defocus values to allow for precise mapping between survey mode and data collection mode; using a large combined beam and image shift relative to current methods to rapidly change the portion of the sample being observed, which allows for a many-fold increase in data collection throughput in SPA applications; also using beam-image shift to survey or preview portions of data collection, including options to perform surveys at high magnification and at high speed; three-dimensional clipping of a grid based on effective, finite sampling tomography (using sample tilts to characterize three-dimensional features to prepare for data collection; our figures are three-dimensional and account for defocus); data collection with beam tilt to achieve high throughput mode collection: 20+ structural information (up to + / -2 - 3 degrees, which will guide hierarchical classification); 20+ sub-tomograms to speed up full range tomography; precession data from dark modes of crystals and circular apertures.

[0069] The present disclosure provides a new method for calibrating an EM instrument. Current manual or semi-automatic calibration relies on analyzing signal modulation in Fourier transform space and the shape of the objective aperture image. This conventional calibration method has problems. First, it only identifies a subset of aberrations. Specifically, it has not been developed to characterize off-axis aberrations. Additionally, the calibration of odd-order aberrations (such as coma) relies on an indirect correlation of aberration features and is thus imprecise and prone to misuse. Second, this conventional procedure requires focusing the source beam on the detector. A highly focused beam has a negative impact on high performance direct electron detectors, and thus this procedure requires attaching a separate detector to the microscope.

[0070] In contrast, the currently disclosed methods rely on analyzing higher-order moments of image intensity (e.g., skewness, entropy, or related moments) that are calculated on an image corrected for contrast transfer function (CTF) acquired with a highly accurate and sensitive detector, and thereby more directly determine aberrations without the limitations of the prior art. Our novel method also uses a global description of off-axis aberrations and calibrates them by using kernel canonical correlation analysis. A second novel aspect is the use of non-reduced (fully described) aberrations during data collection and the reversal of their effects by applying software correction during subsequent data analysis; this allows a substantial increase in the throughput of data collection in cryo-EM without negatively affecting data quality. Another innovation is our use of higher-order aberrations in cryo-EM, which enables us to improve resolution and enable a data collection mode with a tilted beam. Except for diffraction measurements, none of the current data collection methods use variable beam tilt to collect data. A tilted beam speeds up tomography, making it possible to perform very fast tomography over a limited angular range and allowing novel imaging experiments involving crystals.

[0071] The present disclosure provides a more detailed (higher-resolution) structural model; this benefit depends on the level of uncorrected aberrations, which is highly variable between experiments. Experimenters skilled at calibrating microscopes better have lower levels of aberrations in their experimental data. Typical current results in the field have barely acceptable resolution, so improving resolution is very important, even if only moderately. Collecting data using beam tilt allows new experimental modalities. These modalities address scientific and industrial problems such as the determination of the absolute configuration of chemicals that do not produce macroscopic (multi-micron) crystals. Improving resolution is one of the most critical aspects of cryo-EM experiments. Optimizing the resolution of a cryo-EM model can take months or years in an individual experimental project. Our method eliminates one of the unknown unknowns, i.e., the state of instrument alignment during data collection, because current software does not provide any indication of the many types of misalignment that exist. Part of the current problem is that verification of alignment is not saved in any experimental records of standard protocols.

[0072] The currently disclosed methods enable cheaper instruments (e.g., Talos Arctica and Glacios) to produce results that are nearly identical in throughput and quality to those produced by more expensive instruments (e.g., Krios). This can be achieved by accepting non-parallel beam illumination that current data analysis software cannot handle; the Krios can adjust the illuminated area in parallel mode, while cheaper instruments produce a converging beam when reducing the illuminated area on the sample. The cost difference between these instruments can be up to millions of dollars, and other factors contributing to the difference are of less significance in practice.

[0073] The calibration steps used in currently disclosed methods are important novel in terms of their use of maximum entropy related methods and the experimental setups suitable for such methods. It is also novel and innovative in using atomic grids to correct off-axis distortions. One can also use the sample under study in calibration, which is particularly useful for following aberration changes and correcting these variable aberrations, although the sample under study can also provide other calibration information. The global (off-axis) analysis of the variations can also be done using methods less efficient than kernel (non-linear) norm related; however, the kernel norm related is more efficient than other methods as it can rely on a relatively small number of observation / calibration data points to recover a sufficiently accurate (reasonable) predictor.

[0074] (One or more) graphene or graphene oxide layers have multiple uses, including providing a transparent and stable support for heavy atoms to produce high contrast, and the crystal periodicity serves as a distance scale and angle distortion (the technical term is elliptical distortion, the meaning derived from observing distorted circles) marker at high magnifications. Strong heavy atom features, once calibrated at high magnifications with respect to their relative distances, can be used as distance features for magnification and distortion calibration at multiple lower magnifications. At high resolutions, the distortions can be considered linear in the image, and if they are anisotropic, they are sometimes called elliptical distortions. At lower resolutions, a non-linear description can be used to describe their position dependence in the image. At high resolutions, the position dependence within the image may not be observable, but we found that if we apply the beam-image shift method, then the magnification and elliptical distortions do change with the change of position within the microscope, but only for shifts much larger than the size of the observed image. To observe the crystalline lattice, a defocus scan can be performed so that the periodic lattice does not fall on one of the zeros of the contrast transfer function (CTF). Observing the crystalline periodicity of light atoms (carbon, silicon, etc.) is preferably performed at or smaller actual pixel sizes. The actual pixel size refers to the physical pixels of the detector, which can be different from the data pixel size. Due to interpolation / super-resolution methods, the data pixel size can be smaller.

[0075] The next step in calibration involves the determination of the CTF, but it is considered a complex quantity. Depending on the situation, such as the microscope, the design of the calibration conditions in the image and beam-tilt-shift space can follow an incomplete or complete factorial design. For each image position, beam tilt, etc. - analyze the CTF of the image - provide the real component with a Laguerre - Gauss function and the higher moments of the analyzed image, similar to independent component analysis (skewness, kurtosis, maximum entropy) providing the phase shift component (imaginary part) of the CTF. All aberrations obtained through this procedure are local axial aberrations.

[0076] The values obtained from calibration enter into Kernel Canonical Correlation Analysis (KCCA). This method allows the recovery of the non-linear correlation between all parameters (aberrations) in the form of a function - thus it provides a very accurate recovery of the local axial aberration values. This function (predictor) can then be used to calculate the aberration values at all points in the image space, even those that are uncalibrated (calibration is usually sparse).

[0077] The design of electron optical devices inherently induces complex off-axial aberrations within the possible range of beam-image shifts. These aberrations are a result of the electron optical geometry, mainly defined during the manufacturing and microscope assembly steps, although some minor variations do occur. For this reason, calibration will have a large component that is stable over time, plus smaller changes that occur between and during data collection, with larger changes occurring after mechanical realignment of the microscope assembly. Combining calibration with analysis (as above) can provide a prior map of off-axial aberrations (calibration) for a given instrument, where the calibration will have both time-stable and time-varying components. Astigmatism has been observed to be particularly variable, even during a single data collection run, while some complex aberrations (such as spherical aberration or trefoil aberration) are expected to be rather stable. Repeating the full calibration using additional explanatory variables involving time with KCCA identifies the stable and unstable components.

[0078] The present disclosure provides the following aberration corrections, in particular: (1) enabling automatic alignment of microscopes used in cryo-EM for stable and time-related (dynamic) components - which is useful and important in practice and can also be applied to non-cryo-EM, high-resolution applications; (2) enabling high-throughput cryo-EM data collection, as non-axial aberrations due to large beam-image shifts can be corrected (using the beam-image shift method, but not at the scale of beam shifts that our method can correct). The data collection method can involve deliberately coupled beam tilting to compensate for the beam tilting effect caused by beam shifts (which is known), which is particularly important when using a phase plate; (3) enabling continuous data collection from multiple positions (holes) - continuous in creating very long movies containing many frames, where one movie corresponds to several micrographs and is split by software after data collection. The delays present in data collection are related not only to the mechanical movement of the stage in the microscope but also to starting and stopping data acquisition from the detector. The current standard is one micrograph per movie, but at the cost of detector setup time, e.g., 3 seconds for Falcon3. Collecting continuous movies from multiple micrographs can reduce the impact of these delays, but these delays become more significant only if other delays are reduced (e.g., from repositioning the sample, which our method greatly reduces). A single micrograph exposure can be completed in just a few seconds, so avoiding detector setup time is important; (4) enabling better use of the phase plate, especially when used in conjunction with the beam-image shift method. This aspect includes: a) we precisely hit the desired point on the phase plate, and b) we use a novel and non-obvious strategy to reposition the phase plate. This novel strategy has several important benefits, including stable and optimized phase shifts across experimental data and efficient use of the phase plate, which enables high-throughput mode; (5) enabling fast tomography by using a combination of stage tilting and beam tilting, which is faster and more accurate than current methods relying only on stage tilting; (6) enabling data collection from crystals in an imaging mode and adjusting data collection and analysis; (7) enabling the acquisition of additional experimental information to improve high-resolution reconstruction of structures, particularly for macromolecules at the lower size end suitable for cryo-EM (e.g., less than 200 kDa). This aspect of the present disclosure is related to defocus, which, although an aberration, is deliberately used to increase contrast in the absence of a phase plate. The precise value of defocus affects high-resolution reconstruction, which is considered very valuable. Defocus is defined as the distance of the particle centroid in the z-direction (beam direction) from the focus of the optical device. For macromolecules (significantly larger than 200 kDa), defocus can be determined by direct reference-based refinement for each particle separately.For smaller particles, this is not possible due to insufficient signal-to-noise ratio. Instead, signals from multiple particles are used to estimate their z-height. If the z-height varies between particles, for example, because the particles are located at different z-heights in the ice layer and may also be due to the tilt and curvature of this layer, then there are obvious problems. If we have different methods to determine the z-height, then the consequences of inaccurate z-determination can be avoided, and our method is relevant here because it provides such a method. Comparing images of different tilts allows for a more accurate determination of the z-height of individual particles than analyzing defocus. In fact, this is a well-known phase detection method in the optical field, such as used in a camera for taking pictures, and has obvious technical adjustments due to the differences between electrons and visible optics. The experimental data collection needs to be adjusted to collect multiple beam tilts (variable beam directions) in one movie or multiple movies. Current instruments are fully capable of making such adjustments, and this is not complex from a software perspective.

[0079] In one aspect, the present disclosure provides a method for correcting off-axis aberrations in an electron microscope, comprising the steps of: performing kernel canonical correlation analysis (KCCA) with previously calibrated input data from the microscope to recover a global description (predictor) of the off-axis aberrations, i.e., a description of the aberrations at each point of the optical system. The method further comprises the previous step of calibrating the electron microscope with a grid covered with graphene / graphene oxide and sparsely sputtered with a radiation-resistant material (such as gold, platinum, or iridium), the grid providing method-compatible signals and method-compatible computational behavior (e.g., atoms of known size, clusters, diffraction maxima, etc.), wherein the calibration data is collected at multiple but not all points (e.g., hundreds of points) of the optical system, and these points are selected to cover the area that will be used in the beam-image shift method or during tilting.

[0080] The method further comprises the step of applying the predictor to obtain a pattern of aberrations at any position away from the optical axis and at an off-center height to navigate (although calibrating the entire available z-height range is not always required for practicing the currently disclosed technology) the image space away from the optical axis without affecting the data quality.

[0081] The currently disclosed method can be applied to the beam-image shift method and / or automatic calibration of a TEM microscope. The currently disclosed method can be applied to the use of a phase plate, including correcting its special aberrations and making a continuous advancement between micrographs during a data collection session (we do not advance it during one micrograph, but we make a very small advancement between micrographs).

[0082] KCCA can operate in a hierarchical manner, i.e., first obtain a time-invariant description of the instrument and then correct the time-varying components of the aberration each time data is collected. In one aspect, the present disclosure provides a method for correcting off-axis aberrations, comprising the steps of: calibrating with a grid covered with graphene / graphene oxide and sparsely sputtered with a radiation-resistant material (such as gold, platinum, or iridium), the grid providing method-compatible signals and method-compatible computational behavior (e.g., atoms, clusters, diffraction maxima of known size, etc.), wherein the calibration data is collected at multiple but not all points (e.g., hundreds of points) of the optical system, these points being selected to cover the region to be used in the beam-image shift method, including focus adjustment due to different z-heights of the tilted sample (we typically calibrate the aberration for the z-distance range caused by the image shift of the tilted sample, but this is much smaller than the available z-distance range); and performing KCCA (kernel canonical correlation analysis) with the input data from the calibration step, wherein this operation provides a description of the aberration at each point of the optical system.

[0083] Calibration can be used to characterize each cryo-EM instrument. In practice, full calibration is usually rarely performed, such as when purchasing, moving the instrument, or some other potentially calibration-destructive change occurs, however, it can also be performed at other time points, as part of a regular or irregular calibration schedule, or at any time to ensure the normal operation of the instrument.

[0084] KCCA provides a formula specific to the instrument on which calibration is performed based on the input from the calibration in (a), the formula allowing determination of the optical aberration at each point of the optical system that we decide to use, but KCCA also takes into account separating the time-constant distortion (e.g., trefoil) from the time-changing distortion. Thus, data collected during normal experiments (non-calibration) can also be used as input to KCCA to determine these time-varying changes as deviations from the values determined during calibration.

[0085] KCCA uses the input numbers obtained from single micrograph calibration to determine the time-invariant and time-dependent microscope descriptions for the beam-image shift method. The calibration is performed in a different manner from that illustrated herein but should have a combination of features not currently understood / required by standard calibration, such as providing a mixture of two materials with atomic features / diffraction from a defined lattice. Standard calibration at the atomic scale does not characterize the higher-order aberrations present in the full range of the beam-image shift method.

[0086] KCCA can be run on different collected calibration data. In contrast, the points collected by our calibration protocol can be used by other methods and may be slower and less accurate than KCCA and still produce potentially useful global predictors. KCCA runs iteratively, i.e., first on the calibration data to obtain a time-invariant description of the microscope (rarely done), and then the time-varying component of the calibration is corrected again each time during the experiment on "normal" samples (frequently done). The output from calibration and KCCA enables the beam-image shift method to achieve greater offsets, i.e., faster data collection, automated microscope calibration, etc. The presently disclosed methods include all combinations of the specific embodiments recited as if each combination had been painstakingly recited.

Claims

1. A method for correcting one or more image aberrations in an electron microscope (EM) image, characterized in that, The method includes: Obtaining a plurality of EM images of an internal reference grid sample, the plurality of EM images being captured using an electron microscope with respect to a plurality of optical conditions and a plurality of coordinated beam-image shifts; Producing an EM micrograph by correcting sample drift of the plurality of EM images; Generating a deconvolved image by deconvolving a transformed image using one or more deconvolution coefficients, the transformed image being generated by applying a Fourier transform to the EM micrograph; Producing a filtered deconvolved image by applying a high-pass filter to the deconvolved image; Producing an aberration-corrected EM micrograph by calculating an inverse Fourier transform of the filtered deconvolved image; Determining an intensity distribution of the aberration-corrected EM micrograph; Calculating moments of the intensity distribution; and Performing an iterative optimization process using one or more deconvolution coefficients until an optimal deconvolution coefficient is determined based on a maximum value of the moments, the iterative optimization process including selecting one or more deconvolution coefficients from a range of deconvolution coefficient values different from a previous iteration of the iterative optimization process.

2. The method according to claim 1, characterized in that, Further included is: Determining an aberration correction function that can operatively predict aberrations using kernel canonical correlation analysis of the optimal deconvolution coefficient, the plurality of optical conditions, and the plurality of coordinated beam-image shifts.

3. The method according to claim 2, characterized in that, Further included is: Obtaining one or more EM images of a calibration check grid sample, the calibration check grid sample having one or more known characteristics different from at least one known feature of the internal reference grid sample; And Producing an aberration-corrected EM image by applying the aberration correction function to the one or more EM images.

4. The method according to claim 3, characterized in that, Further included is: Comparing one or more features in the aberration-corrected EM image with one or more known characteristics of the calibration check grid sample to determine whether the aberration correction function is applicable.

5. The method according to claim 3, characterized in that, When the aberration correction function is within a range, the aberration correction function is determined to be applicable.

6. The method according to claim 1, characterized in that, The iterative optimization process includes at least repeating the calculation of the inverse Fourier transform until the optimal deconvolution coefficient is determined based on a maximum value of the moments.

7. The method according to claim 1, characterized in that, Producing the EM micrograph includes aligning and motion-correcting the plurality of EM images.

8. The method according to claim 1, characterized in that, The plurality of optical conditions are selected from a plurality of defocuses, a plurality of z-heights, a plurality of beam tilts, a plurality of beam parallels, and any combination thereof.

9. The method according to claim 1, characterized in that, The internal reference grid sample includes an amorphous material distributed on a support.

10. The method according to claim 1, characterized in that, The calculation of the moments includes quantifying the shape of the intensity distribution based on a function suitable for optimization using independent component analysis (ICA).

11. The method according to claim 10, characterized in that, The function is selected from the group, and The group includes negentropy, skewness, and kurtosis.

12. The method according to claim 1, characterized in that,The calculation of the moments includes quantifying the shape of the intensity distribution based on optimizing negentropy.

Images