Automated data acquisition method for transmission electron microscopy.

Abstract

To provide an improved calibration process.SOLUTION: The present invention provides an automatic data acquisition method for a transmission electron microscope. The method contains steps of: acquiring a reference image of a sample by a first scale factor; guiding an electron beam of the transmission electron microscope to a target point in each of a plurality of first target points identified in the reference image; acquiring a calibration image of a sample by a second scale factor larger than the first scale factor; identifying an apparent shift between a prediction position of the target point in the calibration image by using an image processing technique and an observation position of the target point in the calibration image; training a non-linear model by using the apparent shift corresponded to the plurality of first target points; calculating a calibration target point against the next target point on the basis of the non-linear model; guiding the electron beam to the calibration target point; and acquiring an image by a third scale factor higher than the first scale factor.SELECTED DRAWING: Figure 1A

Description

[Technical field]

[0001] The present invention relates to a targeting method for controlling a transmission electron microscope to perform automated data acquisition for single particle analysis (SPA) on a cryogenic electron microscope (cryo-EM). In particular, the present invention provides an improved technique for collecting high magnification images via an image shifting process. [Background technology]

[0002] The present invention relates to improving beam control accuracy while imaging a target area in SPA data acquisition.

[0003] SPA Sample Description: A typical SPA sample contains many biopolymers of the same type frozen on a grid (mostly made of copper or gold with a diameter of about 3 millimeters). Such a grid usually contains about 500 meshes ("grid squares", squares with side lengths of about 75 micrometers). Each such square contains a thin carbon foil with hundreds of regularly distributed holes with a diameter of about 1 micrometer (usually in the range of 0.2-2 micrometers). One suitable freezing method is plunge freezing, ensuring that each such hole is covered by a thin layer of amorphous ice containing multiple biopolymers. Other freezing methods, such as jet freezing and print-based freezing, are also possible.

[0004] SPA data acquisition and beam control scheme: SPA data acquisition is typically performed as follows. (1) Set the microscope to low magnification (LM) mode (typically 200-500x). (2) acquiring navigation images ("grid square images") under the conditions of (1); typically, such navigation images span approximately 10-100 micrometers; (3) using the navigation image to identify and select a subset of the foil holes as target regions for data acquisition; (4) Switch to high magnification (HM) mode (typically 20,000-100,000x), (5) Sequentially acquire HM images of biomolecules in the ice sheet at the locations of the foil holes; typically, such HM images span about 100-1000 nanometers. Multiple HM images can be recorded at a single foil hole, provided that the foil is larger than the extent of the HM images and the irradiation used to record an HM image does not overlap with the next HM image (otherwise, the overlapping irradiation would cause radiation damage to the next HM image).

[0005] Here, "target area" means an area that can be captured on the camera in the HM (typically a selected ice foil or part of a foil hole). "Target position" or "target point" means a central position within the target area.

[0006] These points can be visited by appropriate stage movements or by appropriately deflecting the electron beam using the electron microscope's beam deflection unit. The navigation image is used to determine the magnitude and direction of stage movements and / or electron beam deflections required to center the HM image on the selected subset of foil holes.

[0007] Typically, such electron beam deflection consists of two simultaneous and synchronized actions: a deflection above the sample and a deflection below the sample. The combined effect of these two deflections is that the part of the sample imaged on the camera is shifted away from the optical axis (thus effectively mimicking a mechanical movement of the sample), and therefore such electron beam deflection is usually referred to as "image shift". However, in current microscopes, the deflection range of the electron beam is limited to a few tens of micrometers from the optical axis ("off-axis"), so that larger movements require movement of the sample stage.

[0008] Beam deflection is usually preferred whenever possible, because mechanically moving the sample stage is much slower and introduces undesirable mechanical drift that can lead to a loss of resolution in the recorded images.

[0009] One example of an existing method for acquiring large sets of SPA images is the EPU ("E Pluribus Unum") software from Thermo Fisher Scientific, which includes image shifting techniques. These image shifting techniques allow acquisition of image data from off-axis foil holes without moving the stage to bring the foil holes closer to the optical axis of the objective lens. As a result, the number of stage movements required to image a given point on the grid square can be reduced. Optical aberrations resulting from these image shifting techniques can be minimized by simultaneous refocusing of the objective lens, simultaneous re-stigmatization of the image, and simultaneous application of appropriate tilts of the illumination beam.

[0010] The image shift described above is achieved by adjusting the current in the deflector (the optical module that achieves the beam deflection) in the microscope column. The relationship between the set current and the image shift observed at the sample is theoretically linear and is modeled in the image shift control logic.

[0011] However, the image shift applied by this control logic may not shift the beam exactly to the preselected ice hole. In other words, there may be a mismatch between the position of the ice hole measured from the navigation image and the position of the ice hole represented by the current applied to the deflector coils. These mismatches are usually referred to as "targeting errors". In particular, targeting errors may occur when the required image shift is relatively large (e.g., larger than 12 micrometers). The inventors have hypothesized and identified that such errors may result from the following linear and / or nonlinear contributions to the shift observed in the navigation image and the sample: (1) Distortions in the navigation or HM images. These distortions can be due to optical aberrations inherent to the electron microscope lens, such as third-order image distortion, but they can also be due to optical aberrations that are the result of mechanical imperfections, such as slight out-of-roundness or slight shifts, of the electron microscope lens. (2) Local deviations in the height of the sample from the optical plane that the microscope images onto the image detector. Such deviations, especially in combination with the tilt of the illumination beam, can lead to differences between the position of the sample that is intended to be imaged on the camera and the position of the sample that is actually imaged on the camera.

[0012] If such targeting errors become significant compared to the extent of the HM images (e.g., if the HM images span 200 nm and the targeting errors exceed 50 nm), the HM images will no longer adequately capture the target ice foils and SPA data acquisition will become inefficient. Thus, existing methods for acquiring large sets of SPA images can use a process for calibration of image shift to reduce targeting errors in SPA data acquisition. State-of-the-art image shift calibration processes typically include 1) At medium magnification (MM, approximately 5000x), a predetermined subset of target locations of foils or foil holes to be recorded at high magnification {u i}, where u i is a two-dimensional vector indicating the applied image shift), (2) Using an image processing algorithm, accurately locate the center of the hole in the calibration image acquired in (1) and obtain a set of actual positions on the sample imaged on the camera {u' i} where u' i is a two-dimensional vector that indicates the actual observed image shift), and the targeting error Δ i is the difference between the target position and the actual imaged position, Δ i =u' i -u i is given by, (3) Linear model u' ifit =M∙u i +S to the dataset {u i ,u' i}, where the 2 × 2 matrix M corresponds to possible scaling, rotation, and shear, and the vector S corresponds to possible shifts. Alternatively and equivalently, a linear model can also be fitted to the model Δ i fit =(M-1)∙u i +S (where 1 represents the 2 × 2 identity matrix), we use the data set {u i ,Δ i} can also be applied to (4) The model fitted in (3) is applied to acquire HM images at the calibration positions. u cal =M -1 (uS).

[0013] However, the above-mentioned state-of-the-art calibration process has several drawbacks: (1) The target error has a nonlinear component that is not modeled. (2) The predetermined subset of target areas and the acquisition order cannot be adjusted during calibration, which may lead to measurement errors while locating the centers of the foil holes due to the periodicity of the foil hole pattern. (3) Knowledge of sample height variations is not used. (4) The foil hole center finding algorithm cannot work on samples containing irregular holes because it only recognizes certain predefined geometric shapes. (5) If the calibration targets are not chosen properly, directional bias can be introduced into the fit.

[0014] For this reason, current implementations of SPA acquisition protocols typically limit image shifts to less than about 12 micrometers to prevent excessive targeting errors that would otherwise compromise the effectiveness of the SPA acquisition process. Thus, current implementations of SPA acquisition combine image shifts with mechanical movement of the sample, the latter being typically used to effect shifts greater than 12 micrometers.

[0015] The intent of the present invention is to reduce the targeting error to allow a longer range of image shifts, thereby reducing the number of required stage movements and reducing the time required to perform a complete SPA acquisition. Summary of the Invention [Problem to be solved by the invention]

[0016] The current image shift calibration process for SPA data acquisition described above can suffer from targeting errors that cannot be explained by linear models alone. Moreover, this process does not allow for on-the-fly adjustment of a given point without rerunning the entire calibration procedure. Also, it does not take into account the height of the sample. Therefore, the present invention seeks to provide an improved calibration process. [Means for solving the problem]

[0017] The proposed method improves upon existing methods by considering sample non-flatness and tilt (leading to sample height variations) as well as higher order mismatches between the LM and HM modes. The method also allows for in situ adjustment of a given auxiliary image (described in more detail below). These factors can be considered in addition to the linear shift, scaling, rotation, and shear mismatches between the LM and HM modes (as applied in combination with existing techniques for correcting aberrations).

[0018] The following method for improved image shift calibration is proposed ("Calibration Method 1"). (1) Acquire a grid square image at low magnification; (2) identifying a set of target points for data acquisition in the grid square image; (3) For image shift calibration, a subset of target points {u i}, (4) For each target point u in this subset of target points iAbout a. Using image shift to direct the electron beam to a target point; b. Acquire a calibration image at intermediate magnification; c. Using image processing techniques, the shift Δ between the target points in the grid square image and the actual imaged points in the calibration image i Discover, (5) The nonlinear model Δ(u) is calculated based on the data i ,Δ i ) to (6) Apply the model to the full set of target locations to obtain the predicted targeting error u cal =(Δ+1) -1 Generate a corrected target point that describes (u). (7) Using these corrected targeting points, proceed with data acquisition at high magnification.

[0019] The model in (5) may be based on smooth functions, e.g., Zernike polynomials, to capture the smoothly varying global targeting error behavior. Such a polynomial-based model can be expressed in the general form

[0020]

number

[0021] The model may further account for local variations in targeting error, for example by including parametric approaches such as Gaussian process regression ("kriging") or spline interpolation.

[0022] For example, the closed-form inverse matrix (Δ+1) -1 If Δx does not exist, it may be infeasible to perform the inversion of (6). In this case, the inverse model Δ -1 (u') is the data pair

[0023]

number

[0024]

number

[0025] Alternatively, the following relationship can be used to find the calibration position u' = u + Δ(u) ≒ u + Δ(u') ⇒ u = u' - Δ(u'): cal = u-Δ(u), i.e., calculate the inverse matrix (1+Δ) -1 As a first order approximation, 1-Δ can be used.

[0026] The image processing techniques mentioned in (4)c may be based on feature recognition and matching, e.g., circle detection. Alternatively, pixel-based methods for image registration, e.g., mutual information image registration, may be used to find the shifts between images. The latter kind of methods have the advantage that they do not assume that a particular type of feature is present in the image, making them more general than methods that aim to recognize specific image features (a drawback of feature-based methods mentioned as item (4) in the list of pitfalls with current approaches in the previous section).

[0027] Because foil hole patterns are typically periodic and other large and / or high-contrast features are usually not within the field of view, image registration or other shift measurement techniques tend to fail when the targeting error is too large. For example, when the targeting error is comparable to the distance between foil holes, image registration or feature matching techniques typically align features offset by the distance of one foil hole, resulting in an inaccurate targeting error measurement. To address this issue, a second method is proposed in which successive model fitting and position corrections limit the remaining targeting error ("Calibration Method 2"). (1) Acquire a grid square image of the sample at low magnification; (2) identifying a set of target points for data acquisition in the grid square image; (3) For image shift calibration, a subset of target points {u i}, (4) ordering the subset of target locations such that the targeting error is expected to increase from the beginning to the end of the subset; (5) A set of calibration target points {u i cal}, where the initial value of each calibration target point is equal to the original target point, i.e., for every position i, initially u i cal =u i , (6) For each target point u in the subset of target points i About a. Use image shift to correct the electron beam to the target point u i cal Leading to, b. Acquire a calibration image at intermediate magnification; c. Using image processing techniques, the shift Δ between the target points in the grid square image and the actual imaged points in the calibration image i Discover, d. Nonlinear modeling using previously available data vs. (u i ,Δ i ) to ed to a subset of the target points to obtain a set of calibration target points {u i cal}, update (7) Apply the model to the full set of target points to obtain (using one of the methods described above) corrected target points u that take into account the predicted targeting error. cal Generate (8) Using these correction target points, proceed with data acquisition at higher magnification.

[0028] The selection in (3) is made such that the area of ​​the foil hole (e.g., the convex hull of all target points) is covered with the selected points, and a model with small modeling error (e.g., less than 30 nm at any foil hole location) can be fitted with as few selected points as possible. Typically, for a smooth targeting error field, this can be achieved by spatially uniform coverage using 12 points within a grid square of 50 μm side length. The targeting error Δ i Since typically increases with increasing distance from the optical axis, one possibility to implement the ordering in equation (4) is to order by decreasing distance from the optical axis.

[0029] Note that for both methods 1 and 2, the subset of target locations selected for targeting calibration does not have to coincide with the target points ultimately used for data acquisition. The target locations selected for targeting error calibration can be arbitrarily located within the field of view of the grid square image, as long as: These target locations contain features suitable for image registration within the field of view at the target locations (e.g., the target points should not be in areas opaque to the electron beam). This subset of target locations allows fitting a targeting error model that is accurate for all data acquisition points (but may be inaccurate in areas not used for data acquisition). These target locations are reachable by image shifting.

[0030] A natural choice for this set of target locations for performing target error calibration is the set (or a subset) of target points for data acquisition (e.g., all points are on a straight line, except in pathological situations). However, other choices are possible, such as a set of uniformly spaced points with a selected concentration within the image shift range. Note that it may be advantageous to intentionally minimize the overlap between these two sets in order to minimize the electron dose per point and thereby the radiation damage per point. In addition, independently selecting the two sets can help avoid directional bias in the model fitting that may be introduced by a directionally biased set of target points for imaging (mentioned as item (5) in the list of pitfalls of current state-of-the-art methods in the previous section).

[0031] Data acquisition after calibration takes a significant amount of time per grid square, even if image shifting is used exclusively. This can be problematic for calibration in the presence of sample drift. For example, if there are 500 foil holes in a grid square and the acquisition time per foil hole is 2 seconds, the processing time for a grid square is about 16 minutes. With a linear drift of 2 nanometers / minute, this drift introduces an additional targeting error of more than 30 nanometers in the last acquired foil hole, which may be unacceptable for data acquisition. Furthermore, during data acquisition, more information about the sample height becomes available, for example via contrast transfer function (CTF) measurements, which allows the target error component due to sample height variations to be explicitly modeled.

[0032] Therefore, a dynamic procedure is proposed to update the targeting model during acquisition ("Calibration Method 3"). (1) Fit a targeting model Δ to a set of calibrated target points u calExecuting calibration method 1 or 2 to generate (2) selecting a recalibration criterion to determine whether the targeting model needs to be recalibrated during acquisition; (3) for each target point in the set of target points, a. Determine whether to recalibrate the targeting model using the recalibration criteria in (2), and if no, proceed to g; b. Use image shift to correct target point

[0033]

number

[0034] The recalibration criteria in (2) used to trigger a recalibration may be based on the following: i. The number of acquisitions since the last calibration exceeds some predetermined number of acquisitions, or ii. The time since the last calibration exceeds a certain predetermined time interval, or iii. Measurement targeting error Δ iis a given maximum targeting error Δ max It exceeds or a combination thereof. The recalibration criteria may further take into account whether the optical settings have changed since the last calibration.

[0035] The model for e may use the same form as Method 1 or Method 2. It may further account for sample height variation using an explicit model that relates the (signed) distance from an ideal sample plane to the target error contribution. This relationship is detailed further below.

[0036] As in methods 1 and 2, the target locations used to calibrate the model in (3)b-f do not have to be chosen from the set of points intended for data acquisition. For example, if recalibration is only intended to correct drift, the same region may be used for calibration in each iteration of loop (3). In this case, the model fitted in (3)e may be the original model obtained in (1), with all parameters fixed, except for the shift parameter that is fitted to the newly acquired measurements.

[0037] To avoid the overhead of switching to the intermediate magnification in (3)c, the shift measurement may be performed using the high magnification image acquired in (3)g, resulting in the following procedure ("Calibration Method 4"). (1) Perform calibration method 1 or 2 to fit a targeting model Δ to obtain a set of calibrated target points u cal Generate (2) selecting a recalibration criterion to determine whether the targeting model needs to be recalibrated during acquisition; (3) for each target point in the set of target points, a. Using the recalibration criteria in (2), determine whether to recalibrate the targeting model, and if no, proceed to c; b. Nonlinear model for previously available data vs. (u i ,Δ i )=(u i ,Δ(ui )+Δ i ') to Apply the model in cb to a subset of the target points to obtain a set of compensated target points {u i cal}, update d. Perform data acquisition at high magnification with image shift using the correction target point. e. Using image processing technology, target points u in the grid square image i and the shift Δ between the imaging point in the image acquired in (3)d. i 'Discover.

[0038] Generally, there is provided an automated data acquisition method for a transmission electron microscope, the method comprising: acquiring a reference image of the sample at a first magnification; For each of a first plurality of target points identified in the reference image, directing an electron beam of a transmission electron microscope to a target location; acquiring a calibration image of the sample at a second magnification greater than the first magnification; using image processing techniques to identify apparent shifts between expected positions of the target points in the calibration images and observed positions of the target points in the calibration images; training a nonlinear model using the first plurality of target locations and corresponding apparent shifts; Calculating a calibration target point for a next target point based on the nonlinear model; directing the electron beam to a calibration target point and acquiring an image (of the sample) at a third magnification higher than the first magnification.

[0039] One way in which the proposed method differs from known methods is by training a nonlinear model to explain the apparent shift of features. Here, the term "nonlinear" should be understood as meaning "not necessarily linear". In other words, the nonlinear model includes linear and nonlinear components. Existing methods may consider linear effects (such as those introduced by the first-order optical properties of the lens). However, they do not consider higher-order discrepancies.

[0040] Calculating a calibration target point for the next target point may be considered equivalent to calculating a calibration beam shift. Calculating a calibration "target point" may be conceptually considered as predicting and correcting for the movement of the sample during beam steering (so the beam is steered to a different position). A calibration "beam shift" may be conceptually considered as the sample remains stationary but the beam shifting motion is not accurate (so the beam steering motion itself requires calibration). In either case, the configuration of the model is the same, but the sign of the error is inverted. In other words, if the sample appears to move during the shifting motion, a further beam shift may be used to track the sample. If the targeting model for guiding the beam shift appears inaccurate (because an inaccurate position is imaged), this is detected and a correction is applied to "go off the mark" and correct the targeting error. These two effects and processes have opposite signs. However, each is mathematically similar. Therefore, the true error source may be ignored and the above-mentioned techniques may be used to address both. Sample drift can be misinterpreted as a targeting error, and correction becomes equivalent to tracking the drift.

[0041] Acquiring a reference image of the sample at a first magnification may include acquiring a montage of images at higher magnifications, in this manner, the reference image has a wider range or field of view (FOV) than a single higher magnification image.

[0042] Capturing the images may include capturing one or more images (e.g., at a third magnification), or in other words, capturing multiple images for each target point.

[0043] Some prior art methods constrain the targeting model to predict discrepancies resulting from a preset selection of physical effects. In other words, they create a constrained parameterized model that accounts for some physical causes of discrepancies, and then calibrate the parameters of the model by making some measurements. These methods may ignore discrepancies that have physical causes (e.g., including some higher-order optical distortions) that are not considered when developing the model. In contrast to such prior art methods, examples of the method proposed in this application do not constrain the model in this way. Instead, the proposed method provides a nonlinear model with more degrees of freedom to explain discrepancies (such as higher-order discrepancies with uncertain causes) without requiring that the mathematical components of those discrepancies be specifically added to the model. In this way, the proposed method may provide a more reliable model by allowing data obtained from real images captured and analyzed using image processing to define the model. Advantageously, the model may be continually refined as more measurements are obtained from the analysis of the captured real images.

[0044] Acquiring the calibration image at the second magnification or the image at the third magnification may include acquiring the image immediately after the beam is directed, in other words, the beam is directed to a target point and then an image is captured at the target point.

[0045] Identifying the apparent shift between the expected position of the target point in the calibration image and the observed position of the target point in the calibration image may include identifying one or more features of the target point or the sample surrounding it in the reference image and identifying those same features at the position in the calibration image. When referring to an "expected" position, this may be the position of the target point in the calibration image that would result if the beam steering process worked perfectly. Thus, after steering the beam to the target point, the expected position of the target point in the calibration image may be at the center of the calibration image. When steering to a calibration target point different from the target point, the expected position may be offset from the center of the calibration image by the expected apparent shift.

[0046] Calculating the calibrated target point for a target point may include adding a predicted apparent shift at the target point to the target point, which may estimate and compensate for the predicted targeting error.

[0047] As shown in Figures 2A-2D, by adding an offset d to the target point, the feature should be shifted by -d in the calibration image. Image 2B is acquired at the target point without calibration and offset, and the offset d of the feature from the expected point is observed in the calibration image (illustrated by the arrow in Figure 2B). The targeting error is Δ=-d and the apparent shift is S=d. To (approximately) compensate for this error, a calibration target point (indicated by the arrow in Figure 2C) is acquired by subtracting Δ from the target point (or adding the expected apparent shift to the target point). The predicted and measured targeting errors follow the same sign convention.

[0048] Alternatively, calculating a calibration target point for a given target point may include identifying a calibration target point whose predicted apparent shift results in the target point being at the center of an image captured at the calibration target point. In other words, the predicted apparent shift at the calibration target point is added to the target point. This can provide a calibration target point that more accurately places the target point at the center of the image.

[0049] Calculating the calibration target point according to this method therefore amounts to inverting the model that predicts the target error, i.e., generating a model that takes into account the predicted target error and predicts where to steer the beam ("aim off") so that the desired target point is located at the center of the image thus acquired.

[0050] The points and shifts may be expressed with respect to the coordinate system defined by the reference image, or may be determined in another coordinate system. When adding and subtracting, the points and shifts should be in the same coordinate system.

[0051] In particular, the target point may be expressed in the coordinate system of the low magnification (LM) image as a displacement from a central position in the LM image, which is mathematically convenient since the origin of the coordinate system so chosen typically coincides with the position that would be imaged if no further image shift (beyond that applied when acquiring the LM image) is applied.

[0052] The third magnification may be a useful level for acquiring images of biomolecules embedded in ice. Images acquired at the third magnification for purposes of sample analysis may be referred to as "data acquisition" images.

[0053] The third magnification may be higher than the second magnification. Thus, a data acquisition image at the third magnification (which may be referred to as "high magnification", HM) may show more detail than a calibration image at the second magnification (which may be referred to as "medium magnification", MM). This may be advantageous because a high magnification setting may be needed for the data acquisition image to image the sample in great detail. For calibration, it may be advantageous to have a lower magnification so that the field of view (FOV) is larger. This increases the number of features visible in the image, and therefore image shifts can be determined more accurately or more clearly. Furthermore, if certain features, such as foil hole edges, are not visible in the calibration image (e.g., because the apparent shift of the image is larger than the image FOV), the image may not be useful for calibration. This is less likely when the calibration image is captured at a lower magnification (because the FOV is larger but the target error remains unchanged).

[0054] In some examples, the third magnification and the second magnification may be the same. The calibration image may be used as the data acquisition image, or vice versa.

[0055] The sample may include a perforated foil including a plurality of holes, the target point being a point within a first hole of the plurality of holes, and the next target point being located within a second hole of the plurality of holes. Optionally, the second hole may be directly adjacent to the first hole. In other words, the calibration may be performed using a hole adjacent to the next target point. If calibration data is obtained for holes in the vicinity of the next target point, the target error may be predicted more accurately.

[0056] In a first example, recalibration may be performed for every hole. Thus, model accuracy may be maintained high. In another example, recalibration may be performed only every N holes (e.g., every 10 holes). This may maintain model accuracy while improving efficiency. In another example, calibration may be performed asynchronously in the background using images captured during data acquisition. In another example, recalibration may be triggered when model prediction error exceeds a threshold level. This may keep model accuracy at a required level while further improving processing efficiency. In another example, the method may maintain a map of locations where the model is valid. Recalibration may be performed based on a determination that the next target point (or a subsequent target point) in the acquisition sequence is outside the map of valid points.

[0057] There may be multiple target points per foil hole. To image biomolecules suspended in ice and reduce the proportion of the FOV that is outside the foil holes, a target point can be selected in the center of the foil hole.

[0058] In some cases, the sample may be periodic in nature (e.g., the sample contains a grid of holes). In such cases, it may be impossible to correct for apparent shifts larger than half the distance between the foil holes. This is because image processing techniques may mistake features of adjacent holes for the holes that are intended to be imaged. Targeting errors tend to increase as the distance from the optical axis along which the beam is directed increases. Therefore, to ensure that the model works at larger values ​​of image shift, it is advantageous to calibrate the model using holes close to the optical axis, then use the calibrated model to image holes further away and recalibrate the model using those holes. Then, holes further away are imaged, and so on. In this way, the model can be iteratively refined to accurately target areas with large values ​​of beam image shift.

[0059] Steering the electron beam to the target point may include inputting the target point into a beam steering process (configured to adjust the electron beam by an amount determined by a linear beam steering model).

[0060] A non-linear model can be constructed to predict the apparent shift when a given target point is input into the beam steering process.

[0061] The nonlinear model may be configured to estimate an apparent shift based on the target point (in other words, an apparent shift of an image captured when directing the electron beam to the target point using the linear guidance model). The nonlinear model may be used to determine a calibration target point based on the target point to account for the estimated apparent shift (which is the negative of the "expected targeting error"). The electron beam may then be directed to the calibration target point (using the linear guidance model) rather than to the target point. In this way, the apparent shift is accounted for and the target point appears closer to the center of an image captured after directing the beam to the calibration target point than an image captured after directing the beam to the target point.

[0062] The method may further include ordering the first plurality of target points (e.g., by ordering the target points in order of increasing distance from the optical axis or in a spiral outward from the optical axis) such that a magnitude of a corresponding apparent shift for each target point is expected to increase from first to last.

[0063] The method comprises: ordering the first plurality of target points such that the magnitude of each target point (preferably as a vector from the reference image centre) increases from first to last (e.g. by ordering the target points in order of increasing distance from the optical axis); and / or ordering the first plurality of target points such that the angle of the target points varies smoothly from beginning to end (e.g., by ordering the target points to form a helical shape spiraling outward from the optical axis); For each of the first plurality of target points identified in the reference image, the method further includes calculating a calibration target point (for the next target point) based on the non-linear model (updated after processing the previous calibration image); Steering the electron beam to the target point includes inputting the calibration target point into a beam steering process; The method further includes updating the nonlinear model after identifying each apparent shift based on the calibration target points and the corresponding apparent shift.

[0064] Those skilled in the art will appreciate that the angle of a target point (expressed as a vector relative to the center of the reference image) may loop around a complete circle, since an angle of 2π radians is equivalent to an angle of 0 radians. Thus, the absolute value of the angle may appear to jump as it moves through the ordered list as it passes through the axis along which the angle is measured. Nevertheless, the angle may be smoothly changing as it moves through the ordered list in that it has relatively small angles between adjacent points in the ordered list of target points.

[0065] In some of the above exemplary methods, the calibration image is acquired after directing the electron beam to a calibration target point rather than the target point. In this case, the expected location of the target point may be at the center of the calibration image acquired at the calibration target point (because the expected apparent shift has been accounted for by using the calibration target point). Thus, the feature at the original target point is expected to be at the image center. If the feature is not at the image center, a shift measurement between the observed features may be performed to obtain a residual error. The residual error may be added to the predicted error for the target point to obtain a total target error. Advantageously, using the calibration target point for imaging makes the measurement process easier since the image shift is expected to be smaller.

[0066] In another example, the apparent shift may be calculated as the apparent shift between the expected location of the calibration target point (not the original target point) and the observed location of the calibration target point in the calibration image, in which case the calibration target point may be expected to be at the center of the calibration image (because the "expected" location is based on a guidance model and does not take into account the expected apparent shift).

[0067] The method may further include initializing a set of calibration target points using the original target points and updating the set of calibration target points by replacing each target point with a calibration target point calculated for the next target point based on the non-linear model.

[0068] Training a nonlinear model involves training the nonlinear model against the data (u i ,u' i ), for an i-th target point in the first plurality of target points, i εR 2 is the target point identified in the reference image, and u' i εR 2 is the actual position on the sample identified using image processing techniques, and Δ i =u'i -u i is the corresponding apparent shift (or "targeting error"), which is identified using image processing techniques.

[0069] The next target location may be one of a second plurality of target locations, the method including, for each of the second plurality of target locations: Calculating calibration target points based on the nonlinear model; and directing the electron beam to a calibration target location; and acquiring an image of the sample at a third magnification.

[0070] The second plurality of target points may be identified in the reference image or a second reference image, which may be an image of the sample or a second sample.

[0071] The method comprises: For each of the second plurality of target locations, determining whether the nonlinear model is still valid based on one or more predetermined criteria (determining whether the model is still valid is performed prior to using the nonlinear model to calculate the calibration target points); If recalibration is required, the method may further include acquiring an image of the sample (at a current one of the second plurality of target points) at a second magnification, using image processing techniques to identify an apparent shift between an expected position of the target point in the image and an observed position of the target point in the image, and updating the nonlinear model based on the apparent shift.

[0072] If recalibration is required, the method comprises: Calculating pre-calibration target points based on the non-linear model (old non-linear model that needs recalibration); and directing the electron beam to a pre-calibration target point.

[0073] Updating the nonlinear model based on the apparent shift may include updating the nonlinear model using the pre-calibration target points and the corresponding apparent shift.

[0074] Acquiring an image of the sample at a third magnification may include performing data acquisition at a higher magnification with an image shift using the calibration target points.

[0075] The method comprises: For each of the second plurality of target locations, determining whether the nonlinear model is still valid based on one or more predetermined criteria (prior to calculating the calibration target points based on the nonlinear model); If recalibration is required, the method may further include using image processing techniques to identify apparent shifts between the expected and observed positions of the immediately preceding target points from a second plurality of target points in a corresponding image acquired at a third magnification (acquired during a previous cycle), and updating the non-linear model based on the calibration target points and the corresponding apparent shifts.

[0076] In other words, if recalibration is required, the method using image processing techniques to identify an apparent shift between the expected location and the observed location of the previous target point in an image of the previous target point from the second plurality of target points acquired at a third magnification; and updating the non-linear model based on the calibration target point and the corresponding apparent shift relative to the previous target point.

[0077] The previous image acquired at the immediately preceding target location may be an image of the sample acquired at a third magnification. Advantageously, a separate calibration image at an intermediate magnification may not be required.

[0078] For a first target point of the second plurality of target points, there is no immediately preceding target point, however the model has just been trained and therefore may be assumed to be valid.

[0079] The updated model may be subject to a model quality check. For example, the new model may be rejected if the parameters are outside a selected range of valid parameters. The new model may also be rejected based on a quality metric (e.g., quality of fit to a measurement targeting error against a selected error metric).

[0080] The updated model may also take into account all previous data pairs acquired so far.

[0081] Updating the nonlinear model involves updating the nonlinear model to the data versus (u j ,u' j ), for a jth target point in the second plurality of target points, j εR 2 is the calibration target point, and u' j εR 2 is the actual position on the sample identified using image processing techniques, and Δ j =u' j -u j is the apparent targeting error.

[0082] Calculating the calibration target points based on the non-linear model may include applying an updated model (that more accurately predicts the target error) to the target points to provide the calibration target points.

[0083] The method may further include selecting a recalibration criterion during data acquisition to determine whether the target model needs to be recalibrated. The recalibration criterion may be used when determining whether the nonlinear model is still valid.

[0084] Determining whether the non-linear model remains valid may include determining whether to recalibrate the targeting model using a recalibration criterion.

[0085] The method may further include, for each of the second plurality of target points, using image processing techniques to find an apparent shift of the target point (from the grid square image) between its expected position and the imaged point in an image of the sample acquired at a third magnification.

[0086] The nonlinear model may include multiple model components, each of which may have one or more separate validity criteria. For example, moving the stage to a new grid square may invalidate certain components of the model, but not others. Taking a new reference image may invalidate certain components of the model, but not others. Shifting the beam further than before may invalidate some parts of the model, but not others. Determining whether the model is still valid may include determining whether each model component is still valid. Updating the nonlinear model may include updating model components determined to still not be valid and retaining model components determined to still be valid without updating. Advantageously, selectively updating the model in this manner may be more efficient and may result in a more reliable model.

[0087] Determining whether the model is still valid based on one or more predetermined criteria may include evaluating whether the optical settings have changed compared to the optical settings during acquisition of the image used to fit the current model. The evaluation may further be based on prior knowledge of the current grid square compared to the grid square to which the current model was fitted. For example, this prior knowledge may include estimates of height deviations, contamination levels, cracks, etc. The evaluation may further be based on time since last calibration, total drift estimates, or a combination thereof.

[0088] In general, the points used to calibrate the model (the first plurality of target points) do not need to overlap with the set of points intended for data acquisition (the second set of target points), although in some circumstances they can, as explained below.

[0089] The first plurality of target points may be selected such that the target points and the identified apparent shifts are sufficient to train a nonlinear model such that the nonlinear model is accurate at the second plurality of target points.

[0090] The method comprises: Identifying (receiving / determining) a third plurality of points of interest in the reference image; Selecting the first plurality of target locations as a subset of the third plurality of target locations may further include:

[0091] In other words, the selection of target points can be used to calibrate the model.

[0092] The second plurality of target locations may be the same as the third plurality of target locations. In other words, a "data acquisition" image at the third magnification may be captured for each relevant target location. Alternatively, if the calibration image at the second magnification is acceptable, then a data acquisition image at the third magnification may not be required for the target locations for which the calibration image was acquired. Thus, the second plurality of target locations may be a subset of the third plurality of target locations.

[0093] Each of the first plurality of target locations, the next target location, the second plurality of target locations, and / or the third plurality of target locations comprises: the sample includes one or more features suitable for image registration proximate each target location, one or more of the features being visible in an image acquired at the target location; and / or Each target point may be selected to be located within a threshold distance from the optical axis of the microscope such that the target point is reachable by image shifting.

[0094] The method may further include initializing the model (e.g., with a "sameness" model that predicts no apparent shift at any position). Training the model may include updating the model using the data pairs.

[0095] Steering an electron beam of a transmission electron microscope to a target point can include moving the electron beam using a linear steering model based on a position of the target point in a reference image.

[0096] A "target point" is a point on the sample (feature of interest) and also a point in the reference image (where the feature of interest appears in the reference image). Directing the beam to a target point identified in the reference image estimating a physical location of a target point on the sample within the microscope based on the location of the target point in the reference image; and steering the electron beam to a physical location using a linear steering model.

[0097] Calculating the calibration target point can include using a linear guidance model to determine a point to which the beam should be guided, such that when the target error is added to the calibration target point, the desired target point appears near the center of the image.

[0098] The third magnification may be the same as the second magnification, or may be greater than the second magnification.

[0099] Rather than using the same linear guidance model (known to result in a target error) and steering the image to a corrected position to account for the expected error, the guidance model can be updated to account for the nonlinear behavior, and the correction model can be used to steer the beam to the target point. In other words, the nonlinear target model can be incorporated into the guidance model. The nonlinear model is recalibrated as described above, but the linear guidance model may remain the same. This may be because the linear guidance model is based on physical properties of the microscope that do not change.

[0100] The method may further include acquiring a second reference image of the sample (or a different sample) at the first magnification. The second reference image may show a different point on the same sample, such as a different grid square. Alternatively, the second reference image may show a different sample entirely.

[0101] The method may further include identifying a second plurality of target points in the second reference image.

[0102] The nonlinear model may be configured to estimate the apparent shift of a feature of the sample (the feature located at the target point identified in the reference image) in an image acquired by steering (using the linear steering model) the electron beam to the target point. In other words, the nonlinear model is configured to predict the apparent shift for a given target point, where the apparent shift is the difference between the location that would be expected if the linear steering model were perfect (the center of the image) and the location in the image where one would actually expect to observe the feature if the steering model is imperfect.

[0103] The nonlinear model may account for deviations of a real sample from an ideal flat sample plane. In some instances, the nonlinear model assumes that the real sample is smooth.

[0104] Estimating the apparent shift using the nonlinear model may include estimating the difference between the physical location of the feature and the projection of the feature onto the imaging plane of the microscope. In other words, the model may take into account the z-offset of the sample at the target point. The z-offset at the target point may be estimated based on one or more estimated or determined z-offsets for other features of the sample (at other nearby points), assuming the sample is smooth (assuming the sample height varies smoothly between the sample features). The origin of the z-axis may coincide with the optical surface (often called the "image plane" for simplicity) (at or near the sample) that the microscope images onto the image detector, and the direction of the z-axis may coincide with the main direction of beam propagation (the "optical axis").

[0105] Stage height and tilt may be taken into account when estimating the z-offset. Thus, the z-offset can be split into two components: the difference between an idealized flat sample plane and the image plane, as may be caused by tilting the sample and, optionally, a shift in the height of the stage above the image plane; and The offset of a sample feature from an idealized flat sample plane.

[0106] Such z-offset can be estimated in several ways. One way in which the z-offset can be measured is based on determining the defocus at the imaged data acquisition point. This measurement is typically performed by analyzing the power spectrum of the acquired image and fitting a model of the contrast transfer function (CTF) of the imaging system, where defocus is a key parameter. Since defocus is directly proportional to the z-offset of the imaged point from the imaging plane, the measured defocus can be used to determine this z-offset.

[0107] One other way in which the z-offset can be estimated is to determine the expected position of the feature using already acquired measurements and a model that accounts for optical aberrations and / or stage tilt, assume that any residual difference is due to a height mismatch between the feature's position and an ideal flat sample plane, and use the residual apparent shift to estimate the z-offset.

[0108] Steering the electron beam can include adjusting a tilt and / or a shift of the electron beam. Adjusting the tilt and / or a shift of the electron beam can include: Adjusting the incident electron beam (moving the illumination beam), and / or This may include adjusting the transmitted electron beam (moving the camera FOV).

[0109] The method may further include mapping a position (and / or orientation) of one or more features identified in the reference image to a corresponding position (and / or orientation) in the calibration image using a transformation operation, which may include one or more rotation, translation, and / or scaling operations.

[0110] The transformation operation may take into account the estimated z offset.

[0111] Using image processing techniques to identify apparent shifts between expected positions of target points in the calibration image and observed positions of target points in the calibration image determining expected locations of features in the calibration image using a (linear) guidance model; Identifying features in the calibration image at the observation position (e.g., using feature detection and feature matching processes); and determining the apparent shift as the difference between the expected position and the observed position.

[0112] If the sample includes a perforated foil containing a plurality of holes, the target point may be a point within one of the holes. Identifying the feature / target point in the first image may include identifying an edge of the hole in the first image. Identifying the feature in the calibration image may include identifying an edge of the same hole in the calibration image.

[0113] The method comprises: Obtaining defocus measurements for one or more calibration images; and updating the non-linear model based on the defocus measurements.

[0114] The method comprises: It may further include using the defocus measurement to infer the z-offset of the feature (the difference between the physical location of the feature and the projection of the feature onto the imaging plane of the microscope).

[0115] Training / updating the non-linear model may involve using the target points, the corresponding apparent shifts, and the corresponding defocus measurements.

[0116] The method may include repeatedly acquiring images at a second magnification and refining the nonlinear model by performing one or more cycles, each cycle comprising: identifying a target point in a reference image; directing the electron beam to a target location; acquiring a calibration image at the target location at a second magnification; Identifying a reference feature at the observed position in the calibration image, determining an apparent shift as the difference between the observed position and a predicted position (obtained using the linear guidance model), and updating the non-linear model based on the apparent shift; and / or The method may include obtaining defocus measurements for the calibration image, inferring z-offsets for the features using the defocus measurements, and updating the non-linear model based on the z-offsets.

[0117] The iterative cycle for updating the model may be repeated until a first number of images in which the residual target error is used to update the target model reaches a first threshold and / or until a second number of images in which the defocus measurements are used to update the target model reaches a second threshold.

[0118] In another example, a method for automated data acquisition for a transmission electron microscope is provided, the method comprising: acquiring a reference image of the sample at a first magnification; For each of a first plurality of target points identified in the reference image, shifting a stage of the transmission electron microscope to the target location (preferably to align the target location with the optical axis of the microscope); acquiring a calibration image of the sample at a second magnification greater than the first magnification; using image processing techniques to identify apparent shifts between expected positions of the target points in the calibration images and observed positions of the target points in the calibration images; training a nonlinear model using the first plurality of target locations and corresponding apparent shifts; Calculating a calibration target point for a next target point based on the nonlinear model; shifting the stage to a calibration target location and acquiring an image at a third magnification greater than the first magnification.

[0119] In this way, the proposed method can also be used to calibrate and correct targeting errors in the stage shifting process. The optional features mentioned above can equally be applied to this method.

[0120] There is also provided a transmission electron microscope apparatus configured to perform one or more of the methods described above. The apparatus may comprise a transmission electron microscope, a processor for executing instructions, and a memory for storing instructions.

[0121] Also provided is computer software including instructions that, when executed on a computer processor, cause the computer to perform one or more of the methods described above. The software may be executed on a general-purpose computer system communicatively coupled to the electron microscope. Alternatively, the software may be executed on dedicated hardware coupled to the microscope. [Brief description of the drawings]

[0122] Specific non-limiting examples of the present invention will now be described with reference to some non-limiting examples. [Figure 1A] 1 shows a schematic diagram of a charged particle microscope. [Figure 1B] 1 shows an actual SPA image that suffers from targeting inaccuracies. [Figure 2A] 1 shows schematic images captured during one general exemplary embodiment;FIG. 2 shows a reference image showing target points within the reference image;FIG. [Figure 2B] 1 shows schematic images acquired during one general exemplary embodiment, and a calibration image acquired by directing an electron beam at a target point and the apparent shift of the target point caused by target error. [Figure 2C] 1 shows schematic images acquired during one general exemplary embodiment; 2 shows calibration target points in a reference image; [Figure 2D] 1A-1C show schematic images acquired during one general exemplary embodiment;FIG. 1D show calibration images acquired by directing an electron beam at a calibration target point;FIG. [Figure 3A] Calibration methods 1 and 2 are shown. [Figure 3B] Calibration method 3 is shown. [Figure 3C] Calibration method 4 is shown. [Figure 4] 1 shows a geometric calculation of the targeting imprecision introduced by sample height variations. [Diagram 5] 1 shows a flow diagram of a particular exemplary implementation of the proposed method. [Figure 6]1 shows targeting error on a life science sample. [Figure 7] The targeting error on a 30 degree tilted gold grid sample is shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0123] FIG. 1A (not drawn to scale) is a highly schematic illustration of an example of a charged particle microscope M in which the present invention can be implemented. In one example, the charged particle microscope M may be a transmission microscope, such as a Transmission Electron Microscope (TEM). As shown in FIG. 1A, within a vacuum enclosure 2, an electron source 4 generates a beam B of electrons propagating along an electron optical axis B′ and traversing an electron optical illuminator 6, which serves to direct / focus the electrons onto a selected portion of a sample S, which may be (locally) thinned / flattened. Also shown is a deflector 8, which may be used (among other things) to perform a scanning movement of the beam B.

[0124] The sample S is held on a sample holder H, which can be positioned in multiple degrees of freedom by a positioning device / stage A, which moves a cradle A' to which the holder H is (removably) fixed. For example, the sample holder H may comprise fingers movable (among other things) in the XY plane. A Cartesian coordinate system is also shown in FIG. 1A. Typically, movements parallel to the Z axis and tilts about the X / Y axes are also possible). Such movements allow different parts of the sample S to be illuminated / imaged / inspected by the electron beam B traveling along the axis B' (Z direction) and / or allow a scanning movement to be performed as an alternative to beam scanning. If desired, an optional cooling device (not depicted) can be brought into intimate thermal contact with the sample holder H, thereby maintaining the sample holder H (and the sample S thereon) at, for example, cryogenic temperatures.

[0125] The electron beam B interacts with the sample S in such a way as to cause it to emit various types of "stimulated" radiation, including (for example) secondary electrons, backscattered electrons, X-rays, and light radiation (cathodoluminescence). If necessary, one or more of these radiation types can be detected using an analysis device 22, which can be, for example, a combined scintillator / photomultiplier or an EDX (Energy Dispersive X-ray Spectroscopy) module. In such a case, an image can be constructed using essentially the same principles as in an SEM. However, alternatively or supplementarily, electrons that traverse (pass) the sample S, emerge / emit from the sample S, and continue to propagate (substantially, but generally, deflected / scattered to some extent) along the axis B' can be studied. Such a transmitted electron bundle is incident on an imaging system (projection lens) 24, which broadly includes a wide variety of electrostatic / magnetic lenses, deflectors, correctors (such as astigmatism correctors), etc. In normal (non-scanning) TEM mode, this imaging system 24 can focus the transmitted electron beam onto a phosphor screen 26, which can be retracted / retracted, if required, out of the way of axis B' (schematically indicated by arrow 26'). An image (or diffractogram) of (a part of) the sample S is formed on the screen 26 by the imaging system 24, which image can be viewed through a viewing port 28 provided in a suitable part of the wall of the housing 2. The retraction mechanism of the screen 26 can be, for example, mechanical and / or electrical in nature and is not shown here.

[0126] As an alternative to viewing an image on screen 26, one can instead take advantage of the fact that the focal depth of the electron beam leaving imaging system 24 is typically quite deep (e.g., about one meter). As a result, various other types of analytical devices can be used downstream of screen 26, as will be described in more detail below:

[0127] One other type of analytical device that can be used downstream of screen 26 is a TEM camera 30. At the location of camera 30, the electron beam can form a still image (or Fourier transform diagram) that can be processed by controller / processor 20 and displayed on a display device (not shown), such as a flat panel display. When not needed, camera 30 can be retracted / retracted (as shown diagrammatically by arrow 30') to move the camera off axis B'.

[0128] Another type of analytical device that can be used downstream of screen 26 is a STEM camera 32. The output from camera 32 can be recorded as a function of the (X,Y) scanning position of beam B on sample S, and an image can be constructed that is a "map" of the output from camera 32 as a function of (X,Y). Camera 32 can include a single pixel having a diameter of, for example, 20 mm, as opposed to the matrix of pixels that is typically present in camera 30. Furthermore, camera 32 can typically be a 10 mm pixel, as opposed to a 10 mm pixel. 2 A much higher acquisition speed (e.g., 10 images / sec) 6 points / sec). Again, when not needed, camera 32 can be retracted / retracted (as indicated diagrammatically by arrow 32') so as to be off axis B' (however, such retraction would not be necessary, for example, in the case of a donut-shaped annular darkfield camera 32, in which a central hole allows the passage of the light beam when the camera is not in use).

[0129] As an alternative to performing imaging using the cameras 30 or 32, a spectroscopic device 34 can also be implemented, which can be, for example, an EELS module.

[0130] It should be noted that the order / position of the components 30, 32, and 34 is not strict and many possible variations are possible. For example, the spectroscopic device 34 could be integrated with the imaging system 24.

[0131] It should be noted that a controller (computer processor) 20 is connected to the various illustrated components via control lines (bus) 20'. This controller 20 may provide various functions such as synchronizing actions, providing set points, processing signals, performing calculations, and displaying messages / information on a display device (not shown). Of course, the (schematically depicted) controller 20 may be located (partially) inside or outside the housing 2 and may have a unitary or composite structure, as desired.

[0132] The skilled person will understand that the interior of the housing 2 does not have to be maintained in a strict vacuum. For example, in so-called "environmental TEM / STEM", a background atmosphere of a given gas is deliberately introduced / maintained in the housing 2. The skilled person will also understand that in practice, it may be advantageous, if possible, to limit the volume of the housing 2 so that it essentially envelops the axis B', taking the form of a small diameter tube (e.g. of about 1 cm diameter) through which the electron beam used passes, but widening to accommodate structures such as the source 4, the sample holder H, the screen 26, the camera 30, the camera 32, the spectrometer 34, etc.

[0133] Different parts of the sample can be brought into the field of view of the camera 30 by mechanically shifting certain parts of the sample to the optical axis, or by shifting the electron beam towards these specific parts using the electron microscope's electron beam deflectors. However, such deflections can cause aberrations, which can lead to loss of resolution if these induced aberrations are not properly compensated for. This is explained in the following paragraphs.

[0134] The resolution and field of view achievable in electron beam instruments such as scanning electron microscopes, electron beam microprobes, and electron beam lithography machines are limited by aberrations in the optical system. These aberrations can be classified as parasitic and intrinsic. Parasitic aberrations are caused by imperfections in the lens, such as imperfect circularity or inhomogeneity in the magnetic properties of the material that generates the lens' magnetic field. The best-known parasitic aberration of lenses is (2-fold) astigmatism. Intrinsic aberrations are inherent to the lens and therefore cannot be avoided by careful machining. Intrinsic aberrations are traditionally classified as purely geometrical (the best-known aberration is spherical aberration) or energy-dependent (the best-known aberration is (primary) chromatic aberration). There are eight types of third-order intrinsic geometric aberrations of lenses: isotropic and anisotropic distortion, field curvature, isotropic and anisotropic off-axis astigmatism, isotropic and anisotropic off-axis coma, and spherical aberration. The term "off-axis" is used to distinguish off-axis astigmatism and off-axis coma, which are intrinsic aberrations and are third order, from axial astigmatism and coma, which are parasitic aberrations and are first and second order, respectively. It should be noted that the terms "off-axis" and "axial" are often omitted when it is clear from the context which form of astigmatism or coma is meant.

[0135] For image points on the optical axis of the system, only spherical aberration occurs. As the electron beam is focused to image points further off-axis, the remaining seven aberrations become important in determining the achievable focused spot size and the degree of distortion present in the image.

[0136] Some of these optical aberrations can be corrected with commonly known techniques. The only field-limiting aberrations for which some kind of simultaneous correction is not provided are isotropic and anisotropic coma. As is commonly known, proper placement of the beam-defining aperture allows isotropic coma to be cancelled even in non-scanning electron beam instruments such as conventional electron microscopes. However, there has been no means of completely correcting anisotropic coma. Furthermore, it is a challenge to simultaneously correct anisotropic and isotropic coma while minimizing field curvature.

[0137] In an illustrative example, the best resolution that a 300 kV STEM microscope (without a built-in spherical aberration corrector) can give is about 0.10 nm. Typically, this means that the sample is imaged at about 0.04 × 0.04 nm resolution. 2 Modern STEM controls are capable of collecting images with dimensions of 8000 x 8000 pixels. This is equivalent to a 320 x 320 nm 2 and a maximum off-axis distance of u = 320 nm / √2 = 226 nm. The blur due to off-axis coma can be calculated as follows:

[0138]

number

[0139] To understand the nature of off-axis aberrations, it can be useful to simplify the objective lens to be an infinitely thin lens. Ideally, the refractive strength of the lens increases linearly with the distance of the beam to the center of the lens, with the associated proportionality constant equal to the inverse of the focal length of the objective lens. In practice, however, higher order aberrations cause the refractive strength to deviate from the linear dependence. However, for an infinitely thin lens, a beam directed through the center of the lens does not suffer from these higher order aberrations, since the refractive power of the lens is zero at its center. It is therefore possible to scan a sample at an infinite distance off-axis without imposing higher order aberrations on the beam, provided that the scanning beam is tilted (or oriented) to travel through the center of the objective lens. This approach is known to those skilled in the art as "locating the pivot point at a coma-free plane." Similarly, for real lenses that are not infinitely thin, there is usually a plane fairly close to the center of the lens that has the property that no isotropic off-axis coma is introduced when the scanning beam is directed to intersect the optical axis in this plane.

[0140] By using a complex pivot point, i.e. by adding a shift-dependent tilt correction to the shift, not only isotropic coma can be avoided or corrected, but also anisotropic coma.

[0141] In conventional SPAs, the maximum image shift is set by the aberration introduced by the inherent third-order aberration of the objective lens. The third-order aberration is a relevant quantity since the first-order aberration corresponds to the (axial) astigmatism that can be corrected by an astigmatic device or lens, while the second-order aberration vanishes because the system is rotationally symmetric. The following equation shows this quantitatively: Given the position and angle at the sample in the complex numbers u=x+iy and v=a x +ia y and their complex conjugates are denoted by the additional top bar. The general expression for the shift induced by all third order aberrations is:

[0142]

number

[0143] Introduces a shift u → u + s and a tilt v → v + t in the sample. This introduces additional aberrations

[0144]

number

[0145]

number

[0146] In conventional single particle analysis, no tilt is applied, only a shift s, where the effective coma aberration limits the maximum image shift.

[0147] In this case, a coma aberration B of about 2 μm causes a resolution degradation of about 1 Å, which is the maximum resolution degradation that can be tolerated in practice. Applicant's Krios™ transmission electron microscope has K=0.15+1.42i, so the maximum allowable image shift is about

[0148]

number

[0149] The above-described technique for correcting the aberrations that arise during the image shifting process is sometimes referred to as "aberration-free image shifting" or "AFIS" technology, and is described in U.S. Patent Application Publication No. 2021 / 0272767.

[0150] Aberration-free image shift (AFIS) and the effect of sample height When the electron beam passes through the lower objective lens at an off-axis position, the resulting image experiences undesirable phase aberrations, such as coma, which can be detrimental to the final SPA image quality. This off-axis coma can be counteracted by tilting the beam so that it passes through a fixed point in the lower objective lens (often called the "center of rotation" or "coma-free center") such that the resulting coma is minimal. Pivoting the beam around the coma-free point results in a different shift for each plane perpendicular to the optical axis above the coma-free center. Therefore, to achieve a specific beam shift at the sample, the height of the sample above the coma-free center (the distance between two planes perpendicular to the optical axis, one containing the relevant sample point (target area) and the other containing the coma-free center) must be known. For simplicity, it is common practice to assume that the sample is flat and oriented perpendicular to the optical axis, that the sample is located in the optical plane where the microscope images with the detector, and therefore the height of the sample from the coma-free center is fixed and known. This specimen plane is called the "image plane" or "target plane". The true local height of the specimen above the coma free center is the sum of the height of the target plane above the coma free center and the height of the associated specimen point above the target plane (which can be positive or negative depending on the direction of the height deviation from the target plane). This local deviation of the specimen height above the target plane is called the "specimen height" or "specimen z offset".

[0151] In current microscope systems, the conversion between the desired beam shift and tilt at a selected target plane and the required lens current is established by a calibration procedure that must be performed by an expert. This conversion is stable over time for each acceleration voltage and requires expert knowledge for recalibration, so it is rarely adjusted even when the sample height (e.g., derived from the sample stage z coordinate) is known to deviate from the target plane. In particular, it is not common to dynamically adjust the target plane (by adjusting the strength of the lenses in the imaging system) due to local sample height variations, even when they are predictable, such as when the sample stage is tilted by a known angle.

[0152] The image shifting technique allows the collection of image data from off-axis foil holes in the sample S without moving the stage A to bring these foil holes closer to the optical axis B'.

[0153] In one example, the sample includes a grid with many circular foil holes, each hole containing an amorphous layer of ice with a nearly identical copy of the molecule being imaged. Each ice foil hole is about 2 μm in diameter, with an average separation of about 5 μm. The stage moves to the center of the holes, and two to six different image shifts are used, each measuring approximately 0.5 × 0.5 μm. 2 Two to six images are acquired, covering an area of ​​​​100 mm. The stage then moves to the next hole and the procedure is repeated. This process can be repeated thousands of times, thus generating thousands of images. From these images, particles are extracted, sorted, aligned, and averaged to a specific resolution in the reconstructed particle. An acquisition session can typically take several days.

[0154] The relaxation time for stage shifting can be up to about one minute, and this latency can be a dominant contributor to the total time of a SPA session. If image shifting could be used instead of stage shifting, this total time could be significantly reduced.

[0155] The targeting software may be configured to automatically capture images of desired points by adjusting the deflectors. Existing techniques also aim to predict targeting errors to facilitate improved beam control. This may include a calibration process that compares the apparent positions of foil holes observed using low magnification (LM) with the apparent positions of those same foil holes when observed using high magnification (HM). The discrepancies between the apparent positions are fitted to a model, which is then used to determine the linear rotation and scaling discrepancies between the LM and HM modes. These discrepancies can then be taken into account when targeting to image specific areas at higher magnification during automated operation of the microscope.

[0156] There can be a relatively large mismatch (e.g., in scale, rotation, distortion) between images collected using LM mode and HM mode. One reason for this is that the first imaging (or "intermediate") lens in the projection system 24 of microscope M is switched on in HM mode and switched off in LM mode.

[0157] One example of a single particle analysis (SPA) technique is cryogenic electron microscopy (cryoEM), where a sample S comprises sample particles that have been cryogenically cooled and suspended in glass water. The aqueous sample solution is applied to a foil mesh grid and allowed to freeze. The foil mesh grid contains a number of foil holes in which the frozen solution is suspended. The foil holes in the mesh grid are then analyzed by a microscope M.

[0158] Analysis of the foil holes can be performed automatically to improve the speed at which data is collected. One major bottleneck that limits the throughput of automated data collection is the time spent on stage movements of stage A. To reduce drift, each stage movement must be followed by a settling period. Furthermore, extra tracking steps may be required to compensate for inaccuracies in the stage movements. The image shifting technique described above is designed to reduce the number of required stage movements, thereby increasing throughput during automated data collection.

[0159] However, the experimental targeting accuracy of the image shift in SPA can be found to be worse than 100 nm when the shift distance on the sample plane is larger than 10 μm. An example of a typical bad image is shown in Figure 1B, where more than 50% of the field of view (FOV) is unusable (for imaging the foil itself, not the ice floating in the foil hole of the sample S).

[0160] Figure 1B shows actual SPA images that suffer from targeting imprecision. The FOV of these images is approximately 300 nm.

[0161] To ensure that the beam still contacts some of the conductive support film (to help reduce charging effects on the sample) without this illuminated portion of the film appearing within the camera FOV (or at least without the film dominating most of the FOV), the target accuracy needs to be better than 50 nm. Thus, the current target accuracy limits the effective number of foil holes that are reachable from a single imaging center (in other words, the target accuracy limits the number of holes that can be imaged without moving stage A).

[0162] The intention of the present invention is to improve the target accuracy to better than 50 nm for shift distances of the order of 25 μm (the range limit of current image deflectors). Such an improvement allows the image shifting technique to cover more area, thereby reducing the number of stage movements per square acquisition. Also, the throughput in tilted SPA acquisition is significantly improved (at least 2-fold), which proves to be crucial for proteins with orientation preferences induced by the air-water interface, such as the COVID-19 spike protein.

[0163] The main types of error sources for targeting include: 1. Unevenness of the sample, 2. Defocus in LM mode, which causes linear rotation and scaling for the HM mode, suggesting a mismatch between the LM and HM maps. 3. Other error sources such as higher order distortions and mechanical imperfections in the deflector.

[0164] Existing techniques only address error source 2: defocus causing linear rotation and scaling between LM and HM modes. At the start of the collection of a series of images, known techniques collect several images and apply linear (scaling and rotation) corrections to the observed targeting error. In this way, the possible influence of sample non-flatness on the targeting error is ignored, along with possible higher order distortions.

[0165] To address error source 1, the method proposed in this example takes into account that variations in sample height, in combination with the beam tilt selected for acquisition (as beam tilt is inherent to the illumination scheme), can generate an illumination shift that can account for a significant portion of the inaccuracies currently affecting known image shifting techniques.

[0166] In addition to compensating for error source 1, to address error sources 2 and 3, in a particular example, the proposed method fits a smooth quadratic function on the residual error in-place to enable accurate error prediction for nearby acquisition regions, thereby further improving targeting. The physical root causes of the higher order error contributions may be unknown. However, it may not be necessary to attribute the error contributions to physical root causes. The present invention may work well by modeling these errors as higher order terms.

[0167] One general example implementation is illustrated with reference to FIGS. 2A-2D and can be summarized as follows. 1) Acquire an overview image I(u) of the sample at low magnification (LM) as shown in Figure 2A (here,

[0168]

number

[0169] In an ideal scenario (as assumed by the prior art), the center of this HM image corresponds to the center of the region of interest u1. However, for various reasons, there may be an apparent shift between these centers. This can be caused by: i. Incorrect calibration of the magnification of the LM and / or HM images, ii. Incorrect orientation of the LM and / or HM images; iii. Distortion in the LM image, which can be due to: a. Optical aberrations, and / or b. Camera imperfections, iv. Inaccurate targeting due to sample non-flatness, as explained below. v. Stage drift, and vi. Other unspecified factors.

[0170] To account for such targeting errors, the following steps can be added to the method. 5) Using cross-correlation, neural networks, or AI, determine the apparent shift or “targeting error” Δ1 between the center of the HM image and the center of the first target region u1, as indicated by the arrow in FIG. 2B.

[0171] All target regions u i Once visited and recorded, an error map Δ is provided for the overview image. This error map should be smooth (in other words, free of abrupt changes) since the underlying causes i, ii, iii, iv, v all produce a smoothly varying error map. Due to this smoothness, the map can be created using already recorded HM images and their errors, and this map can be interpolated and / or extrapolated to areas not yet acquired. Thus, while image acquisition is still in progress, already recorded HM images can be used to predict what corrections should be applied to the beam shift to record the next HM image. 6) Refine the error map using the latest apparent shift / targeting error Δ1; 7) Using the refined error map, the next target point Δ i+1 Targeting error u i+1 Predict the calibration target point

[0172]

number

[0173] Some exemplary methods are: i Change the order in which the target regions u i may be visited along a spiral path that spirals out from the center of the LM overview (or "reference") image.

[0174] The error map Δ(u) is the basis function B k (u). In other words, Δ(u)=Σ k c k B k (u). Coefficient c k are all zero initially (so there is no error), and the measurement error Δ i is used to determine and fine-tune.

[0175] In one example, Zernike polynomials may be used for the basis functions. These functions may be useful for describing optical aberrations. Alternatively or additionally, 2D spline functions may be used for the basis functions. These functions may be more suitable for describing the non-flatness of the sample.

[0176] The method is extended by an additional step 2a (taking an exterior image after step 2 and a final image at HM after step 3) in which maps at some intermediate magnification MM are recorded (e.g. LM=100x, MM=6,000x, HM=100,000x) to measure and construct a first approximation of Δ(u).

[0177] 3A shows a flowchart of "calibration method 1" and "calibration method 2." Below, the steps shown in the flowchart will be described individually.

[0178] In step S301, a targeting model is initialized. This initialization can be an identity map, i.e., a model that, given any target point, returns a calibration target point that is the same as the given target point. Alternatively, initialization can be done by copying parameters from a model fit to targets from another grid square.

[0179] In step S302, a grid square image is acquired.

[0180] Based on the grid square image acquired in step S302, target points are identified in step S303, which is typically done by a hole-finding algorithm, but can be assisted or performed entirely by a human operator.

[0181] In step S304, a subset of target points is selected from the target points identified in step S303. This selection is made such that the area of ​​the foil holes (e.g., the convex hull of all target points) is covered by the selected points so that a model with small modeling error (e.g., less than 50 nm at any foil hole location) can be fitted to as few selected points as possible. Typically, for a smooth targeting error field, this can be achieved by spatially uniform coverage using at least 10 points within a grid square with a side length of 50 μm. Alternatively, the selection may be made randomly from each of a selected number of bins, each bin containing a selected number of points in a selected order. For example, the order of points may be the order of acquisition set by the acquisition software.

[0182] In step S305, the calibration criteria determines whether the current target model is still valid. This evaluation may be based on whether the optical settings have changed compared to the acquisition of the image used to fit the current model. The evaluation may also be based on prior knowledge of the current grid square compared to the grid square to which the current model was fitted. For example, this prior knowledge may include estimates of height deviations, contamination levels, cracks, etc.

[0183] If the targeting model evaluated in step S305 is invalid, the method enters a loop. At the start of the loop, a stopping criterion is checked in step S306. This stopping criterion may be the logical OR of "Is there no next target available?" and further criteria. These further criteria may be based on a confidence value that the current target model has an error less than a selected value.

[0184] In step S307, a next target point is selected from the subset 4 of target points as the current target point.

[0185] In "Calibration Method 2," the current target model is applied to the current target point to generate a calibration target point in step S308. In "Calibration Method 1," the calibration target point is selected as the current target point.

[0186] In step S309, a medium (MM) or high (HM) magnification image is acquired at the calibration target point using an image shift that deflects the beam so that it is centered on the calibration target point.

[0187] The targeting error at the calibration target point is measured in step S310 by comparing a crop of the grid square image at the calibration target point to the image acquired in step S309. This measurement may be performed using image processing techniques, such as feature tracking, optical flow, feature recognition, or image registration, to detect shifts between the two images. The measured targeting error at the calibration target point is stored.

[0188] The new targeting model uses previously available data vs. i ,Δ i ) and for the i-th position,

[0189]

number

[0190]

number

[0191] In step S312, which is reached either if the model evaluated in step S305 is still valid or if the stopping criterion checked in step S306 is reached, the current target model is applied to all target points selected in step S304, resulting in calibration positions for all target points.

[0192] A conventional data acquisition method is performed in step S313 using the calibration target points from step S312.

[0193] 3B shows a flowchart of "Calibration Method 3." Below, the steps shown in the flowchart will be explained individually.

[0194] In step S321, the targeting model is initialized. This initialization can be an identity map, i.e., a model that, given any target point, returns a calibration target point that is the same as the given target point. Alternatively, initialization can be done by copying parameters from a model fit to targets from another grid square.

[0195] In step S322, a grid square image is acquired.

[0196] Based on the grid square image acquired in step S322, target points are identified in step S323. This is typically done by a hole-finding algorithm, but can be assisted or performed entirely by a human operator.

[0197] At step S324 it is checked whether further targets remain to be processed for acquisition. If not, data acquisition for the current grid square ends and the method proceeds to step S313. The application may decide to continue acquisition in the next grid square.

[0198] If more targets remain to be processed, the next target is selected in step S325.

[0199] In step S326, the calibration criteria determines whether the current target model is still valid. This evaluation may be based on whether the optical settings have changed compared to the acquisition of the image used to fit the current model. The evaluation may also be based on prior knowledge of the current grid square compared to the grid square to which the current model was fitted. For example, this prior knowledge may include estimates of height deviations, contamination levels, cracks, etc. The evaluation may further be based on the time since the last calibration, a total drift estimate, or a combination thereof.

[0200] If the current model is valid, it is immediately applied to calibrate the current target points in step S331. If not, the current model is applied to the current target to generate pre-calibration target points in step S327.

[0201] In step S328, an image at medium magnification (MM) is acquired at the pre-calibration target point using an image shift that deflects the beam so that it is centered on the pre-calibration target point.

[0202] The targeting error at the pre-calibration target point is measured in step S329 by comparing a crop of the grid square image at the pre-calibration target point with the image acquired in step S328. This measurement may be performed using image processing techniques, such as feature tracking, optical flow, feature recognition, or image registration, to detect shifts between the two images. The measured targeting error at the pre-calibration target point is stored.

[0203] In step S330, a new targeting model is generated based on the previously available data (u i ,Δ i ) and for the i-th position,

[0204]

number

[0205]

number

[0206] In step S332, a high magnification (HM) image is acquired at the calibration target point.

[0207] 3C shows a flowchart of “Calibration Method 4.” Below, the steps shown in the flowchart will be explained individually.

[0208] In step S341, the target model is initialized. This initialization can be an identity map, i.e., a model that, given any target point, returns a calibration target point that is the same as the given target point. Alternatively, initialization can be done by copying parameters from a model fit to targets from another grid square.

[0209] In step S342, a grid square image is acquired.

[0210] Based on the grid square image acquired in step S342, target points are identified in step S343. This is typically done by a hole-finding algorithm, but can be assisted or performed entirely by a human operator.

[0211] At step S344 it is checked whether more targets remain to be processed for acquisition. If not, data acquisition for the current grid square ends at step S351. The application may decide to continue acquisition at the next grid square.

[0212] If more targets remain to be processed, the next target is selected in step S345.

[0213] In step S346, the calibration criteria determines whether the current target model is still valid. This evaluation may be based on whether the optical settings have changed compared to the acquisition of the image used to fit the current model. The evaluation may also be based on prior knowledge of the current grid square compared to the grid square to which the current model was fitted. For example, this prior knowledge may include estimates of height deviations, contamination levels, cracks, etc. The evaluation may further be based on the time since the last calibration, a total drift estimate, or a combination thereof.

[0214] If the current model is valid, then in step S348 it is immediately applied to calibrate the current target point. If not, in step S347 a new targeting model is applied to the previously available data pairs (u i ,Δ i ) and for the i-th position,

[0215]

number

[0216]

number

[0217] In step S348, the current target model is applied to the target points to generate calibrated target points.

[0218] In step S349, a high magnification (HM) image is acquired at the calibration target point.

[0219] The targeting error at the calibration target point is measured in step S350 by comparing a crop of the grid square image at the calibration target point with the image acquired in step S349. This measurement may be performed using image processing techniques, such as feature tracking, optical flow, hole recognition, or image registration, to detect shifts between the two images. The measured targeting error at the pre-calibration target point is stored.

[0220] In the proposed method shown in the further specific examples, the variation in sample height can be accounted for by the model. One way this can be achieved is by adding a dimension z to the targeting algorithm (sample height deviation from a "virtual targeting plane", as explained below).

[0221] A geometric model for determining the targeting error is shown in Figure 4. The model takes into account the sample height variation. For simplicity, the figure assumes that the coma free pivot point coincides with the center of the objective lens.

[0222] The apparent targeting error vectors are defined as follows: Δ=u'-u. The x and y components of these vectors are Δ=(e x ,e y ), u = (x, y), and u' = (x', y'). When the sample is flat and overlaps with the "ideal flat sample surface", the irradiation position is given by equations (1a) and (1b). x'=x+e x (1a) z'=h+xtan(α)+e x tan(α) (1b)

[0223] Comparing similar triangles, we can conclude that in equation (2) f / x=(h+xtan(α)+e x tan(α)) / e x (2)

[0224] Therefore, the targeting error e x can be calculated using equation (3) as follows: e x =(xh+x 2 tan(α)) / (f-xtan(α)) (3)

[0225] In the particular example, the proposed method first approximates the sample height using the above formula, assuming that the sample is flat and overlaps with the "ideal flat sample plane". The method switches to a more accurate height extrapolation scheme that takes sample height variation into account once there are enough defocus values ​​from neighboring positions available. This approach assumes local smoothness of the SPA sample. The particular proposed method keeps monitoring the error (using neural network-based image recognition) and refines the sample height prediction on the fly using a smooth function.

[0226] Compared to conventional techniques, improvements are offered by the proposed method (sometimes referred to as 3D-AFIS), including: Taking into account sample height variations; Using extrapolation for sample height prediction; On-the-fly monitoring and correction of residual errors.

[0227] A detailed example is shown in the flow chart of Figure 5. The method begins in step S501.

[0228] In step S502, a conventional beam control method is performed.

[0229] In step S503, the system acquires the next HM image (eg, using the "Acquire Data" preset function as provided in the EPU software mentioned above).

[0230] In step S504, the system determines whether the foil hole edge is visible. If yes, the method proceeds to step S505. If no, the method proceeds to step S506.

[0231] In step S505, the system first determines u' by: 1. Image segmentation for edge contouring 2.Geometric analysis

[0232] Second, the system calculates the error vector using equation (4). e = (e x ,e y )=u'-u-predicted_optical_error(u') (4)

[0233] Here, the expected optical errors are set by default to (0,0) at the beginning of the session, and we also assume that the optical errors introduced by changes in z' are negligible, which is valid considering the small beam tilt angles.

[0234] In step S506, the system determines whether the following two criteria are met: A number of HM images exceeding a threshold "M" are acquired with measured defocus. HM images with a number of foil hole edges exceeding a threshold "N" are analyzed.

[0235] M and N are positive integer parameters set by the user to ensure robustness. They can be determined based on the experiment type.

[0236] If both criteria are met, the method proceeds to step S509. If one or both criteria are not met, the method proceeds to step S507.

[0237] In step S507, the system assumes the sample is flat and estimates z using α.

[0238] In step S508, the system performs an improved beam control where x, y, z and the function predict_optical_error(.,.) are used to predict the beam / image tilt / shift for the next HM region, including the case where z=0.

[0239] In step S509, the system uses all x, y, b, and z from the pre-acquired HM image to extrapolate and determine z using f and α as a priori known parameters, where b is defined as the difference between the Thon ring measured defocus and the objective lens defocus. It measures the sample height deviation from a "virtual targeting plane" using the local deviation of the sample height from an ideal flat sample plane and the contribution from the sample tilt.

[0240] Aberrations that are odd orders in the beam angle (e.g., focus and astigmatism, which are first order, and spherical aberration, which is third order) can be observed because these odd order aberrations affect so-called "Thon rings" in the Fourier transform of the image (Thon rings are rings or ellipses of zero intensity in the Fourier transform). The Fourier transform of the image may be observed in real time, and focus and astigmatism controls may be adjusted to optimize image resolution. The difference between the Thon ring measured defocus and the objective lens defocus can be determined by analyzing the position and ellipticity of the Thon ring.

[0241] In step S510, the system fits and updates the predict_optical_error(.,.) function with the difference between the measured e and the predicted e using equation (3) above.

[0242] In this way, optical errors are addressed and accounted for in the model without the need to know the physical root causes.

[0243] Experimental Results The targeting imprecision was investigated for a typical SPA sample and a tilted gold grid sample. First, "raw" data was collected for the entire grid square (no correction). To do so, the following steps were performed: Set the image shift range so that the entire grid is covered, Switch off the scaling and rotation calibration in the software, Collect HM images of all foil holes in the entire grid square. Evaluate the "raw" targeting error (i.e., the difference between the position in the sample map recorded in the LM and the position measured from the HM image).

[0244] Figure 6 shows the targeting error experimentally measured on a life science sample. Error measurements were obtained by targeting all reachable positions on a grid square using image shift without applying any correction. Each arrow in the vector field plot indicates one error measurement, where the start of the vector is placed at the target point for that measurement and the tip of the vector is displaced from the start of the vector by the measurement error.

[0245] Figure 6A shows the raw error vector field (no corrections applied), Figure 6B shows the residual error vector field after emulating a scaling and rotation calibration as performed in known techniques (about 50% of the area has an error of 50 nm or more), and Figure 6C shows the residual error vector field after emulating a particular exemplary technique proposed in this application (about 7% of the area has an error of 50 nm or more).

[0246] Figure 7 shows the targeting error on a gold grid sample tilted by 30 degrees. Figure 7A shows the raw error vector field (without correction). Figure 7B shows the residual error vector field after emulating a scaling and rotation calibration as performed in known techniques (about 52% of the area has an error of 50 nm or more). Figure 7C shows the residual error vector field after emulating a specific exemplary technique proposed in this application (less than 1% of the area has an error of 50 nm or more).

[0247] The "raw" targeting errors are shown in Figures 6A and 7A.

[0248] The performance of known approaches is illustrated in Figures 6B and 7B, which show the correction of raw target error with rotation and linear scaling using known techniques.

[0249] Finally, the same raw target error is corrected using a specific method described in this application by looping through the foil holes in acquisition order and predicting the target error from the nearest already acquired foil hole. The results are shown in Figures 6C and 7C. In all cases, the current approach proved insufficient to reach an overall accuracy of 50 nm or better, but for the majority of foil holes the accuracy was better than 50 nm using the proposed technique.

[0250] It is worth noting that part of the residual error shown in Figures 6C and 7C may be due to inaccurate measurement of the foil hole center, rather than the actual error after correction using the proposed method.

[0251] The phase aberration and apparent astigmatism of an image can be determined using a technique known as "Thon ring fitting", which measures the location and ellipticity of the Thon rings in the Fourier transform of the image. The contrast transfer function (CTF) mathematically describes how aberrations affect an image and can be used to determine parameters that quantify the phase aberration and apparent astigmatism (among other parameters such as defocus).

[0252] It is desirable that the conductive support film does not appear inside the camera FOV so that all of the captured images are useful. However, the software refines the model based on the residual targeting error determined by observing the discrepancies between the expected and observed positions of the foil hole edges. In order to observe / measure these discrepancies, the foil hole edges need to be within the FOV. One solution is to take more images at the same HM magnification: initial images with the foil hole edges to refine the model, and subsequent images without the foil hole edges. In another solution, auxiliary HM images can be acquired at slightly lower magnification simultaneously with the high magnification HM images. In this way, the camera FOV is widened to see the foil hole edges at the same time as capturing an image without the foil hole edges. The high magnification HM image (without foil hole edges) and the low magnification HM image (with foil hole edges) share the same targeting error since most of the microscope settings are the same. Only the final magnification is adjusted, and only slightly.

[0253] As used herein, including in the claims, singular forms of terms herein are to be construed as including the plural, and vice versa, unless the context indicates otherwise. For example, a singular reference, including in the claims, such as "a" or "an" (e.g., an analog-to-digital converter) means "one or more" (e.g., one or more analog-to-digital converters), unless the context indicates otherwise. Throughout the specification and claims of this disclosure, words such as "comprise," "including," "having," and "contain," as well as variations of words such as "comprising" and "comprises" or the like, mean "including, but not limited to," and are not intended to exclude other components.

[0254] Although embodiments according to the present disclosure have been described with reference to particular types of devices and applications (particularly transmission electron microscopy, single particle analysis, and cryo-electron microscopy), and the embodiments have particular advantages in such cases, as discussed herein, the approaches according to the present disclosure may be applied to other types of devices and / or applications. The specific construction details of the microscope, while potentially advantageous (particularly considering the constraints and capabilities of known electron microscope systems), may be significantly modified to arrive at a device having similar or identical operation. Each feature disclosed herein may be replaced by an alternative feature serving the same, equivalent, or similar purpose, unless otherwise specified. Thus, unless otherwise specified, each feature disclosed is merely an example of a generic series of equivalent or similar features.

[0255] The above techniques are described in the context of a transmission electron microscope (TEM). It should be understood that the techniques may be useful when applied to sample alignment in other charged particle beam microscope systems, such as scanning tunneling electron microscope (STEM) systems, scanning electron microscope (SEM) systems, dual beam microscope systems, and / or ion-based microscopes. This discussion of TEM imaging is provided merely as an example of one suitable imaging modality. These techniques may be used with a collimated incident beam.

[0256] Any and all examples or exemplary language provided herein (for example, "such as," "for instance," The use of "examples" and the like is merely to better illustrate the invention and does not denote a limitation of scope, which is within the scope of the invention unless otherwise asserted. No language in the specification should be construed as indicating any element not claimed as essential to the practice of the invention.

[0257] Any steps described herein can be performed in any order, or simultaneously, unless otherwise stated or required otherwise by context.

[0258] All aspects and / or features disclosed herein may be combined in any combination, except combinations in which at least some of such features and / or steps are mutually exclusive. There may be certain combinations of aspects that are of further benefit, such as the aspect of determining a set of compensation parameters and applying the set of compensation parameters to measurements as described herein. In particular, preferred features of the present invention are applicable to all aspects of the present invention and may be used in any combination. Similarly, features described in non-essential combinations may be used separately (not in combination).

[0259] When this application refers to "odd" and "even" order aberrations, this refers to orders that are angle-dependent (as opposed to phase-dependent). In other words, odd order aberrations include first order focal and astigmatism, and third order spherical aberration. Even order aberrations can include on-axis and off-axis coma.

Claims

1. 1. A method of automated data acquisition for a transmission electron microscope, comprising: acquiring a reference image of the sample at a first magnification; For each of a first plurality of target points identified in the reference image, directing an electron beam of the transmission electron microscope to the target location; acquiring a calibration image of the sample at a second magnification greater than the first magnification; using image processing techniques to identify apparent shifts between expected positions of the target points in the calibration images and observed positions of the target points in the calibration images; training a non-linear model using the first plurality of target points and corresponding apparent shifts; calculating a calibration target point for a next target point based on the nonlinear model; directing the electron beam to the calibration target point and acquiring an image at a third magnification greater than the first magnification.

2. ordering the first plurality of target points such that the length of the vector from the center of the reference image to each of the target points increases from first to last; and / or ordering the first plurality of target points such that a deflection angle of a vector from the center of the reference image to each of the target points varies smoothly from first to last; For each of the first plurality of target points identified in the reference image, the method further includes calculating a calibration target point based on the non-linear model; Steering the electron beam to the target point includes inputting the calibration target point into a beam steering process; The method of claim 1 , further comprising updating the nonlinear model after identifying each apparent shift based on the calibration target points and the corresponding apparent shift.

3. The next target location is one of a second plurality of target locations, and the method includes, for each of the second plurality of target locations: Calculating calibration target points based on the nonlinear model; and directing the electron beam to the calibration target point; and acquiring an image of the sample at the third magnification.

4. For each of the second plurality of target locations, determining whether the non-linear model is still valid based on one or more predetermined criteria; 4. The method of claim 3, further comprising: if recalibration is required, acquiring an image of the sample at the second magnification, using image processing techniques to identify an apparent shift between an expected position of the target point in the image and an observed position of the target point in the image, and updating the non-linear model based on the apparent shift.

5. For each of the second plurality of target locations, determining whether the non-linear model is still valid based on one or more predetermined criteria; 4. The method of claim 3, further comprising: if recalibration is required, using image processing techniques to identify apparent shifts between expected and observed positions of previous target points from the second plurality of target points in the corresponding images acquired at the third magnification; and updating the nonlinear model based on the calibration target points and the corresponding apparent shifts.

6. 4. The method of claim 3, wherein the first plurality of target points are selected such that the target points and the identified apparent shifts are sufficient to train the nonlinear model such that the nonlinear model is accurate at the second plurality of target points.

7. identifying a third plurality of associated target points in the reference image; The method of claim 3 , further comprising: selecting the first plurality of target locations as a subset of the third plurality of target locations.

8. For each of the first plurality of target locations, the next target location, the second plurality of target locations, and / or the third plurality of target locations, the sample includes one or more features suitable for image registration proximate each target location, one or more of the features being visible in images acquired at the target locations; and / or 8. The method of claim 7, wherein each target point is located within a threshold distance from the optical axis of the microscope such that the target point is reachable by image shifting.

9. acquiring a second reference image of the sample at the first magnification; The method of claim 1 , further comprising: identifying a second plurality of target points in the second reference image.

10. 2. The method of claim 1, wherein the nonlinear model is configured to estimate an apparent shift of a feature of the specimen in an image acquired by directing the electron beam at the target point.

11. Steering the electron beam preferably comprises tilting and / or shifting the electron beam. Adjusting the incident electron beam; and / or The method of claim 1 , comprising adjusting by adjusting a transmitted electron beam.

12. using image processing techniques to identify apparent shifts between the expected positions of the target points in the calibration images and the observed positions of the target points in the calibration images; determining the expected locations of features in the calibration image using a guidance model; identifying a feature in the calibration image at a viewing position; and determining the apparent shift as a difference between the expected position and the observed position.

13. Obtaining defocus measurements for one or more calibration images; The method of claim 1 , further comprising: updating the nonlinear model based on the defocus measurements.

14. A transmission electron microscope apparatus configured to carry out the method according to any one of claims 1 to 13.

15. Computer software comprising instructions which, when executed by a processor of a computer, cause the computer to carry out a method according to any one of claims 1 to 13.

Images