Method for autofocus and astigmatism correction in a microscope

Abstract

A method for autofocus and astigmatism correction for a particular microscope setup, in particular an electron microscope, the microscope being capable of adjusting at least the microscope parameters working distance, the stigmata in the X-direction and the stigmata in the Y-direction, the method comprising: a) capturing a first image of a sample at a first working distance perturbation and capturing a second image of the sample at a second working distance perturbation at a current working distance and a current stigmator setting; b) selecting n subregions of the first image and n subregions of the second image, where n≧1, and the i-th subregion of the first image and the i-th subregion of the second image form an i-th input patch pair, where 1≦i≦n; c) processing each i-th input patch pair and receiving an i-th correction term having corrections for the current working distance, the x-direction stigmata, and the y-direction stigmata; d) receiving an output correction term in response to all of the correction terms; e) adjusting the current working distance and stigmator settings by applying the output correction term to the current working distance and stigmator settings; Includes.

Description

[Technical field]

[0001] The present invention relates to an autofocus and astigmatism correction method for a microscope. [Background technology]

[0002] For microscopes, especially electron microscopes, the microscope parameters, especially those related to image quality, are often manually controlled by the microscope user, but adjusting the parameters for each probe is a very time-consuming task.

[0003] As an improvement, electron microscopes may rely on automation and control algorithms to ensure higher image quality with minimal human intervention.

[0004] However, currently used focus and astigmatism correction algorithms typically attempt to explicitly model image formation and subsequent aberration correction, often resulting in long processing times and poor generalizability to untested settings.

[0005] The high resolution of electron microscopes (EM) and their ability to image even the smallest details of a sample are unmatched. A key component of automated EM is maintaining high quality images throughout the entire imaging process, which may involve millions of individual images and prolonged operation. This makes manual tuning of microscope parameters nearly impossible. Tight constraints on algorithm run-time budgets, convergence speed of aberration correction, and low electron dose budgets to avoid sample artifacts pose significant challenges to automated defocus and astigmatism correction algorithms, despite their necessity, especially in high-throughput electron microscopes.

[0006] Existing solutions in the field of scanning electron microscopy (SEM) are typically based on explicit physical models of the interaction of the electron beam with the sample. Measurements (in this case images) with known perturbations are made, followed by focus and StigmatorParameters are inferred and wavefront aberrations are estimated. Although such a physically based approach should in theory lead to good generalization properties, i.e. the ability to work well with different microscopes and samples with no or minimal adaptation, in practice it contains many parameters that require careful tuning.

[0007] Here we present a primarily data-driven method for aberration correction in scanning electron microscopes that works under very low signal-to-noise conditions, reduces processing times by more than an order of magnitude compared to the state of the art, converges rapidly over a wide aberration range, and can be recalibrated automatically by non-experts for different microscopes and challenging samples.

[0008] In particular, machine learning-based focusing for scanning electron microscopes and Stigmator The proposed algorithm is nearly instantaneous inference, has fast convergence, works on noisy images at low electron doses, does not require expert knowledge, has a ready-to-use procedure for recalibrating to new machines or samples, and guarantees convergence in all usage scenarios. Summary of the Invention

[0009] It is an object of the present invention to provide a method for autofocus and astigmatism correction in an electron microscope, which method is fast and robust in use.

[0010] The problem is solved by a method according to claim 1. The further dependent claims provide preferred embodiments. Also according to claim 13 a method for generating training data for a machine learning modelling model used in a microscope setup is provided. The problem is also solved by a computing system according to claim 15.

[0011] According to the invention, a method for autofocus and astigmatism correction for a particular microscope setup, in particular an electron microscope, is provided. The microscope has at least the following microscope parameters: working distance, X-direction Stigmator , and in the Y direction Stigmator and the method further comprises: a) At the current working distance and the current stigmator setting, First working distance perturbation have Take a first image of the sample and perform a second working distance perturbation. have capturing a second image of the sample; b) selecting n sub-regions of the first image and n sub-regions of the second image, where n≧1, and the i-th sub-region of the first image and the i-th sub-region of the second image form an i-th input patch pair, where 1≦i≦n; c) Process each i-th input patch pair and compute the current working distance, Stigmator and in the y direction Stigmator receiving an i-th correction term having a correction for d) receiving an output correction term responsive to all of said correction terms; e) The output correction term is multiplied by the current working distance. Stigmator By applying the setting, the current working distance and Stigmator adjusting the settings; Includes.

[0012] In general, images of flat specimens or samples in microscopes, particularly scanning electron microscopes, are best imaged when the spot size of the electron beam is equal to or smaller than the sampling distance. Stigmator (herein referred to as stig x and stig y) allow the microscope operator to directly control the spot shape, making it smaller than or equal to the pixel size at the beam-sample interaction point, resulting in a sharp image.

[0013] Advantageously, the microscope or microscope set-up may be a scanning electron microscope (SEM) or an environmental scanning electron microscope (ESEM). It may also be a scanning transmission electron microscope (STEM) or a transmission electron microscope (TEM), but the imaging principles are different, so the working distance and Stigmator Instead of the parameters of the setup, different microscope parameters need to be controlled in addition or alternatively.

[0014] The proposed method takes as input the current microscope working distance and Stigmator Have two images with known working distance perturbations in your setup. Current working distance and Stigmator The settings relate to the particular images that may be taken by the microscope. The current working distance and Stigmator Note that the setup typically provides an out-of-focus image of the sample.

[0015] Furthermore, the method requires at least two perturbed test images to be able to retrieve the magnitude and direction of the correction term. According to a preferred embodiment, the correction term includes a correction direction.

[0016] According to a preferred embodiment of the invention, the perturbations may be of different types with respect to each other, in particular the first perturbation and the second perturbation are both positive perturbations, or both negative perturbations, or the first perturbation is a positive perturbation and the second perturbation is a negative perturbation.

[0017] More preferably, the absolute values ​​of the first and second perturbations are either identical or different from each other. In particular, if one perturbation is positive and the other negative, the perturbations preferably have the same absolute value. However, it is also possible that one perturbation is positive and the other negative, and the perturbations have different absolute values. If one perturbation is positive and the other negative, the important fact is that these perturbations introduce an asymmetry in terms of sharpness of the subregions of the image pair, meaning that one subregion of the image patch pair is sharper than the other, thereby resulting in the correction direction of the correction term. If the initial setting is a focused image and there is an asymmetric perturbation in the working distance parameter, the test image inherits the same blur distortion (due to the symmetry of pure defocus). In the case of an unfocused initial setting, the asymmetry of the distortions in the test image can be exploited to determine the difference between the in-focus parameter settings (working distance and Stigmator A correction vector for the set of

[0018] Asymmetry can also be exploited when the perturbations have the same sign, i.e. both perturbations are positive or negative. However, if both perturbations have the same sign, the absolute values ​​of the perturbations must be different from each other. If the perturbations have the same absolute value, the same perturbed image is captured twice, so no image patch pair is formed and no directionality is considered.

[0019] However, there are some requirements for the perturbations from the point of view of the present invention. First, it should be noted that the present invention is not limited to two perturbations. In fact, multiple perturbations may be used, the total number of which may exceed two. Preferably, the total number of perturbations is fixed, and more preferably, is pre-defined for the method of the present invention. Furthermore, the magnitude and direction may also be fixed. The direction may be given by the sign of the perturbation.

[0020] Furthermore, the absolute value of the perturbation must be comparable to the target parameter range, which means that the order of magnitude of the perturbation is the working distance. StigmatorThis means that the perturbations must be on the order of magnitude of the corrections in the settings, or less. Furthermore, the perturbations must be sufficiently different for the target parameter range.

[0021] According to a preferred embodiment of the present invention, the size of the i-th partial region of the first image and the i-th partial region of the second image are substantially identical. In the sense of the present invention, "substantially" relates to a difference of at most 10%, preferably 5%, in the size of the respective partial regions. Most preferably, the size of the i-th partial region is identical. As an example, the test image has a size of 1024x768, a pixel size of 10 nm, and the size of the partial region is 512x512, 256x256 or even 128x128 square, and larger sizes are also possible. However, it has been found that a smaller size of the partial region may lead to a slower convergence speed. It should be understood that the size of the image, the pixel size and the size of the partial region may have values ​​different from those presented. However, it should be understood that the size of the partial region is smaller than the size of the entire image.

[0022] According to a further preferred embodiment, the position of the i-th partial region of the first image relative to the first image is substantially identical or different to the i-th partial region of the second image relative to the second image.

[0023] According to a further preferred embodiment of the present invention, the i-th sub-region of the first image and the (i+1)-th sub-region of the second image may partially overlap. However, it is also possible that the sub-regions do not overlap or only partially overlap, since the sub-regions may be selected randomly and independently.

[0024] According to a further preferred embodiment of the invention, the output correction term is the return value of a function. More preferably, the function combines every i-th correction term into one output correction term.

[0025] As an example, such a function can take into account the average value of all correction terms, or the median value of all correction terms. As yet another alternative, the function can be a computational function that is highly robust to outliers, such as the RANSAC algorithm.

[0026] According to a further preferred embodiment of the invention, the method steps a) to e) are repeated at least one time. Advantageously, the method is stopped when a termination condition is met.

[0027] Preferably, the termination condition is the current working distance and Stigmator Settings and adjusted working distance and Stigmator The termination condition may be that the absolute difference between the settings is smaller than a first threshold value. Also preferably, the termination condition is that the sharpness measure indicating the sharpness of the image exceeds a second threshold value. Or, preferably, after a predetermined number of repetitions of the method steps a) to e). The termination condition may also be a combination of the possibilities offered for further specifying the termination condition.

[0028] During the development of the method of the present invention, it was shown that the method converges typically after 3-4 iterative steps, or iterations of the method.

[0029] According to a further preferred embodiment of the invention, method step c) is implemented using a machine learning modeling method having machine learning modeling method settings. Advantageously, the machine learning modeling method is a deep learning method. Preferably, at least one machine learning modeling method setting is a weight of the machine learning modeling method.

[0030] The machine learning modeling method receives a number of input patch pairs as input and converts the input into the actuation distance and the Stigmator The machine learning modeling method preferably consists of a sequence of convolutional and fully connected layers for nonlinear feature extraction of the input and transformation to the target domain.

[0031] According to a further preferred embodiment of the present invention, the machine learning modeling method weights the i-th output term for each i-th input patch pair with an i-th weighting coefficient, the i-th weighting coefficient depending on the importance predicted by the machine learning model of the i-th input patch pair to the final output correction term.

[0032] During the development of the method, it was realized that samples and specimens sometimes contain sites or regions where there is little information available to the autofocus algorithm, such as blood vessels in tissue. For example, blood vessels in tissue may only exhibit a blank of epoxy resin in the sample holder, which does not present contrast that the AF algorithm can use, resulting in suboptimal results. It was therefore reasoned that these regions should have less weighting in the autofocus decision. This is achieved by incorporating an ith weighting factor into the architecture, leading to a new set of outputs of the method that independently weight each autofocus estimate. The output of these new methods is a loosely regularized score, preferably regularized only in terms of weight decay, which is already used as a weighting factor during the machine learning modeling method training.

[0033] Two different granularities of weighting were tested. First, at the level of input image patches, which are cut-outs of pairs of patches of a larger input image acquired by the microscope. Second, at the level of individual pixels, which leads to the scoring of all locations in the input image. Both approaches are more robust to object regions with less contrast information, showing that the method does not require traditional image processing to pre-filter low-contrast regions.

[0034] As an example, the output of the processing step is a correction term extended by a weighting factor:

number

[0035] This can be treated as an output correction term:

number

[0036] According to a preferred embodiment, the learning method comprises: i) Working distance and Stigmator capturing a focused image with known settings; ii) storing a set of parameter settings and perturbed test images (e.g., two for each out-of-focus parameter setting if the working distance perturbation is asymmetric) for the k out-of-focus parameter settings; iii) repeating steps i) and ii) j times for different subregions of the sample, targeting known out-of-focus parameters to generate a total of j k image pairs that are used as training inputs for the model; iv) adjusting the correction term by the known working distance and Stigmator comparing the settings to determine a comparison error, and iteratively adjusting the settings of the machine learning modeling method in response to the comparison error; Includes.

[0037] In particular, the weights of the machine learning modeling method are adjusted using, inter alia, the backpropagation method.

[0038] It may also be possible to use techniques / methods other than backpropagation, or to combine different methods.

[0039] It should be noted that the above learning method produces a model that is specialized for the microscope used to generate the learning data.

[0040] However, when transferred to a different microscope setup with significantly different imaging settings, such as landing energy, beam current, working distance range, and rotational image acquisition, the method may fail and lead to divergence instead of convergence.

[0041] Here, we present a method to create training data for machine learning modeling models that is independent of microscope setup and microscope settings.

[0042] A method for generating training data for a machine learning modeling model for use in a microscope setup, in particular an electron microscope setup, said method being independent of the microscope setup or microscope settings, and the microscope being configured to have the following microscope parameters: working distance, x-direction Stigmator , and in the Y direction Stigmator and the method further comprises: a) Reference working distance and Stigmator Take a single focused image of the sample with the settings assigning a first score value to the single in-focus image representative of sharpness of the in-focus image; b) generating N out-of-focus images for the Lth position of the sample, where L=1, determining a score value corresponding to each out-of-focus image, and determining a directional correction term using an optimization method; c) Working distance and Stigmator By applying the directional correction term to the setting, the working distance at the Lth position of the sample and Stigmator Adjust the settings to suit your working distance and Stigmator Obtaining a new image with the setting and a score value associated with the obtained image; d) Score value, working distance, Stigmator Using the setting, determining a directed correction term from an optimization technique; e) Repeating c) and d) until the score value of the obtained image is substantially equal to or less than the first score value of a), and determining a reference working distance and Stigmator obtaining a configuration; f) The obtained reference working distance and Stigmator acquiring M defocused image pairs along with defocus parameters based on the setting; g) changing the Lth position of the sample by moving the sample in the x and y directions; h) repeating steps b) to g) while increasing L; Includes.

[0043] The training data generation method obtains images based on the movement of samples only in the x and y directions.

[0044] According to a further preferred embodiment of the present invention, the optimization method is classical optimization. Examples of optimization methods include the Simplex method or the Downhill Simplex method. The advantage of such a Downhill Simplex method is that it is very robust and simple. The disadvantage is that it has a slow convergence speed. However, the method of generating training data creates a minimal training data set for a new microscope setup so that the presented autofocus method is correctly adjusted for the new microscope setup so that it converges based on a set of training data.

[0045] Furthermore, the presented method for generating training data uses a score value that represents the sharpness of the image, which is directionless and therefore does not depend on the microscope setup parameters, except for the scaling factor, which is taken into account by adjusting the termination threshold according to a).

[0046] Further advantages, objects and features of the present invention will be explained by the accompanying drawings and the following description. [Brief description of the drawings]

[0047] [Figure 1] An example of a microscope setup is shown. [Diagram 2] Overview of the autofocus method. [Diagram 3] A method for generating training data is shown below. [Figure 4A] Convergence plot with dwell time 200ns. [Figure 4B] Images acquired according to the convergence plot in Figure 4A. [Figure 4C]Convergence plot with dwell time of 50ns. [Figure 4D] Images acquired according to the convergence plot in Figure 4C. [Diagram 5] Explain processing time. [Figure 6A] The sample contains areas where there is little information useful to the autofocus algorithm. [Figure 6B] Convergence plot in Figure 6A. [Figure 6C] Image patch crops and corresponding scores. [Figure 6D] Score map. [Figure 7A] 4 is a convergence plot of the method according to the invention on a particular microscope. [Figure 7B] Image autofocus. [Figure 7C] 4 is a convergence plot of the method according to the invention on another microscope. [Figure 7D] 4. Convergence plots of the method according to the invention on different microscopes after tuning. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0048] Typically, an image of a flat sample in a microscope setup 1, preferably a scanning electron microscope, is optimally imaged when the spot size of the electron beam is equal to or smaller than the sampling distance. The microscope setup 1 also comprises a sample holder 1' in which the sample 1'' can be placed.

[0049] In general, there are three parameters of the microscope setup: working distance, wd, and orthogonal Stigmator stigx, stigy (hereinafter, x-direction and y-direction Stigmator (also called a holographic microscope) allows the microscope operator to directly control the electron beam spot shape and adjust it to be less than or equal to the pixel size at the interaction point of the beam with the specimen, resulting in a sharp image.

[0050] Figure 1 briefly shows the three parameters wd, stigx, and stigy of microscope setup 1.

[0051] As shown simply in FIG. 2, the method according to the present invention is similar to the current microscope working distance. Stigmator Assume that two images 2 and 3 are input with known working distance perturbations 4 and 5 in the setup. The current microscope working distance and Stigmator The setting allows you to set the current microscope working distance and Stigmator Image 6 of the setting is obtained. Current microscope working distance and Stigmator The settings can be represented by a settings vector F.

number

[0052] In FIG. 2, a first image 2 has a positive working distance perturbation 4 and a second image 3 has a negative working distance perturbation 5, where the working distance perturbations 4, 5 are ±σ wd The perturbations 4 and 5 introduce an asymmetry in the sharpness of the images 2 and 3.

[0053] In the method according to the invention, firstly a first image 2 and a second image 3 are captured, which form the basis for the further method steps.

[0054] According to this method, n sub-regions 7, 9 of the first image 2 and n sub-regions 8, 10 of the second image 3 are selected, where n≧1, and the i-th sub-region 7, 9 of the first image 2 and the i-th sub-region 8, 10 of the second image 3 form the i-th input patch pair, where 1≦i≦n.

[0055] 2, by way of example, only two sub-regions are shown for each image 2, 3. However, preferably, many sub-regions are selected from the images 2, 3.

[0056] Preferably, some or all sub-areas of one image 2, 3 overlap.

[0057] In FIG. 2, preferably, a first partial region 7 of the first image 2 and a first partial region 8 of the second image 3 form a first input patch pair, and preferably, a second partial region 9 of the first image 2 and a second partial region 10 of the second image 3 form a second input patch pair.

[0058] The image patch pairs form the input data for the processing step, and based on each i-th image patch pair, we calculate the current working distance and Stigmator Correction term for setting ΔF i 12 is estimated.

[0059] The processing steps are performed by a processing unit 11, which preferably comprises at least one processor 13 and a memory 14, which preferably is coupled to the at least one processor 13, the memory being designed to store processor-executable commands, which are commands executable by the at least one processor 13. In particular, the at least one processor 13 can be a CPU or at least one of a CPU and a GPU.

[0060] Particularly preferably, the processing steps are performed by use of a machine learning modelling method, and even more preferably, the machine learning modelling method is a deep learning method, and the machine learning method is trainable by means of training data.

[0061] As an output of the processing step, a correction term ΔF for each image patch pair is i and all the correction terms ΔF i are combined by a function, and the output of the function is the output correction term ΔF. The output correction term is the working distance and Stigmator Including corrections for , ΔF = (Δwd, Δstigx, Δstigy).

[0062] Then, the output correction term is calculated based on the current working distance. Stigmator Apply to the settings and correct the current settings.

[0063] More preferably, the method steps are repeated at least once to improve the overall result of the method, and the method is stopped or terminated when a termination condition is met.

[0064] Preferably, the termination condition is the current working distance and Stigmator Settings and adjusted working distance and Stigmator The termination condition may be that the absolute difference between the settings is less than a first threshold value; or, preferably, that the sharpness measure indicating the sharpness of the image exceeds a second threshold value; or after a predetermined number of repetitions of the method steps.

[0065] Working distance and Stigmator Adjusting the settings results in a focused image 15.

[0066] To evaluate the model, the modeling method is trained by a learning method, which i) Working distance and Stigmator capturing a focused image with known settings; ii) storing a set of parameter settings and perturbed test images (e.g., two for each out-of-focus parameter setting if the working distance perturbation is asymmetric) for the k out-of-focus parameter settings; iii) repeating steps i) and ii) j times for different subregions of the sample, targeting known out-of-focus parameters to generate a total of j k image pairs that are used as training inputs for the model; iv) adjusting the correction term by the known working distance and Stigmator comparing the settings to determine a comparison error, and iteratively adjusting the settings of the machine learning modeling method in response to the comparison error; Includes.

[0067] The learning method is simply shown in Figure 3.

[0068] To evaluate different sub-regions of the sample, the sample is moved in the x and y directions to image different regions of the sample, preferably the sample regions are different from each other, and preferably the regions at most partially overlap.

[0069] Working distance and Stigmator The set (parameter i), or in-focus or out-of-focus image values, are uniformly and independently sampled delta Δ i ~U(a i ,b i ) to generate a set of distorted images with corresponding target values. More specifically, -Δ i are the target values ​​for each parameter, returning the in-focus image setting.

[0070] As an example, to train our modeling method, we trained it on a single GPU for approximately 44 hours on a set of 32 sample positions with different aberration parameters, for a total of n = 32 * 10 = 320 input image pairs (where 10 is the number of aberration parameters).

[0071] The model was tested on position-aberration image pairs that were not included in the training data. As can be seen in Figure 4A (convergence plot) and Figure 4B (initial image with current settings on the left, autofocused image on the right), the autofocus method rapidly converges towards the target value ΔF within three iterations, even for image pairs with low signal-to-noise (SNR) ratios, as seen in Figure 4C (convergence plot) and Figure 4D (initial image with current settings on the left, autofocused image on the right), and for small input patches, as shown by the small squares in Figure 4B.

[0072] In particular, Figure 4A shows a convergence plot where each parameter update was calculated as the average of N predicted values ​​for a patch shape of 2 × H × W (H is the height and W is the width in pixels taken from the two perturbed images) using a 5 × 2 × 512 × 512 input patch and a 200 ns pixel dwell time. The Y-axis shows the initial focus values ​​after each iteration with an initial aberration of 30 μm, +6, -6 (wd, stigx, stigy) (dashed and dotted horizontal lines are Stigmator The remaining difference is shown for the initial aberration and the working distance margins of 0.25 μm and 1 μm. The input image size is 1024 × 768 pixels. The numbers in the upper right corner of the sample images indicate the number of iterations. The SNR was calculated relative to the final focused image after 10 iterations. The images acquired with the initial aberration and the image after applying our method are shown in Figure 4B. The scale bar is 1 μm.

[0073] Figures 4C and 4D are the same as Figures 4A and 4B, but with a pixel dwell time of 50 ns.

[0074] We also investigated the effect of input alignment and found that the model performed well even in the extreme case where the patches in an input pair did not share the same offset, chosen randomly. To quantify the error for a wider range of initial defocus (working distance) values, we measured the remaining ΔF after the first iteration. As expected, StigmatorThe values ​​were stable, and the smaller the initial deviation, the closer the residual working distance error was to the target value. Apart from the ability to correct image aberrations with high accuracy, a good autofocus method should add as little computational overhead as possible to the test image acquisition time. Therefore, we compared the processing time of our method with respect to the microscope image acquisition time for CPU-only and GPU-based inference, running it directly on the microscope control computer. As examples of such control computers, we used an Intel Xeon CPU E5-2609 v2 and an NVIDIA T1000 with 2.50 GHz, 4 threads, and 16 GB RAM. The GPU-based inference was about an order of magnitude faster than CPU-only processing, especially for large input image patches. Importantly, the processing time of our method did not add any substantial overhead, even in the optimized CPU-only mode.

[0075] Figure 5 shows the ratio of the average autofocus processing time per input patch pair to the acquisition time of two input images (2 × 220 ms at 1024 × 768, dwell time 200 ns) on the microscope PC for different patch side lengths (10 iterations, output correction terms and standard deviations for 5 input patches). Error bars indicate the uncorrected standard deviation (s.d.).

[0076] However, during development of the present invention, it was discovered that many specimens have areas that have little usable information for the autofocus algorithm, e.g., areas that merely show blank epoxy resin and no usable contrast for the autofocus method, as shown in Figures 6A and 6B, thereby producing less than optimal results.

[0077] This is achieved by incorporating an ith weighting factor into the architecture, which provides independent weighting to each autofocus estimate.

[0078] These new outputs are the loosely regularized scores that are already used as weighting factors when training machine learning models.

[0079] The new output is ΔF i =(Δ i,wd , Δ i,stigx , Δ i,stigy , S i )

[0080] Weights were tested at two different granularities: first, at the level of input image patch pairs, which are cutouts of larger input image pairs acquired by a microscope; Figure 6C shows the weights and scores.

[0081] The second is at the level of individual pixels, leading to the scoring of all locations in the input image, as shown in Figure 6D. Figure 6D shows the score map for an example patch. The left column shows an example input patch, and the right column shows the scores of the corresponding pixels in iterations 0 and 2. The scale bar indicates 0.5 μm.

[0082] Both approaches are more robust to object regions with little contrast information, indicating that the method according to the invention does not require conventional image processing to pre-filter low contrast regions.

[0083] To test the extent to which our method suffers from overfitting on its significantly smaller training set, we evaluated it firstly on unseen samples and secondly on an entirely different microscope with different image settings.

[0084] Surprisingly, our method generalizes almost perfectly to new samples, even when trained only on image data of a single specimen, as shown in Figures 7A and 7B. However, when moving to a different microscope setup with significantly different image settings such as landing energy, beam current, working distance range, and rotational image acquisition, our method can fail and diverge, as shown in Figure 7C. However, fine-tuning our method on a different microscope setup leads to convergence, as shown in Figure 7D.

[0085] Here, we introduce a method for generating training data for a machine learning modeling model that is independent of microscope setup and microscope settings.

[0086] When the original model is applied to a new setup, its correction output is transformed into the coordinate frame of the new setup, preferably without additional re-learning.

[0087] We present a method to generate training data for machine learning modeling models used in microscope setups, in particular electron microscope setups. The method is independent of the microscope setup and microscope settings, and the microscope is configured with the microscope parameters, working distance, and X-axis. Stigmator , and in the Y direction Stigmator The method is at least adjustable. a) Reference working distance and Stigmator Take a single focused image of the sample with the settings assigning a first score value representative of the sharpness of the in-focus image to the single in-focus image; b) generating N out-of-focus images for the Lth position of the sample, where L=1, determining a score value corresponding to each out-of-focus image, and determining a directional correction term using an optimization method; c) Working distance and Stigmator By applying the directional correction term to the setting, the working distance at the Lth position of the sample and Stigmator Adjust the settings to suit your working distance and StigmatorObtaining a new image with the setting and a score value associated with the obtained image; d) Score value, working distance, Stigmator Using the setting, determining a directed correction term from an optimization technique; e) repeating steps c) and d) until the score value of the obtained image is substantially equal to or less than the first score value; Stigmator obtaining a configuration; f) The obtained reference working distance and Stigmator acquiring M defocused image pairs along with defocus parameters based on the setting; g) changing the Lth position of the sample by moving the sample in the x and y directions; h) repeating steps b) to g) while increasing L; Includes.

[0088] The training data generation method obtains images based on the movement of samples only in the x and y directions.

[0089] Using such a method to generate new training data for a different microscope setup, a new minimal training data set of n=10 positions was created, corresponding to 31% of the original training set for the microscope setup for which the method according to the invention diverged. Fine-tuning (recalibration) of the model was performed in a reasonably short time interval. For example, in one example, fine-tuning was performed in less than 2 hours on a single GPU, returning the ability to estimate ΔF at the original convergence rate, as shown in Figure 7D.

[0090] All features disclosed in the application are claimed to be essential to the invention if, individually or in any combination, they are novel over the prior art. [Explanation of symbols]

[0091] 1. Microscope Setup 1' Sample 1'' Sample Holder 2. First image 3. Second image 4 First Perturbation 5. Second Perturbation 6 Current working distance and Stigmator Image by Settings 7 First partial region of first image 8 First partial region of the second image 9 Second partial region of the first image 10. Second partial region of second image 11 Processing Unit 12 Output correction term 13 Functions of all correction terms 14. Memory 15. In-focus image

Claims

1. A method for autofocus and astigmatism correction for a particular microscope setup, the microscope being capable of adjusting at least the microscope parameters working distance, X-stigmator and Y-stigmator, The method comprises: a) capturing a first image of a sample with a first working distance perturbation and a second image of the sample with a second working distance perturbation at a current working distance and a current stigmator setting; b) selecting n sub-regions of the first image and n sub-regions of the second image, where n≧1, and the i-th sub-region of the first image and the i-th sub-region of the second image form an i-th input patch pair, where 1≦i≦n; c) processing each i-th input patch pair and receiving an i-th correction term having a correction for the current working distance, x-direction stigmator, and y-direction stigmator, where c) is performed using a machine learning modeling method in a machine learning modeling setting, the machine learning modeling method being optimized by a training method that creates a training data set; d) calculating an output correction term based on all of said correction terms; e) adjusting the current working distance and stigmator settings by applying said output correction term to the current working distance and stigmator settings; A method comprising:

2. The method of claim 1 , wherein the correction term includes a correction direction.

3. The method of claim 1 , wherein a size of the i-th subregion of the first image and a size of the i-th subregion of the second image are substantially identical.

4. The method of claim 1 , wherein a position of the i-th sub-region of the first image relative to the first image is either substantially the same as or different from the i-th sub-region of the second image relative to the second image.

5. The method of claim 1 , wherein the output correction term is a return value of a function, the function combining all of the correction terms into the output correction term.

6. The steps a) to e) are repeated at least once; The method is stopped when a termination condition is met; The termination condition may be that an absolute difference between the current working distance and stigmator setting and the adjusted working distance and stigmator setting is less than a first threshold; or The method according to claim 1 , wherein the termination condition is that a sharpness index indicating the sharpness of the image exceeds a second threshold value, or after steps a) to e) have been repeated a predetermined number of times.

7. The method of claim 1 , wherein the i-th sub-region of the first image and the (i+1)-th sub-region of the first image overlap.

8. 2. The method of claim 1, wherein the machine learning modeling method weights each i-th input patch pair with an i-th weighting factor, the i-th weighting factor being dependent on the importance of the i-th input patch pair to the output correction term as predicted by a machine learning model.

9. The learning method includes: i) capturing a focused image with known working distance and stigmator settings; ii) generating k defocused image patch pairs, where each image has a known different parameter setting; iii) repeating steps i) and ii) j times for different sub-regions of the sample; iv) comparing said correction terms to known working distances and stigmator settings to determine a comparison error, and adjusting settings of said machine learning modeling method in response to said comparison error; The method of claim 1 , comprising:

10. The first working distance perturbation and the second working distance perturbation are both positive working distance perturbations, or both negative working distance perturbations, or the first working distance perturbation is a positive working distance perturbation and the second working distance perturbation is a negative working distance perturbation; The method of claim 1 , wherein absolute values ​​of the first working distance perturbation and the second working distance perturbation are i) identical or ii) different from each other.

11. A method for generating training data for the machine learning model of claim 1, the method being independent of microscope setup or microscope settings, The method comprises: a) taking a single in-focus image of the sample at a reference working distance and stigmator setting; assigning a first score value to the single in-focus image representative of sharpness of the in-focus image; b) generating N out-of-focus images for the Lth position of the sample, where L=1, determining a score value corresponding to each out-of-focus image, and determining a directional correction term using an optimization method; c) adjusting the working distance and stigmator setting at the Lth position of the sample by applying the directional correction term to the working distance and stigmator setting to obtain a new image having the working distance and stigmator setting and a score value associated with the obtained image; d) determining a directional correction term from an optimization technique using the score value, working distance, and stigmator setting of c); e) repeating c) and d) to obtain a reference working distance and stigmator setting until the score value of the obtained image is substantially equal to or less than the first score value of a); f) acquiring M defocused image pairs along with defocus parameters based on the obtained reference working distance and the stigmator setting focused on the Lth position; g) changing the Lth position of the sample by moving the sample in the x and y directions; h) repeating steps b) to g) while increasing L; and A method comprising:

12. The method of claim 11 , wherein the training data set is formed using L*M images.

13. 1. A computing system comprising: a memory coupled to at least one processor and storing processor-executable instructions, A computing system, wherein the instructions, when executed by the at least one processor, cause the computing system to perform a method for autofocus and astigmatism correction as described in any one of claims 1 to 10 and / or a method for generating training data as described in claim 11.

Images