Autofocus and astigmatism correction method for microscope

Abstract

A method for autofocus and astigmatism correction for a particular microscope setup, in particular an electron microscope, the microscope being capable of adjusting at least the microscope parameters working distance, the stigmata in the X-direction and the stigmata in the Y-direction, the method comprising: a) capturing a first image of a sample at a first working distance perturbation and capturing a second image of the sample at a second working distance perturbation at a current working distance and a current stigmator setting; b) selecting n subregions of the first image and n subregions of the second image, where n≧1, and the i-th subregion of the first image and the i-th subregion of the second image form an i-th input patch pair, where 1≦i≦n; c) processing each i-th input patch pair and receiving an i-th correction term having corrections for the current working distance, the x-direction stigmata, and the y-direction stigmata; d) receiving an output correction term in response to all of the correction terms; e) adjusting the current working distance and stigmator settings by applying the output correction term to the current working distance and stigmator settings; Includes.

Description

Technical Field

[0001] The present invention relates to an autofocus and aberration correction method for a microscope.

Background Art

[0002] In the case of microscopes, particularly electron microscopes, parameters of the microscope, particularly parameters related to image quality, are often manually controlled by the user of the microscope. However, adjusting the parameters for each probe is a very time-consuming task.

[0003] As an improvement measure, electron microscopes may rely on automation and control algorithms to guarantee higher image quality with minimal human intervention.

[0004] However, currently used focus aberration correction and astigmatism correction algorithms usually attempt to explicitly model image formation and subsequent aberration correction, and often have a long processing time and problems with generalization for unvalidated settings.

[0005] The high resolution of an electron microscope (EM) and the ability to image down to the details of a sample are unrivaled. An important component of an automated EM is to maintain high-quality images throughout the entire imaging process, which may include millions of individual images and long-term operation. For this reason, it is almost impossible to manually adjust microscope parameters. Severe constraints on the execution time budget of the algorithm, the convergence speed of aberration correction, and the low electron dose budget to avoid sample artifacts pose significant challenges to autofocus and astigmatism correction algorithms, particularly in high-throughput electron microscopes, despite the necessity.

[0006] Existing solutions in the field of scanning electron microscopes (SEM) typically rely on an explicit physical model of the interaction between the electron beam and the sample. Measurements (in this case, images) are made with known perturbations, followed by focus and StigmatorParameters are inferred and wavefront aberrations are estimated. Such an approach based on physical grounds should theoretically lead to excellent generalization characteristics, i.e., the ability to work well with little or minimal adaptation for different microscopes and samples, but in practice it involves many parameters that require careful tuning.

[0007] Here, a mainly data-driven method for aberration correction in a scanning electron microscope is introduced. This method functions under very low signal-to-noise ratio conditions, shortens the processing time by more than one order of magnitude compared to the state of the art, converges rapidly over a wide range of aberrations, and can be automatically recalibrated by non-experts for different microscopes and difficult samples.

[0008] In particular, it is a machine learning-based focusing and Stigmator correction method for a scanning electron microscope. The proposed algorithm enables inference almost instantaneously, converges quickly, operates on low electron-dose and noisy images, requires no expertise, has an immediately usable procedure for recalibration for new machines and samples, and guarantees convergence in all usage scenarios. SUMMARY OF THE INVENTION

[0009] An object of the present invention is to provide a method for autofocusing and aberration correction of an electron microscope, which is fast and robust during use.

[0010] This problem is solved by the method according to claim 1. Further dependent claims provide preferred embodiments. Also, according to claim 13, a method for generating training data for a machine learning modeling model used in a microscope setup is provided. Also, the problem is solved by the computing system according to claim 15.

[0011] According to the present invention, a method for autofocus and aberration correction for a specific microscope setup, particularly for an electron microscope, is provided. The microscope is adjustable at least with respect to the following microscope parameters: working distance, in the X direction Stigmator , and in the Y direction Stigmator , and the method comprises a) With the current operating distance and the current stigmator setting imaging a first image of the sample with a first working distance perturbation Having , and imaging a second image of the sample with a second working distance perturbation Having ; b) selecting n sub-regions of the first image and n sub-regions of the second image, where n ≧ 1, and the i-th sub-region of the first image and the i-th sub-region of the second image form an i-th input patch pair, with 1 ≦ i ≦ n; c) processing each i-th input patch pair and receiving an i-th correction term having a correction for the current working distance, in the x direction Stigmator , and in the y direction Stigmator ; d) receiving an output correction term according to all of the correction terms; e) adjusting the current working distance and Stigmator settings by applying the output correction term to the current working distance and Stigmator settings, and includes.

[0012] Generally, an image of a flat specimen or sample in a microscope, particularly a scanning electron microscope, is optimally imaged when the size of the spot of the electron beam is below the sampling distance. Generally, by adjusting three parameters: working distance, and the orthogonal Stigmator (referred to herein as stig x and stig y), the electron microscope operator can directly control the spot shape and make it smaller than the pixel size at the interaction point of the beam and the sample, and as a result, a sharp image can be formed.

[0013] Advantageously, the microscope or microscope setup may be a scanning electron microscope (SEM) or an environmental scanning electron microscope (ESEM). It may also be a scanning transmission electron microscope (STEM) or a transmission electron microscope (TEM), but due to different image formation principles, the working distance and Stigmator Instead of the set parameters, it is necessary to control different microscope parameters additionally or alternatively.

[0014] The proposed method has, as input, two images with the current working distance of the microscope and Stigmator known working distance perturbations in the setting. The current working distance and Stigmator settings are related to a specific image that may be taken by the microscope. Note that the current working distance and Stigmator settings typically provide an image of an out-of-focus sample.

[0015] Furthermore, this method requires at least two perturbation test images to be able to search for the magnitude and direction of the correction term. According to a preferred embodiment, the correction term includes a correction direction.

[0016] According to a preferred embodiment of the present invention, the perturbations may be of different types with respect to each other. In particular, the first perturbation and the second perturbation are both positive perturbations, or both are negative perturbations, or the first perturbation is a positive perturbation and the first perturbation is a negative perturbation.

[0017] More preferably, the absolute values of the first perturbation and the second perturbation are either the same as each other or different from each other. In particular, when one perturbation is positive and the other perturbation is negative, the perturbations preferably have the same absolute value. However, there may be cases where one perturbation is positive and the other perturbation is negative, and the perturbations have different absolute values. When one perturbation is positive and the other perturbation is negative, the important fact is that these perturbations bring about asymmetry with respect to the sharpness of the partial regions of the image pair, meaning that one partial region of the image patch pair is sharper than the other partial region, thereby bringing about the correction direction of the correction term. In the case of an initially focused image with an asymmetric perturbation in the working distance parameter, the test image inherits the same blurring distortion (due to the symmetry of pure defocus). In the case of an initially unfocused setting, the asymmetry of the distortion of the test image can be utilized to infer a correction vector towards the focused parameter setting (working distance and Stigmator settings).

[0018] The asymmetry can also be utilized when the perturbations have the same sign, that is, when both perturbations are positive or negative. However, if both perturbations have the same sign, the absolute values of the perturbations must be different from each other. If the perturbations have the same absolute value, the same perturbed image is captured twice, so that no image patch pair is formed and no directionality can be considered.

[0019] However, it should be noted that there are several requirements for the perturbations from the perspective of the present invention. First, note that the present invention is not limited to two perturbations. In fact, a plurality of perturbations may be used, and the total number thereof may exceed two. Preferably, the total number of perturbations is constant, and more preferably, it is preset for the method of the present invention. Furthermore, it is possible for the magnitude and direction to also be constant. The direction may be given by the sign of the perturbation.

[0020] Furthermore, the absolute value of the perturbation must be equivalent to the target parameter range, which means that the order of magnitude of the perturbation is the working distance and StigmatorIt means that it is of the same order as or less than the magnitude of the correction of the setting. Further, the perturbation must be sufficiently different with respect to the target parameter range.

[0021] According to a preferred embodiment of the present invention, the sizes of the i-th partial regions of the first image and the second image are substantially the same. In the sense of the present invention, "substantially" relates to a difference of at most 10%, preferably 5% in the sizes of the respective partial regions. Most preferably, the sizes of the i-th partial regions are the same. As an example, the size of the test image is 1024x768, the pixel size is 10 nm, and the size of the partial region is a square of 512x512, 256x256, or 128x128, and larger sizes are also possible. However, it has been found that reducing the size of the partial region may reduce the convergence speed. It should be understood that the size of the image, the pixel size, and the size of the partial region may have values different from the presented values. However, it should be understood that the size of the partial region is smaller than the size of the entire image.

[0022] According to a more preferred embodiment, the position of the i-th partial region of the first image with respect to the first image is substantially the same as or different from the i-th partial region of the second image with respect to the second image.

[0023] According to a further preferred embodiment of the present invention, the i-th partial region of the first image and the (i + 1)-th partial region of the first image may partially overlap. However, it is also possible that the partial regions do not overlap or only a part of the partial regions overlaps. This is because the partial regions may be randomly and independently selected.

[0024] According to a further preferred embodiment of the present invention, the output correction term is the return value of a function. More preferably, the function combines all the i-th correction terms into one output correction term.

[0025] As an example, in response to such a function, the average value of all correction terms, or the median value of all correction terms, can be considered. As yet another method, the function may be a computer function that is very robust to outliers, such as, for example, the RANSAC algorithm.

[0026] According to a further preferred embodiment of the present invention, method steps a) to e) are repeated at least once. Advantageously, the method is stopped when an end condition is met.

[0027] Preferably, the end condition can be that the absolute difference between the current operating distance and the Stigmator setting and the adjusted operating distance and the Stigmator setting is less than a first threshold value. Also preferably, the end condition is that the sharpness degree indicating the sharpness of the image exceeds a second threshold value. Or preferably, it is after a predetermined number of repetitions of method steps a) to e). The end condition may be a combination of possibilities presented to further specify the end condition.

[0028] During the development of the method of the present invention, typically 3 to 4 iterative steps, that is, it was shown that the method converges after iteration of the method.

[0029] According to a further preferred embodiment of the present invention, method step c) is performed using a machine learning modeling method having a machine learning modeling method setting. Advantageously, the machine learning modeling method is a deep learning method. Preferably, at least one machine learning modeling method setting is the weight of the machine learning modeling method.

[0030] The machine learning modeling method receives a plurality of input patch pairs as inputs and, using adjustable weights, converts the inputs into one correction term for the operating distance and the Stigmator setting. The machine learning modeling method preferably consists of a sequence of convolutional layers and fully connected layers for non-linear feature extraction of the inputs and conversion to the target area.

[0031] According to a more preferred embodiment of the present invention, in the machine learning modeling method, the i-th output term is weighted for each i-th input patch pair by the i-th weighting coefficient, and the i-th weighting coefficient depends on the importance predicted by the machine learning modeling model of the i-th input patch pair with respect to the final output correction term.

[0032] During the development of the method, it was noticed that samples and specimens may include sites and regions with little information available for use in autofocus algorithms, such as blood vessels in tissues. For example, like blood vessels in a tissue, it only shows the blank of the epoxy resin of the sample holder and does not show the contrast available for the AF algorithm, and as a result, optimal results may not be obtained. Therefore, it was inferred that these regions should have less weight in the determination of autofocus. This is achieved by incorporating the i-th weighting coefficient into the architecture, leading to a new output set of methods that independently weight each autofocus estimate. The outputs of these new methods are loosely regularized scores, preferably regularized only in terms of weight decay, and are already used as weighting coefficients during the machine learning modeling methodology learning.

[0033] Two different granularities of weighting were tested. First, it is the level of input image patches, which is the cropping of patch pairs of large input images acquired by a microscope. Second, it is the level of individual pixels leading to the scoring of all positions within the input image. Both approaches are more robust to subject regions with less contrast information, indicating that this method does not require conventional image processing to pre-filter low-contrast regions.

[0034] As an example, the output of the processing step is a correction term extended by a weighting coefficient:

Number

[0035] This can be processed as an output correction term: [Number]

[0036] According to a preferred embodiment, the learning method is: i) imaging a focused image in a state where the operating distance and Stigmator settings are known; and ii) storing a set of parameter settings and perturbed test images (e.g., two for each unfocused parameter setting when the operating distance perturbation is not the target) for k unfocused parameter settings; and iii) repeating steps i) and ii) j times for different sub-regions of the sample, and generating a total of j×k image pairs used as the learning input of the model with the unfocused parameters known as the target; and iv) comparing and processing the correction term with the known operating distance and Stigmator settings, obtaining a comparison error, and iteratively adjusting the settings of the machine learning modeling method according to the comparison error. It includes.

[0037] In particular, the weights of the machine learning modeling method are adjusted using, in particular, the backpropagation method.

[0038] Also, it may be possible to use techniques / methods other than the backpropagation method, or different methods can be combined.

[0039] Note that in the above learning method, a model specialized for the microscope used to generate the learning data is obtained.

[0040] However, when migrating to another microscope setup with significantly different image settings, such as landing energy, beam current, operating distance range, rotational image acquisition, etc., the method may fail and may lead to divergence instead of convergence.

[0041] Therefore, a method for creating learning data for a machine learning modeling model that is independent of microscope setup and microscope settings will be introduced.

[0042] A method for generating learning data for a machine learning modeling model used in microscope setup, particularly electron microscope setup, wherein the method is independent of microscope setup and microscope settings, and the microscope can adjust at least the following microscope parameters: working distance, in the X direction Stigmator , and in the Y direction Stigmator , and the method includes: a) Imaging a single focused image of the sample at a reference working distance and Stigmator settings, and assigning a first score value representing the sharpness of the focused image to the single focused image; b) With L = 1, generating N unfocused images of the sample at the L-th position of the sample, obtaining a score value corresponding to each unfocused image, and obtaining a directed correction term using an optimization method; c) By applying the directed correction term to the working distance and settings, adjusting the working distance and Stigmator settings at the L-th position of the sample, and obtaining a new image with the adjusted working distance and Stigmator settings and a score value associated with the obtained image; Stigmator d) Using the score value, working distance, settings to obtain a directed correction term from an optimization method; Stigmator e) Repeating steps c) and d) until the score value of the obtained image is substantially less than or equal to the first score value in step a) to obtain a reference working distance and settings; Stigmator f) Based on the obtained reference working distance and settings, obtaining M pairs of unfocused images together with unfocused parameters; Stigmator g) Changing the L-th position of the sample by moving the sample in the x direction and y direction. h) Repeating steps b) to g) for different positions of the sample until all positions of the sample have been processed. h) Repeat steps b) to g) while increasing L, and including.

[0043] According to the method for generating learning data, an image is obtained based on the movement of the sample only in the x-direction and y-direction.

[0044] According to a more preferred embodiment of the present invention, the optimization method is classical optimization. Examples of optimization methods include the simplex method or the downhill simplex method. The advantage of such a downhill simplex method is that it is very robust and simple. The disadvantage is that the convergence speed is slow. However, the method for generating learning data creates a minimal learning data set for a new microscope setup so that the presented autofocus method converges based on the set of learning data and is correctly adjusted for the new microscope setup.

[0045] Furthermore, the presented method for generating learning data uses a score value representing the sharpness of the image, and since the score value has no directionality, it does not depend on the microscope setup parameters except for the scaling factor. The scale is considered by adjusting the end threshold according to a).

[0046] Further advantages, objects and characteristics of the present invention are explained by the accompanying drawings and the following description.

Brief Description of the Drawings

[0047]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 4C

Figure 4D

Figure 5

Figure 6A

Figure 6B

Figure 6C

Figure 6D

Figure 7A

Figure 7B

Figure 7C

Figure 7D

DETAILED DESCRIPTION OF THE INVENTION

[0048] Generally, an image of a flat sample in a microscope setup 1, preferably a scanning electron microscope, is optimally imaged when the size of the spot of the electron beam is less than or equal to the sampling distance. Also, the microscope setup 1 includes a sample holder 1' on which the sample 1'' can be placed.

[0049] Generally, three parameters of the microscope setup 1, namely the working distance wd and the orthogonal Stigmator stigx, stigy (hereinafter also referred to as the Stigmator x-direction and y-direction) can be adjusted by the microscope operator to directly control the spot shape of the electron beam and make it less than or equal to the pixel size of the interaction point between the beam and the specimen, resulting in sharp image formation.

[0050] Figure 1 briefly shows the three parameters wd, stigx, and stigy of the microscope setup 1.

[0051] As briefly shown in Figure 2, the method according to the present invention takes as input two images 2, 3 with the current microscope working distance and Stigmator the known working distance perturbations 4, 5 in the setting. The current microscope working distance and Stigmator by the setting, the current microscope working distance and Stigmator the image 6 of the setting are obtained. The current microscope working distance and Stigmator the setting can be represented by the setting vector F.

Number

[0052] In Figure 2, the first image 2 has a positive working distance perturbation 4, the second image 3 has a negative working distance perturbation 5, and the working distance perturbations 4, 5 are represented by ±σ wd The perturbations 4, 5 bring about asymmetry with respect to the sharpness of the images 2, 3.

[0053] In the method according to the present invention, first, the first image 2 and the second image 3 are captured, which serve as the basis for subsequent method steps.

[0054] According to this method, n partial regions 7, 9 of the first image 2 and n partial regions 8, 10 of the second image 3 are selected, where n ≥ 1, and the i-th partial region 7, 9 of the first image 2 and the i-th partial region 8, 10 of the second image 3 form the i-th input patch pair, where 1 ≤ i ≤ n.

[0055] In Figure 2, only two partial regions are shown for each of the images 2, 3 as an example. However, preferably, many partial regions are selected from the images 2, 3.

[0056] Preferably, some or all of the partial regions of one image 2, 3 overlap.

[0057] In FIG. 2, preferably, the first partial region 7 of the first image 2 and the first partial region 8 of the second image 3 form a first input patch pair, and preferably, the second partial region 9 of the first image 2 and the second partial region 10 of the second image 3 form a second input patch pair.

[0058] The image patch pairs form the input data of the processing step, and based on each i-th image patch pair, the current operating distance and Stigmator a correction term ΔF for the setting i 12 are estimated.

[0059] The processing step is executed by the processing unit 11. Preferably, the processing unit 11 includes at least one processor 13 and a memory 14. Preferably, the memory 14 is coupled to at least one processor 13, and the memory is designed to store processor-executable commands that are commands executable by at least one processor 13. In particular, at least one processor 13 can be a CPU, or at least one of a CPU and a GPU.

[0060] Particularly preferably, the processing step is implemented by using a machine learning modeling method. More preferably, the machine learning modeling method is a deep learning method, and the machine learning method is learnable by learning data.

[0061] As the output of the processing step, a correction term ΔF for each image patch pair i is obtained, all the correction terms ΔF i are combined by a function, and the output of the function becomes the output correction term ΔF. The output correction term includes the correction for the operating distance and Stigmator the setting, and ΔF = (Δwd, Δstigx, Δstigy).

[0062] Then, the output correction term is applied to the current operating distance and Stigmator the setting to correct the current setting.

[0063] More preferably, in order to improve the overall result of the method, the method steps are repeated at least once, and the method stops or ends when the end condition is met.

[0064] Preferably, the end condition is that the absolute difference between the current operating distance and Stigmator the setting and the adjusted operating distance and Stigmator the setting is less than a first threshold value; or, preferably, the end condition is that the sharpness degree indicating the sharpness of the image exceeds a second threshold value; or, it may be after the method steps are repeated a predetermined number of times.

[0065] As a result of adjusting the operating distance and Stigmator the setting, a focused image 15 is obtained.

[0066] To evaluate the model, the modeling method is trained by a learning method, and the learning method i) taking a focused image with the operating distance and Stigmator the setting in a known state; ii) storing a set of parameter settings and perturbed test images (for example, two for each unfocused parameter setting when the operating distance perturbation is not the target) for k unfocused parameter settings; iii) repeating steps i) and ii) j times for different partial regions of the sample, and generating a total of j×k image pairs used as the learning input of the model with the known unfocused parameters as the target; iv) comparing the correction term with the known operating distance and Stigmator the setting, obtaining a comparison error, and repeatedly adjusting the setting of the machine learning modeling method according to the comparison error. including.

[0067] The learning method is briefly shown in FIG. 3.

[0068] To evaluate different partial regions of the sample, the sample is moved in the x - direction and the y - direction to image different regions of the sample. Preferably, the sample regions are different from each other. Also preferably, the regions overlap at most partially.

[0069] The working distance and Stigmator the setting (parameter i), or the value of the in - focus image or the value of the out - of - focus image, is uniformly and independently sampled as a delta Δ i ~U(a i ,b i ) and is changed by addition to generate a set of distorted images together with the corresponding target values. More specifically, - Δ i is the target value of each parameter and returns the setting of the in - focus image.

[0070] As an example, to train the modeling method, for a set of 32 sample positions with different aberration parameters, a total of n = 32 * 10 = 320 input image pairs (10 is the number of aberration parameters), it was trained on one GPU for about 44 hours.

[0071] This model was tested with position - aberration image pairs at positions not included in the training data. As can be seen in Figure 4A (convergence plot) and Figure 4B (left is the initial image with the current setting, right is the autofocused image), even for image pairs with a low signal - to - noise (SNR) ratio as seen in Figure 4C (convergence plot) and Figure 4D (left is the initial image with the current setting, right is the autofocus image), and even for small input patches as shown by the small squares in Figure 4B, the autofocus method converges rapidly towards the target value ΔF in 3 iterations.

[0072] In particular, Fig. 4A shows a convergence plot where each parameter update was calculated as the average of N predicted values of a patch shape of 2×H×W (H is the height and W is the width of the pixels taken from two perturbed images) using a 5×2×512×512 input patch and a 200 ns pixel dwell time. The Y-axis shows the remaining difference with respect to the initial focus value after each iteration of the initial aberration of 30 μm, +6, -6 (wd, stigx, stigy) (the dashed and dotted horizontal lines indicate Stigmator the margin of the working distance of 0.25 μm and 1 μm). The input image size is 1024×768 pixels. The number in the upper right corner of the sample image indicates the number of iterations. The SNR was calculated as a relative value with respect to the final focus image after 10 iterations. Fig. 4B shows the image obtained with the initial aberration and the image after applying this method. The scale bar is 1 μm.

[0073] Figs. 4C and 4D are the same as Figs. 4A and 4B, but with a pixel dwell time of 50 ns.

[0074] Also, the influence of the input alignment was investigated, and it was found that this model works well even in the extreme case where the input pair of patches do not share the same offset and are randomly selected. To quantify the error for a wider range of initial defocus (working distance) values, the remaining ΔF was measured after the first iteration. As expected, StigmatorThe value was stable, and the smaller the initial deviation, the closer the residual actuation distance error approached the target value. Apart from the ability to correct image aberration with high precision, a good autofocus method needs to minimize the computational overhead with respect to the acquisition time of test images. Therefore, the processing time of this method with respect to the microscope image acquisition time was compared for CPU-only and GPU-based inference, and it was executed directly on the microscope control computer. As an example of such a control computer, an Intel Xeon CPU E5-2609 v2 with 2.50 GHz, 4 threads, and 16 GB of RAM and an NVIDIA T1000 were used. GPU-based inference was approximately one order of magnitude faster than CPU-only processing, especially for large input image patches. Importantly, the processing time of the method according to the present invention did not add a substantial overhead even in the optimized CPU-only mode.

[0075] Figure 5 shows the ratio of the average autofocus processing time per input patch pair with respect to the imaging time of two input images (2×220 ms at 1024×768, dwell time 200 ns) when changing the patch side length (10 repetitions, output correction term and standard deviation of 5 input patches) on the microscope PC. Error bars indicate the uncorrected standard deviation (s.d.).

[0076] However, during the development of the present invention, as shown in FIGS. 6A and 6B, many specimens have regions with little information available for the autofocus algorithm, for example, regions that only show blank epoxy resin and do not show the contrast available for the autofocus method, and thus have little information available for the autofocus algorithm, resulting in suboptimal results.

[0077] This is achieved by incorporating the i-th weighting coefficient into the architecture, and independent weighting is performed on each autofocus estimate.

[0078] These new outputs are loosely regularized scores that were already used as weighting coefficients during the training of the machine learning model.

[0079] The new output is ΔF i =(Δ i,wd 、Δ i,stigx 、Δ i,stigy 、S i ).

[0080] The weighting was tested at two different granularities. The first is at the level of input image patch pairs, which are cutouts of large input image pairs obtained by the microscope. Figure 6C shows the weighting and scores.

[0081] The second is at the level of individual pixels, which leads to scoring of all positions within the input image as shown in Figure 6D. Figure 6D is a score map for an example of a patch. The left column shows examples of input patches, and the right column shows the scores of the corresponding pixels at iterations 0 and 2. The scale bar indicates 0.5 μm.

[0082] Both approaches are more robust to subject regions with little contrast information, indicating that the method according to the present invention does not require conventional image processing to pre-filter low-contrast regions.

[0083] To test to what extent the method according to the present invention is troubled by overfitting to its extremely small training set, it was evaluated first with unseen samples and second with a completely different microscope and different image settings.

[0084] Surprisingly, the method according to the present invention generalizes almost perfectly to new samples even when trained with only the image data of a single specimen, as shown in FIGS. 7A and 7B. However, when migrating to different microscope setups with significantly different image settings such as landing energy, beam current, operating distance range, rotational image acquisition, etc., as shown in FIG. 7C, the method according to the present invention may fail and diverge. However, if the method according to the present invention is fine-tuned for different microscope setups, it converges as shown in FIG. 7D.

[0085] Therefore, a method for generating training data for a machine learning modeling model that is independent of the microscope setup and microscope settings is introduced.

[0086] When the original model is applied to a new setup, its corrected output is preferably converted to the coordinate frame of the new setup without additional retraining.

[0087] A method for generating training data for a machine learning modeling model used in a microscope setup, particularly an electron microscope setup, is proposed. The method is independent of the microscope setup and microscope settings, and the microscope is at least adjustable for microscope parameters, operating distance, in the X direction Stigmator , and in the Y direction Stigmator . The method is a) Imaging a single focused image of the sample at a reference operating distance and Stigmator settings, and assigning a first score value representing the sharpness of the focused image to the single focused image; and b) With L = 1, for the L-th position of the sample, generating N out-of-focus images of the sample, obtaining the score value corresponding to each out-of-focus image, and obtaining a directed correction term using an optimization method; and c) By applying the directed correction term to the operating distance and settings, adjusting the operating distance and Stigmator settings at the L-th position of the sample, and the operating distance and Stigmator settings, and the operating distance and StigmatorObtaining a new image having settings and a score value associated with the obtained image; d) The score value, the operating distance, Stigmator Using the settings, obtaining a directed correction term from an optimization method; e) Repeating steps c) and d) until the score value of the obtained image is substantially less than or equal to the first score value, and obtaining a reference operating distance and Stigmator settings; f) Based on the obtained reference operating distance and Stigmator settings, obtaining M pairs of out-of-focus images together with out-of-focus parameters; g) Changing the L-th position of the sample by moving the sample in the x-direction and the y-direction; h) Repeating steps b) to g) while increasing L, and including.

[0088] According to the method for generating learning data, an image is obtained based only on the movement of the sample in the x-direction and the y-direction.

[0089] When such a method is used to generate new learning data for different microscope setups, n = 10 new minimum learning data sets corresponding to 31% of the original learning set for the divergent microscope setup according to the method of the present invention were created. Fine-tuning (recalibration) of the model was performed at reasonable short time intervals. For example, in one example, the fine-tuning was performed in less than 2 hours on a single GPU and, as shown in FIG. 7D, the ability to estimate ΔF at the original convergence rate was restored.

[0090] All features disclosed in the application documents are claimed to be the essence of the invention if they are novel over the prior art, either individually or in combination.

Explanation of Reference Numerals

[0091] 1 Microscope setup 1’ Sample 1’’ Sample holder 2 First image 3 Second Image 4 First Perturbation 5 Second Perturbation 6 Current Operating Distance and Stigmator Image According to Settings 7 First Sub-region of the First Image 8 First Sub-region of the Second Image 9 Second Sub-region of the First Image 10 Second Sub-region of the Second Image 11 Processing Unit 12 Output Correction Term 13 Function of All Correction Terms 14 Memory 15 Focused Image

Claims

1. A method for autofocusing and aberration correction for a specific microscope setup, wherein the microscope is at least adjustable for microscope parameters, working distance, stigmator in the X direction, and stigmator in the Y direction, The method comprises: a) imaging a first image of a sample with a first working distance perturbation and a second image of the sample with a second working distance perturbation at the current working distance and current stigmator settings; b) selecting n partial regions of the first image and n partial regions of the second image, where n ≥ 1, and the i-th partial region of the first image and the i-th partial region of the second image form an i-th input patch pair, where 1 ≤ i ≤ n; c) processing each i-th input patch pair to receive an i-th correction term having a correction for the current working distance, stigmator in the x direction, and stigmator in the y direction, where the step c) is performed using a machine learning modeling method in the setting of the machine learning modeling method, and the machine learning modeling method is optimized by a learning method for creating a learning dataset; d) calculating an output correction term based on all the correction terms; e) adjusting the current working distance and stigmator settings by applying the output correction term to the current working distance and stigmator settings, A method comprising the above steps.

2. The method according to claim 1, wherein the correction term includes a correction direction.

3. The method according to claim 1, wherein the size of the i-th partial region of the first image and the size of the i-th partial region of the second image are substantially the same.

4. The method according to claim 1, wherein the position of the i-th partial region of the first image with respect to the first image is either substantially the same as or different from the i-th partial region of the second image with respect to the second image.

5. The method according to claim 1, wherein the output correction term is a return value of a function, and the function combines all the correction terms into the output correction term.

6. The steps a) to e) are repeated at least once, The method is stopped when an end condition is satisfied, The end condition may be that the absolute difference between the current working distance and stigmator settings and the adjusted working distance and stigmator settings is smaller than a first threshold; Or, The end condition is that the sharpness degree indicating the sharpness of the image exceeds a second threshold value, or after the steps a) to e) are repeated a predetermined number of times. The method according to claim 1.

7. The method according to claim 1, wherein the i-th partial region of the first image and the (i + 1)-th partial region of the first image partially overlap.

8. The machine learning modeling method weights each i-th input patch pair with an i-th weighting coefficient, and the i-th weighting coefficient depends on the importance of the i-th input patch pair with respect to the output correction term predicted by the machine learning modeling model. The method according to claim 1.

9. The learning method is i) imaging a focused image in a state where the operating distance and the stigmator setting are known; ii) generating k unfocused image patch pairs, where each image has different known parameter settings; iii) repeating the steps i) and ii) j times for different partial regions of the sample; iv) comparing and processing the correction term with the known operating distance and stigmator setting, obtaining a comparison error, and adjusting the setting of the machine learning modeling method according to the comparison error. The method according to claim 1, comprising

10. The first operating distance perturbation and the second operating distance perturbation are both positive operating distance perturbations, or both are negative operating distance perturbations, or the first operating distance perturbation is a positive operating distance perturbation and the second operating distance perturbation is a negative operating distance perturbation, The absolute values of the first operating distance perturbation and the second operating distance perturbation are i) the same, or ii) different from each other. The method according to claim 1.

11. A method for generating learning data of the machine learning modeling model according to claim 1, wherein the method does not depend on the setup of the microscope or the settings of the microscope. The method is a) imaging a single focused image of the sample with a reference operating distance and a stigmator setting, assigning a first score value representing the sharpness of the focused image to the single focused image. b) With L = 1, for the L-th position of the sample, generate N out-of-focus images of the sample, obtain the score values corresponding to each out-of-focus image, and obtain the directed correction term using an optimization method; c) By applying the directed correction term to the operating distance and stigmator setting, adjust the operating distance and stigmator setting at the L-th position of the sample, and obtain a new image with the operating distance and stigmator setting and the score value associated with the obtained image; d) Using the score value, operating distance, and stigmator setting in c), obtain the directed correction term from the optimization method; e) Repeat c) and d) until the score value of the obtained image is substantially less than or equal to the first score value in a) to obtain the reference operating distance and stigmator setting; f) Based on the obtained reference operating distance and the stigmator setting focused on the L-th position, obtain M pairs of out-of-focus images together with the out-of-focus parameters; g) By moving the sample in the x and y directions, change the L-th position of the sample; h) Repeat steps b) to g) while increasing L, A method comprising.

12. A method of forming the training dataset using the L * M images according to Claim 11.

13. A computing system comprising a memory coupled to at least one processor and storing processor-executable instructions, When the instructions are executed by the at least one processor, the computing system is caused to execute the method for autofocus and aberration correction according to any one of Claims 1 to 10 and / or the method for generating the training data according to Claim 11. A computing system.

Images