Electron microscope automatic focusing method and system based on deep learning model

Abstract

The invention provides an electron microscope automatic focusing method and system based on a deep learning model, and relates to the technical field of electron microscope automatic focusing, and the method comprises the steps: collecting a plurality of images at different focusing distances through an electron microscope, and recording the focusing distance of each image; calculating the characteristic value of each collected image, combining the focusing distance and the characteristic value of the image into a characteristic vector, and generating a characteristic sequence according to the characteristic vectors obtained under different focusing distances; training a pre-constructed neural network model by using the feature sequence to obtain a trained neural network model; and inputting a to-be-tested image into the trained neural network model, calculating a difference value between the predicted optimal focusing distance and the focusing distance of the collected image, and judging whether the difference value is within a set range or not. According to the invention, the microscopic image acquisition efficiency and quality are obviously improved.

Description

Technical Field

[0001] The present invention relates to the technical field of electron microscope autofocusing, and in particular to an electron microscope autofocusing method and system based on a deep learning model. Background Art

[0002] As an important tool in the field of modern science and technology, electron microscopes play an indispensable role in many fields such as materials science, semiconductor technology, biology, etc. They can provide high-resolution images, helping researchers to deeply observe and analyze the microstructure of materials, thereby promoting technological progress in these fields.

[0003] However, a significant challenge in the autofocus process of electron microscopes is the need to frequently shoot electron beams at the target area to capture images. This process poses a potential threat to some sensitive materials, especially semiconductor products. Since these products can often only withstand a limited number of electron beam irradiations, how to ensure autofocus accuracy while minimizing the number of shots to reduce damage to the product has become a technical problem that needs to be solved urgently.

[0004] There are many electron microscope autofocus methods on the market, but they all have certain limitations in dealing with the above challenges. The following is an introduction to several existing methods and their shortcomings:

[0005] The autofocus method based on machine vision, such as Chinese patent CN115330859A, adopts a strategy of continuous stepping photography, and continuously adjusts the focus distance by monitoring the changes in the image "clarity index" until the best focus position is found. Its limitation is that it takes multiple shots to determine the best focus distance, and the average number of shots is far more than 7 times. For semiconductor products, such a shooting frequency undoubtedly increases the risk of damage.

[0006] The autofocus method with different search step lengths, such as Chinese patent CN112525909A, optimizes the focusing process by setting different search step lengths, first coarse adjustment with a large step length, narrowing the search range, and then fine adjustment with a small step length. Its limitation is that it is difficult to set a fixed and appropriate search step length for all products due to the large differences between products. Therefore, in actual applications, the number of shots is still unstable, and the average number is still greater than 7 times.

[0007] The autofocus method based on image feature index and function fitting, such as Chinese patent CN117555123A, attempts to predict the optimal focus position through image feature index and function fitting, designs two stages, takes at least 3 photos in each stage, and then extracts the feature index of the image and fits the focus distance and the direct function curve to find the extreme point of the function in the interval as the optimal focal length position. Its limitations are: changes in the product may cause the fixed fitting function to no longer be applicable, and repeated experiments and manual design of suitable fitting functions are required. In addition, too few points may lead to large errors in function fitting, which may easily cause a large difference between the inferred optimal focus distance and the actual position. If the function fitting result is not monotonous, the number of photos should be increased to more than 6 times.

[0008] In summary, the existing electron microscope autofocus method still has shortcomings in reducing the number of shots and reducing damage to semiconductor products, and the singleness and lack of adaptability of image clarity evaluation indicators also limit its effectiveness in practical applications. Therefore, developing a method that can accurately and efficiently achieve autofocus with a controllable number of focus shots is of great significance for protecting sensitive materials and improving the efficiency of electron microscopes. Summary of the invention

[0009] To this end, an embodiment of the present invention provides an electron microscope autofocus method and system based on a deep learning model, which is used to solve the problems in the prior art of electron microscope autofocus methods, such as large number of shooting times, great damage to sensitive materials, single image clarity evaluation index and insufficient adaptability.

[0010] In order to solve the above problems, an embodiment of the present invention provides an electron microscope autofocus method based on a deep learning model, the method comprising:

[0011] Step S1: using an electron microscope to collect multiple images at different focus distances, and recording the focus distance of each image;

[0012] Step S2: for each captured image, calculate its eigenvalue, and combine the focus distance and eigenvalue of the image into a feature vector, and generate a feature sequence according to the feature vectors obtained at different focus distances;

[0013] Step S3: using the feature sequence to train the pre-built neural network model to obtain a trained neural network model;

[0014] Step S4: Input the image to be tested into the trained neural network model, calculate the difference between the predicted optimal focus distance and the focus distance of the captured image, and determine whether the difference is within the set range. If so, use the image as the focus result without reshooting; if not, adjust the focus distance of the electron microscope according to the predicted optimal focus distance, and reshoot an image for verification.

[0015] Preferably, in step S1, a plurality of images are collected at different focus distances using an electron microscope, and the focus distance of each image is recorded, which specifically includes:

[0016] By setting a reasonable focus range, adjusting the focus distance in a certain step within the range and taking images with an electron microscope, the focus distance of each image is recorded.

[0017] Preferably, in step S2, for each captured image, its eigenvalue is calculated, and the focus distance and the eigenvalue of the image are combined into a feature vector, and a feature sequence is generated according to the feature vectors obtained at different focus distances, specifically including:

[0018] For each collected image, multiple feature calculation units are used to calculate multiple feature values, wherein each feature calculation unit is used to calculate a specific image feature value;

[0019] After the eigenvalue calculation is completed, the focus distance of each image and all eigenvalues ​​calculated by the feature calculation unit at the focus distance are arranged in a predetermined order to obtain a complete eigenvector;

[0020] The feature vectors obtained at different focus distances are arranged into a feature sequence.

[0021] Preferably, the feature calculation unit is used to calculate a specific image feature value, and the image feature value includes Brenner gradient function value, Tenengrad gradient function value, Laplacian operator value, grayscale variance function value, grayscale variance product function value, energy gradient function value and stdDev image grayscale standard deviation value.

[0022] Preferably, the method of arranging the feature vectors obtained at different focus distances into a feature sequence is:

[0023] The feature vectors obtained at different focus distances are sorted in ascending or descending order according to the focus distance to generate an ordered feature sequence.

[0024] Preferably, in step S3, the pre-built neural network model consists of a long short-term memory network and a multi-layer perceptron network.

[0025] Preferably, in step S3, the pre-built neural network model is trained using the feature sequence to obtain a trained neural network model, which specifically includes:

[0026] Using an end-to-end training method, the feature sequence is input into a pre-built neural network model for training. The predicted value is calculated through forward propagation, and the network parameters are adjusted through the back-propagation algorithm to minimize the error between the predicted optimal focus distance value and the actual focus distance value, and the predicted optimal focus distance is output. At the same time, a suitable optimization algorithm is used to update the network parameters, and the L1 loss function is used as the optimization target.

[0027] The embodiment of the present invention further provides an electron microscope autofocus system based on a deep learning model, which is used to implement the electron microscope autofocus method based on a deep learning model described above, and specifically includes:

[0028] An image acquisition module, used to acquire multiple images at different focus distances using an electron microscope, and record the focus distance of each image;

[0029] The feature extraction module is used to calculate the eigenvalue of each collected image, and combine the focus distance and the eigenvalue of the image into a feature vector, and generate a feature sequence according to the feature vectors obtained at different focus distances;

[0030] A neural network training module is used to train a pre-built neural network model using a feature sequence to obtain a trained neural network model;

[0031] The focus prediction and verification module is used to input the image to be tested into the trained neural network model, calculate the difference between the predicted optimal focus distance and the focus distance of the collected image, and determine whether the difference is within the set range. If so, the image is used as the focus result without reshooting; if not, the focus distance of the electron microscope is adjusted according to the predicted optimal focus distance, and a new image is shot for verification.

[0032] An embodiment of the present invention also provides an electronic device, comprising a processor, a memory and a bus system, wherein the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned electron microscope autofocus method based on the deep learning model.

[0033] An embodiment of the present invention also provides a computer storage medium, which stores a computer software product. The computer software product includes several instructions for enabling a computer device to execute the above-mentioned electron microscope autofocus method based on the deep learning model.

[0034] It can be seen from the above technical solutions that the present invention has the following beneficial effects:

[0035] (1) High precision and flexibility: Through a deep learning model, the present invention can achieve high-precision prediction of the electron microscope autofocus distance (average error less than 0.005 mm) with only 3 or fewer shots. It also supports flexible adjustment of the number of shots according to actual needs, thus balancing the needs of protecting sensitive materials and obtaining high-quality images.

[0036] (2) Adaptability and easy maintenance: In the face of the problem of increased prediction error caused by product changes or focus range changes, the present invention does not need to change the design method. It only needs to re-collect a small number of new samples and train the neural network model parameters to quickly adapt to new situations, thereby reducing maintenance costs and improving the system's adaptability and long-term stability.

[0037] (2) High efficiency: By combining the long short-term memory network (LSTM) and the multi-layer perceptron neural network (MLP), the present invention constructs a high-efficiency prediction model that can accurately calculate and predict the optimal focusing distance, significantly improving the acquisition efficiency and quality of electron microscope images, and bringing technological innovation to the field of microscopic analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the implementation cases of the present invention or the technical solutions in the prior art, the following is a brief description of the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention will be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0039] Figure 1 A flowchart of an electron microscope autofocus method based on a deep learning model provided in an embodiment;

[0040] Figure 2 It is a structural diagram of the automatic focusing method of the present invention in an embodiment;

[0041] Figure 3 A schematic diagram for generating a feature vector (sequence) in an embodiment;

[0042] Figure 4 is a histogram of absolute values ​​of errors when a single input (N=1) is used in the embodiment;

[0043] Figure 5 is a histogram of absolute values ​​of errors when double input (N=2) is used in the embodiment;

[0044] Figure 6is a histogram of absolute values ​​of errors when three inputs (N=3) are used in the embodiment;

[0045] Figure 7 is a histogram of absolute values ​​of errors when four inputs (N=4) are made in the embodiment;

[0046] Figure 8 A block diagram of an electron microscope autofocus system based on a deep learning model provided in an embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Embodiment 1

[0049] In order to solve the problems of the existing electron microscope autofocus method, such as the large number of shooting times, great damage to sensitive materials, single image clarity evaluation index and insufficient adaptability. Figure 1 As shown, an embodiment of the present invention proposes an electron microscope autofocus method based on a deep learning model, the method comprising:

[0050] Step S1: using an electron microscope to collect multiple images at different focus distances, and recording the focus distance of each image;

[0051] Step S2: for each captured image, calculate its eigenvalue, and combine the focus distance and eigenvalue of the image into a feature vector, and generate a feature sequence according to the feature vectors obtained at different focus distances;

[0052] Step S3: using the feature sequence to train the pre-built neural network model to obtain a trained neural network model;

[0053] Step S4: Input the image to be tested into the trained neural network model, calculate the difference between the predicted optimal focus distance and the focus distance of the captured image, and determine whether the difference is within the set range. If so, use the image as the focus result without reshooting; if not, adjust the focus distance of the electron microscope according to the predicted optimal focus distance, and reshoot an image for verification.

[0054] From the above technical solution, it can be seen that the present invention proposes an electron microscope autofocus method based on a deep learning model. Through the deep learning model, high-precision autofocus of the electron microscope is achieved. By taking multiple photos to collect images and calculating feature values, combined with neural network training optimization, the efficiency and quality of microscopic image acquisition are significantly improved. Its advantage is that it can flexibly specify the number of photos to balance accuracy and efficiency, and when facing product changes or changes in the focusing environment, only a small amount of new samples are needed to retrain the model to quickly adapt, showing strong adaptability and efficiency.

[0055] like Figure 2 As shown, in step S1, an electron microscope is used to collect multiple images at different focus distances, and the focus distance of each image is recorded.

[0056] Specifically, the present invention sets a reasonable focus interval (generally 0.5 mm to 5 mm), adjusts the focus distance within the interval with a certain step size (the step size is 0.001 mm to 0.005 mm), uses an electron microscope to capture images, and records the focus distance of each image.

[0057] In step S2, for each captured image, its eigenvalue is calculated, and the focus distance and the eigenvalue of the image are combined into a feature vector, and a feature sequence is generated according to the feature vectors obtained at different focus distances, specifically including:

[0058] First, for each collected image, multiple feature calculation units are used to calculate multiple feature values, where each feature calculation unit is used to calculate a specific image feature value, where the image feature values ​​include Brenner gradient function value, Tenengrad gradient function value, Laplacian operator value, grayscale variance function value, grayscale variance product function value, energy gradient function value and stdDev image grayscale standard deviation value

[0059] Then, after the eigenvalue calculation is completed, the focus distance of each image and all the eigenvalues ​​calculated by the feature calculation unit at the focus distance are arranged in a predetermined order to obtain a complete feature vector, such as Figure 3 shown.

[0060] Finally, the feature vectors obtained at different focus distances are sorted in ascending or descending order according to the focus distance to generate an ordered feature sequence.

[0061] As an optional specific implementation manner, the characteristic value calculated by each characteristic calculation unit in this embodiment and the calculation method are as follows:

[0062] The feature calculation unit 1 is used to calculate the Brenner gradient function value. The Brenner gradient function is a simple method for evaluating image clarity. It is calculated based on the square of the grayscale difference between adjacent pixels and is expressed as follows:

[0063] D1(f)=∑ y ∑ x |f(x+2,y)-f(x,y)| 2 ;

[0064] Where D1(f) represents the Brenner gradient function value, which is used to quantify the clarity of the image; (x, y) represents the pixel point in the image, and f(x, y) represents the grayscale value at the pixel point (x, y) in the image.

[0065] Feature calculation unit 2 is used to calculate the Tenengrad gradient function value. The Sobel operator is used to extract the gradient values ​​in the horizontal and vertical directions respectively. The Tenengrad gradient function defines an image clarity D2(f), which is expressed as follows:

[0066] D2(f)=∑ y ∑ x |G(x,y)|(G(x,y)>T);

[0067] The form of G(x,y) is as follows:

[0068]

[0069] In the formula, D2(f) represents the Tenengrad gradient function value, which is used to evaluate the clarity of the image, G(x,y) represents the gradient amplitude, T represents the set edge detection threshold, and G x (x, y) represents the horizontal gradient value of the image at the pixel point (x, y), which is obtained by convolving the Sobel horizontal operator with the image; G y (x, y) represents the vertical gradient value of the image at the pixel point (x, y), which is obtained by convolving the Sobel vertical operator with the image; g x and g y Represent the Sobel horizontal operator and Sobel vertical operator respectively.

[0070] The feature calculation unit 3 is used to calculate the Laplacian gradient function value, and uses the Laplacian operator (the Laplacian operator is a second-order derivative operator, which is used to calculate the second-order derivative of each pixel in the image. In image processing, the second-order derivative will produce zero crossing at the edge, so it is often used for edge detection.) to perform convolution operation on the image, and detect the edge in the image through threshold processing, which is expressed as follows:

[0071] D3(f)=∑ y ∑ x |G1(x,y)|(G1(x,y)>T1);

[0072]

[0073] Where D3(f) is the sum of the edge strengths of the image after Laplacian edge detection and threshold processing. This value can be used to evaluate the richness or significance of the edge in the image. G1(x,y) represents the gradient value obtained after the image is convolved with the Laplacian operator at the pixel point (x,y). T1 is the set threshold used to distinguish edge pixels from non-edge pixels. The Laplacian operator L is a 3×3 matrix.

[0074] The feature calculation unit 4 is used to calculate the grayscale variance (Sum of Modulus of Differences, SMD) function value. The grayscale variance method is based on an assumption: when the image is fully focused, the details in the image are clearest, so the grayscale changes in the image (i.e., high-frequency components) are also the richest. By calculating the sum of the absolute values ​​of the grayscale differences between adjacent pixels in the image, this grayscale change can be quantified, which can be used as a basis for focusing evaluation. The formula of the grayscale variance method is as follows:

[0075] D4(f)=∑ y ∑ x (|f(x,y)-f(x,y-1)|+|f(x,y)-f(x+1,y)|);

[0076] Where D4(f) represents the grayscale variance function value of the image, which is the accumulation of the absolute value of the grayscale difference between all pixels in the image and their horizontally adjacent pixels (i.e., the pixels adjacent to the left and right in the same row) and vertically adjacent pixels (i.e., the pixels at the corresponding position in the previous row). The larger the D4(f) value, the more drastic the grayscale change in the image, that is, the clearer the image is and the higher the degree of focus. Therefore, the grayscale variance method is an effective automatic focusing evaluation method.

[0077] The feature calculation unit 5 is used to calculate the grayscale variance product (SMD2) function value to evaluate the clarity of the image. The grayscale variance product method quantifies the clarity of the image by multiplying the grayscale difference between each pixel and its horizontal and vertical adjacent pixels and accumulating these products. The function is defined as follows:

[0078] D5(f)=∑ y ∑ x |f(x,y)-f(x+1,y)|*|f(x,y)-f(x,y+1)|;

[0079] In the formula, D5(f) represents the grayscale variance product function value of the image. The larger the D5(f) value, the more inconsistent the grayscale changes between adjacent pixels in the image, which usually means that the image contains more details and edge information, that is, the image is clearer. Therefore, the grayscale variance product method is also an effective autofocus evaluation method, especially in scenarios that require more precise distinction between different focus levels.

[0080] The feature calculation unit 6 is used to calculate the energy gradient function value. The energy gradient method quantifies the clarity or energy change of the image by accumulating the square of the grayscale difference between each pixel in the image and its horizontal and vertical adjacent pixels. This method emphasizes the intensity of the grayscale change in the image, rather than just the existence of the change, so it can better reflect the details and edge information of the image. The function is defined as follows:

[0081] D6(f)=∑ y ∑ x (|f(x+1,y)-f(x,y)| 2 +|f(x,y+1)-f(x,y)| 2 );

[0082] In the formula, D6(f) represents the energy gradient function value of the image. The larger the D6(f) value is, the more dramatic the grayscale changes between adjacent pixels in the image are, that is, the larger the energy gradient of the image is, which usually means that the image contains more details and edge information, so the image is clearer. Therefore, the energy gradient function value is an effective image clarity evaluation indicator, especially suitable for scenes that need to quantify the intensity of grayscale changes in images.

[0083] The feature calculation unit 7 is used to calculate the image grayscale standard deviation value (stdDev). The image grayscale standard deviation value is a statistic that measures the degree of discreteness of the image grayscale distribution. The larger the grayscale standard deviation, the wider the distribution of grayscale values ​​in the image, that is, the higher the contrast of the image, and the more detailed information it may contain. Its calculation formula is:

[0084]

[0085] Where D7(f) represents the grayscale standard deviation of the image, M×N represents the size of the image, that is, the image has M rows and N columns, and μ represents the grayscale mean of the image.

[0086] Furthermore, for example, when the image focus distance is 4.77 mm, D1(f)=406.12487, D2(f)=88.63143, D3(f)=21.20257, D4(f)=10.30307, ​​D5(f)=27.975, D6(f)=94.76401, D7(f)=20.95349; then the generated feature sequence is [4.77, 406.12487, 88.63143, 21.20257, 10.30307, ​​27.975, 94.76401,

[0087] 20.95349].

[0088] In this embodiment, in step S3, the pre-built neural network model is trained using the feature sequence to obtain a trained neural network model.

[0089] Specifically, in the development of the electron microscope autofocus system, one of the core steps is to train the pre-built neural network model using feature sequences. This process aims to enable the neural network model to learn the mapping relationship from image features to the optimal focus distance through a large amount of microscopic image data with annotated focus distances. The specific training process is as follows:

[0090] The neural network model in this implementation consists of two parts: a long short-term memory network (LSTM) and a multi-layer perceptron network (MLP).

[0091] 1. Long Short-Term Memory Network (LSTM)

[0092] Input: A sequence of features sorted by focus distance.

[0093] Processing flow:

[0094] The feature sequence is fed into the LSTM network and processed layer by layer through multiple layers of LSTM units.

[0095] The LSTM unit uses the forget gate, input gate, and output gate mechanism to capture the long-term dependencies between feature vectors.

[0096] Finally, a latent feature vector is output, which contains the key information in the sequence.

[0097] Output: Hidden feature vector output by the LSTM network.

[0098] 2. Multilayer Perceptron Network (MLP)

[0099] Input: Hidden feature vector output by LSTM.

[0100] Processing flow:

[0101] The latent feature vector is transformed nonlinearly through multiple fully connected layers.

[0102] Each layer of neurons uses the Sigmoid activation function to enhance the nonlinear expression ability of the network.

[0103] Output: Predicted optimal focus distance.

[0104] 3. Model training

[0105] Model initialization: Load the pre-built LSTM and MLP network model structures and initialize the network parameters (weights and bias terms).

[0106] Training process:

[0107] Forward propagation: Input the feature sequence into the LSTM network to calculate the latent feature vector. Then, the latent feature vector is sent to the MLP network to calculate the predicted optimal focus distance.

[0108] Loss calculation: The error between the predicted focus distance and the actual focus distance is calculated using the L1 loss function.

[0109] Back Propagation: Based on the loss value, the parameters (weights and bias terms) in the LSTM and MLP networks are adjusted through the back propagation algorithm to minimize the error.

[0110] Parameter update: The Adam optimization algorithm is used to update network parameters to accelerate the training process and improve model convergence performance.

[0111] Iterative training: Repeat the process of forward propagation, loss calculation, back propagation and parameter update until the training stop condition is met (such as reaching the preset number of iterations, the loss value is less than the threshold, etc.).

[0112] Model evaluation: Use an independent test dataset to evaluate the performance of the trained neural network model and verify its generalization ability and accuracy.

[0113] Model output: Finally, a trained neural network model is obtained, which can receive a new image feature sequence as input, predict and output the optimal focus distance, and realize the autofocus function of the electron microscope.

[0114] In step S4, the image to be tested is input into the trained neural network model, and the difference between the predicted optimal focusing distance and the focusing distance of the captured image is calculated to determine whether the difference is within the set range. If so, the image is used as the focusing result without reshooting; if not, the focusing distance of the electron microscope is adjusted according to the predicted optimal focusing distance, and a new image is shot for verification.

[0115] In order to further illustrate the advantages of the present invention, a specific experimental case is provided below for illustration.

[0116] This experiment is based on 100 sets of focus image data sets, each set contains 30 to 120 images, and the focus adjustment step is set in the range of 0.001mm to 0.005mm. For each set of images, we conducted 100 independent tests, and each test randomly selected N different images and their corresponding focus distances as inputs of the present invention. A total of 10,000 tests were conducted to comprehensively evaluate the performance of the method of the present invention.

[0117] When inputting a single time (N=1), the average absolute value of the error is 0.030014mm. The absolute value histogram of the error is as follows: Figure 4 Figure 2 shows the distribution of prediction errors when only a single image is used as input, showing a relatively large error range.

[0118] When double input (N=2), the average absolute value of the error is 0.005747mm. The absolute value histogram of the error is as follows: Figure 5 As shown, as the number of input images increases, the error distribution narrows significantly, indicating that the prediction accuracy of the present invention is significantly improved.

[0119] When three inputs are made (N=3), the average absolute value of the error is 0.004716 mm. The absolute value histogram of the error is shown in Figure 6 As shown, the advantages of the present invention in high-precision prediction are continued to be demonstrated, and the error range is further reduced.

[0120] When input four times (N=4), the average absolute value of the error is 0.004547mm. The absolute value histogram of the error is as follows: Figure 7 As shown, even if the number of input images is increased, the improvement of the prediction error of the present invention tends to be stable, indicating that when N≥3, the present invention can achieve a higher prediction accuracy.

[0121] In terms of effect, with only three shots, the present invention can achieve the best autofocus distance prediction with an average error of less than 0.005mm, which is significantly better than the traditional method and shows extremely high prediction accuracy. The present invention supports adjusting the number of shots according to actual needs, balancing the needs of reducing the number of shots to protect sensitive materials and increasing the number of shots to obtain clear images. In addition, when the prediction error increases due to product change or focus adjustment interval changes, it is only necessary to re-collect and train new samples without modifying the model design, which reflects strong flexibility and adaptability.

[0122] Embodiment 2

[0123] like Figure 8As shown, the present invention provides an electron microscope autofocus system based on a deep learning model, which is used to implement the electron microscope autofocus method based on a deep learning model of the above-mentioned embodiment 1, and specifically includes:

[0124] The image acquisition module 100 is used to acquire multiple images at different focus distances using an electron microscope and record the focus distance of each image;

[0125] The feature extraction module 200 is used to calculate the feature value of each collected image, and combine the focus distance and the feature value of the image into a feature vector, and generate a feature sequence according to the feature vectors obtained at different focus distances;

[0126] A neural network training module 300 is used to train a pre-built neural network model using a feature sequence to obtain a trained neural network model;

[0127] The focus prediction and verification module 400 is used to input the image to be tested into the trained neural network model, calculate the difference between the predicted optimal focus distance and the focus distance of the captured image, and determine whether the difference is within a set range. If so, the image is used as the focus result without reshooting; if not, the focus distance of the electron microscope is adjusted according to the predicted optimal focus distance, and a new image is shot for verification.

[0128] An electron microscope autofocus system based on a deep learning model in this embodiment is used to implement the aforementioned electron microscope autofocus method based on a deep learning model. Therefore, the specific implementation method of the electron microscope autofocus system based on a deep learning model can be seen in the embodiment part of the electron microscope autofocus method based on a deep learning model in the previous text. For example, the image acquisition module 100, the feature extraction module 200, the neural network training module 300, and the focus prediction and verification module 400 are respectively used to implement steps S1, S2, S3, and S4 in the above-mentioned electron microscope autofocus method based on a deep learning model. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part. In order to avoid redundancy, it will not be repeated here.

[0129] Embodiment 3

[0130] An embodiment of the present invention provides an electronic device, which includes a processor, a memory and a bus system. The processor and the memory are connected through the bus system. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned electron microscope autofocus method based on the deep learning model.

[0131] Embodiment 4

[0132] An embodiment of the present invention provides a computer storage medium, which stores a computer software product. The computer software product includes several instructions for enabling a computer device to execute the above-mentioned electron microscope autofocus method based on the deep learning model.

[0133] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0134] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0135] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0136] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from them are still within the protection scope of the invention.

Claims

1. An electron microscope autofocus method based on a deep learning model, characterized in that: include: Step S1: using an electron microscope to collect multiple images at different focus distances, and recording the focus distance of each image; Step S2: for each captured image, calculate its eigenvalue, and combine the focus distance and eigenvalue of the image into a feature vector, and generate a feature sequence according to the feature vectors obtained at different focus distances; Step S3: using the feature sequence to train the pre-built neural network model to obtain a trained neural network model; Step S4: Input the image to be tested into the trained neural network model, calculate the difference between the predicted optimal focus distance and the focus distance of the captured image, and determine whether the difference is within the set range. If so, use the image as the focus result without reshooting; if not, adjust the focus distance of the electron microscope according to the predicted optimal focus distance, and reshoot an image for verification.

2. The electron microscope autofocus method based on the deep learning model according to claim 1, characterized in that: In step S1, multiple images are collected at different focus distances using an electron microscope, and the focus distance of each image is recorded, specifically including: By setting a reasonable focus range, adjusting the focus distance in a certain step within the range and taking images with an electron microscope, the focus distance of each image is recorded.

3. The electron microscope autofocus method based on the deep learning model according to claim 1, characterized in that: In step S2, for each captured image, its eigenvalue is calculated, and the focus distance and the eigenvalue of the image are combined into a feature vector, and a feature sequence is generated according to the feature vectors obtained at different focus distances, specifically including: For each collected image, multiple feature calculation units are used to calculate multiple feature values, wherein each feature calculation unit is used to calculate a specific image feature value; After the eigenvalue calculation is completed, the focus distance of each image and all eigenvalues ​​calculated by the feature calculation unit at the focus distance are arranged in a predetermined order to obtain a complete eigenvector; The feature vectors obtained at different focus distances are arranged into a feature sequence.

4. The electron microscope autofocus method based on deep learning model according to claim 3, characterized in that: The feature calculation unit is used to calculate a specific image feature value, which includes a Brenner gradient function value, a Tenengrad gradient function value, a Laplacian operator value, a grayscale variance function value, a grayscale variance product function value, an energy gradient function value and a stdDev image grayscale standard deviation value.

5. The electron microscope autofocus method based on deep learning model according to claim 3, characterized in that: The method of arranging the feature vectors obtained at different focus distances into a feature sequence is: The feature vectors obtained at different focus distances are sorted in ascending or descending order according to the focus distance to generate an ordered feature sequence.

6. The electron microscope autofocus method based on deep learning model according to claim 1, characterized in that: In step S3, the pre-built neural network model consists of a long short-term memory network and a multi-layer perceptron network.

7. The electron microscope autofocus method based on deep learning model according to claim 1, characterized in that: In step S3, the pre-built neural network model is trained using the feature sequence to obtain a trained neural network model, which specifically includes: Using an end-to-end training method, the feature sequence is input into a pre-built neural network model for training. The predicted value is calculated through forward propagation, and the network parameters are adjusted through the back-propagation algorithm to minimize the error between the predicted optimal focus distance value and the actual focus distance value, and the predicted optimal focus distance is output. At the same time, a suitable optimization algorithm is used to update the network parameters, and the L1 loss function is used as the optimization target.

8. An electron microscope autofocus system based on a deep learning model, characterized in that: The system is used to implement the electron microscope autofocus method based on the deep learning model according to any one of claims 1 to 7, specifically comprising: An image acquisition module, used to acquire multiple images at different focus distances using an electron microscope, and record the focus distance of each image; The feature extraction module is used to calculate the eigenvalue of each collected image, and combine the focus distance and the eigenvalue of the image into a feature vector, and generate a feature sequence according to the feature vectors obtained at different focus distances; A neural network training module is used to train a pre-built neural network model using a feature sequence to obtain a trained neural network model; The focus prediction and verification module is used to input the image to be tested into the trained neural network model, calculate the difference between the predicted optimal focus distance and the focus distance of the collected image, and determine whether the difference is within the set range. If so, the image is used as the focus result without reshooting; if not, the focus distance of the electron microscope is adjusted according to the predicted optimal focus distance, and a new image is shot for verification.

9. An electronic device, characterized in that: The electronic device includes a processor, a memory and a bus system, the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the electron microscope autofocus method based on the deep learning model as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer software product, which includes several instructions for enabling a computer device to execute the electron microscope autofocus method based on a deep learning model as described in any one of claims 1 to 7.

Images