Electron microscope rapid automatic focusing method and system based on CNN and RNN

Abstract

The invention provides an electron microscope rapid automatic focusing method and system based on CNN and RNN, and relates to the technical field of electron microscope automatic focusing, and the method comprises the steps: collecting an electron microscope image, and carrying out the preprocessing of the image, and training a pre-constructed CNN model; inputting a real-time image into the trained CNN model, outputting an offset and a characteristic value, and calculating a definition value; determining a focusing distance of next shooting based on the predicted offset and the definition value; focusing shooting is adjusted, and multiple sets of focusing distance and definition value data are collected until the preset number of times is reached; splicing the feature value and the focusing distance difference into a feature vector, sorting the feature vector, inputting the sorted feature vector into the trained RNN, and predicting the optimal focusing offset; and adjusting the electron microscope according to the predicted optimal focusing distance to complete automatic focusing. According to the invention, shooting times are reduced, materials are protected, image definition is flexibly evaluated, high-precision automatic focusing is realized, and the model can be adaptively retrained and cope with changes and is flexible and practical.

Description

Technical Field

[0001] The present invention relates to the technical field of electron microscope autofocus, and in particular to a method and system for fast autofocusing of an electron microscope based on CNN (Convolutional Neural Networks) and RNN (Recurrent Neural Networks). Background Art

[0002] As a key tool of modern science and technology, electron microscopes occupy a core position in many fields such as materials science, semiconductor technology and biology. However, in its autofocus mechanism, the process of frequently emitting electron beams to capture images of the target area poses a potential risk of damage to sensitive materials such as semiconductor products, because these materials often have a limited number of electron beam exposure tolerances. Therefore, how to minimize the number of shots to reduce damage while ensuring autofocus accuracy has become an urgent technical challenge.

[0003] Although there are many electron microscope autofocus technologies on the market, they all show limitations in dealing with the above challenges. For example, Chinese patent CN115330859A proposes an autofocus and centering method based on machine vision, which relies on continuous stepping photography and image clarity index monitoring, and adjusts the focus distance multiple times to find the best position, but the number of shots is usually far more than seven, which significantly increases the risk of damage to semiconductor products.

[0004] Another Chinese patent CN112525909A uses different search step length strategies to optimize the focusing process, with a large step length for rough adjustment at the beginning and then a small step length for fine adjustment. This method can theoretically reduce the number of shots, but due to the large differences between products, it is difficult to set a universally applicable fixed step length, resulting in an unstable and high number of shots in actual applications.

[0005] Furthermore, the autofocus method based on deep learning and image processing (such as Chinese patent CN117555123A) attempts to combine deep learning models, but its sampling strategy is mostly based on preset strokes and step sizes, and fails to fully utilize the dynamic adjustment capabilities of deep learning in image clarity prediction. In addition, this method uses traditional clarity evaluation indicators and deep learning models in parallel, and the weighting and fitting of the results of the two may introduce errors, especially when the evaluations of the two are inconsistent, which affects the focusing accuracy and efficiency.

[0006] In summary, the existing technology is still insufficient in reducing the number of shots and protecting sensitive materials, and the diversity and adaptability of image clarity evaluation indicators are limited, which limits the actual application effect. Therefore, developing a method that can accurately control the number of shots and efficiently achieve autofocus is of great significance for protecting sensitive materials and improving the performance of electron microscopes. Summary of the invention

[0007] To this end, an embodiment of the present invention provides a method and system for fast autofocusing of an electron microscope based on CNN and RNN, which is used to solve the problem that the existing electron microscope autofocus technology in the prior art is difficult to effectively control and reduce the number of shots while ensuring the focusing accuracy, thereby increasing the potential risk of damage to sensitive materials.

[0008] In order to solve the above problems, an embodiment of the present invention provides a fast autofocus method for an electron microscope based on CNN and RNN, the method comprising:

[0009] Step S1: collecting electron microscope image data, preprocessing the collected image data, and using the preprocessed image data to train a convolutional neural network model constructed in advance;

[0010] Step S2: input the real-time electron microscope image into the trained convolutional neural network model, output the predicted offset between the focus distance of the current image and the optimal focus distance and F eigenvalues, and calculate the clarity value of the current image;

[0011] Step S3: Obtaining the focus distance of the next captured image according to the predicted offset and clarity value;

[0012] Step S4: according to the focus distance obtained in step S3, adjusting the focus system of the electron microscope, taking images, and collecting the focus distance and the corresponding image clarity value data until a preset number of shots is reached;

[0013] Step S5: After multiple shots, the F eigenvalues ​​and relative focus distances of each image are spliced ​​into a feature vector, the feature vectors of all images are sorted, and the sorted feature vector sequence is input into a pre-trained recurrent neural network, and the predicted offset between the optimal autofocus distance of the electron microscope and the focus distance with the highest clarity in the captured image is output to obtain the predicted optimal focus distance;

[0014] Step S6: adjusting the focus distance of the electron microscope according to the predicted optimal focus distance to achieve automatic focus.

[0015] Preferably, the method of collecting electron microscope image data and preprocessing the collected image data is:

[0016] Continuously focus and take photos with an electron microscope to obtain a series of image data, and each image records the focus distance when it was taken;

[0017] Manually evaluate the clarity of each image in each data group, determine the focus distance when the visual clarity is optimal, and calculate the difference between the focus distance of each image and the focus distance when the clarity is optimal in the group, and take the absolute value as the training label;

[0018] Normalize the image to ensure that the pixel values ​​are within a uniform range; adjust the image size to meet the input requirements of the neural network.

[0019] Preferably, the method of training the pre-built convolutional neural network model using the pre-processed image data is:

[0020] Construct a convolutional neural network model, including convolutional layers, activation layers, pooling layers, and fully connected layers;

[0021] The preprocessed image data is input into the pre-built convolutional neural network model. The input image data passes through the convolution layer, activation layer, pooling layer, and fully connected layer. Through forward propagation, the predicted offset between the focus distance of the current captured image and the optimal focus distance is finally output.

[0022] During the training process, the L1 loss function is used to calculate the error between the predicted offset and the actual offset, and minimizing the error between the predicted offset and the actual offset is used as the optimization goal; the error is back-propagated to each layer of the network through the back-propagation algorithm, and the network parameters are updated using a suitable optimization algorithm, including Adam and SGD; until the preset training rounds are reached or the loss value converges, the training is completed.

[0023] Preferably, when the predicted offset output by the trained convolutional neural network model is greater than a set threshold, new image data is collected and the convolutional neural network model is retrained.

[0024] Preferably, the method for calculating the definition value of the current captured image is:

[0025]

[0026] Where D(f) represents the clarity value, M×N represents the size of the image, f(w,h) represents the grayscale value of the image pixel (w,h), and μ represents the mean grayscale value of the image.

[0027] Preferably, the method for obtaining the focus distance of the next captured image according to the predicted offset and the clarity value is:

[0028] Step S31: Calculate two possible focus distances based on the predicted offset offset output by the convolutional neural network:

[0029] WD1 = WD0 + offset;

[0030] WD2 = WD0 – offset;

[0031] where WD0 represents the current focus distance, WD1 represents the first possible focus distance, and WD2 represents the second possible focus distance;

[0032] Step S32: Ensure that the calculated focus distances are within the physical focus range of the electron microscope. If WD1 or WD2 is not within the valid range, adjust it to the closest valid value;

[0033] Step S33: If WD1 < min_WD or WD1 > max_WD, that is, WD1 can expand the range of the captured focus distances, first capture the image at WD1 and calculate the sharpness value D1, where min_WD represents the minimum value of all captured focus distances, and max_WD represents the maximum value of all captured focus distances;

[0034] Otherwise, first capture the image at WD2 and calculate the sharpness value D2;

[0035] Step S34: When the image captured first is at WD1:

[0036] If D1 > D0, update WD0 = WD1 and D0 = D1, where D0 represents the sharpness value of the current best - focused image;

[0037] Otherwise, capture the image at WD2 and calculate the sharpness value D2; if D2 > D0, update WD0 = WD2 and D0 = D2, otherwise keep the current focus distance WD0;

[0038] When the image captured first is at WD2:

[0039] If D2 > D0, update WD0 = WD2 and D0 = D2;

[0040] Otherwise, capture the image at WD1 and calculate the sharpness value D1; if D1 > D0, update WD0 = WD1 and D0 = D1, otherwise keep the current focus distance WD0;

[0041] Step S35: Based on the updated WD0, obtain the focus distance WD_next for the next image capture.

[0042] Preferably, the method of splicing the F eigenvalues ​​and the relative focus distance of each image into a feature vector, sorting the feature vectors of all images, and inputting the sorted feature vector sequence into a pre-trained recurrent neural network, outputting the predicted offset between the optimal autofocus distance of the electron microscope and the focus distance with the highest clarity in the captured image, and finally obtaining the predicted optimal focus distance is:

[0043] Step S51: splicing the F eigenvalues ​​and relative focus distance of each image into a eigenvector, wherein the relative focus distance = focus distance - the focus distance with the highest definition in the captured image;

[0044] Step S52: sorting the feature vectors of all images according to the focus distance to form a feature vector sequence of length N, and inputting the sorted feature vector sequence into a pre-trained recurrent neural network to output a predicted offset between the optimal autofocus distance of the electron microscope and the focus distance with the highest clarity in the captured image, wherein the recurrent neural network is composed of a long short-term memory network and a multi-layer perceptron network;

[0045] Step S53: According to the predicted optimal auto-focus distance = the focus distance of the highest definition image + the predicted offset output by the recurrent neural network, the final predicted optimal focus distance is obtained.

[0046] The embodiment of the present invention further provides an electron microscope fast autofocus system based on CNN and RNN, which is used to implement the above-mentioned electron microscope fast autofocus method based on CNN and RNN, and specifically includes:

[0047] The data preprocessing and model training module is used to collect electron microscope image data, preprocess the collected image data, and use the preprocessed image data to train the pre-built convolutional neural network model;

[0048] The convolutional neural network prediction module is used to input the real-time electron microscope image into the trained convolutional neural network model, output the predicted offset between the focus distance of the current image and the optimal focus distance; and calculate the clarity value of the current image at the same time;

[0049] A clarity calculation module, used to calculate the clarity value of the current captured image;

[0050] A focus distance sampling decision module is used to obtain the focus distance of the next captured image based on the predicted offset and clarity value;

[0051] An image capture and data collection module is used to adjust the focus system of the electron microscope according to the focus distance obtained by the focus distance sampling decision module, capture images, and collect the focus distance and the corresponding image clarity value data until a preset number of captures is reached;

[0052] The recurrent neural network prediction module is used to, after multiple shots, concatenate the F eigenvalues ​​and relative focus distance of each image into a feature vector, sort the feature vectors of all images, and input the sorted feature vector sequence into a pre-trained recurrent neural network, output the predicted offset between the optimal autofocus distance of the electron microscope and the focus distance with the highest clarity in the shot image, and obtain the predicted optimal focus distance;

[0053] The autofocus control module is used to adjust the focus distance of the electron microscope according to the predicted optimal focus distance to achieve autofocus.

[0054] An embodiment of the present invention also provides an electronic device, which includes a processor, a memory and a bus system, wherein the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned CNN and RNN-based electron microscope fast autofocus method.

[0055] An embodiment of the present invention also provides a computer storage medium, which stores a computer software product. The computer software product includes several instructions for enabling a computer device to execute the above-mentioned CNN and RNN-based electron microscope fast autofocus method.

[0056] It can be seen from the above technical solutions that the present invention has the following beneficial effects:

[0057] (1) High efficiency and high precision: The present invention combines the convolutional neural network (CNN) and the recurrent neural network (RNN) method to achieve high-precision autofocus in a relatively small number of shots (such as 5 or even 3). This not only reduces the number of shots and the potential risk of damage to sensitive materials, but also significantly improves the accuracy and efficiency of focusing.

[0058] (3) Flexibility and adaptability: This method allows users to specify the number of shots according to actual needs, thereby finding the best balance between protecting sensitive materials and obtaining clear images. In addition, when the product changes or the focus adjustment range changes significantly, the parameters of the neural network model can be retrained to adapt to the new situation and maintain the focus accuracy instead of changing the entire method design, which reflects good flexibility and adaptability.

[0059] (5) Intelligence and automation: By automatically processing image data, predicting focus offset, adjusting the focus system and continuously optimizing the prediction results, the present invention realizes the intelligence and automation of electron microscope autofocus. This not only simplifies the operation process, but also reduces the errors caused by human intervention and improves the stability and reliability of the entire focusing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the implementation cases of the present invention or the technical solutions in the prior art, the following is a brief description of the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention will be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0061] Figure 1 A flowchart of a fast autofocus method for an electron microscope based on CNN and RNN provided in an embodiment;

[0062] Figure 2 A block diagram of a fast autofocus system for an electron microscope based on CNN and RNN provided in an embodiment;

[0063] Figure 3 It is a flow chart of the system structure of the present invention in the embodiment;

[0064] Figure 4 A schematic diagram of an image 1 collected in the embodiment;

[0065] Figure 5 A schematic diagram of an image 2 collected in the embodiment;

[0066] Figure 6 A schematic diagram of image 3 collected in the embodiment;

[0067] Figure 7 A schematic diagram of an image 4 collected in the embodiment;

[0068] Figure 8 A schematic diagram of an image 5 collected in the embodiment;

[0069] Fig. 9 1000 auto-focus tests are performed using the present invention when the maximum number of photographing times N=3 in the embodiment, and the statistical histogram of the test results;

[0070] Fig.10 1000 auto-focus tests are performed using the present invention when the maximum number of photographing times N=4 in the embodiment, and the statistical histogram of the test results is shown;

[0071] Fig.11The figure is a statistical histogram of the test results of 1000 auto-focus tests using the present invention when the maximum number of photographing times N=5 in the embodiment. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] Embodiment 1

[0074] In order to solve the problem that the existing electron microscope autofocus technology in the prior art is difficult to effectively control and reduce the number of shots while ensuring the focusing accuracy, thereby increasing the potential risk of damage to sensitive materials. Figure 1 As shown, an embodiment of the present invention proposes a fast autofocus method for an electron microscope based on CNN and RNN, the method comprising:

[0075] Step S1: collecting electron microscope image data, preprocessing the collected image data, and using the preprocessed image data to train a convolutional neural network model constructed in advance;

[0076] Step S2: input the real-time electron microscope image into the trained convolutional neural network model, output the predicted offset between the focus distance of the current image and the optimal focus distance and F eigenvalues, and calculate the clarity value of the current image;

[0077] Step S3: Obtaining the focus distance of the next captured image according to the predicted offset and clarity value;

[0078] Step S4: according to the focus distance obtained in step S3, adjusting the focus system of the electron microscope, taking images, and collecting the focus distance and the corresponding image clarity value data until a preset number of shots is reached;

[0079] Step S5: After multiple shots, the F eigenvalues ​​and relative focus distances of each image are spliced ​​into a feature vector, the feature vectors of all images are sorted, and the sorted feature vector sequence is input into a pre-trained recurrent neural network, and the predicted offset between the optimal autofocus distance of the electron microscope and the focus distance with the highest clarity in the captured image is output to obtain the predicted optimal focus distance;

[0080] Step S6: adjusting the focus distance of the electron microscope according to the predicted optimal focus distance to achieve automatic focus.

[0081] It can be seen from the above technical scheme that the present invention provides a fast autofocus method for an electron microscope based on CNN and RNN, which combines the advantages of convolutional neural network (CNN) and recurrent neural network (RNN) to achieve high-precision prediction and adjustment of the optimal focus distance within a limited number of shots. The method first uses CNN to preprocess and train the image to predict the focus offset and extract eigenvalues ​​of a single shot image; then, based on the predicted offset and image clarity, the focus distance of the next shot is intelligently adjusted to reduce blind shooting. By taking multiple shots and integrating the feature vectors of each shot, RNN further fuses this information and accurately predicts the optimal autofocus distance. This solution can not only achieve high focus accuracy within 5 or even 3 shots, but also has a high degree of flexibility and adaptability. When the product is changed or the focus range changes, high-precision focus can be quickly restored by simply retraining the model, effectively reducing the potential risk of damage to sensitive materials, while improving the efficiency of experiments and production.

[0082] In step S1, electron microscope image data is collected and preprocessed, specifically including:

[0083] The electron microscope is used to continuously focus and take pictures to obtain a series of image data. Each image records the focus distance when it was taken.

[0084] Manually evaluate the clarity of each image in each data set to determine the focus distance at which visual clarity is optimal. At the same time, calculate the difference between the focus distance of each image and the focus distance at which clarity is optimal in the group, and take the absolute value as the training label.

[0085] Normalize the image to ensure that the pixel values ​​are within a uniform range (such as 0-1). Adjust the image size to meet the input requirements of the neural network. In addition, the amount of data can be increased by rotating, flipping, scaling, etc. to improve the generalization ability of the model.

[0086] Furthermore, a convolutional neural network model is constructed, including convolutional layers, activation layers, pooling layers, and fully connected layers. The convolutional layer is the core of the convolutional neural network. It extracts local features of the image through convolution operations. The input of each node is only a small piece of the previous neural network layer. The activation layer is used to enhance the nonlinear ability of the network and improve the network's expressiveness. Common activation functions include Sigmoid, ReLU, LeakyReLU, etc. The pooling layer is used to reduce the size of the matrix and reduce the number of parameters required for subsequent network layers while retaining important features. The fully connected layer is responsible for summarizing the learned features.

[0087] The preprocessed image data is input into the pre-built convolutional neural network model. The input image data passes through the convolution layer, activation layer, pooling layer, and fully connected layer. Through forward propagation, the predicted offset between the focus distance of the current captured image and the optimal focus distance is finally output.

[0088] During the training process, the error between the predicted offset and the true offset is calculated using the L1 loss function, and minimizing the error between the predicted offset and the true offset is used as the optimization goal; the error is back-propagated to each layer of the network through the back-propagation algorithm, and the network parameters are updated using a suitable optimization algorithm, wherein the optimization algorithm includes Adam and SGD; until the preset training rounds are reached or the loss value converges, the training is completed. This embodiment uses the Adam optimization algorithm for parameter update, because it can adaptively adjust the learning rate, which can effectively accelerate training and improve convergence performance.

[0089] In step S2, the electron microscope image captured in real time is input into the trained convolutional neural network model, and the predicted offset between the focus distance of the current captured image and the optimal focus distance is output.

[0090] It should be noted here that when the product is changed or the focus adjustment interval changes significantly, resulting in an increase in the prediction result error (when the prediction offset output by the trained convolutional neural network model is greater than the set threshold), it is only necessary to collect image data under the new situation and retrain the parameters of the neural network module in the present invention without changing the method designed by the present invention.

[0091] Furthermore, the grayscale standard deviation of the current captured image is calculated as the clarity value, and the larger the value, the higher the clarity. The calculation formula is:

[0092]

[0093] Where D(f) represents the clarity value, M×N represents the size of the image, f(w,h) represents the grayscale value of the image pixel (w,h), and μ represents the mean grayscale value of the image.

[0094] In step S3, the focus distance of the next captured image is obtained according to the predicted offset and the clarity value, which specifically includes:

[0095] Step S31: Calculate two possible focus distances according to the predicted offset output by the convolutional neural network:

[0096] WD1=WD0+offset;

[0097] WD2 = WD0 – offset;

[0098] Among them, WD0 represents the current focusing distance, WD1 represents the first possible focusing distance, and WD2 represents the second possible focusing distance.

[0099] Step S32: Ensure that the calculated focusing distance is within the physical focusing range of the electron microscope. If WD1 or WD2 is not within the effective range, adjust it to the closest valid value.

[0100] Step S33: If WD1 < min_WD or WD1 > max_WD, that is, WD1 can expand the range of the captured focusing distance, first capture an image at WD1 and calculate the sharpness value D1, where min_WD represents the minimum value of all captured focusing distances, and max_WD represents the maximum value of all captured focusing distances;

[0101] Otherwise, first capture an image at WD2 and calculate the sharpness value D2.

[0102] Step S34: When the image captured first is at WD1:

[0103] If D1 > D0, update WD0 = WD1 and D0 = D1, where D0 represents the sharpness value of the current best focused image;

[0104] Otherwise, capture an image at WD2 and calculate the sharpness value D2; if D2 > D0, update WD0 = WD2 and D0 = D2, otherwise keep the current focusing distance WD0, or adjust WD0 according to other strategies (such as average, median, etc.).

[0105] When the image captured first is at WD2:

[0106] If D2 > D0, update WD0 = WD2 and D0 = D2;

[0107] Otherwise, capture an image at WD1 and calculate the sharpness value D1; if D1 > D0, update WD0 = WD1 and D0 = D1, otherwise keep the current focusing distance WD0, or adjust WD0 according to other strategies (such as average, median, etc.).

[0108] Step S35: According to the updated WD0, obtain the focusing distance WD_next for the next image capture.

[0109] In step S4, according to the focusing distance obtained in step S3, adjust the focusing system of the electron microscope, capture images, and collect the data of the focusing distance and the corresponding image sharpness value until the preset number of captures is reached, where the minimum number of captures can be set to 3 times. The present invention can specify the number of captures according to actual needs, and achieve a balance between reducing the number of captures for protecting sensitive material products and increasing the number of captures for obtaining clear images.

[0110] In step S5, after multiple shots, the F eigenvalues ​​and relative focus distances of each image are spliced ​​into a feature vector, the feature vectors of all images are sorted, and the sorted feature vector sequence is input into a pre-trained recurrent neural network, and the predicted offset between the optimal autofocus distance of the electron microscope and the focus distance with the highest clarity in the captured image is output to obtain the predicted optimal focus distance, which specifically includes:

[0111] Step S51: splicing the F eigenvalues ​​and relative focus distance of each image into a eigenvector, wherein the relative focus distance = focus distance - the focus distance with the highest definition in the captured image;

[0112] Step S52: sorting the feature vectors of all images according to the focus distance to form a feature vector sequence of length N, and inputting the sorted feature vector sequence into a pre-trained recurrent neural network to output a predicted offset between the optimal autofocus distance of the electron microscope and the focus distance with the highest clarity in the captured image;

[0113] Step S53: According to the predicted optimal auto-focus distance = the focus distance of the highest definition image + the predicted offset output by the recurrent neural network, the final predicted optimal focus distance is obtained.

[0114] Furthermore, in this embodiment, the recurrent neural network is composed of a long short-term memory network and a multi-layer perceptron network. The recurrent neural network module is the core of the electron microscope autofocus system, which is responsible for fusing the feature information extracted from images taken at N different positions to accurately predict the optimal focus distance. This process starts with a feature vector sequence composed of F eigenvalues ​​and relative focus distances, where the F eigenvalues ​​come from the deep analysis of the image by the convolutional neural network, and the relative focus distance is calculated by subtracting the focus distance with the highest clarity in the captured image from the focus distance. The sequence is arranged in ascending or descending order of the focus distance to ensure that LSTM can effectively capture the time dependency and feature association therein.

[0115] As the core component of the recurrent neural network, the long short-term memory network (LSTM) is composed of multiple LSTM unit layers stacked together. Each unit is designed with a forget gate, input gate, and output gate to flexibly handle the flow of information in the sequence. These gating mechanisms allow LSTM to forget irrelevant information while transmitting important features, thereby capturing the long-term dependencies between feature vectors. Ultimately, LSTM outputs a latent feature vector that contains the key information of the entire sequence.

[0116] Next, the multi-layer perceptron network (MLP) receives the latent feature vector output by the LSTM as input, and further maps the latent feature vector to the prediction result through multiple fully connected layers and nonlinear activation functions inside it. In this embodiment, the output of the MLP is the predicted offset between the best autofocus distance of the electron microscope and the focus distance with the highest clarity in the captured image. By simply adding this predicted offset to the focus distance of the highest clarity image, the predicted best focus distance can be obtained.

[0117] In order to train this entire network, the present invention adopts an end-to-end training method, that is, directly taking the feature vector as input, taking the predicted optimal focus distance value as output, and using the electron microscopy image data set marked with the focus distance for training (the data set consists of images obtained by artificial continuous focus photography, each image records the corresponding focus distance, and marks the focus distance when the human eye visual clarity is best in each data set.). The predicted value is calculated by forward propagation, and the network parameters are continuously adjusted using the back propagation algorithm and a suitable optimization algorithm (such as Adam) to minimize the error between the predicted value and the true value. In this process, the present invention selects the L1 loss function as the optimization target because it can directly reflect the absolute value of the predicted offset, which helps the model to learn small changes in the focus distance more accurately.

[0118] In step S6, the focus distance of the electron microscope is adjusted according to the predicted optimal focus distance to achieve automatic focusing.

[0119] Embodiment 2

[0120] like Figure 2 As shown, the present invention provides an electron microscope fast autofocus system based on CNN and RNN, which is used to implement the electron microscope fast autofocus method based on CNN and RNN of the above embodiment 1, and specifically includes:

[0121] The data preprocessing and model training module is used to collect electron microscope image data, preprocess the collected image data, and use the preprocessed image data to train the pre-built convolutional neural network model;

[0122] The convolutional neural network prediction module is used to input the real-time electron microscope image into the trained convolutional neural network model, output the predicted offset between the focus distance of the current image and the optimal focus distance; and calculate the clarity value of the current image at the same time;

[0123] A clarity calculation module, used to calculate the clarity value of the current captured image;

[0124] A focus distance sampling decision module is used to obtain the focus distance of the next captured image based on the predicted offset and clarity value;

[0125] An image capture and data collection module is used to adjust the focus system of the electron microscope according to the focus distance obtained by the focus distance sampling decision module, capture images, and collect the focus distance and the corresponding image clarity value data until a preset number of captures is reached;

[0126] The recurrent neural network prediction module is used to, after multiple shots, concatenate the F eigenvalues ​​and relative focus distance of each image into a feature vector, sort the feature vectors of all images, and input the sorted feature vector sequence into a pre-trained recurrent neural network, output the predicted offset between the optimal autofocus distance of the electron microscope and the focus distance with the highest clarity in the shot image, and obtain the predicted optimal focus distance;

[0127] The autofocus control module is used to adjust the focus distance of the electron microscope according to the predicted optimal focus distance to achieve autofocus.

[0128] A CNN- and RNN-based electron microscope fast autofocus system of the present embodiment is used to implement the aforementioned CNN- and RNN-based electron microscope fast autofocus method. Therefore, the specific implementation method of the CNN- and RNN-based electron microscope fast autofocus system can be found in the embodiment part of the CNN- and RNN-based electron microscope fast autofocus method. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part. In order to avoid redundancy, it will not be repeated here.

[0129] In order to further illustrate the advantages of the present invention, a specific case is described below.

[0130] In this case, within the focus distance range of (0.5mm, 1.2mm), the number of shots N = 5, automatic focus is achieved, such as Figure 3 As shown, the specific steps include:

[0131] First, image 1 is acquired at WD0 = (0.5 + 1.2) / 2 = 0.85 mm, as shown in FIG. Figure 4As shown. The sharpness calculation module calculates the sharpness value D0 = 6.77715. The processed image 1 is input into the trained convolutional neural network model, and the predicted offset offset = 0.0908 mm and 128 eigenvalue [2.1065e-05, 5.7691e-06, 4.7169e-06, ……, 7.1942e-04] are output. The feature vector (0.85, [2.1065e-05, 5.7691e-06, 4.7169e-06, ……, 7.1942e-04]) is added to the record R.

[0132] Next, it is judged whether 0.85 + 0.0908 = 0.9408 > max(0.85) holds. If so, image 2 is collected at the focusing distance of 0.9408 mm, as Figure 5 shown. The calculated sharpness value is 29.47953 > D0 = 6.77715, and WD0 = 0.9408 mm and D0 = 29.47953 are updated. The processed image 2 is input into the trained convolutional neural network model, and the predicted offset offset = 0.0067 mm and 128 eigenvalue [1.2212e-09, 1.3088e-10, 8.5680e-11, ……, 1.3171e-05] are output. The feature vector (0.9408, [1.2212e-09, 1.3088e-10, 8.5680e-11, ……, 1.3171e-05]) is added to the record R.

[0133] Next, it is judged whether 0.9408 + 0.0067 = 0.9475 > max(0.85, 0.9408) holds. If so, image 3 is collected at the focusing distance of 0.9475 mm, as Figure 6 shown. The calculated sharpness value is 25.48799 < D0 = 29.47953 (it is necessary to capture the image at the other side position). The processed image 2 is input into the trained convolutional neural network model, and the predicted offset offset = 0.0190 mm and 128 eigenvalue [3.8073e-08, 4.2729e-09, 3.0802e-09, ……, 7.0159e-05] are output. The feature vector (0.9475, [3.8073e-08, 4.2729e-09, 3.0802e-09, ……, 7.0159e-05]) is added to the record R.

[0134] Next, image 4 is collected at the focusing distance of (0.9408 - 0.0067) = 0.9341, as Figure 7As shown. The calculated clarity value is 32.07731, 32.07731>D0=29.47953, update WD0=0.9341mm, D0=32.07731. Input the processed image 4 into the trained convolutional neural network model, and output the predicted offset offset=0.0049mm and 128 eigenvalues ​​[1.6205e-10,1.9540e-11,1.1602e-11,……,4.6018e-06]. Add the eigenvector (0.9341,[1.6205e-10,1.9540e-11,1.1602e-11,……,4.6018e-06]) to the record R.

[0135] Finally, determine whether 0.9341+0.0049=0.939>max(0.85, 0.9408, 0.9475, 0.9341) is true. If not, capture image 5 at the focus distance (0.9341-0.0049)=0.9292 mm. Figure 8 As shown. The calculated clarity value is 33.61079>D0=32.07731, and WD0=0.9292mm and D0=33.61079 are updated. The processed image 4 is input into the trained convolutional neural network model, and the output prediction offset offset=0.0048mm and 128 eigenvalues ​​[5.9202e-11, 6.8553e-12, 3.8949e-12, …, 2.9062e-06] are added to the record R.

[0136] Finally, R = [

[0137] (0.85,[2.1065e-05,5.7691e-06,4.7169e-06,……,7.1942e-04])

[0138] (0.9408,[1.2212e-09,1.3088e-10,8.5680e-11,……,1.3171e-05])

[0139] (0.9475,[3.8073e-08,4.2729e-09,3.0802e-09,……,7.0159e-05])

[0140] (0.9341,[1.6205e-10,1.9540e-11,1.1602e-11,……,4.6018e-06])

[0141] (0.9292,[5.9202e-11,6.8553e-12,3.8949e-12,……,2.9062e-06]) ];

[0143] The maximum image clarity D0 = 34.61079 corresponds to WD0 = 0.9292 mm. Subtract WD0 from the first column value in R to obtain:

[0144] R=[

[0145] (0.85-0.9292,[2.1065e-05,5.7691e-06,4.7169e-06,……,7.1942e-04])

[0146] (0.9408-0.9292,[1.2212e-09,1.3088e-10,8.5680e-11,……,1.3171e-05])

[0147] (0.9475-0.9292,[3.8073e-08,4.2729e-09,3.0802e-09,……,7.0159e-05])

[0148] (0.9341-0.9292,[1.6205e-10,1.9540e-11,1.1602e-11,……,4.6018e-06])

[0149] (0.9292-0.9292,[5.9202e-11,6.8553e-12,3.8949e-12,……,2.9062e-06]) ];

[0151] Then concatenate the first column value and the following eigenvalues ​​together to get:

[0152] R=[

[0153] [-0.0792,2.1065e-05,5.7691e-06,4.7169e-06,……,7.1942e-04],

[0154] [0.0116,1.2212e-09,1.3088e-10,8.5680e-11,……,1.3171e-05],

[0155] [0.0183,3.8073e-08,4.2729e-09,3.0802e-09,……,7.0159e-05],

[0156] [0.0049,1.6205e-10,1.9540e-11,1.1602e-11,……,4.6018e-06],

[0157] [0,5.9202e-11,6.8553e-12,3.8949e-12,……,2.9062e-06] ];

[0159] Input R into the recurrent neural network, and get the output prediction offset offset as -0.0015, and calculate the predicted optimal focus distance = WD0 + offset = 0.9292 - 0.0015 = 0.9277. In this case, the actual optimal focus distance is 0.927, and the prediction error is 0.0007.

[0160] In addition, the prediction error of the CNN model output and the error between the actual offset were evaluated on a dataset containing 10,000 focused images. The average error of the model prediction was 0.0114 mm for a single shot.

[0161] In order to further improve the focusing accuracy, when the maximum number of photographing times N=3, the present invention is used to perform 1000 auto-focus tests, and the statistical histogram of the test results is as follows: Fig. 9 As shown in the figure, the average error between the predicted optimal focus distance and the actual focus distance is reduced to 0.0087mm, which is a significant improvement compared to a single shot.

[0162] In order to further improve the focusing accuracy, when the maximum number of photographing times N=4, the present invention is used to perform 1000 auto-focus tests, and the statistical histogram of the test results is as follows: Fig.10 As shown in the figure, the average error between the predicted optimal focus distance and the actual focus distance is reduced to 0.0076mm, which is a significant improvement compared to a single shot.

[0163] In order to further improve the focusing accuracy, when the maximum number of photographing times N=5, the present invention is used to perform 1000 auto-focus tests, and the statistical histogram of the test results is as follows: Fig.11 The average error between the predicted optimal focus distance and the actual focus distance is reduced to 0.0069mm, which is a significant improvement compared to a single shot.

[0164] Embodiment 3

[0165] An embodiment of the present invention provides an electronic device, which includes a processor, a memory and a bus system. The processor and the memory are connected through the bus system. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned CNN and RNN-based electron microscope fast autofocus method.

[0166] Embodiment 4

[0167] An embodiment of the present invention provides a computer storage medium, which stores a computer software product. The computer software product includes several instructions for enabling a computer device to execute the above-mentioned CNN- and RNN-based electron microscope fast autofocus method.

[0168] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0169] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0170] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0171] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.

Claims

1. A fast autofocus method for electron microscope based on CNN and RNN, characterized in that: include: Step S1: collecting electron microscope image data, preprocessing the collected image data, and using the preprocessed image data to train a convolutional neural network model constructed in advance; Step S2: input the real-time electron microscope image into the trained convolutional neural network model, output the predicted offset between the focus distance of the current image and the optimal focus distance and F eigenvalues, and calculate the clarity value of the current image; Step S3: Obtaining the focus distance of the next captured image according to the predicted offset and clarity value; Step S4: according to the focus distance obtained in step S3, adjusting the focus system of the electron microscope, taking images, and collecting the focus distance and the corresponding image clarity value data until a preset number of shots is reached; Step S5: After multiple shots, the F eigenvalues ​​and relative focus distances of each image are spliced ​​into a feature vector, the feature vectors of all images are sorted, and the sorted feature vector sequence is input into a pre-trained recurrent neural network, and the predicted offset between the optimal autofocus distance of the electron microscope and the focus distance with the highest clarity in the captured image is output to obtain the predicted optimal focus distance; Step S6: adjusting the focus distance of the electron microscope according to the predicted optimal focus distance to achieve automatic focus.

2. The electron microscope fast autofocus method based on CNN and RNN according to claim 1, characterized in that: The method for collecting electron microscope image data and preprocessing the collected image data is: Continuously focus and take photos with an electron microscope to obtain a series of image data, and each image records the focus distance when it was taken; Manually evaluate the clarity of each image in each data group, determine the focus distance when the visual clarity is optimal, and calculate the difference between the focus distance of each image and the focus distance when the clarity is optimal in the group, and take the absolute value as the training label; Normalize the image to ensure that the pixel values ​​are within a uniform range; adjust the image size to meet the input requirements of the neural network.

3. The electron microscope fast autofocus method based on CNN and RNN according to claim 1, characterized in that: The method of training the pre-built convolutional neural network model using the pre-processed image data is: Construct a convolutional neural network model, including convolutional layers, activation layers, pooling layers, and fully connected layers; The preprocessed image data is input into the pre-built convolutional neural network model. The input image data passes through the convolution layer, activation layer, pooling layer, and fully connected layer. Through forward propagation, the predicted offset between the focus distance of the current captured image and the optimal focus distance is finally output. During the training process, the L1 loss function is used to calculate the error between the predicted offset and the actual offset, and minimizing the error between the predicted offset and the actual offset is used as the optimization goal; the error is back-propagated to each layer of the network through the back-propagation algorithm, and the network parameters are updated using a suitable optimization algorithm, including Adam and SGD; until the preset training rounds are reached or the loss value converges, the training is completed.

4. The electron microscope fast autofocus method based on CNN and RNN according to claim 1, characterized in that: When the predicted offset output by the trained convolutional neural network model is greater than the set threshold, new image data is collected and the convolutional neural network model is retrained.

5. The electron microscope fast autofocus method based on CNN and RNN according to claim 1, characterized in that: The method for calculating the definition value of the current captured image is: Wherein, D(f) represents the sharpness value, M×N represents the size of the image, f(w,h) represents the gray value of the image pixel at point (w,h), and μ represents the average gray value of the image.

6. The electron microscope fast autofocus method based on CNN and RNN according to claim 1, characterized in that: The method for obtaining the focusing distance of the next captured image based on the predicted offset and the sharpness value is as follows: Step S31: Calculate two possible focusing distances according to the predicted offset offset output by the convolutional neural network: WD1 = WD0 + offset; WD2 = WD0 – offset; Wherein, WD0 represents the current focusing distance, WD1 represents the first possible focusing distance, and WD2 represents the second possible focusing distance; Step S32: Ensure that the calculated focusing distance is within the physical focusing range of the electron microscope. If WD1 or WD2 is not within the effective range, adjust it to the closest valid value; Step S33: If WD1 < min_WD or WD1 > max_WD, that is, WD1 can expand the range of the captured focusing distance, first capture the image at WD1 and calculate the sharpness value D1, where min_WD represents the minimum value of all captured focusing distances, and max_WD represents the maximum value of all captured focusing distances; Otherwise, first capture the image at WD2 and calculate the sharpness value D2; Step S34: When the image captured first is the image at WD1: If D1 > D0, update WD0 = WD1 and D0 = D1, where D0 represents the sharpness value of the current best focused image; Otherwise, capture the image at WD2 and calculate the sharpness value D2; if D2 > D0, update WD0 = WD2 and D0 = D2, otherwise keep the current focusing distance WD0; When the image captured first is the image at WD2: If D2 > D0, update WD0 = WD2 and D0 = D2; Otherwise, capture the image at WD1 and calculate the sharpness value D1; if D1 > D0, update WD0 = WD1 and D0 = D1, otherwise keep the current focusing distance WD0; Step S35: Obtain the focusing distance WD_next of the next captured image according to the updated WD0.

7. The electron microscope fast autofocus method based on CNN and RNN according to claim 1, characterized in that: The method of splicing the F eigenvalue and the relative focusing distance of each image into a feature vector, sorting the feature vectors of all images, and inputting the sorted feature vector sequence into a pre-trained recurrent neural network to output the predicted offset between the best autofocus distance of the electron microscope and the focusing distance with the highest sharpness in the captured image, and finally obtaining the predicted best focusing distance is as follows: Step S51: Splice the F eigenvalue and the relative focusing distance of each image into a feature vector, where the relative focusing distance = focusing distance - the focusing distance with the highest sharpness in the captured image; Step S52: sorting the feature vectors of all images according to the focus distance to form a feature vector sequence of length N, and inputting the sorted feature vector sequence into a pre-trained recurrent neural network to output a predicted offset between the optimal autofocus distance of the electron microscope and the focus distance with the highest clarity in the captured image, wherein the recurrent neural network is composed of a long short-term memory network and a multi-layer perceptron network; Step S53: According to the predicted optimal auto-focus distance = the focus distance of the highest definition image + the predicted offset output by the recurrent neural network, the final predicted optimal focus distance is obtained.

8. A fast autofocus system for electron microscope based on CNN and RNN, characterized in that: The system is used to implement the electron microscope fast autofocus method based on CNN and RNN according to any one of claims 1 to 7, specifically comprising: The data preprocessing and model training module is used to collect electron microscope image data, preprocess the collected image data, and use the preprocessed image data to train the pre-built convolutional neural network model; The convolutional neural network prediction module is used to input the real-time electron microscope image into the trained convolutional neural network model, output the predicted offset between the focus distance of the current image and the optimal focus distance; and calculate the clarity value of the current image at the same time; A clarity calculation module, used to calculate the clarity value of the current captured image; A focus distance sampling decision module is used to obtain the focus distance of the next captured image based on the predicted offset and clarity value; An image capture and data collection module is used to adjust the focus system of the electron microscope according to the focus distance obtained by the focus distance sampling decision module, capture images, and collect the focus distance and the corresponding image clarity value data until a preset number of captures is reached; The recurrent neural network prediction module is used to, after multiple shots, concatenate the F eigenvalues ​​and relative focus distance of each image into a feature vector, sort the feature vectors of all images, and input the sorted feature vector sequence into a pre-trained recurrent neural network, output the predicted offset between the optimal autofocus distance of the electron microscope and the focus distance with the highest clarity in the shot image, and obtain the predicted optimal focus distance; The autofocus control module is used to adjust the focus distance of the electron microscope according to the predicted optimal focus distance to achieve autofocus.

9. An electronic device, characterized in that: The electronic device includes a processor, a memory and a bus system, the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the CNN and RNN-based electron microscope fast autofocus method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer software product, and the computer software product includes several instructions for enabling a computer device to execute the CNN and RNN-based electron microscope fast autofocus method as described in any one of claims 1 to 7.

Images