CNN-based electron microscope rapid automatic focusing method and system

Abstract

The invention provides a CNN-based electron microscope rapid automatic focusing method and system, and relates to the technical field of electron microscope automatic focusing, and the method comprises the steps: collecting an electron microscope image, and carrying out the preprocessing of the image, and training a pre-constructed CNN model; inputting a real-time image into the trained CNN model, predicting an offset, and calculating a definition value; determining a focusing distance of next shooting based on the predicted offset and the definition value; focusing shooting is adjusted, and multiple sets of focusing distance and definition value data are collected until the preset number of times is reached; converting the data into coordinate points, fitting a Gaussian curve, and solving to obtain an optimal focusing distance; and adjusting the electron microscope according to the predicted optimal focusing distance to complete automatic focusing. According to the invention, shooting times are reduced, materials are protected, image definition is flexibly evaluated, high-precision automatic focusing is realized, and the model can be adaptively retrained and cope with changes and is flexible and practical.

Description

Technical Field

[0001] The present invention relates to the technical field of electron microscope autofocusing, and in particular to a CNN (Convolutional Neural Networks)-based electron microscope fast autofocusing method and system. Background Art

[0002] As a key tool of modern science and technology, electron microscopes play an indispensable role in fields such as materials science, semiconductor technology, and biology. However, in the autofocus process, frequently projecting electron beams to the target area to capture images poses a potential risk of damage to sensitive materials such as semiconductor products, because these materials often have strict restrictions on the number of electron beam irradiations. Therefore, while ensuring the accuracy of autofocus, how to minimize the number of photos to reduce damage to the sample has become a technical bottleneck that needs to be broken through.

[0003] Although there are a variety of electron microscope autofocus technologies on the market, they all show limitations in dealing with the above challenges. For example, Chinese patent CN115330859A adopts a continuous stepping photography strategy based on machine vision, which adjusts the focus by monitoring changes in image clarity, but requires multiple shots to determine the best focus. The average number of shots exceeds seven, increasing the potential damage to semiconductor products.

[0004] Another Chinese patent CN112525909A optimizes the focusing process by setting different search step sizes. Although this can theoretically reduce the number of shots, it is difficult to set a uniform and appropriate step size due to large differences between products. In actual applications, the number of shots is still unstable and high.

[0005] In addition, there is also Chinese patent CN117555123A which combines defocus analysis with curve fitting technology, collects images at the initial position and preset proportional points on both sides, and determines the focus position through evaluation value correction and calculation. However, the fixed position sampling method of this method fails to fully utilize the previous image features, the fitting accuracy is limited, and the total number of shots is also six.

[0006] In summary, the existing electron microscope autofocus technology still has shortcomings in reducing the number of shots, protecting sensitive materials and improving focusing efficiency. The lack of uniformity and flexibility of the image clarity evaluation system also limits its practical effect. Therefore, developing a technology that can accurately control the number of shots and achieve efficient autofocus with very few times is of great significance for protecting sensitive materials and optimizing the performance of electron microscopes. Summary of the invention

[0007] To this end, an embodiment of the present invention provides a CNN-based electron microscope fast autofocus method and system, which is used to solve the problems that the electron microscope autofocus technology in the prior art is insufficient in reducing the number of shots to protect sensitive materials, and the image clarity evaluation system lacks flexibility and diversity, which limits its practical effectiveness.

[0008] In order to solve the above problems, an embodiment of the present invention provides a CNN-based electron microscope fast autofocus method, the method comprising:

[0009] Step S1: collecting electron microscope image data, preprocessing the collected image data, and using the preprocessed image data to train a convolutional neural network model constructed in advance;

[0010] Step S2: inputting the real-time electron microscope image into the trained convolutional neural network model, outputting the predicted offset between the focus distance of the current image and the optimal focus distance, and calculating the clarity value of the current image;

[0011] Step S3: Obtaining the focus distance of the next captured image according to the predicted offset and clarity value;

[0012] Step S4: according to the focus distance obtained in step S3, adjusting the focus system of the electron microscope, taking images, and collecting the focus distance and the corresponding image clarity value data until a preset number of shots is reached;

[0013] Step S5: after multiple shots, the focus distance and clarity value of the shot image are converted into multiple coordinate points, a Gaussian function curve is fitted according to the coordinate points, and the parameters of the Gaussian function curve are solved to obtain the predicted optimal focus distance;

[0014] Step S6: adjusting the focus distance of the electron microscope according to the predicted optimal focus distance to achieve automatic focus.

[0015] Preferably, the method of collecting electron microscope image data and preprocessing the collected image data is:

[0016] Continuously focus and take photos with an electron microscope to obtain a series of image data, and each image records the focus distance when it was taken;

[0017] Manually evaluate the clarity of each image in each data group, determine the focus distance when the visual clarity is optimal, and calculate the difference between the focus distance of each image and the focus distance when the clarity is optimal in the group, and take the absolute value as the training label;

[0018] Normalize the image to ensure that the pixel values ​​are within a uniform range; adjust the image size to meet the input requirements of the neural network.

[0019] Preferably, the method of training the pre-built convolutional neural network model using the pre-processed image data is:

[0020] Construct a convolutional neural network model, including convolutional layers, activation layers, pooling layers, and fully connected layers;

[0021] The preprocessed image data is input into the pre-built convolutional neural network model. The input image data passes through the convolution layer, activation layer, pooling layer, and fully connected layer. Through forward propagation, the predicted offset between the focus distance of the current captured image and the optimal focus distance is finally output.

[0022] During the training process, the L1 loss function is used to calculate the error between the predicted offset and the actual offset, and minimizing the error between the predicted offset and the actual offset is used as the optimization goal; the error is back-propagated to each layer of the network through the back-propagation algorithm, and the network parameters are updated using a suitable optimization algorithm, including Adam and SGD; until the preset training rounds are reached or the loss value converges, the training is completed.

[0023] Preferably, when the predicted offset output by the trained convolutional neural network model is greater than a set threshold, new image data is collected and the convolutional neural network model is retrained.

[0024] Preferably, the method for calculating the definition value of the current captured image is:

[0025]

[0026] Where D(f) represents the clarity value, M×N represents the size of the image, f(w,h) represents the grayscale value of the image pixel (w,h), and μ represents the mean grayscale value of the image.

[0027] Preferably, the method for obtaining the focus distance of the next captured image according to the predicted offset and the clarity value is:

[0028] Step S31: Calculate two possible focus distances according to the predicted offset output by the convolutional neural network:

[0029] WD1=WD0+offset;

[0030] WD2 = WD0 – offset;

[0031] Among them, WD0 represents the current focus distance, WD1 represents the first possible focus distance, and WD2 represents the second possible focus distance;

[0032] Step S32: Ensure that the calculated focus distance is within the physical focus range of the electron microscope. If WD1 or WD2 is not within the valid range, adjust it to the closest valid value;

[0033] Step S33: If WD1 < min_WD or WD1 > max_WD, that is, WD1 can expand the range of the captured focus distance, first capture an image at WD1 and calculate the sharpness value D1, where min_WD represents the minimum value of all captured focus distances, and max_WD represents the maximum value of all captured focus distances;

[0034] Otherwise, first capture an image at WD2 and calculate the sharpness value D2;

[0035] Step S34: When the image captured first is at WD1:

[0036] If D1 > D0, update WD0 = WD1 and D0 = D1, where D0 represents the sharpness value of the current best - focused image;

[0037] Otherwise, capture an image at WD2 and calculate the sharpness value D2; if D2 > D0, update WD0 = WD2 and D0 = D2, otherwise keep the current focus distance WD0;

[0038] When the image captured first is at WD2:

[0039] If D2 > D0, update WD0 = WD2 and D0 = D2;

[0040] Otherwise, capture an image at WD1 and calculate the sharpness value D1; if D1 > D0, update WD0 = WD1 and D0 = D1, otherwise keep the current focus distance WD0;

[0041] Step S35: Obtain the focus distance WD_next for the next image capture according to the updated WD0.

[0042] Preferably, the method of converting the focus distance and sharpness value of the sampled image into multiple coordinate points, fitting a Gaussian function curve according to the coordinate points, and solving the parameters of the Gaussian function curve to obtain the predicted best - focus distance is as follows:

[0043] Convert the focus distance and sharpness value of the captured image into multiple coordinate points (x, y), and obtain one coordinate point each time an image is captured; use all the coordinate points as the input of the Gaussian function, fit a Gaussian function curve according to the coordinate points, and solve the parameters of the Gaussian function curve, where the Gaussian function curve is expressed as:

[0044]

[0045] Wherein, x represents the focus distance, y represents the clarity value, a is the height of the Gaussian function curve, b is the symmetry axis of the Gaussian function curve, and c is the standard deviation. The solved b is the predicted optimal focus distance.

[0046] The embodiment of the present invention further provides a CNN-based electron microscope fast autofocus system, which is used to implement the above-mentioned CNN-based electron microscope fast autofocus method, specifically comprising:

[0047] The data preprocessing and model training module is used to collect electron microscope image data, preprocess the collected image data, and use the preprocessed image data to train the pre-built convolutional neural network model;

[0048] The convolutional neural network prediction module is used to input the real-time electron microscope image into the trained convolutional neural network model, output the predicted offset between the focus distance of the current image and the optimal focus distance; and calculate the clarity value of the current image at the same time;

[0049] A clarity calculation module, used to calculate the clarity value of the current captured image;

[0050] A focus distance sampling decision module is used to obtain the focus distance of the next captured image based on the predicted offset and clarity value;

[0051] An image capture and data collection module is used to adjust the focus system of the electron microscope according to the focus distance obtained by the focus distance sampling decision module, capture images, and collect the focus distance and the corresponding image clarity value data until a preset number of captures is reached;

[0052] A Gaussian function fitting module is used to convert the focus distance and clarity value of the captured image into multiple coordinate points after multiple shots, fit a Gaussian function curve according to the coordinate points, solve the parameters of the Gaussian function curve, and obtain the predicted optimal focus distance;

[0053] The autofocus control module is used to adjust the focus distance of the electron microscope according to the predicted optimal focus distance to achieve autofocus.

[0054] An embodiment of the present invention also provides an electronic device, which includes a processor, a memory and a bus system, wherein the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned CNN-based electron microscope fast autofocus method.

[0055] An embodiment of the present invention further provides a computer storage medium, which stores a computer software product. The computer software product includes several instructions for enabling a computer device to execute the above-mentioned CNN-based electron microscope fast autofocus method.

[0056] It can be seen from the above technical solutions that the present invention has the following beneficial effects:

[0057] (1) Efficiency and accuracy: By quickly processing a single shot image using a convolutional neural network (CNN), the offset between the current image focal length and the optimal focus position can be accurately predicted, reducing the number of shots during the autofocus process, thereby effectively protecting sensitive materials from excessive exposure. In the test, the average error of the offset predicted by a single shot was only 0.0114 mm, demonstrating the efficiency and accuracy of the method.

[0058] (2) Flexibility and adaptability: The present invention allows users to set the maximum number of shots according to actual needs, thereby achieving a flexible balance between reducing the number of shots to protect the sample and increasing the number of shots to improve the focus accuracy. In addition, when the product is changed or the focus range changes significantly, there is no need to change the overall method design. Only the CNN model parameters need to be retrained to quickly adapt to the new situation, showing good adaptability and flexibility.

[0059] (3) Intelligent decision-making and optimization: During the autofocus process, the present invention combines the offset predicted by CNN and image clarity evaluation (such as grayscale variance) to determine the focus distance for the next shot through an intelligent decision-making algorithm. After multiple shots, the Gaussian function fitting technology is used to solve the optimal focus distance, further improving the focus accuracy. This intelligent decision-making and optimization mechanism ensures the accuracy and efficiency of the autofocus process. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the implementation cases of the present invention or the technical solutions in the prior art, the following is a brief description of the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention will be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0061] Figure 1 A flowchart of a CNN-based electron microscope fast autofocus method provided in an embodiment;

[0062] Figure 2 A block diagram of a CNN-based electron microscope fast autofocus system provided in an embodiment;

[0063] Figure 3 It is a flow chart of the system structure of the present invention in the embodiment;

[0064] Figure 4 A schematic diagram of an image 1 collected in the embodiment;

[0065] Figure 5 A schematic diagram of an image 2 collected in the embodiment;

[0066] Figure 6 A schematic diagram of image 3 collected in the embodiment;

[0067] Figure 7 A schematic diagram of an image 4 collected in the embodiment;

[0068] Figure 8 A schematic diagram of an image 5 collected in the embodiment;

[0069] Fig. 9 A schematic diagram of fitting a Gaussian function curve using coordinate points in an embodiment;

[0070] Fig.10 The figure is a statistical histogram of the test results of 1000 auto-focus tests using the present invention when the maximum number of photographing times N=5 in the embodiment. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] Embodiment 1

[0073] In order to solve the problem that the existing electron microscope autofocus technology has shortcomings in reducing the number of shots to protect sensitive materials, and the image clarity evaluation system lacks flexibility and diversity, which limits its practical effectiveness. Figure 1 As shown, an embodiment of the present invention proposes a CNN-based electron microscope fast autofocus method, the method comprising:

[0074] Step S1: collecting electron microscope image data, preprocessing the collected image data, and using the preprocessed image data to train a convolutional neural network model constructed in advance;

[0075] Step S2: inputting the real-time electron microscope image into the trained convolutional neural network model, outputting the predicted offset between the focus distance of the current image and the optimal focus distance, and calculating the clarity value of the current image;

[0076] Step S3: Obtaining the focus distance of the next captured image according to the predicted offset and clarity value;

[0077] Step S4: according to the focus distance obtained in step S3, adjusting the focus system of the electron microscope, taking images, and collecting the focus distance and the corresponding image clarity value data until a preset number of shots is reached;

[0078] Step S5: after multiple shots, the focus distance and clarity value of the shot image are converted into multiple coordinate points, a Gaussian function curve is fitted according to the coordinate points, and the parameters of the Gaussian function curve are solved to obtain the predicted optimal focus distance;

[0079] Step S6: adjusting the focus distance of the electron microscope according to the predicted optimal focus distance to achieve automatic focus.

[0080] From the above technical solution, it can be seen that the present invention proposes a CNN-based electron microscope fast autofocus method, which evaluates the image clarity by predicting the offset of the focus distance and the grayscale variance, and combines multiple shooting strategies with Gaussian function fitting technology to accurately solve the optimal focus distance. This method not only reduces the number of shots to protect sensitive materials, but also improves the practical efficiency of autofocus through a flexible image clarity evaluation system. Under the limit of a specific number of shots, high-precision focusing is achieved, and the model has the ability to adaptively retrain to cope with product changes or changes in the focus range, showing strong flexibility and practicality.

[0081] In step S1, electron microscope image data is collected and preprocessed, specifically including:

[0082] The electron microscope is used to continuously focus and take pictures to obtain a series of image data. Each image records the focus distance when it was taken.

[0083] Manually evaluate the clarity of each image in each data set to determine the focus distance at which visual clarity is optimal. At the same time, calculate the difference between the focus distance of each image and the focus distance at which clarity is optimal in the group, and take the absolute value as the training label.

[0084] Normalize the image to ensure that the pixel values ​​are within a uniform range (such as 0-1). Adjust the image size to meet the input requirements of the neural network. In addition, the amount of data can be increased by rotating, flipping, scaling, etc. to improve the generalization ability of the model.

[0085] Furthermore, a convolutional neural network model is constructed, including convolutional layers, activation layers, pooling layers, and fully connected layers. The convolutional layer is the core of the convolutional neural network. It extracts local features of the image through convolution operations. The input of each node is only a small piece of the previous neural network layer. The activation layer is used to enhance the nonlinear ability of the network and improve the network's expressiveness. Common activation functions include Sigmoid, ReLU, LeakyReLU, etc. The pooling layer is used to reduce the size of the matrix and reduce the number of parameters required for subsequent network layers while retaining important features. The fully connected layer is responsible for summarizing the learned features.

[0086] The preprocessed image data is input into the pre-built convolutional neural network model. The input image data passes through the convolution layer, activation layer, pooling layer, and fully connected layer. Through forward propagation, the predicted offset between the focus distance of the current captured image and the optimal focus distance is finally output.

[0087] During the training process, the error between the predicted offset and the true offset is calculated using the L1 loss function, and minimizing the error between the predicted offset and the true offset is used as the optimization goal; the error is back-propagated to each layer of the network through the back-propagation algorithm, and the network parameters are updated using a suitable optimization algorithm, wherein the optimization algorithm includes Adam and SGD; until the preset training rounds are reached or the loss value converges, the training is completed. This embodiment uses the Adam optimization algorithm for parameter update, because it can adaptively adjust the learning rate, which can effectively accelerate training and improve convergence performance.

[0088] In step S2, the electron microscope image captured in real time is input into the trained convolutional neural network model, and the predicted offset between the focus distance of the current captured image and the optimal focus distance is output.

[0089] It should be noted here that when the product is changed or the focus adjustment interval changes significantly, resulting in an increase in the prediction result error (when the prediction offset output by the trained convolutional neural network model is greater than the set threshold), it is only necessary to collect image data under the new situation and retrain the parameters of the neural network module in the present invention without changing the method designed by the present invention.

[0090] Furthermore, the grayscale standard deviation of the current captured image is calculated as the clarity value, and the larger the value, the higher the clarity. The calculation formula is:

[0091]

[0092] Where D(f) represents the clarity value, M×N represents the size of the image, f(w,h) represents the grayscale value of the image pixel (w,h), and μ represents the mean grayscale value of the image.

[0093] In step S3, according to the predicted offset and the clarity value, the focusing distance for the next captured image is obtained, which specifically includes:

[0094] Step S31: According to the predicted offset offset output by the convolutional neural network, calculate two possible focusing distances:

[0095] WD1 = WD0 + offset;

[0096] WD2 = WD0 – offset;

[0097] Among them, WD0 represents the current focusing distance, WD1 represents the first possible focusing distance, and WD2 represents the second possible focusing distance.

[0098] Step S32: Ensure that the calculated focusing distance is within the physical focusing range of the electron microscope. If WD1 or WD2 is not within the effective range, adjust it to the closest valid value.

[0099] Step S33: If WD1 < min_WD or WD1 > max_WD, that is, WD1 can expand the range of the captured focusing distance, first capture the image at WD1 and calculate the clarity value D1, where min_WD represents the minimum value of all captured focusing distances, and max_WD represents the maximum value of all captured focusing distances;

[0100] Otherwise, first capture the image at WD2 and calculate the clarity value D2.

[0101] Step S34: When the image at WD1 is captured first:

[0102] If D1 > D0, update WD0 = WD1 and D0 = D1, where D0 represents the clarity value of the current best focused image;

[0103] Otherwise, capture the image at WD2 and calculate the clarity value D2; if D2 > D0, update WD0 = WD2 and D0 = D2, otherwise keep the current focusing distance WD0, or adjust WD0 according to other strategies (such as average, median, etc.).

[0104] When the image at WD2 is captured first:

[0105] If D2 > D0, update WD0 = WD2 and D0 = D2;

[0106] Otherwise, capture the image at WD1 and calculate the clarity value D1; if D1 > D0, update WD0 = WD1 and D0 = D1, otherwise keep the current focusing distance WD0, or adjust WD0 according to other strategies (such as average, median, etc.).

[0107] Step S35: Obtain the focus distance WD_next for the next image capture according to the updated WD0.

[0108] In step S4, according to the focus distance obtained in step S3, the focus system of the electron microscope is adjusted, an image is captured, and the focus distance and the corresponding image clarity value data are collected until a preset number of shots is reached, wherein the minimum number of shots can be set to 3. The present invention can specify the number of shots according to actual needs, and strike a balance between reducing the number of shots required to protect sensitive material products and increasing the number of photos required to obtain clear images.

[0109] In step S5, after multiple shots, the focus distance and clarity value of the shot image are converted into multiple coordinate points, a Gaussian function curve is fitted according to the coordinate points, and the parameters of the Gaussian function curve are solved to obtain the predicted optimal focus distance, which specifically includes:

[0110] The focus distance and clarity value of the captured image are converted into multiple coordinate points (x, y), and one coordinate point is obtained each time an image is captured; all coordinate points are used as inputs of the Gaussian function, and the Gaussian function curve is fitted according to the coordinate points to solve the parameters of the Gaussian function curve, where the Gaussian function curve is expressed as:

[0111]

[0112] Wherein, x represents the focus distance, y represents the clarity value, a is the height of the Gaussian function curve, b is the symmetry axis of the Gaussian function curve, and c is the standard deviation. The solved b is the predicted optimal focus distance.

[0113] In step S6, the focus distance of the electron microscope is adjusted according to the predicted optimal focus distance to achieve automatic focusing.

[0114] Embodiment 2

[0115] like Figure 2 As shown, the present invention provides a CNN-based electron microscope fast autofocus system, which is used to implement the CNN-based electron microscope fast autofocus method of the first embodiment, specifically comprising:

[0116] The data preprocessing and model training module is used to collect electron microscope image data, preprocess the collected image data, and use the preprocessed image data to train the pre-built convolutional neural network model;

[0117] The convolutional neural network prediction module is used to input the real-time electron microscope image into the trained convolutional neural network model, output the predicted offset between the focus distance of the current image and the optimal focus distance; and calculate the clarity value of the current image at the same time;

[0118] A clarity calculation module, used to calculate the clarity value of the current captured image;

[0119] A focus distance sampling decision module is used to obtain the focus distance of the next captured image based on the predicted offset and clarity value;

[0120] An image capture and data collection module is used to adjust the focus system of the electron microscope according to the focus distance obtained by the focus distance sampling decision module, capture images, and collect the focus distance and the corresponding image clarity value data until a preset number of captures is reached;

[0121] A Gaussian function fitting module is used to convert the focus distance and clarity value of the captured image into multiple coordinate points after multiple shots, fit a Gaussian function curve according to the coordinate points, solve the parameters of the Gaussian function curve, and obtain the predicted optimal focus distance;

[0122] The autofocus control module is used to adjust the focus distance of the electron microscope according to the predicted optimal focus distance to achieve autofocus.

[0123] A CNN-based electron microscope fast autofocus system of the present embodiment is used to implement the aforementioned CNN-based electron microscope fast autofocus method. Therefore, the specific implementation method of the CNN-based electron microscope fast autofocus system can be seen in the embodiment part of the CNN-based electron microscope fast autofocus method above. Therefore, its specific implementation method can refer to the description of the corresponding various parts of the embodiment. In order to avoid redundancy, it will not be repeated here.

[0124] In order to further illustrate the advantages of the present invention, a specific case is described below.

[0125] In this case, within the focus distance range of (0.5mm, 1.2mm), the number of shots N = 5, automatic focus is achieved, such as Figure 3 As shown, the specific steps include:

[0126] First, image 1 is acquired at WD0 = (0.5 + 1.2) / 2 = 0.85 mm, as shown in FIG. Figure 4 As shown. Calculate the clarity value D0 = 6.77715, input the processed image 1 into the trained convolutional neural network model, and output the predicted offset offset = 0.0908mm. Add the coordinates (0.85, 6.77715) to the record R.

[0127] Secondly, determine whether 0.85+0.0908=0.9408>max(0.85) holds. If it does, capture image 2 at the focus distance of 0.9408 mm. Figure 5As shown. The calculated sharpness value is 29.47953 > D0 = 6.77715, update WD0 = 0.9408 mm, D0 = 29.47953. Input the processed Image 2 into the trained convolutional neural network model, and the output predicted offset is offset = 0.0067 mm. Add the coordinate (0.9408, 29.47953) to the record R.

[0128] Next, determine whether 0.9408 + 0.0067 = 0.9475 > max(0.85, 0.9408) holds. If it does, capture Image 3 at the focus distance of 0.9475 mm, as Figure 6 shown. The calculated sharpness value is 25.48799 < D0 = 29.47953 (it is necessary to capture an image of the other side position). Input the processed Image 3 into the trained convolutional neural network model, the output predicted offset is offset = 0.0190 mm, and add the coordinate (0.9475, 25.48799) to the record R.

[0129] Next, capture Image 4 at the focus distance of (0.9408 - 0.0067) = 0.9341, as Figure 7 shown. The calculated sharpness value is 32.07731, 32.07731 > D0 = 29.47953, update WD0 = 0.9341 mm, D0 = 32.07731. Input the processed Image 4 into the trained convolutional neural network model, the output predicted offset is offset = 0.0049 mm, and add the coordinate (0.9341, 32.07731) to the record R.

[0130] Finally, determine whether 0.9341 + 0.0049 = 0.939 > max(0.85, 0.9408, 0.9475, 0.9341) holds. If it does not, capture Image 5 at the focus distance of (0.9341 - 0.0049) = 0.9292 mm, as Figure 8 shown. The calculated sharpness value is 33.61079 > D0 = 32.07731, update WD0 = 0.9292 mm, D0 = 33.61079. Input the processed Image 5 into the trained convolutional neural network model, the output predicted offset is offset = 0.0048 mm, and add the coordinate (0.9292, 33.61079) to the record R.

[0131] Finally, R = [(0.85, 6.77715), (0.9408, 29.47953), (0.9475, 25.48799), (0.9341, 32.07731), (0.9292, 33.61079)]. Fit a Gaussian function curve through the coordinate points in R like Fig. 9 As shown. a=35.3429,b=0.9177,c=-0.0373 are calculated, and the predicted optimal focus distance b=0.9177 is obtained. The actual optimal focus distance in this case is 0.927, and the prediction error is 0.0093.

[0132] In addition, the prediction error of the CNN model output and the error between the actual offset were evaluated on a dataset containing 10,000 focused images. The average error of the model prediction was 0.0114 mm for a single shot.

[0133] In order to further improve the focusing accuracy, when the maximum number of photographing times N=5, the present invention is used to perform 1000 auto-focus tests, and the statistical histogram of the test results is as follows: Fig.10 The average error between the predicted optimal focus distance and the actual focus distance is reduced to 0.0098mm, which is a significant improvement compared to a single shot.

[0134] Embodiment 3

[0135] An embodiment of the present invention provides an electronic device, which includes a processor, a memory and a bus system. The processor and the memory are connected through the bus system. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned CNN-based electron microscope fast autofocus method.

[0136] Embodiment 4

[0137] An embodiment of the present invention provides a computer storage medium, which stores a computer software product. The computer software product includes several instructions for enabling a computer device to execute the above-mentioned CNN-based electron microscope fast autofocus method.

[0138] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0139] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0140] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0141] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from them are still within the protection scope of the invention.

Claims

1. A fast autofocus method for an electron microscope based on CNN, characterized in that: include: Step S1: collecting electron microscope image data, preprocessing the collected image data, and using the preprocessed image data to train a convolutional neural network model constructed in advance; Step S2: inputting the real-time electron microscope image into the trained convolutional neural network model, outputting the predicted offset between the focus distance of the current image and the optimal focus distance, and calculating the clarity value of the current image; Step S3: Obtaining the focus distance of the next captured image according to the predicted offset and clarity value; Step S4: according to the focus distance obtained in step S3, adjusting the focus system of the electron microscope, taking images, and collecting the focus distance and the corresponding image clarity value data until a preset number of shots is reached; Step S5: after multiple shots, the focus distance and clarity value of the shot image are converted into multiple coordinate points, a Gaussian function curve is fitted according to the coordinate points, and the parameters of the Gaussian function curve are solved to obtain the predicted optimal focus distance; Step S6: adjusting the focus distance of the electron microscope according to the predicted optimal focus distance to achieve automatic focus.

2. The CNN-based electron microscope fast autofocus method according to claim 1, characterized in that: The method for collecting electron microscope image data and preprocessing the collected image data is: Continuously focus and take photos with an electron microscope to obtain a series of image data, and each image records the focus distance when it was taken; Manually evaluate the clarity of each image in each data group, determine the focus distance when the visual clarity is optimal, and calculate the difference between the focus distance of each image and the focus distance when the clarity is optimal in the group, and take the absolute value as the training label; Normalize the image to ensure that the pixel values ​​are within a uniform range; adjust the image size to meet the input requirements of the neural network.

3. The CNN-based electron microscope fast autofocus method according to claim 1, characterized in that: The method of training the pre-built convolutional neural network model using the pre-processed image data is: Construct a convolutional neural network model, including convolutional layers, activation layers, pooling layers, and fully connected layers; The preprocessed image data is input into the pre-built convolutional neural network model. The input image data passes through the convolution layer, activation layer, pooling layer, and fully connected layer. Through forward propagation, the predicted offset between the focus distance of the current captured image and the optimal focus distance is finally output. During the training process, the L1 loss function is used to calculate the error between the predicted offset and the actual offset, and minimizing the error between the predicted offset and the actual offset is used as the optimization goal; the error is back-propagated to each layer of the network through the back-propagation algorithm, and the network parameters are updated using a suitable optimization algorithm, including Adam and SGD; until the preset training rounds are reached or the loss value converges, the training is completed.

4. The CNN-based electron microscope fast autofocus method according to claim 1, characterized in that: When the predicted offset output by the trained convolutional neural network model is greater than the set threshold, new image data is collected and the convolutional neural network model is retrained.

5. The CNN-based electron microscope fast autofocus method according to claim 1, characterized in that: The method for calculating the definition value of the current captured image is: Where D(f) represents the clarity value, M×N represents the size of the image, f(w,h) represents the grayscale value of the image pixel (w,h), and μ represents the mean grayscale value of the image.

6. The CNN-based electron microscope fast autofocus method according to claim 1, characterized in that: The method for obtaining the focusing distance of the next captured image according to the predicted offset and clarity value is as follows: Step S31: Calculate two possible focusing distances according to the predicted offset offset output by the convolutional neural network: WD1 = WD0 + offset; WD2 = WD0 – offset; where WD0 represents the current focusing distance, WD1 represents the first possible focusing distance, and WD2 represents the second possible focusing distance; Step S32: Ensure that the calculated focusing distance is within the physical focusing range of the electron microscope. If WD1 or WD2 is not within the valid range, adjust it to the closest valid value; Step S33: If WD1 < min_WD or WD1 > max_WD, that is, WD1 can expand the range of the captured focusing distances, first capture the image at WD1 and calculate the clarity value D1, where min_WD represents the minimum value of all captured focusing distances, and max_WD represents the maximum value of all captured focusing distances; Otherwise, first capture the image at WD2 and calculate the clarity value D2; Step S34: When the image at WD1 is captured first: If D1 > D0, update WD0 = WD1 and D0 = D1, where D0 represents the clarity value of the current best focused image; Otherwise, capture the image at WD2 and calculate the clarity value D2; if D2 > D0, update WD0 = WD2 and D0 = D2, otherwise keep the current focusing distance WD0; When the image at WD2 is captured first: If D2 > D0, update WD0 = WD2 and D0 = D2; Otherwise, capture the image at WD1 and calculate the clarity value D1; if D1 > D0, update WD0 = WD1 and D0 = D1, otherwise keep the current focusing distance WD0; Step S35: Obtain the focusing distance WD_next of the next captured image according to the updated WD0.

7. The method for converting the focusing distance and clarity value of the sampled image into multiple coordinate points, fitting a Gaussian function curve according to the coordinate points, and solving the parameters of the Gaussian function curve to obtain the predicted best focusing distance is as follows: Convert the focusing distance and clarity value of the captured image into multiple coordinate points (x, y), and obtain one coordinate point each time an image is captured; use all the coordinate points as the input of the Gaussian function, and fit a Gaussian function curve according to the coordinate points to solve the parameters of the Gaussian function curve, where the Gaussian function curve is expressed as: In the formula, x represents the focusing distance, y represents the clarity value, a is the height of the Gaussian function curve, b is the symmetry axis of the Gaussian function curve, and c is the standard deviation. The solved b is the predicted best focusing distance.

8. A CNN-based electron microscope fast autofocus system, characterized in that: The system for implementing the CNN-based fast automatic focusing method for electron microscopes according to any one of claims 1 to 7 specifically includes: A data preprocessing and model training module, which is used to collect electron microscope image data, preprocess the collected image data, and use the preprocessed image data to train a pre-constructed convolutional neural network model; The convolutional neural network prediction module is used to input the real-time electron microscope image into the trained convolutional neural network model, output the predicted offset between the focus distance of the current image and the optimal focus distance; and calculate the clarity value of the current image at the same time; A clarity calculation module, used to calculate the clarity value of the current captured image; A focus distance sampling decision module is used to obtain the focus distance of the next captured image based on the predicted offset and clarity value; An image capture and data collection module is used to adjust the focus system of the electron microscope according to the focus distance obtained by the focus distance sampling decision module, capture images, and collect the focus distance and the corresponding image clarity value data until a preset number of captures is reached; A Gaussian function fitting module is used to convert the focus distance and clarity value of the captured image into multiple coordinate points after multiple shots, fit a Gaussian function curve according to the coordinate points, solve the parameters of the Gaussian function curve, and obtain the predicted optimal focus distance; The autofocus control module is used to adjust the focus distance of the electron microscope according to the predicted optimal focus distance to achieve autofocus.

9. An electronic device, characterized in that: The electronic device includes a processor, a memory and a bus system, the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the CNN-based electron microscope fast autofocus method described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer software product, and the computer software product includes several instructions for enabling a computer device to execute the CNN-based electron microscope fast autofocus method described in any one of claims 1 to 7.

Images