Electronic microscopic imaging automatic focusing method, device, equipment, medium and product

Abstract

The invention relates to the technical field of electron microscopic imaging, in particular to an electron microscopic imaging automatic focusing method and device, computer equipment, a storage medium and a computer program product. The method comprises: acquiring an initial image; taking the initial image as a starting point, searching a to-be-detected object by using a current positioning model, and determining a focusing search starting point; the focusing search starting point is adjusted based on preset search parameters, a detection image corresponding to the adjusted imaging parameters is obtained, and the search step length and the search direction corresponding to the search parameters are dynamically adjusted according to the definition evaluation result of the historical detection image; in response to the condition that the adjusted detection image does not meet the definition evaluation index, updating the search parameter; and adjusting the imaging parameter based on the updated search parameter until the detection image meets a preset focusing requirement. By adopting the method, the efficiency of automatic focusing can be improved, and the accuracy of focus point positioning and the robustness of algorithm implementation can be improved.

Description

Technical Field

[0001] The present application relates to the field of electron microscopic imaging technology, and in particular to an electron microscopic imaging automatic focusing method, device, computer equipment, storage medium and computer program product. Background Art

[0002] The electron microscope is an important instrument for measuring critical dimensions in the semiconductor manufacturing process. It can measure the size of the pattern formed during the photolithography process with high precision, which is crucial to ensuring the performance and stability of semiconductor chips. The main functions and applications of the electron microscope include: high-resolution imaging, critical dimension measurement, defect detection, process control, and three-dimensional measurement. Electron microscopes are used as front-end inspection equipment in the field of semiconductor integrated circuits. They can realize the fast and accurate automatic focusing function of the equipment, and provide image quality assurance for subsequent visual positioning, critical dimension measurement, defect detection and other functions.

[0003] In the related art, autofocus in electron microscopes is an important function used to ensure the clarity of the image and the accuracy of the measurement during the scanning process. Autofocus is the ability to adjust the focusing parameters of the electron microscope during its operation to ensure that the image remains clear at different depths and when the sample surface changes. This is especially important for high-precision critical dimension measurements, because image blur will affect the calculation and recognition of dimensions. The currently commonly used autofocus technology usually requires the sample stage of the electron microscope to move back and forth, so that the imaging area can be quickly searched on the sample, and the focus area is determined based on the clarity of different imaging areas, and finally the position of the focus point is determined.

[0004] However, the current automatic focusing method has the following technical problems: the existing automatic focusing method has low focusing efficiency, poor stability and universality, and it is difficult to quickly and accurately locate the focus point. Summary of the invention

[0005] Based on this, it is necessary to provide an electron microscopy imaging autofocus method, device, computer equipment, computer-readable storage medium and computer program product that can improve the efficiency of autofocus, improve the accuracy of focus point positioning and the robustness of algorithm implementation in order to solve the above-mentioned technical problems.

[0006] In the first aspect, the present application provides an electron microscopy imaging autofocus method. The method includes: in response to the startup of the target device, obtaining an initial image of the object to be tested based on the preset initial startup parameters; using the initial image as the starting point, applying the pre-trained current positioning model to search the object to be tested, and determining the focus search starting point, the focus search starting point is the current value corresponding to the best output result of the current positioning model; adjusting the focus search starting point based on the preset search parameters, obtaining the detection image corresponding to the adjusted imaging parameters, and dynamically adjusting the search step and search direction corresponding to the search parameters according to the clarity evaluation results of the historical detection images; in response to the detection image after adjustment not meeting the preset clarity evaluation index, updating the search parameters based on the historical detection image before adjustment and the detection image after adjustment; adjusting the imaging parameters based on the updated search parameters until the detection image meets the preset focus requirements.

[0007] In one of the embodiments, in response to the adjusted detection image not satisfying a preset clarity evaluation index, based on the historical detection image before adjustment and the adjusted detection image, before updating the search parameters, it also includes: determining a focus area in the detection image, dividing the focus area into a number of sub-areas, and further dividing each of the sub-areas into a number of calculation units; traversing and calculating the maximum difference of the accumulated grayscale values ​​of the calculation units in the sub-areas as the contrast of the sub-areas, and accumulating to obtain the overall contrast of the focus area; and obtaining a clarity evaluation result of the detection image according to the overall contrast of the focus area.

[0008] In one of the embodiments, the updating of the search parameters based on the historical detection image before adjustment and the detection image after adjustment includes: comparing the clarity of the detection image with that of the historical detection image, and selecting the search point corresponding to the detection image with higher clarity as a new search starting point; and determining the search step size and search direction based on the historical detection image and the imaging parameters corresponding to the detection image.

[0009] In one of the embodiments, the method takes the initial image as the starting point, applies a pre-trained current positioning model to search the object to be measured, and before determining the focus search starting point, it also includes: constructing an initial neural network model, calling a sample set to train the initial neural network model to obtain a classification network model; based on the detection requirements of the target features, optimizing the classification network model to obtain the current positioning model.

[0010] In one embodiment, based on the detection requirement of the target feature, the classification network model is optimized to obtain the current positioning model, including: selecting a target convolution kernel in the classification network model, wherein the target convolution kernel is a multidimensional convolution kernel for feature dimensionality upgrading; calling a plurality of low-dimensional convolution kernels to replace the target convolution kernel to obtain the current positioning model; and / or adding a spatial attention module to the classification network model, wherein the spatial attention module is used to perform weighted processing on the target area in the detection image.

[0011] In one embodiment, the constructing of the initial neural network model and calling the sample set to train the initial neural network model to obtain the classification network model include: preprocessing the sample data in the sample set, and training the classification network model based on the preprocessed sample data, wherein the preprocessing includes data enhancement and random noise injection.

[0012] In the second aspect, the present application also provides an electron microscopy imaging autofocus device. The device includes: an initial image module, which is used to obtain an initial image of the object to be measured based on a preset initial startup parameter in response to the startup of the target device; a focus starting point module, which is used to use the initial image as the starting point, apply a pre-trained current positioning model to search the object to be measured, and determine the focus search starting point, wherein the focus search starting point is the current value corresponding to the best output result of the current positioning model; a detection image module, which is used to adjust the focus search starting point based on the preset search parameters, and obtain the detection image corresponding to the adjusted imaging parameters, and the search step size and search direction corresponding to the search parameters are dynamically adjusted according to the clarity evaluation results of the historical detection images; a dynamic search module, which is used to update the search parameters based on the historical detection images before adjustment and the detection images after adjustment in response to the detection images after adjustment not meeting the preset clarity evaluation index; an autofocus module, which is used to adjust the imaging parameters based on the updated search parameters until the detection image meets the preset focus requirements.

[0013] In one of the embodiments, the dynamic search module further includes: a region segmentation module, used to determine the focus area in the detection image, divide the focus area into a number of sub-areas, and further divide each of the sub-areas into a number of calculation units; a contrast module, used to traverse and calculate the maximum difference of the cumulative grayscale values ​​of the calculation units in the sub-areas as the contrast of the sub-areas, and accumulate to obtain the overall contrast of the focus area; a clarity evaluation module, used to obtain a clarity evaluation result of the detection image according to the overall contrast of the focus area.

[0014] In one embodiment, the dynamic search module includes: a search starting point module, used to compare the clarity of the detection image with the historical detection image, and select the search point corresponding to the detection image with higher clarity as a new search starting point; a step direction module, used to determine the search step and search direction based on the historical detection image and the imaging parameters corresponding to the detection image.

[0015] In one of the embodiments, before the focusing starting point module, it also includes: a classification network model module, which is used to construct an initial neural network model, call a sample set to train the initial neural network model, and obtain a classification network model; a model optimization module, which is used to optimize the classification network model based on the detection requirements of the target features to obtain the current positioning model.

[0016] In one embodiment, the model optimization module includes: a dimensionality-increasing convolution kernel module, which is used to select a target convolution kernel in the classification network model, and the target convolution kernel is a multidimensional convolution kernel used for feature dimensionality increasing; a convolution kernel replacement module, which is used to call a number of low-dimensional convolution kernels to replace the target convolution kernel to obtain the current positioning model; and / or a spatial attention module, which is used to add a spatial attention module to the classification network model, and the spatial attention module is used to perform weighted processing on the target area in the detection image.

[0017] In one embodiment, the classification network model module includes: a sample preprocessing module, which is used to preprocess the sample data in the sample set, and train the classification network model based on the preprocessed sample data, and the preprocessing includes data enhancement and random noise injection.

[0018] In a third aspect, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in the electron microscopy imaging automatic focusing method as described in any one of the embodiments in the first aspect are implemented.

[0019] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the electron microscopy imaging automatic focusing method as described in any one of the embodiments in the first aspect are implemented.

[0020] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps in the electron microscopy imaging automatic focusing method as described in any one of the embodiments in the first aspect are implemented.

[0021] The above-mentioned electron microscopy imaging automatic focusing method, device, computer equipment, storage medium and computer program product can achieve the beneficial effect of the technical problem in the corresponding background technology by deducing the technical features in the claims: the present application provides an electron microscopy imaging automatic focusing method, including obtaining an initial image of the object to be measured based on preset initial startup parameters in response to the startup of the target device; using the initial image as the starting point, applying a pre-trained current positioning model to search the object to be measured, and determining a focus search starting point, wherein the focus search starting point is the current value corresponding to the best output result of the current positioning model; adjusting the focus search starting point based on the preset search parameters to obtain the detection image corresponding to the adjusted imaging parameters, and the search step size and search direction corresponding to the search parameters are dynamically adjusted according to the clarity evaluation results of the historical detection images; in response to the adjusted detection image not meeting the preset clarity evaluation index, updating the search parameters based on the historical detection images before adjustment and the adjusted detection image; adjusting the imaging parameters based on the updated search parameters until the detection image meets the preset focus requirements. In implementation, by presetting the initial startup parameters, the electron microscopy imaging can have an initial startup parameter, so as to perform subsequent focus search. In the search process, the pre-trained current positioning model can improve the search efficiency of the focus search starting point and improve the accuracy of target recognition in the detection image. Finally, the search and detection of the best focus point is started from this focus search starting point. In the detection process, the search step size and search direction corresponding to the search parameters are dynamically adjusted according to the clarity evaluation results of the historical detection images. In this way, dynamic search can be performed according to the actual search situation, rather than mechanically searching according to a fixed step size and direction. This helps to improve the calculation efficiency of the best focus point, while reducing the possibility of reciprocating calculations, avoiding problems such as agitation and image jitter caused by frequent current changes, and helps to improve hardware accuracy and image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The first flowchart of an electron microscopy autofocus method in one embodiment is shown.

[0023] Figure 2 FIG. 4 is a second flow chart of an electron microscopy imaging autofocus method in another embodiment.

[0024] Figure 3a A schematic diagram of focus area target selection in one embodiment.

[0025] Figure 3b A schematic diagram of focus area target selection in another embodiment.

[0026] Figure 4 FIG. 4 is a schematic diagram of a third flow chart of an electron microscopy imaging automatic focusing method in another embodiment.

[0027] Figure 5 FIG. 4 is a schematic diagram of a fourth process flow of an electron microscopy imaging automatic focusing method in another embodiment.

[0028] Figure 6 FIG. 5 is a fifth flow chart of an electron microscopy imaging autofocus method in another embodiment.

[0029] Figure 7 Schematic diagram of the structure of an improved classification network model in an embodiment.

[0030] Figure 8 FIG. 6 is a sixth flow chart of an electron microscopy imaging autofocus method in another embodiment.

[0031] Fig. 9 The figure is a structural block diagram of an electron microscopic imaging automatic focusing device in one embodiment.

[0032] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0034] In the related art, autofocus in electron microscopes is an important function used to ensure the clarity of the image and the accuracy of the measurement during the scanning process. Autofocus is the ability to adjust the focusing parameters of the electron microscope during its operation to ensure that the image remains clear at different depths and when the sample surface changes. This is especially important for high-precision critical dimension measurements, because image blur will affect the calculation and recognition of dimensions. The currently commonly used autofocus technology usually requires the sample stage of the electron microscope to move back and forth, so that the imaging area can be quickly searched on the sample, and the focus area is determined based on the clarity of different imaging areas, and finally the position of the focus point is determined.

[0035] However, the current automatic focusing method has the following technical problems: the existing automatic focusing method has low focusing efficiency and is difficult to locate the focus point quickly and accurately.

[0036] Based on the above problems, the embodiments of the present application provide an electron microscopy imaging autofocus method, apparatus, computer equipment, storage medium and computer program product.

[0037] In one embodiment, Figure 1 As shown, an electron microscopy imaging autofocus method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps.

[0038] Step 101: in response to the startup of the target device, an initial image of the object to be tested is acquired based on preset initial startup parameters.

[0039] The initial startup parameters may refer to the imaging parameters set when the device is first focused after startup, and the imaging parameters may include the coordinates of the imaging area and the magnitude of the imaging current value, etc. The initial startup parameters may be pre-set by a technician.

[0040] Exemplarily, the terminal may acquire an initial image of the object to be tested based on a preset initial startup parameter in response to the startup of the target device. Specifically, the initial startup parameter may be 50% of the total adjustable current range of the target device, where the adjustable current of the target device may include Coarse Current and Fine Current.

[0041] Step 103: Taking the initial image as a starting point, the pre-trained current localization model is used to search the object to be detected, and a focus search starting point is determined.

[0042] Among them, the focus search starting point can refer to the imaging parameters corresponding to the detection image with the preset target feature, which can enable the search for the best point to start from the vicinity of the determined target feature, thus improving the search efficiency of the best focus point.

[0043] The current localization model may refer to a deep learning algorithm feature and its evaluation score used to perform feature detection and classification on an image to determine whether a target exists in the image. The current localization model may be obtained by technicians through training with a large amount of sample data in advance.

[0044] Exemplarily, the terminal may use the initial image and the initial current value as a starting point, apply a pre-trained current positioning model to search the object to be detected, and determine a focus search starting point.

[0045] Step 105: The focus search starting point is adjusted based on preset search parameters to obtain the detection image corresponding to the adjusted imaging parameters. The search step size and search direction corresponding to the search parameters are dynamically adjusted according to the clarity evaluation results of historical detection images.

[0046] Among them, search can refer to adjusting relevant parameters through preset rules, and continuously using the adjusted relevant parameters to perform specific processing until the processing that meets the target requirements is achieved. Search parameters can refer to parameters that control the search process, such as search step length, that is, the adjustment amplitude; search direction, that is, the parameter adjustment trend is to enlarge or reduce, etc.

[0047] Exemplarily, the terminal may adjust the focus search starting point based on preset search parameters, and obtain the detection image corresponding to the adjusted imaging parameters.

[0048] Step 107: In response to the adjusted detection image not satisfying a preset clarity evaluation index, the search parameter is updated based on the historical detection image before adjustment and the adjusted detection image.

[0049] The clarity evaluation index may refer to a quantitative clarity evaluation result obtained by analyzing the detection image through a unified clarity evaluation method.

[0050] Exemplarily, in response to the adjusted detection image not satisfying a preset clarity evaluation index, the terminal may update the search parameter based on the historical detection image before adjustment and the adjusted detection image.

[0051] In another embodiment, if the acquired detection image can meet the clarity evaluation index, it can be determined that the current imaging parameters meet the focus requirements, and the current imaging parameters can be used as the search result to end the search process.

[0052] Step 109: Adjust the imaging parameters based on the updated search parameters until the detected image meets the preset focus requirement.

[0053] Exemplarily, the terminal can adjust the imaging parameters based on the updated search parameters until the detected image meets the preset focus requirements. In this way, the terminal continuously and dynamically adjusts the search parameters and performs corresponding imaging tests, and finally obtains the imaging parameters of the best focus point.

[0054] In the above-mentioned electron microscopy imaging automatic focusing method, reasonable deduction is performed in combination with the technical features in the embodiment, which can achieve the following beneficial effects of solving the technical problems raised in the background technology.

[0055] The present application provides an electron microscopy imaging autofocus method, comprising: in response to the startup of a target device, acquiring an initial image of an object to be tested based on preset initial startup parameters; taking the initial image and the initial current value as a starting point, applying a pre-trained current positioning model to search the object to be tested, and determining a focus search starting point, wherein the focus search starting point is a current value corresponding to an optimal output result of the current positioning model; adjusting the focus search starting point based on preset search parameters, acquiring the detection image corresponding to the adjusted imaging parameters, and dynamically adjusting the search step size and search direction corresponding to the search parameters according to a clarity evaluation result of a historical detection image; in response to the detection image after adjustment not meeting a preset clarity evaluation index, updating the search parameters based on the historical detection image before adjustment and the detection image after adjustment; and adjusting the imaging parameters based on the updated search parameters until the detection image meets the preset focus requirement. In implementation, by presetting the initial startup parameters, the electron microscopy imaging can have an initial startup parameter, so as to perform subsequent focus search. In the search process, the pre-trained current positioning model can improve the search efficiency of the focus search starting point and improve the accuracy of target recognition in the detection image. Finally, the search and detection of the best focus point is started from this focus search starting point. In the detection process, the search step size and search direction corresponding to the search parameters are dynamically adjusted according to the clarity evaluation results of the historical detection images. In this way, dynamic search can be performed according to the actual search situation, rather than mechanically searching according to a fixed step size and direction. This helps to improve the calculation efficiency of the best focus point, while reducing the possibility of reciprocating calculations, avoiding problems such as agitation and image jitter caused by frequent current changes, and helps to improve hardware accuracy and image quality.

[0056] In one embodiment, the Figure 2 As shown, before step 107, the method further includes: step 201: determining a focus area in the detection image, dividing the focus area into a plurality of sub-areas, and further dividing each of the sub-areas into a plurality of computing units.

[0057] For example, the terminal may determine a focus area in the detection image, divide the focus area into a plurality of sub-areas, and further divide each of the sub-areas into a plurality of computing units. Figure 3a , 3b As shown, the focus area can be one or more, and the entire image can also be used as the focus area.

[0058] Step 203: traverse and calculate the maximum difference of the accumulated grayscale values ​​of the calculation units in the sub-region as the contrast of the sub-region, and accumulate to obtain the overall contrast of the focus area.

[0059] Exemplarily, the terminal may traverse and calculate the maximum difference of the accumulated grayscale values ​​of the calculation units in the sub-area as the contrast of the sub-area, and accumulate to obtain the overall contrast of the focus area.

[0060] Specifically, the terminal can first calculate the cumulative grayscale value of the pixels in each minimum unit for a single sub-area; secondly, record the maximum and minimum cumulative grayscale values ​​of the minimum unit in the sub-area, and use the difference as the contrast of the sub-area; finally, traverse all sub-areas, accumulate their contrasts as the overall contrast of the focus area, and use it to judge the image clarity.

[0061] Step 205: Obtain a clarity evaluation result of the detection image according to the overall contrast of the focus area.

[0062] In this embodiment, by dividing the focus area into multiple levels, the overall contrast calculation of the focus area based on the grayscale value difference is realized. This method has unimodal and unbiased properties, improves the robustness to noise and brightness changes, can overcome interference in complex environments to accurately judge the image clarity, and does not require complex calculations, which helps to improve the computational efficiency of clarity evaluation.

[0063] In one embodiment, the search strategy plays a vital role in the auto-focus algorithm. A good search strategy can quickly approach the focus peak point under the condition of a small number of iterations, and then determine the best focus position. However, the commonly used search strategies such as the binary search method, the Fibonacci method, the hill climbing method, etc. have problems such as many adjustments, large amount of calculations, and difficulty in determining the step length. Based on the above problems, it can be as follows Figure 4 As shown, the step 107 includes: Step 401: comparing the clarity of the detection image with that of the historical detection image, and selecting the search point corresponding to the detection image with higher clarity as a new search starting point.

[0064] Exemplarily, the terminal may compare the clarity of the detection image with that of the historical detection image, and select a search point corresponding to the detection image with higher clarity as a new search starting point.

[0065] Step 403: Determine the search step length and search direction based on the historical detection image and the imaging parameters corresponding to the detection image.

[0066] Exemplarily, the terminal may determine the search step size and the search direction based on the historical detection image and the imaging parameters corresponding to the detection image.

[0067] Specifically, the terminal can obtain the corresponding detection image at the search starting point, calculate the image clarity using a preset clarity evaluation algorithm, and record the current value and the corresponding image clarity. Subsequently, the first search point can be determined according to the set initial search step and initial search direction, the corresponding current value can be set, the image can be acquired and the clarity can be calculated, and the result can be recorded at the same time. Then, the terminal can compare the image clarity twice, take the current value corresponding to the higher clarity as the current point, and search in another direction with the current step, and record the result; acquire the image again and compare the clarity, take the higher current value as the current point, record the result, and dynamically adjust the step and direction according to the calculation results of the first three times, and then determine the next search point. Finally, repeat the above steps until the minimum step requirement is met and the current point is better than the points on the left and right sides, then the current point is the best focus position.

[0068] For example, suppose the currents at the last search point, the current search point, and the next search point are , and , the corresponding image resolutions are , and , the adjustment of dynamic step size is divided into two cases for discussion: (1) If the current search point Corresponding image clarity Maximum, then the next time you need to and For the larger direction search, the dynamic step size can be expressed as follows: ; (2) If the next search point Corresponding image clarity Maximum, you need to judge , and Size: If Greater than and , then the current search direction needs to be maintained next time; if Between and The next time you need to and The larger direction of the two is searched, and the dynamic step size is shown as follows: .

[0069] In this embodiment, the specific adjustment method of dynamic search includes the determination of new search points and the adjustment of search steps and search directions, which helps to quickly and accurately obtain the focus position through fewer iterations. In addition, the significance of recording each search current value and the corresponding image clarity is that before setting the current to obtain the image and calculate the clarity each time, the algorithm will determine whether the current point has been calculated. If it has not been calculated, the operations of setting the current, obtaining the image, and calculating the clarity are performed; and if it has been calculated, this series of operations is not required, and the image clarity corresponding to the current value can be directly read. In this way, on the one hand, the number of times the hardware sets the current is reduced, and the problems such as agitation and image jitter caused by frequent current changes are avoided, and the hardware accuracy and image quality are improved. On the other hand, setting the current and obtaining the image are both time-consuming operations, and calculating the clarity will also consume time and computing power. Reducing this series of operations will greatly improve the overall efficiency of automatic focusing and avoid redundant operations and calculations.

[0070] In one embodiment, the Figure 5 As shown, before step 103, the process also includes: step 501: constructing an initial neural network model, calling a sample set to train the initial neural network model, and obtaining a classification network model.

[0071] Step 503: Based on the detection requirements for the target features, the classification network model is optimized to obtain the current location model.

[0072] In this embodiment, in the construction of the current localization model, the neural network model is applied and adaptive optimization processing is performed on the neural network model, so that the final current localization model is more suitable for the engineering application of automatic focusing, thereby improving the overall efficiency of automatic focusing.

[0073] In one embodiment, the Figure 6 As shown, the step 503 includes: Step 601: selecting a target convolution kernel in the classification network model, wherein the target convolution kernel is a multi-dimensional convolution kernel for feature dimensionality enhancement.

[0074] Among them, convolution kernel means that convolution is to generate output by sliding the convolution kernel (also called filter) on the input data and calculating the weighted sum of the local area.

[0075] Step 603: call several low-dimensional convolution kernels to replace the target convolution kernel to obtain the current localization model; and / or step 605: add a spatial attention module to the classification network model, and the spatial attention module is used to perform weighted processing on the target area in the detection image.

[0076] Specifically, for example, Figure 7As shown, in order to balance efficiency and functionality, the terminal can use a lightweight ResNet18 network and improve the network. First, the 7×7 convolution kernel used for feature dimension upgrading in ResNet18 is replaced by three stacked 3×3 convolution kernels, which deepens the network depth while reducing the number of network parameters; secondly, considering that the target of the focused image is generally located in the center of the image, a spatial attention mechanism is added to the network to focus on the features of key areas, enhance useful information and suppress useless information, and improve network performance. In addition, ResNet18 uses the ReLU activation function, as shown in the following formula: ; When the input is less than 0, the hard saturation phenomenon of neuron "death" will occur, so the Mish activation function can be used instead, as shown in the following formula: .

[0077] When the input is negative, a relatively small negative gradient is allowed to flow in, and the smoothness and gradient descent effects are better than ReLU. Finally, in order to enable the network to adapt to changes in brightness and noise, the data is enhanced online, the brightness and contrast of the training data are adjusted, and random noise is injected. Two types of images are used for the training of the improved ResNet18 network, namely, full noise images of out-of-focus areas and images of different clarity and different targets in the focused areas. The network is trained based on the PyTorch framework, and the trained model is converted through openvino to facilitate actual engineering calls.

[0078] In this embodiment, the specific processing steps in the optimization of the current localization model include replacing the convolution kernel, adding a spatial attention module, etc. By making the above improvements to the model, the engineering application effect of the model can be improved.

[0079] In one embodiment, the Figure 8 As shown, the step 501 includes: Step 801: preprocessing the sample data in the sample set, and training the classification network model based on the preprocessed sample data, wherein the preprocessing includes data enhancement and random noise injection.

[0080] In this embodiment, the optimization process in training mainly includes adaptive adjustment of sample data. By preprocessing the sample data, the engineering application effect of the model can be improved.

[0081] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0082] Based on the same inventive concept, the embodiment of the present application also provides an electron microscopic imaging automatic focusing device for implementing the electron microscopic imaging automatic focusing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations of one or more embodiments of the electron microscopic imaging automatic focusing device provided below can refer to the limitations of the electron microscopic imaging automatic focusing method above, and will not be repeated here.

[0083] In one embodiment, Fig. 9 As shown, an electron microscopic imaging autofocus device is provided, comprising: an initial image module, a focus starting point module, a detection image module, a dynamic search module and an autofocus module, wherein: the initial image module is used to obtain an initial image of the object to be measured based on a preset initial startup parameter in response to the startup of the target device; the focus starting point module is used to use the initial image as a starting point, apply a pre-trained current positioning model to search the object to be measured, and determine a focus search starting point, wherein the focus search starting point is a current value corresponding to the best output result of the current positioning model; the detection image module is used to adjust the focus search starting point based on preset search parameters, and obtain the detection image corresponding to the adjusted imaging parameters, and the search step size and search direction corresponding to the search parameters are dynamically adjusted according to the clarity evaluation result of the historical detection image; the dynamic search module is used to update the search parameters based on the historical detection image before adjustment and the detection image after adjustment in response to the detection image after adjustment not meeting the preset clarity evaluation index; the autofocus module is used to adjust the imaging parameters based on the updated search parameters until the detection image meets the preset focus requirement.

[0084] In one of the embodiments, the dynamic search module further includes: a region segmentation module, used to determine the focus area in the detection image, divide the focus area into a number of sub-areas, and further divide each of the sub-areas into a number of calculation units; a contrast module, used to traverse and calculate the maximum difference of the cumulative grayscale values ​​of the calculation units in the sub-areas as the contrast of the sub-areas, and accumulate to obtain the overall contrast of the focus area; a clarity evaluation module, used to obtain a clarity evaluation result of the detection image according to the overall contrast of the focus area.

[0085] In one embodiment, the dynamic search module includes: a search starting point module, used to compare the clarity of the detection image with the historical detection image, and select the search point corresponding to the detection image with higher clarity as a new search starting point; a step direction module, used to determine the search step and search direction based on the historical detection image and the imaging parameters corresponding to the detection image.

[0086] In one of the embodiments, before the focusing starting point module, it also includes: a classification network model module, which is used to construct an initial neural network model, call a sample set to train the initial neural network model, and obtain a classification network model; a model optimization module, which is used to optimize the classification network model based on the detection requirements of the target features to obtain the current positioning model.

[0087] In one embodiment, the model optimization module includes: a dimensionality-increasing convolution kernel module, which is used to select a target convolution kernel in the classification network model, and the target convolution kernel is a multidimensional convolution kernel used for feature dimensionality increasing; a convolution kernel replacement module, which is used to call a number of low-dimensional convolution kernels to replace the target convolution kernel to obtain the current positioning model; and / or a spatial attention module, which is used to add a spatial attention module to the classification network model, and the spatial attention module is used to perform weighted processing on the target area in the detection image.

[0088] In one embodiment, the classification network model module includes: a sample preprocessing module, which is used to preprocess the sample data in the sample set, and train the classification network model based on the preprocessed sample data, and the preprocessing includes data enhancement and random noise injection.

[0089] Each module in the above-mentioned electron microscopy imaging autofocus device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0090] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an electron microscopy imaging autofocus method is realized. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0091] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0092] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0093] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0094] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0095] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0096] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0097] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0098] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An electron microscopy imaging automatic focusing method, characterized in that: The method comprises: In response to the start-up of the target device, acquiring an initial image of the object to be tested based on preset initial start-up parameters; Taking the initial image as a starting point, applying a pre-trained current localization model to search the object to be measured, and determining a focus search starting point, where the focus search starting point is a current value corresponding to the best output result of the current localization model; The focus search starting point is adjusted based on preset search parameters to obtain a detection image corresponding to the adjusted imaging parameters, and the search step size and search direction corresponding to the search parameters are dynamically adjusted according to the clarity evaluation results of the historical detection images; In response to the adjusted detection image not satisfying a preset clarity evaluation index, updating the search parameter based on the historical detection image before adjustment and the adjusted detection image; The imaging parameters are adjusted based on the updated search parameters until the detected image meets a preset focus requirement.

2. The method according to claim 1, characterized in that In response to the adjusted detection image not satisfying a preset definition evaluation index, before updating the search parameter based on the historical detection image before adjustment and the adjusted detection image, the method further includes: Determine a focus area in the detection image, divide the focus area into a plurality of sub-areas, and further divide each of the sub-areas into a plurality of computing units; traversing and calculating the maximum difference of the accumulated grayscale values ​​of the calculation units in the sub-region as the contrast of the sub-region, and accumulating to obtain the overall contrast of the focus region; A clarity evaluation result of the detection image is obtained according to the overall contrast of the focus area.

3. The method according to claim 1, characterized in that: The updating of the search parameter based on the historical detection image before adjustment and the detection image after adjustment comprises: Comparing the clarity of the detection image with that of the historical detection image, and selecting the search point corresponding to the detection image with higher clarity as a new search starting point; Based on the historical detection image and the imaging parameters corresponding to the detection image, a search step length and a search direction are determined.

4. The method according to claim 1, characterized in that The method of using the initial image as a starting point and applying a pre-trained current localization model to search the object to be detected, and determining a focus search starting point, further includes: Constructing an initial neural network model, calling the sample set to train the initial neural network model, and obtaining a classification network model; Based on the detection requirements for target features, the classification network model is optimized to obtain the current positioning model.

5. The method according to claim 4, characterized in that Based on the detection requirement of the target feature, the classification network model is optimized to obtain the current location model, including: Selecting a target convolution kernel in the classification network model, wherein the target convolution kernel is a multidimensional convolution kernel for feature dimension upgrading; Calling a plurality of low-dimensional convolution kernels to replace the target convolution kernel to obtain the current localization model; and / or; A spatial attention module is added to the classification network model, and the spatial attention module is used to perform weighted processing on the target area in the detection image.

6. The method according to claim 4, characterized in that The constructing of the initial neural network model, calling the sample set to train the initial neural network model, and obtaining the classification network model comprises: The sample data in the sample set is preprocessed, and the classification network model is trained based on the preprocessed sample data, wherein the preprocessing includes data enhancement and random noise injection.

7. An electron microscopic imaging automatic focusing device, characterized in that: The device comprises: An initial image module, used for acquiring an initial image of the object to be tested based on preset initial startup parameters in response to startup of the target device; A focus starting point module, used to use the initial image as a starting point, apply a pre-trained current localization model to search the object to be measured, and determine a focus search starting point, where the focus search starting point is a current value corresponding to the best output result of the current localization model; A detection image module, used to adjust the focus search starting point based on preset search parameters, and obtain a detection image corresponding to the adjusted imaging parameters, wherein the search step size and search direction corresponding to the search parameters are dynamically adjusted according to the clarity evaluation results of historical detection images; A dynamic search module, configured to update the search parameters based on the historical detection image before adjustment and the detection image after adjustment in response to the detection image after adjustment not satisfying a preset clarity evaluation index; The auto-focus module is used to adjust the imaging parameters based on the updated search parameters until the detected image meets the preset focusing requirements.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images