An electron microscope image automatic focusing method, electronic device, storage medium and program product

Abstract

The application relates to an automatic focusing method, device, medium and product of an electron microscope image, and the method comprises the following steps: obtaining a current image by imaging a sample; pre-processing the obtained image; carrying out block processing on the pre-processed image to obtain a plurality of subblocks, and calculating the texture richness degree of each subblock; selecting a plurality of subblocks with the maximum texture richness degree as a focusing window; collecting an electron microscope image under the objective lens current, calculating the definition evaluation value of the electron microscope image, controlling the objective lens current of the electron microscope through a search algorithm, and positioning the best focus point in combination with the definition evaluation value. The focusing evaluation function provided by the application has excellent unimodality, and can effectively avoid the system from falling into local optimization during the automatic focusing process. Based on the unimodality advantage of the focusing evaluation function, the focusing process is divided into coarse focusing and fine focusing, so that the focusing efficiency is greatly improved, and the time consumption is extremely low. The focusing evaluation function and the focusing method have better effects on different electron microscope samples.

Description

Technical Field

[0001] This invention relates to the field of image analysis technology, and in particular to an autofocus method, electronic device, storage medium, and program product for electron microscope images. Background Technology

[0002] Scanning electron microscopes (SEMs) encounter various defocusing problems when photographing samples. Changes in the electron beam accelerating voltage alter the speed at which the electron beam passes through the objective lens coil. If the original current in the objective lens coil remains constant, the Lorentz force acting on the electron beam during its passage will lengthen or shorten, resulting in a shift of the focal plane in the electron microscope system, thus causing the observed sample image to become out of focus.

[0003] Current technologies for solving the above problems generally suffer from the following issues:

[0004] 1. The focus evaluation function is not unimodal, which will cause the focus to get stuck in a local optimum and make it impossible to find the true focal plane position.

[0005] 2. The selection of the focus window is not very adaptable and cannot be applied in all scenarios;

[0006] 3. The extensive traversal search method avoids the single-peak problem of the focus evaluation function to some extent, but the accuracy is directly proportional to the time consumption. To obtain better accuracy, it is necessary to spend more time, even longer than the manual focus time, which greatly reduces the user experience. Summary of the Invention

[0007] In order to achieve the above-mentioned objectives and other advantages of the present invention, a first objective of the present invention is to provide an autofocusing method for electron microscope images, comprising the following steps:

[0008] The current image is obtained by imaging the sample;

[0009] The acquired images are preprocessed;

[0010] The preprocessed image is divided into blocks to obtain several sub-blocks, and the texture richness of each sub-block is calculated.

[0011] Select the sub-blocks with the richest texture as the focus window;

[0012] Electron microscope images under objective current are acquired, and the sharpness evaluation value of the electron microscope images is calculated. The objective current of the electron microscope is controlled by a search algorithm, and the optimal focusing point is located by combining the sharpness evaluation value.

[0013] Furthermore, the preprocessing of the acquired image involves using a median filtering algorithm to remove salt-and-pepper noise from the acquired image.

[0014] Furthermore, the calculation of the texture richness of each sub-block is performed by calculating the entropy value of the gray-level co-occurrence matrix of each sub-block, as shown in the formula:

[0015] ,

[0016] in, Sub-block image coordinates Pixel value at that location, It is the entropy value;

[0017] The selection of the multiple sub-blocks with the highest texture richness as the focus window specifically refers to selecting the multiple sub-blocks with the highest entropy value as the focus window.

[0018] Furthermore, the step of controlling the objective current of the electron microscope through a search algorithm and locating the optimal focal point by combining the sharpness evaluation value includes using a variance evaluation function for coarse focusing.

[0019] Furthermore, the step of controlling the objective current of the electron microscope through a search algorithm and locating the optimal focal point by combining the sharpness evaluation value also includes using the MDGS evaluation function for fine focusing.

[0020] Furthermore, the coarse focusing using the variance evaluation function includes the following steps:

[0021] Set the step size, let , Used to record the number of times a local maximum is found, and to determine the search direction;

[0022] Acquire electron microscope images under the current objective current and calculate the current image sharpness. ;

[0023] Determine the sharpness of the current image Is the image sharpness greater than the previous frame? ;

[0024] If so, adjust the objective current in the original direction and jump to the step of acquiring the electron microscope image under the current objective current;

[0025] Otherwise, acquire two frames of images consecutively at the set step size, calculate the sharpness evaluation value for each frame, and obtain the first sharpness evaluation value. Second Sharpness Rating ;

[0026] Compare the current image sharpness, the first sharpness evaluation value, and the second sharpness evaluation value;

[0027] Determine the sharpness of the previous frame image Is it greater than the first sharpness evaluation value? And the clarity of the previous frame image Is it greater than the second sharpness evaluation value? ,and ;

[0028] If so, adjust the objective lens current in the opposite direction, so that ;

[0029] Otherwise, determine the sharpness of the previous frame image. Is it greater than the first sharpness evaluation value? And the clarity of the previous frame image Is it greater than the second sharpness evaluation value? ,and ;

[0030] If so, the search ends, and the resolution of the previous frame image is... The corresponding objective current is the search result;

[0031] Otherwise, if the objective current is locally optimal, it will be adjusted to the objective current corresponding to the maximum value between the first and second sharpness evaluation values, and the process will jump to the step of adjusting the objective current in the original direction.

[0032] Furthermore, the fine focusing using the MDGS evaluation function includes the following steps:

[0033] Set the step size, let , Used to record the number of times a local maximum is found, and to determine the search direction;

[0034] Acquire electron microscope images under the current objective current and calculate the current image sharpness. The calculation formula is:

[0035] ,

[0036] ,

[0037] ,

[0038] ,

[0039] ,

[0040] ,

[0041] in, To measure the corresponding pixel point within a pixel block of an image A local evaluation function for sharpness. This represents the number of pixels in each row of the image. This represents the number of pixels in each column of the image. The result of the image sharpness calculation. The side length of the selected pixel block in the image. Represents the corresponding pixel point within a pixel block. grayscale value, This represents the sum of the grayscale values ​​of the pixels within the currently calculated pixel block. This represents the sum of the grayscale values ​​of the pixels in the pixel block to the right of the current pixel block. This represents the sum of the grayscale values ​​of the pixels in the pixel block below the current pixel block. This represents the sum of the grayscale values ​​of the pixels within the current pixel block along the diagonal direction.

[0042] Determine the sharpness of the current image Is the image sharpness greater than the previous frame? ;

[0043] If so, adjust the objective current in the original direction and jump to the step of acquiring the electron microscope image under the current objective current;

[0044] Otherwise, if Then adjust the objective lens current in the opposite direction, so that... ;like The clarity of the previous frame image The corresponding objective current is the current search result. Set the step size to half of the original step size, and let... ;

[0045] Determine if the set step size is less than the search threshold;

[0046] If so, the search ends, and the resolution of the previous frame image is... The corresponding objective current is the search result;

[0047] Otherwise, proceed to the step of adjusting the objective lens current in the original direction.

[0048] A second objective of the present invention is to provide an electronic device comprising: a memory having program code stored thereon; and a processor connected to the memory, wherein when the program code is executed by the processor, the aforementioned method for autofocusing electron microscope images is implemented.

[0049] A third objective of this invention is to provide a computer-readable storage medium having program instructions stored thereon, which, when executed, implement the aforementioned method for autofocusing electron microscope images.

[0050] A fourth objective of this invention is to provide a computer program product, comprising a computer program / instruction, characterized in that, when the computer program / instruction is executed by a processor, it implements the aforementioned autofocusing method for electron microscope images.

[0051] Compared with the prior art, the beneficial effects of the present invention are:

[0052] This invention provides a real-time and fast automatic focusing method for electron microscopy images. By dividing the image into several sub-blocks, it adaptively acquires the areas with rich image details and uses these as focusing windows to evaluate sharpness. The optimal focusing point is located by combining a designed search algorithm with the sharpness evaluation value.

[0053] The focus evaluation function proposed in this invention exhibits excellent unimodality, effectively preventing the system from getting trapped in local optima during autofocus. Based on the superior unimodality of the focus evaluation function, the focusing process is divided into two stages: coarse focusing and fine focusing, significantly improving focusing efficiency and reducing time consumption. The focus evaluation function and focusing method proposed in this invention demonstrate excellent results for various electron microscope samples.

[0054] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it according to the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Specific embodiments of the present invention are given in detail below with reference to the accompanying drawings. Attached Figure Description

[0055] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0056] Figure 1 This is a flowchart of an autofocusing method for electron microscope images according to Example 1;

[0057] Figure 2 The image is a blurred electron microscope image;

[0058] Figure 3 Original image;

[0059] Figure 4 This is the filtered image;

[0060] Figure 5 This is a schematic diagram of the image sharpness evaluation function curves before and after filtering;

[0061] Figure 6 Select a schematic diagram to focus on the window;

[0062] Figure 7 This is a schematic diagram of image pixel blocks;

[0063] Figure 8 This is a schematic diagram of the electron microscope imaging principle.

[0064] Figure 9 The graph shows the results of the variance evaluation function test.

[0065] Figure 10 The graph shows the results of the MDGS evaluation function test.

[0066] Figure 11 This is a flowchart of the rough adjustment phase;

[0067] Figure 12 To refine the stage flowchart;

[0068] Figure 13 This is a block diagram of an electronic device.

[0069] Figure 14 This is a block diagram illustrating the principle of a computer-readable storage medium. Detailed Implementation

[0070] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0071] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0072] Example 1

[0073] An autofocusing method for electron microscope images, such as Figure 1 As shown, it includes the following steps:

[0074] By imaging the sample, the current image is obtained, such as... Figure 2 As shown.

[0075] During autofocus, to meet real-time requirements, the image scanning speed is set relatively fast. This inevitably increases salt-and-pepper noise in the image. These alternating black and white highlights and shadows negatively impact the sharpness evaluation results. Figure 3 As shown, the acquired image needs to be preprocessed. Specifically, a median filtering algorithm is used to remove salt-and-pepper noise from the acquired image, such as... Figure 4 As shown, this involves calculating the median value of the eight neighbors of a pixel in a digital image, replacing the original value of that pixel with this median value, and then calculating the sharpness evaluation function curve after filtering out salt-and-pepper noise, as shown in the figure. Figure 5 As shown. Figure 5The upper line represents the sharpness evaluation function curve of the original image, while the lower line represents the sharpness evaluation function curve of the filtered image. The local minima in the rectangular region disappear after filtering, which to some extent eliminates the risk of subsequent searches getting trapped in local optima.

[0076] The filtered image is divided into blocks to obtain several sub-blocks, such as dividing the image into... The sub-blocks are then processed. The texture richness of each sub-block is calculated; in this embodiment, the entropy value of the 16-level gray-level co-occurrence matrix of each sub-block is calculated, such as... Figure 6 As shown, the formula used to describe the texture richness of a sub-block is:

[0077] ,

[0078] in, Sub-block image coordinates Pixel value at that location, Entropy is the value of entropy; entropy represents a measure of the randomness of the information contained in an image. Entropy is at its maximum when all values ​​in the co-occurrence matrix are equal or when pixel values ​​exhibit maximum randomness. In other words, the more complex the grayscale distribution of an image and the more information it contains, the greater the entropy value.

[0079] This embodiment performs uniform sampling on the entire image, rather than sampling within a certain window, thus preserving the overall grayscale distribution characteristics of the image. Multiple sub-blocks (e.g., five sub-blocks) with the highest entropy values ​​are selected as the focusing window.

[0080] Electron microscopy (EM) images are acquired under objective current, and the sharpness evaluation value of the EEM images is calculated. The objective current of the EEM is controlled using a search algorithm, and the optimal focusing point is located based on the sharpness evaluation value. In this embodiment, a multi-directional gradient square function (MDGS) is proposed for calculating the sharpness evaluation function value of the focusing window. The calculation formula is as follows:

[0081] ,

[0082] in, The expression is:

[0083] ,

[0084] in, This represents the number of pixels in each row of the image. This represents the number of pixels in each column of the image. Representing an image Corresponding pixel grayscale value, This represents the result of the image sharpness calculation.

[0085] In electron microscopy imaging, the greater the distance from the focal point, the larger the area of ​​blur on the image, and consequently, the larger the region with similar gray values. Based on this principle, pixel blocks are used as pixels in the sharpness evaluation function, and downsampling is employed in the calculation. Figure 7 As shown. The calculation expression for a single image pixel block is as follows:

[0086] ,

[0087] ,

[0088] ,

[0089] ,

[0090] in, The side length of the selected pixel block in the image. Represents the corresponding pixel point within a pixel block. grayscale value, It represents the sum of the grayscale values ​​of the pixels within the currently calculated pixel block. It represents the sum of the grayscale values ​​of the pixels in the pixel block to the right of the current pixel block. It represents the sum of the grayscale values ​​of the pixels in the pixel block below the current pixel block. This represents the sum of the pixel grayscale values ​​within the current pixel block along its diagonal direction. Combined with the improved pixel block... Represented as:

[0091] .

[0092] like Figure 8 As shown, the focusing principle of an electron microscope is as follows: the objective lens focuses the electron beam onto the sample surface. The distance between the focusing plane and the objective lens is defined as the working distance, which is equivalent to the focal length of the electron microscope. The position near the peak point of the sharpness evaluation function, i.e., the area where the image remains sharp, is called the depth of field. A sharp image is obtained by controlling the objective lens current of the electron microscope through a search algorithm. The search process is divided into two stages: coarse adjustment and fine adjustment. In the coarse adjustment stage, the classic variance evaluation function is used, such as... Figure 9 As shown; this function has high real-time performance and good noise resistance. During the fine-tuning stage, the proposed MDGS evaluation function is used, such as... Figure 10 As shown, this function has high real-time performance and sensitivity.

[0093] like Figure 11 As shown, coarse focusing using the variance evaluation function includes the following steps:

[0094] Set the step size, let , Used to record the number of times a local maximum is found, and to determine the search direction;

[0095] Acquire electron microscope images under the current objective current and calculate the current image sharpness. ;

[0096] Determine the sharpness of the current image Is the image sharpness greater than the previous frame? ;

[0097] If so, adjust the objective current in the original direction and jump to the step of acquiring the electron microscope image under the current objective current;

[0098] After obtaining results through the hill-climbing search algorithm in the coarse-tuning phase, a verification phase is added. This verification phase is mainly to prevent the search from getting trapped in local optima. When the sharpness evaluation curve reaches an extreme point, the search process does not immediately end, but instead adjusts the current objective lens current value. Corresponding sharpness rating Save the file and continue searching forward, comparing it with the previous entries. Clarity rating per step Is it greater than .like This indicates the objective lens current. The corresponding position is a local optimum; if This indicates the objective lens current. The corresponding position is the optimal focal point. The specific steps are as follows:

[0099] Otherwise, acquire two frames of images consecutively at the set step size, calculate the sharpness evaluation value for each frame, and obtain the first sharpness evaluation value. Second Sharpness Rating ;

[0100] Compare the current image sharpness with the first sharpness rating and the second sharpness rating;

[0101] Determine the sharpness of the previous frame. Is it greater than the first sharpness rating value? And the clarity of the previous frame image Is it greater than the second sharpness rating? ,and ;

[0102] If so, adjust the objective lens current in the opposite direction, so that ;

[0103] Otherwise, judge the sharpness of the previous frame. Is it greater than the first sharpness rating value? And the clarity of the previous frame image Is it greater than the second sharpness rating? ,and ;

[0104] If yes, the search ends; the previous frame's image clarity... The corresponding objective current is the search result;

[0105] Otherwise, if the objective current is locally optimal, it will be adjusted to the objective current corresponding to the maximum value between the first and second sharpness evaluation values, and the process will jump to the step of adjusting the objective current in the original direction.

[0106] After the coarse focusing process is completed, the fine focusing process begins. For example... Figure 12 As shown, fine focusing using the MDGS evaluation function includes the following steps:

[0107] Set the step size, let , Used to record the number of times a local maximum is found, and to determine the search direction;

[0108] Acquire electron microscope images under the current objective current and calculate the current image sharpness. The calculation formula is:

[0109] ,

[0110] ,

[0111] ,

[0112] ,

[0113] ,

[0114] ,

[0115] in, To measure the corresponding pixel point within a pixel block of an image A local evaluation function for sharpness. This represents the number of pixels in each row of the image. This represents the number of pixels in each column of the image. The result of the image sharpness calculation. The side length of the selected pixel block in the image. Represents the corresponding pixel point within a pixel block. grayscale value, This represents the sum of the grayscale values ​​of the pixels within the currently calculated pixel block. This represents the sum of the grayscale values ​​of the pixels in the pixel block to the right of the current pixel block. This represents the sum of the grayscale values ​​of the pixels in the pixel block below the current pixel block. This represents the sum of the grayscale values ​​of the pixels within the current pixel block along the diagonal direction.

[0116] Determine the sharpness of the current image Is the image sharpness greater than the previous frame? ;

[0117] If so, adjust the objective current in the original direction and jump to the step of acquiring the electron microscope image under the current objective current;

[0118] Otherwise, if Then adjust the objective lens current in the opposite direction, so that... ;

[0119] In the fine-tuning stage, a variable step size search method is used. Because the depth of field of a scanning electron microscope (SEM) is short, even slight changes in the objective lens current can cause image blurring. Therefore, using a fixed step size that is too long can easily miss the optimal focusing point, while using a fixed step size that is too short will lead to more searches and increase time consumption. Therefore, a binary variable step size method is used to define the step size in the fine-focusing stage. Assuming the current step size... The stop search threshold is set according to the depth of field range. After finding the optimal focal point, the step size will be adjusted. Set to Continue the search, iterating until... Stop the search and return to the optimal focal point location. The specific steps are as follows:

[0120] like The clarity of the previous frame image The corresponding objective current is the current search result. Set the step size to half of the original step size, i.e., step = step / 2. ;

[0121] Determine if the set step size is less than the search threshold;

[0122] If yes, the search ends; the previous frame's image clarity... The corresponding objective current is the search result;

[0123] Otherwise, proceed to the step of adjusting the objective current in the original direction.

[0124] Example 2

[0125] An electronic device 200, such as Figure 13As shown, including but not limited to: a memory 201 storing program code; and a processor 202 connected to the memory, which, when the program code is executed by the processor, implements an autofocusing method for electron microscope images.

[0126] Example 3

[0127] A computer-readable storage medium, such as Figure 14 As shown, it stores program instructions, which, when executed, implement an autofocusing method for electron microscope images.

[0128] Example 4

[0129] A computer program product includes a computer program / instructions, characterized in that the computer program / instructions, when executed by a processor, implement an autofocusing method for electron microscope images.

[0130] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0131] The above description is merely an embodiment of this specification and is not intended to limit the scope of one or more embodiments of this specification. For those skilled in the art, various modifications and variations can be made to one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of one or more embodiments of this specification.

Claims

1. An autofocusing method for electron microscope images, characterized in that, Includes the following steps: The current image is obtained by imaging the sample; The acquired images are preprocessed; The preprocessed image is divided into blocks to obtain several sub-blocks, and the texture richness of each sub-block is calculated. Select the sub-blocks with the richest texture as the focus window; Electron microscope images under objective current are acquired, the sharpness evaluation value of the electron microscope images is calculated, the objective current of the electron microscope is controlled by a search algorithm, and the optimal focal point is located by combining the sharpness evaluation value; The method of controlling the objective current of the electron microscope through a search algorithm and locating the optimal focal point by combining the sharpness evaluation value includes using a variance evaluation function for coarse focusing. The coarse focusing using the variance evaluation function includes the following steps: Set the step size, let , Used to record the number of times a local maximum is found, and to determine the search direction; Acquire electron microscope images under the current objective current and calculate the current image sharpness. ; Determine the sharpness of the current image Is the image sharpness greater than the previous frame? ; If so, adjust the objective current in the original direction and jump to the step of acquiring the electron microscope image under the current objective current; Otherwise, acquire two frames of images consecutively at the set step size, calculate the sharpness evaluation value for each frame, and obtain the first sharpness evaluation value. Second Sharpness Rating ; Compare the current image sharpness, the first sharpness evaluation value, and the second sharpness evaluation value; Determine the sharpness of the previous frame image Is it greater than the first sharpness evaluation value? And the clarity of the previous frame image Is it greater than the second sharpness evaluation value? ,and ; If so, adjust the objective lens current in the opposite direction, so that ; Otherwise, determine the sharpness of the previous frame image. Is it greater than the first sharpness evaluation value? And the clarity of the previous frame image Is it greater than the second sharpness evaluation value? ,and ; If so, the search ends, and the resolution of the previous frame image is... The corresponding objective current is the search result; Otherwise, if the objective current is locally optimal, it will be adjusted to the objective current corresponding to the maximum value between the first and second sharpness evaluation values, and the process will jump to the step of adjusting the objective current in the original direction.

2. The method for autofocusing electron microscope images according to claim 1, characterized in that: The preprocessing of the acquired image involves using a median filtering algorithm to remove salt-and-pepper noise from the acquired image.

3. The method for autofocusing electron microscope images according to claim 1, characterized in that: The calculation of the texture richness of each sub-block is performed by calculating the entropy value of the gray-level co-occurrence matrix of each sub-block, using the following formula: ， in, Sub-block image coordinates Pixel value at that location, It is the entropy value; The selection of the multiple sub-blocks with the highest texture richness as the focus window specifically refers to selecting the multiple sub-blocks with the highest entropy value as the focus window.

4. The method for autofocusing electron microscope images according to claim 1, characterized in that, The method of controlling the objective current of the electron microscope through a search algorithm and locating the optimal focal point by combining the sharpness evaluation value also includes using a multi-directional gradient square function for fine focusing.

5. The method for autofocusing electron microscope images according to claim 4, characterized in that, The fine focusing using a multi-directional gradient squared function includes the following steps: Set the step size, let , Used to record the number of times a local maximum is found, and to determine the search direction; Acquire electron microscope images under the current objective current and calculate the current image sharpness. The calculation formula is: ， ， ， ， ， ， in, To measure the corresponding pixel point within a pixel block of an image A local evaluation function for sharpness. This represents the number of pixels in each row of the image. This represents the number of pixels in each column of the image. The result of the image sharpness calculation. The side length of the selected pixel block in the image. Represents the corresponding pixel point within a pixel block. grayscale value, This represents the sum of the grayscale values ​​of the pixels within the currently calculated pixel block. This represents the sum of the grayscale values ​​of the pixels in the pixel block to the right of the current pixel block. This represents the sum of the grayscale values ​​of the pixels in the pixel block below the current pixel block. This represents the sum of the grayscale values ​​of the pixels within the current pixel block along the diagonal direction. Determine the sharpness of the current image Is the image sharpness greater than the previous frame? ; If so, adjust the objective current in the original direction and jump to the step of acquiring the electron microscope image under the current objective current; Otherwise, if Then adjust the objective current in the opposite direction, so that ;like The clarity of the previous frame image The corresponding objective current is the current search result. Set the step size to half of the original step size, and let... ; Determine if the set step size is less than the search threshold; If so, the search ends, and the resolution of the previous frame image is... The corresponding objective current is the search result; Otherwise, proceed to the step of adjusting the objective lens current in the original direction.

6. An electronic device, characterized in that, include: A memory that stores program code; A processor, which is connected to the memory, and which, when the program code is executed by the processor, implements the method as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, It stores program instructions that, when executed, implement the method as described in any one of claims 1 to 5.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the method as described in any one of claims 1 to 5.

Images