Method, device and storage medium for generating image set in scanning electron microscope

Abstract

The present application relates to the technical field of scanning electron microscopes, and discloses a method, device, and storage medium for generating an image set in a scanning electron microscope. The method comprises: using a scanning electron microscope to collect multiple electron microscope images of a sample to be tested according to target image shooting conditions to form an initial image set; calculating at least one reference index for each electron microscope image in the initial image set; screening abnormal images in the initial image set based on the reference index; performing grayscale histogram matching on the electron microscope images in the image set after screening abnormal images to obtain a target image set; and calculating the average image of the target image set to generate a final image set. Using the scheme of the present application, a high-quality noise-free image set that is close to reality can be obtained, thereby improving the usability of deep learning in the field of SEM image denoising.

Description

Technical Field

[0001] The present application generally relates to the field of scanning electron microscopy technology. More specifically, the present application relates to a method, an apparatus and a computer-readable storage medium for generating an image set in a scanning electron microscope. Background Art

[0002] Scanning Electron Microscope (SEM) is a microscopic technique that uses an electron beam to scan the surface of a sample. Because it can provide high-resolution surface morphology images, SEM has become an indispensable tool in scientific research and industrial inspection. In practical applications, SEM images are often interfered by various noises, which may come from the equipment itself, the sample preparation process, the instability of the electron beam and other factors. Image denoising can significantly improve the quality of SEM images, reduce the residence time of the electron beam during shooting, save time costs, and the denoised image is convenient for more accurate image segmentation and feature extraction and other downstream tasks, making image denoising occupy an important position in SEM image processing.

[0003] Traditional SEM image denoising includes a variety of methods, such as traditional filtering techniques (such as Gaussian filtering, median filtering) and denoising based on wavelet transform, etc., and these denoising methods will blur image details while denoising. In contrast, deep learning models can effectively remove complex and nonlinear noise while maintaining image details, without the need to redesign or adjust algorithm parameters for each specific noise type, and are widely used in a variety of SEM images. However, the dependence of deep learning methods on data is one of their main limitations. It is usually costly to obtain a large amount of high-quality data, especially in the field of SEM images that require professional equipment or complex programs to generate data. In addition, the data annotation process usually requires a lot of manual participation, which further increases the cost. Furthermore, in the field of SEM, the image shooting process is easily affected by the actual machine shooting capabilities and conditions, and the existing denoising methods cannot fully cover the specific problems encountered, so they are fully applicable to SEM image acquisition. Therefore, how to generate SEM denoised image sets, reduce worker costs, and better apply them to deep learning has become a technical problem that needs to be solved.

[0004] In view of this, there is an urgent need to provide a solution for generating image sets in scanning electron microscopes, so as to obtain high-quality noise-free image sets that are close to reality and improve the usability of deep learning in the field of SEM image denoising. Summary of the invention

[0005] In order to at least solve one or more of the above-mentioned technical problems, the present application proposes a solution for generating an image set in a scanning electron microscope in multiple aspects.

[0006] In a first aspect, the present application provides a method for generating an image set in a scanning electron microscope, comprising: using a scanning electron microscope to collect multiple electron microscope images of a sample to be tested according to target imaging conditions to form an initial image set; calculating at least one reference index for each electron microscope image in the initial image set; screening abnormal images in the initial image set based on the reference index; determining a registration reference image and a cropping reference image according to the image offset of the electron microscope image in the image set after screening the abnormal image; performing image cropping based on whether a first offset determined by the registration between the electron microscope image in the image set after screening the abnormal image and the registration reference image exceeds a buffer pixel range to obtain an initial cropped image; and,

[0007] Based on the size relationship between the second offset determined by the registration between the initial cropped image and the cropped reference image and the offset threshold, a final cropped image is determined; grayscale histogram matching is performed on the final cropped image in the image set after the abnormal images are filtered out to obtain a target image set; and an average image of the target image set is calculated to generate a final image set.

[0008] In some embodiments, the target image capturing conditions include at least one or more of the capturing point, horizontal field of view width, brightness contrast, number of shots or image size.

[0009] In other embodiments, the reference indicators include at least one or more of image offset, image grayscale mean, image grayscale maximum, image grayscale minimum, image grayscale range, Pearson correlation coefficient, image structure similarity or brightness contrast excess value.

[0010] In some other embodiments, screening abnormal images in the initial image set based on the reference indicator includes: calculating an abnormal screening score based on the reference indicator; and eliminating electron microscope images whose abnormal screening scores do not meet a score threshold range from the initial image set, so as to screen abnormal images in the initial image set.

[0011] In some other embodiments, the abnormal screening score includes a first abnormal screening score and a second abnormal screening score, and removing electron microscopy images whose abnormal screening scores do not meet a score threshold range from the initial image set to screen abnormal images in the initial image set includes: removing electron microscopy images whose first abnormal screening score and / or the second abnormal screening score do not meet a corresponding score threshold range from the initial image set to screen abnormal images in the initial image set.

[0012] In yet other embodiments, the first abnormality screening score comprises a z-score, and the second abnormality screening score comprises an interquartile range score.

[0013] In some other embodiments, image cropping is performed based on whether the first offset determined by the registration between the electron microscope image in the image set after filtering out the abnormal images and the registration reference image exceeds the buffer pixel range to obtain the initial cropped image, which includes: in response to the first offset determined by the registration between the electron microscope image in the image set after filtering out the abnormal images and the registration reference image not exceeding the buffer pixel range, image cropping is performed on the electron microscope image in the image set after filtering out the abnormal images to obtain the initial cropped image.

[0014] In some further embodiments, determining the final cropped image based on the relationship between the second offset determined by the registration between the initial cropped image and the cropped reference image and an offset threshold includes: in response to the second offset determined by the registration between the initial cropped image and the cropped reference image being less than or equal to the offset threshold, determining the initial cropped image as the final cropped image.

[0015] In some other embodiments, it also includes: removing electron microscope images that have not achieved image registration and image cropping from the image set after abnormal images are screened; and manually screening abnormal images from the remaining electron microscope images based on corresponding reference indicators.

[0016] In some further embodiments, it also includes: determining the number of images of the final cropped image; and in response to the number of images not satisfying a quantity threshold, recalculating the reference index based on the screened out abnormal images to restore the erroneously screened out abnormal images.

[0017] In some other embodiments, grayscale histogram matching is performed on the final cropped image in the image set after filtering out abnormal images, and obtaining the target image set includes: calculating the grayscale mean of the final cropped image in the image set after filtering out abnormal images; selecting a grayscale reference image based on the grayscale mean; and performing grayscale histogram matching on the electron microscope image whose difference between the final cropped image in the image set after filtering out abnormal images and the grayscale reference image is greater than a grayscale threshold, so as to obtain the target image set.

[0018] In some further embodiments, calculating the average image of the target image set to generate the final image set includes: selecting a cropped image with the smallest offset distance under a target number from the target image set; and calculating the average image based on the cropped image with the smallest offset distance under the target number to generate the final image set.

[0019] In a second aspect, the present application provides a device for generating an image set in a scanning electron microscope, comprising: a processor; and a memory, in which program instructions for generating an image set in a scanning electron microscope are stored, and when the program instructions are executed by the processor, the device implements one or more embodiments of the first aspect.

[0020] In a third aspect, the present application provides a computer-readable storage medium having stored thereon computer-readable instructions for generating an image set in a scanning electron microscope, wherein when the computer-readable instructions are executed by one or more processors, one or more embodiments of the first aspect are implemented.

[0021] By the scheme for generating an image set in a scanning electron microscope as provided above, the embodiment of the present application performs abnormal image screening by calculating at least one reference index of each electron microscope image in the initial image set collected, and performs image registration, cropping, grayscale histogram matching on the electron microscope images in the image set after screening the abnormal images, and calculates the average map. Based on this, the construction error of the image set can be alleviated to the greatest extent, and a high-quality noise-free image set that is approximately true can be obtained, thereby improving the availability of deep learning in the field of scanning electron microscope image denoising. At the same time, the computational efficiency is improved, and the time cost and labor cost are saved. Further, the embodiment of the present application also combines manual abnormal image screening, abnormal image restoration, and the average map under the calculation target number, etc., which greatly improves the utilization rate of electron microscope images, solves the problems of scarcity of public data in the field of scanning electron microscope image denoising, insufficient scene diversity, and low quality of data labels, and has a high engineering application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become easy to understand. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0023] Figure 1 is an exemplary flowchart showing a method for generating an image set in a scanning electron microscope according to an embodiment of the present application;

[0024] Figure 2 is an exemplary flowchart showing a method of screening abnormal images in an initial image set according to an embodiment of the present application;

[0025] Figure 3 is an exemplary flowchart showing image registration and image cropping according to an embodiment of the present application;

[0026] Figure 4is an exemplary flowchart showing grayscale histogram matching according to an embodiment of the present application;

[0027] Figure 5 is an exemplary flowchart showing the process of calculating the average image of the target image set to generate the final image set according to an embodiment of the present application;

[0028] Figure 6 is an exemplary flowchart showing an overall process for generating an image set in a scanning electron microscope according to an embodiment of the present application;

[0029] Figure 7 is an exemplary schematic diagram showing a result graph obtained according to an embodiment of the present application and a result graph obtained by adjusting hardware;

[0030] Fig. 8A is an exemplary schematic diagram showing a result graph of different times of calculation under different samples according to an embodiment of the present application;

[0031] Figure 8B is an exemplary schematic diagram showing original images of different samples and corresponding result images according to an embodiment of the present application;

[0032] Fig. 9 is an exemplary structural block diagram showing a device for generating an image set in a scanning electron microscope according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0034] It should be understood that the terms "include" and "comprising" used in the specification and claims of the present application indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0035] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this application specification and claims, unless the context clearly indicates otherwise, the singular forms of "a", "an" and "the" are intended to include plural forms. It should also be further understood that the term "and / or" used in this application specification and claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0036] As used in this specification and claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0037] The specific implementation of the present application is described in detail below with reference to the accompanying drawings.

[0038] Figure 1 FIG. 1 is an exemplary flowchart of a method 100 for generating an image set in a scanning electron microscope according to an embodiment of the present application. Figure 1 As shown in , at step S101, a scanning electron microscope is used to collect multiple electron microscope images of the sample to be tested according to the target image capturing conditions to form an initial image set. In some embodiments, the target image capturing conditions may include but are not limited to one or more of the shooting point, horizontal field of view width, brightness contrast, number of shots, or image size.

[0039] Specifically, there should be significant changes in the shooting points and horizontal field of view width ("HFW") of the sample to be tested to ensure the diversity of the electron microscope images. In some implementation scenarios, multiple electron microscope images collected at the same shooting point and horizontal field of view width of the sample to be tested can constitute an image group (or image group), that is, an initial image set. As an example, the aforementioned horizontal field of view width can be, for example, a horizontal field of view width of 25um, 50um or 100um, and the present application does not impose any restrictions in this regard. In some implementation scenarios, the brightness contrast can be adjusted to minimize the number of 0s and 255s in the grayscale histogram to avoid excessive pixel cropping in subsequent image cropping, which may lead to abnormal calculation of the average image.

[0040] In other implementation scenarios, the aforementioned number of shots may exceed about 50% of the highest average number. For example, if the maximum number of electron microscope images for the subsequent calculation of the average image is 80, then the number of electron microscope images shot is not less than 120. In other implementation scenarios, in order to enhance the robustness against image pixel drift (that is, the pixels are offset in the x-direction and y-direction), the size of the electron microscope image shot is slightly larger than the image size actually required. As an example, if the size of the electron microscope image actually required to be generated is 4096×4096, a buffer pixel range of, for example, 100 can be reserved around, that is, the size of the electron microscope image shot is 4296×4296. In addition, in some other implementation scenarios, the aforementioned target shooting conditions may also include focusing and adjusting other parameters, locking the machine, etc., and ensuring that there are no jittery edges, object distortion, movement or telescoping in the electron microscope image. During the shooting process, slight blur is allowed, but excessive blur is avoided to aggravate the error of subsequent image registration and avoid abnormalities such as black spots and distortion in the initial reference image, and it does not produce excessive displacement with the two adjacent images.

[0041] Based on the initial image set collected above, at step S102, at least one reference index of each electron microscope image in the initial image set is calculated. In some embodiments, the aforementioned reference index may include but is not limited to one or more of image offset (including x-direction displacement offset and y-direction displacement offset), image grayscale mean, image grayscale maximum value, image grayscale minimum value, image grayscale range value, Pearson correlation coefficient, image structure similarity, or brightness contrast overlimit value.

[0042] Next, at step S103, the abnormal images in the initial image set are screened based on the reference index. In some embodiments, the abnormal screening score can be calculated based on the reference index, and the electron microscope images whose abnormal screening scores do not meet the score threshold range are removed from the initial image set to screen the abnormal images in the initial image set. In some implementation scenarios, the aforementioned abnormal screening score may include a first abnormal screening score and a second abnormal screening score. Specifically, in screening abnormal images, the electron microscope images whose first abnormal screening score and / or the second abnormal screening score do not meet the corresponding score threshold range are removed from the initial image set to screen the abnormal images in the initial image set. In some embodiments, the aforementioned first abnormal screening score includes a z-score ("z-score"), and the aforementioned second abnormal screening score includes an interquartile range score ("IQR").

[0043] That is to say, the embodiment of the present application first calculates the z score and the interquartile range score according to the reference index of each electron microscope image, and when one or both of the z score and the interquartile range score do not meet the corresponding score threshold range, the corresponding electron microscope image is removed as an abnormal image. When the z score and the interquartile range score both meet the corresponding score threshold range, the corresponding electron microscope image is retained. In some implementation scenarios, the score threshold range corresponding to the z score can be, for example, less than or equal to 3, and the score threshold range corresponding to the interquartile range score can be, for example, the upper quartile of the upper boundary of the image plus 1.5 times IQR and the lower quartile of the lower boundary of the image minus 1.5 times IQR. For example, in an exemplary scenario, when the z score of the electron microscope image is greater than the threshold value 3 and / or the upper and lower quartile scores of the upper and lower boundaries of the electron microscope image do not meet the addition of 1.5 times IQR and minus 1.5 times IQR, the corresponding electron microscope image is removed as an abnormal image. Otherwise, the corresponding electron microscope image is retained.

[0044] It is understood that the aforementioned score threshold range is merely exemplary and non-limiting and can be adjusted according to the actual needs of generating an image set. In addition, regarding the above-mentioned multiple reference indicators, the image offset, image grayscale mean, Pearson correlation coefficient and image structure similarity can be used to calculate the z-score and interquartile range score for abnormal image screening, while the image grayscale maximum value, image grayscale minimum value, image grayscale range value and brightness contrast over-limit value can be used for subsequent manual screening of abnormal images.

[0045] In some embodiments, image registration and image cropping can also be performed on the electron microscope images in the image set after the abnormal images are filtered out to ensure pixel-level matching of multiple electron microscope images in the same scene. Specifically, at step S104, the registration reference image and the cropping reference image are determined according to the image offset of the electron microscope images in the image set after the abnormal images are filtered out, and at step S105, image cropping is performed based on whether the first offset determined by the registration between the electron microscope images in the image set after the abnormal images are filtered out and the registration reference image exceeds the buffer pixel range to obtain an initial cropped image, and at step S106, the final cropped image is determined based on the size relationship between the second offset determined by the registration between the initial cropped image and the cropped reference image and the offset threshold.

[0046] More specifically, in response to the first offset determined by the registration between the electron microscope image in the image set after the abnormal image is screened and the registration reference image does not exceed the buffer pixel range, the electron microscope image in the image set after the abnormal image is screened is cropped to obtain an initial cropped image. In response to the second offset determined by the registration between the initial cropped image and the cropped reference image being less than or equal to an offset threshold, the initial cropped image is determined as the final cropped image. In some embodiments, the buffer pixel range is a pixel range that exceeds the actual image size, such as the buffer pixel range 100 mentioned above. The aforementioned offset threshold may be, for example, 1.

[0047] In some embodiments, the electron microscope images that have not achieved image registration and image cropping can also be eliminated from the image set after the abnormal images are screened, so as to manually screen the abnormal images in the remaining electron microscope images based on the corresponding reference indicators. That is, after eliminating the images that have not been successfully registered and cropped, the abnormal images are manually screened. In some implementation scenarios, for example, a picture viewer can be used to directly traverse and output the image set that is successfully matched and cropped, and the abnormalities such as jitter, distortion and black spots that appear during the quick browsing process can be clearly captured by the naked eye. This is because the ideal image set has only random noise attached to the surface "jumping" on the scene, and the scene should not have any displacement or jitter changes, but a small scene displacement can be allowed. Further, it can be combined with parameter indicators such as the maximum value of image grayscale, the minimum value of image grayscale, the image grayscale range value, and the brightness contrast over-limit value, and the parameter indicators that do not meet the requirements are manually screened out. After the abnormal images, the aforementioned image registration and image cropping are performed. In some implementation scenarios, the aforementioned registration can be achieved by a method such as fast Fourier transform.

[0048] In some embodiments, based on the cropped images, the number of images of the final cropped images can be determined, and in response to the number of images not meeting the number threshold, the reference index is recalculated based on the screened abnormal images to restore the abnormal images that were screened out incorrectly. In other words, when the number of images after cropping cannot meet, for example, the maximum average number of images or exceeds it by a small amount, in order to increase the utilization rate of the images, the abnormal images that were misjudged can be re-performed with the aforementioned image registration and cropping operations, and the successfully cropped images can be added to the image set.

[0049] Further, at step S107, the final cropped image in the image set after the abnormal image is filtered out is matched with a grayscale histogram to obtain a target image set. In some embodiments, the grayscale mean of the final cropped image in the image set after the abnormal image is filtered out can be calculated, and a grayscale reference image can be selected according to the grayscale mean, and then the electron microscope image whose difference between the final cropped image in the image set after the abnormal image is filtered out and the grayscale reference image is greater than the grayscale threshold is matched with a grayscale histogram to obtain a target image set. In some implementation scenarios, the grayscale reference image can be selected according to the mean extreme difference of the grayscale mean. As an example, a grayscale center image can be selected as a grayscale reference image based on a cropped image whose mean extreme difference is greater than 1. Then, the final cropped image is traversed, and a grayscale histogram matching is performed on the cropped image whose difference with the grayscale reference image is greater than the grayscale threshold (for example, 0.5), so that the grayscale of the overall image is close to the grayscale reference image, so as to calibrate the grayscale mean and contrast in the image.

[0050] After obtaining the target image set, at step S108, the average map of the target image set is calculated to generate the final image set. In some embodiments, the cropped image with the smallest offset distance under the target number can be selected from the target image set, and then the average map is calculated based on the cropped image with the smallest offset distance under the target number to generate the final image set. In an exemplary scenario, the aforementioned cropped image with the smallest offset distance can be, for example, a cropped image with the smallest pixel offset (i.e., the Euclidean distance calculated from the origin in the x and y directions). In some implementation scenarios, the average map under different target numbers can be calculated and saved.

[0051] In combination with the above description, it can be known that the embodiment of the present application screens the abnormal images in the initial image set by screening based on parameter indicators, and aligns and cuts the electron microscope images in the image set after the abnormal images need to be screened, and the grayscale histogram is matched and the average graph is calculated. Thus, the construction error of the image set can be alleviated, and an image set containing high-quality electron microscope images can be obtained. In addition, the embodiment of the present application can also save the electron microscope images averaged at different times to obtain more abundant noise level data. It can be used as a data benchmark for image denoising in deep learning later, further enriching the data diversity in the field of image denoising.

[0052] Figure 2 FIG. 1 is an exemplary flowchart showing the process of screening abnormal images in an initial image set according to an embodiment of the present application. It should be understood that Figure 2 is the above Figure 1 is a specific embodiment of step S103 in method 100, so the above Figure 1 The description also applies to Figure 2 .

[0053] like Figure 2As shown in, at step S201, at least one reference index of each electron microscope image in the initial image set is calculated. As can be seen from the foregoing, parameter indices may include multiple ones, among which, for example, image offset, image grayscale mean, Pearson correlation coefficient and image structure similarity can be used to calculate z-scores and interquartile range scores to achieve automatic screening of abnormal images. Specifically, for calculating the Z-score, at step S202, the mean and standard deviation of each electron microscope image under the corresponding parameter index are calculated. Then, at step S203, the aforementioned mean and standard deviation are normalized to obtain the Z-score, and then at step S204, the electron microscope images corresponding to the absolute values ​​greater than 3 in the normalized data are eliminated. That is, the electron microscope images whose z-scores do not meet the corresponding score threshold range are eliminated.

[0054] On the other hand, at step S205, the upper and lower quartile scores of each electron microscope image under the corresponding parameter index are calculated, and at step S206, the IQR value is obtained according to the upper and lower quartile scores. In some implementation scenarios, the upper quantile of the upper boundary of the image plus 1.5 times the IQR and the lower quantile of the lower boundary of the image minus 1.5 times the IQR are set as the corresponding score threshold ranges. Thus, at step S207, the electron microscope images whose upper and lower quartile scores of the upper and lower boundaries of the electron microscope image do not satisfy the corresponding values ​​of adding 1.5 times the IQR and minus 1.5 times the IQR are eliminated, so as to obtain abnormal images in the screened initial image set at step S208. As mentioned above, in order to ensure pixel-level matching of multiple electron microscope images in the same scene, the embodiment of the present application can also perform image registration and image cropping on the electron microscope images in the image set after the abnormal images are screened. The following will be combined with Figure 3 The aforementioned image registration and image cropping operations are described in detail.

[0055] Figure 3 FIG. 1 is an exemplary flowchart showing image registration and image cropping according to an embodiment of the present application. It should be understood that Figure 3 is the above Figure 1 Another specific embodiment of the method 100, therefore, the above Figure 1 The description also applies to Figure 3 .

[0056] like Figure 3As shown in, at step S301, a pixel drift index is determined according to the image offset. That is, the offset of the pixel in the x-direction and the y-direction. Based on the pixel drift index, at step S302, the index of the drift center image is calculated using the pixel drift index to obtain a registration reference image and a cropped reference image. Then, at step S303, the first offset between the electron microscope image after filtering abnormal images and the registration reference image is calculated. At step S304, it is determined whether the first offset exceeds the buffer pixel range. If it exceeds the buffer pixel range, at step S305, the image is discarded. If it does not exceed the buffer pixel range, at step S306, the image is cropped to obtain an initial cropped image.

[0057] Based on the initial cropped image obtained, at step S307, a second offset is calculated based on the initial cropped image and the cropped reference image, and at step S308, it is determined whether the second offset is greater than 1. If the second offset is greater than 1, jump to step S305 and discard the image. If the second offset is less than or equal to 1, at step S309, the cropped image is retained. After traversing all electron microscope images without abnormalities, the figure further shows that at step S310, it is determined whether the number of images of the cropped image meets the quantity threshold. If satisfied, at step S311, the final cropped image is obtained. If not satisfied, at step S312, the abnormal image that was mistakenly screened is restored, and then adjusted to the aforementioned step S301, and the aforementioned operation is repeated until the number of images of the cropped image meets the quantity threshold and stops.

[0058] Figure 4 is an exemplary flowchart showing grayscale histogram matching according to an embodiment of the present application. It should be understood that: Figure 4 is the above Figure 1 is a specific embodiment of step S104 in method 100, so the above Figure 1 The description also applies to Figure 4 .

[0059] like Figure 4 As shown in, at step S401, the final cropped image is traversed to calculate the grayscale mean. Then, at step S402, it is determined whether the mean range is greater than 1. If the mean range is greater than 1, then at step S403, the grayscale center image is selected as the grayscale reference image. Further, the cropped images are traversed again, and at step S404, grayscale histogram matching is performed on the cropped images whose difference with the grayscale reference image is greater than 0.5 to calibrate the grayscale mean and contrast in the image to obtain the target image set. If the mean range is less than or equal to 1, then at step S405, the grayscale histogram matching process is directly exited.

[0060] Figure 5FIG. 1 is an exemplary flowchart showing how to calculate the average image of a target image set to generate a final image set according to an embodiment of the present application. It should be understood that Figure 5 is the above Figure 1 is a specific embodiment of step S105 in method 100, so the above Figure 1 The description also applies to Figure 5 .

[0061] like Figure 5 As shown in , at step S501, the pixel drift index of the cropped image in the target image set is calculated, and at step S502, the cropped image with the smallest offset distance (for example, the Euclidean distance calculated from the origin in the x and y directions) is determined. After determining the cropped image with the smallest offset distance, at step S503, the cropped images under the target number are selected to calculate the average graph. That is, the average graph of images under different numbers is calculated. At step S504, the average graph under the corresponding target number is saved to generate the final image set. In some embodiments, before calculating the average graph, one or more of the aforementioned parameter indicators can be calculated again for the cropped images in the target image set to manually determine whether there are abnormal images and improve the image quality in the final image set.

[0062] Figure 6 FIG. 1 is an exemplary flow chart showing the overall process of generating an image set in a scanning electron microscope according to an embodiment of the present application. Figure 6 As shown in , at step S601, multiple electron microscope images under target shooting conditions are collected. In some embodiments, the target shooting conditions may include, for example, shooting points, horizontal field of view width, brightness contrast, number of shots or image size, locking machine, etc. At step S602, reference indicators are calculated to screen abnormal images. Specifically, abnormal judgment can be achieved by calculating z-scores and / or interquartile range scores, and abnormal images can be eliminated. Then, at step S603, image registration and image cropping are performed on the electron microscope images in the image set after the abnormal images are filtered. For more details on image registration and image cropping, please refer to the above Figure 1 and Figure 3 The content of the above description will not be repeated in this application. For images that are not successfully registered and cropped, in step S604, abnormal images can be manually screened before image registration and image cropping.

[0063] Next, at step S605, it is determined whether the number of images of the cropped image meets the number threshold. If not, at step S606, the abnormal images that were mistakenly screened out are restored, and then the above operation is repeated at step S603. If satisfied, at step S607, grayscale histogram matching is performed, and at step S607, the average image of the image with the smallest offset distance under different target quantities is calculated. In some embodiments, the image quality in the image set can also be evaluated by manual inspection at step S608, and for qualified images, a final image set is generated at step S609.

[0064] Based on the above scheme of the embodiment of the present application, semi-automatic abnormal image screening is performed by calculating multiple indicators, and after image registration based on fast Fourier transform, the required images are matched with grayscale histograms, which alleviates the image set construction error caused by the above problems to the greatest extent, while improving the calculation efficiency and reducing the labor cost in the process of making the image set. In addition, in order to reduce the impact of misjudgment of abnormal images, the embodiment of the present application also includes the recovery of misjudged abnormal images, which improves the utilization rate of captured images. This solves the problems of scarcity of public data, insufficient scene diversity, and low quality of data labels in the field of SEM image denoising, and greatly improves the availability of supervised deep learning methods in the field of SEM image denoising, and has high engineering application value.

[0065] In addition, the embodiment of the present application can also save images averaged at different times, and can also obtain data with richer noise levels. Furthermore, since the clarity of the label has a crucial impact on the training of the deep learning model, the SEM denoised image set generated by the embodiment of the present application can be used as a data benchmark for various subsequent deep learning denoising models, further enriching the data diversity in the field of image denoising.

[0066] Figure 7 is an exemplary schematic diagram showing a result diagram obtained according to an embodiment of the present application and a result diagram obtained by adjusting the hardware. Figure 7 The left side of the middle shows the SEM image obtained by adjusting the hardware. Figure 7 The right side of the figure shows an SEM image calculated 80 times based on the solution of the embodiment of the present application. The upper left and right sides of the figure are SEM image results, and the lower left and right sides are corresponding enlarged images. It can be seen from the figure that the SEM image obtained by using the embodiment of the present application can further reduce noise.

[0067] Fig. 8A is an exemplary schematic diagram showing the result graph of different times of calculation under different samples according to the embodiment of the present application. Fig. 8AThe two rows in the upper middle correspond to the result graph of the shot put and the magnified image of the target area in the result graph, and the two rows in the lower middle correspond to the result graph of the silicon wafer and the magnified image of the target area in the result graph. Furthermore, each column in the figure corresponds to the result graph of 1, 2, 4, 8, 16 and 80 averages.

[0068] Figure 8B is an exemplary schematic diagram showing original images of different samples and corresponding result images according to an embodiment of the present application. Figure 8B The two rows in the middle and upper part are the original images corresponding to each sample and the enlarged images of the target area in the original image. Figure 8B The two rows in the middle and lower parts are the result images corresponding to each sample and the magnified images of the target area in the result images. Furthermore, the first two columns in the figure are the original images and result images corresponding to the silicon wafer, and the last four columns are the original images and result images of the solder ball, copper mesh, metal and paint. Fig. 8A and Figure 8B It can be seen that the embodiment of the present application can obtain data with richer noise levels by saving images averaged at different times.

[0069] Fig. 9 FIG. 9 is an exemplary structural block diagram showing a device 900 for generating an image set in a scanning electron microscope according to an embodiment of the present application. Fig. 9 As shown in , the device 900 of the present application may include a processor 901 and a memory 902, wherein the processor 901 and the memory 902 communicate with each other via a bus. The memory 902 stores program instructions for generating an image set in a scanning electron microscope. When the program instructions are executed by the processor 901, the method steps described in the above text in combination with the accompanying drawings are implemented: using a scanning electron microscope to collect multiple electron microscope images of the sample to be tested according to the target image shooting conditions to form an initial image set; calculating at least one reference index of each electron microscope image in the initial image set; screening abnormal images in the initial image set based on the reference index; performing grayscale histogram matching on the electron microscope images in the image set after screening abnormal images to obtain a target image set; and calculating the average image of the target image set to generate a final image set.

[0070] According to the above description in combination with the accompanying drawings, those skilled in the art can also understand that the embodiments of the present application can also be implemented by a software program. Therefore, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores computer-readable instructions for collision monitoring in a scanning electron microscope. When the computer-readable instructions are executed by one or more processors, the present application in combination with the accompanying drawings can be implemented. Figure 1 A method for generating an image set in a scanning electron microscope is described.

[0071] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0072] It should be noted that although the operations of the method of the present application are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flow chart can be performed in a different order. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0073] It should be understood that when the terms "first", "second", "third" and "fourth" are used in the claims, the specification and the drawings of the present application, they are only used to distinguish different objects, rather than to describe a specific order. The terms "include" and "comprise" used in the specification and claims of the present application indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections.

[0074] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this application specification and claims, unless the context clearly indicates otherwise, the singular forms of "a", "an" and "the" are intended to include plural forms. It should also be further understood that the term "and / or" used in this application specification and claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0075] Although the implementation methods of the present application are as above, the contents described are only examples adopted to facilitate the understanding of the present application, and are not intended to limit the scope and application scenarios of the present application. Any technician in the technical field described in the present application can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in the present application, but the scope of patent protection of the present application shall still be subject to the scope defined in the attached claims.

Claims

1. A method for generating an image set in a scanning electron microscope, characterized in that include: Using a scanning electron microscope to collect multiple electron microscope images of the sample to be tested according to target imaging conditions to form an initial image set; Calculating at least one reference index for each electron microscope image in the initial image set; Eliminate abnormal images in the initial image set based on the reference index to obtain an image set after filtering out abnormal images; Determine a pixel drift index according to an image offset of an electron microscope image in an image set after screening abnormal images; Using the pixel drift index to calculate the index of the drift center image to obtain a registration reference image and a cropping reference image; Perform image cropping based on whether a first offset determined by registration between the electron microscope image in the image set after filtering abnormal images and the registration reference image exceeds a buffer pixel range to obtain an initial cropped image; as well as, Determine a final cropped image based on a magnitude relationship between a second offset determined by registration between the initial cropped image and the cropped reference image and an offset threshold; Perform grayscale histogram matching on the final cropped images in the image set after filtering out abnormal images to obtain a target image set; as well as, Calculate the average image of the target image set to generate a final image set, The image is cropped based on whether the first offset determined by the registration between the electron microscope image in the image set after filtering the abnormal image and the registration reference image exceeds the buffer pixel range to obtain the initial cropped image, which includes: When a first offset determined by registration between an electron microscope image in the image set after screening abnormal images and the registration reference image exceeds a buffer pixel range, discarding the corresponding image; When the first offset determined by the registration between the electron microscope image in the image set after the abnormal image is screened and the registration reference image does not exceed the buffer pixel range, the electron microscope image in the image set after the abnormal image is screened is cropped to obtain the initial cropped image; The determining of the final cropped image based on the size relationship between the second offset determined by the registration between the initial cropped image and the cropped reference image and the offset threshold comprises: When a second offset determined by registration between the initial cropped image and the cropped reference image is greater than an offset threshold, discarding the corresponding image; When a second offset determined by registration between the initial cropped image and the cropped reference image is less than or equal to the offset threshold, determining the initial cropped image as the final cropped image; The grayscale histogram matching is performed on the final cropped image in the image set after filtering the abnormal images, and the target image set is obtained including: Calculate the grayscale mean of the final cropped image in the image set after filtering out abnormal images; Selecting a grayscale reference image according to the grayscale mean; and, Grayscale histogram matching is performed between the electron microscope images whose grayscale difference between the final cropped image in the image set after filtering out abnormal images and the grayscale reference image is greater than the grayscale threshold, so as to obtain the target image set.

2. The method according to claim 1, characterized in that The target image capturing conditions include at least one or more of the shooting point, horizontal field of view width, brightness contrast, number of shots or image size.

3. The method according to claim 1, characterized in that The reference indicators include at least one or more of image offset, image grayscale mean, image grayscale maximum, image grayscale minimum, image grayscale range, Pearson correlation coefficient, image structure similarity or brightness contrast excess value.

4. The method according to claim 3, characterized in that Wherein screening abnormal images in the initial image set based on the reference index comprises: Calculating an abnormal screening score based on the reference index; and, Electron microscope images whose abnormal screening scores do not meet the score threshold range are eliminated from the initial image set to screen abnormal images in the initial image set.

5. The method according to claim 4, characterized in that The abnormal screening score includes a first abnormal screening score and a second abnormal screening score, and the electron microscope images whose abnormal screening scores do not meet the score threshold range are eliminated from the initial image set, so as to screen the abnormal images in the initial image set including: Electron microscope images whose first abnormal screening score and / or second abnormal screening score do not meet the corresponding score threshold range are eliminated from the initial image set to screen abnormal images in the initial image set.

6. The method according to claim 5, characterized in that The first abnormal screening score includes a z score, and the second abnormal screening score includes an interquartile range score.

7. The method according to claim 1, characterized in that It also includes: Eliminate the electron microscope images that have not achieved image registration and image cropping from the image set after the abnormal images are screened; and Abnormal images in the remaining electron microscopy images were manually screened based on the corresponding reference indicators.

8. The method according to claim 7, characterized in that It also includes: determining the number of images of the final cropped image; and, In response to the number of images not satisfying the number threshold, the reference index is recalculated based on the screened abnormal images to restore the erroneously screened abnormal images.

9. The method according to claim 1, characterized in that: Calculating the average image of the target image set to generate a final image set includes: Selecting a cropped image with the smallest offset distance under the target number from the target image set; and, An average image is calculated based on the cropped images with the smallest offset distance under the target number to generate the final image set.

10. A device for generating an image set in a scanning electron microscope, characterized in that include: processor; as well as, A memory storing program instructions for generating an image set in a scanning electron microscope, wherein when the program instructions are executed by the processor, the device implements the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that: Computer-readable instructions for generating an image set in a scanning electron microscope are stored thereon, and when the computer-readable instructions are executed by one or more processors, the method according to any one of claims 1 to 9 is implemented.

Images